summerstudent.tex

\documentclass[a4paper]{jpconf}
\usepackage{graphicx}
\begin{document}
\title{INFN CNAF log analysis: a first experience with summer students}

\author{Daniele Bonacorsi$^1$, Andrea Ceccanti$^2$, Tommaso Diotalevi$^1$, Antonio Falabella$^2$, Luca Giommi$^2$, Barbara Martelli$^2$, Diego Michelotto$^2$, Lucia Morganti$^2$, Simone Rossi Tisbeni$^1$, Enrico Vianello$^2$}

\address{$^1$University of Bologna, $^2$INFN CNAF}

\ead{barbara.martelli@cnaf.infn.it}

\begin{abstract}
In 2018 the INFN CNAF computing center has started to investigate predictive and preventive maintenance solutions in order to improve fault diagnosis by applying machine learning techniques to hardware and service logs. An excellent experience has been carried out by three students who dedicated three summer months to collect logs of the StoRM services and the resources that host them, to preprocess these logs in order to remove all bias information and to perform initial data analysis \cite{seminario}. Here we are going to present the activities fulfilled by these students, the initial outcome and the ongoing work at the INFN CNAF data center.
\end{abstract}

\section{Introduction}
In recent years INFN CNAF has put a great effort to define and implement a common monitoring infrastructure based on Sensu, InfluxDB and Grafana and to centralize logs from the most relevant services \cite{bovina2015, bovina2017}. Nowadays, this unified infrastructure has been fully integrated in the data center \cite{fattibene2018} and there is the intention to face the new challenge/opportunity to correlate this vast volume of data and extract actionable insights.

During the summer 2018 a first investigation has been exploited with the help of three summer student. Once identified a specific system to analyze, i. e. StoRM, the following activities have been addressed:
\begin{itemize}
\item Log collection and harmonization
\item Log parsing of various services, such as StoRMfrontend, StoRMbackend, heartbeat, messages, GridFTP and GPFS (not covered in our study, but potentially interesting)
\item Metrics data adding (from Tier1 InfluxDB)
\end{itemize}

However, to provide a first proof of concept for the predictive and preventive maintenance, data categorization and machine learning techniques application represent two key points that have been conducted from the end 2018 and the middle 2019.

\section{Log collection and harmonization}
The first part of the work consisted in the collection of StoRM logs from the StoRM servers dedicated to the Atlas experiment. 

Subsequently, most relevant information was extracted from the logs using the ELK Stack suite \cite{elk}. The ELK stack consists of four components: Beats used for data collection from multiple sources, Logstash used for data aggregation and processing, Elasticsearch used for store and index data, Kibana for data analysis and visualization. In particular, Logstash has been used to ingest data from Beats in a continuous live-feed streaming, filter relevant entries and parse each event, identifying named fields to build a user defined structure and ship parsed data to the Elasticsearch engine. Most data was filtered using a \textit{grok} filter which is based on regular expressions and provides predefined filters together with the ability of defining customized ones. 
Finally, several dashboards were created using Kibana in order to show in a human-friendly way a summary of the most relevant information derived from StoRM logs (ee for example \ref{fig3}).
\begin{figure}[h]
\includegraphics[width=20pc]{kibana.png}\hspace{2pc}
\begin{minipage}[b]{14pc}\caption{\label{fig3}An example of Kibana dashboard created.}
\end{minipage}
\end{figure}


\section{Log parsing}
Among the INFN Tier 1 services hosted at the INFN CNAF computing center, there are efficient storage systems, like StoRM that is a grid Storage Resource Manager (SRM) solution. Figure \ref{fig1} shows the StoRM architecture: the frontend service manages user authentication and stores requests data, while the backend service executes SRM functionalities and takes care of space and authorization.
The log files contains basically three types of information: timestamp, metrics, and messages.
\begin{figure}[h]
\includegraphics[width=20pc]{StoRM-full-picture.png}\hspace{2pc}%
\begin{minipage}[b]{14pc}\caption{\label{fig1}The StoRM architecture.}
\end{minipage}
\end{figure}

At the beginning of this work (mid 2018), StoRM at Tier1 was monitored by InfluxDB and Grafana. Metrics monitored included CPU, RAM, network and disk usage; number of sync SRM request per minute per host; duration of async PTG and PTP per host (avg). We wanted to add information derived from the analysis of StoRM logs to already available monitoring information, in order to derive new insights potentially useful to enhance service availability and efficiency with the long-term intent of implementing a global predictive maintenance solution for Tier-1. In order to build a Machine Learning model for anomaly prediction, logs from two different period were analyzed: a normal behavior period and a critical behavior period (due to wrong configuration of the file system and wrong configuration of the queues coming from the farm). 
A four-steps activity has been carried out: 
\begin{enumerate}
\item Parsing: log files were parsed and deconstructed, converting them to CSV format
\item Feature selection: was done grouping messages based on their common content (core part of the message). The grouping phase resulted in 20 \textit{Request Types} (Connection, Run, Ping, Ls, Check permission, PTG, PTG status, Get space tokens, PTP, PTP status, BOL status, Put don, Release files, Mv, Mkdir, BOL, Abort request, Abort files, Get space metadata, nan) and 15 \textit{Result Types} (SRM\_SUCCESS, SRM\_FAILURE, SRM\_NOT\_SUPPORTED, SRM\_REQUEST\_QUEUED, SRM\_REQUEST\_INPROGRESS, Protocol check failed, Received 4 protocols, Some protocols supported, SRM\_DUPLICATION\_ERROR, rpcResponseHandler\_AbortFiles, SRM\_INVALID\_REQUEST, SRM\_INVALID\_PATH, Received 5 protocols, SRM\_INTERNAL\_ERROR, nan). A first data exploration phase was performed by counting occurrencies of messages in each group. 
Techniques used for the feature selection procedure were: SelectKBest with the chi-squared statistical test, Recursive Feature Elimination, Principal Component Analysis (PCA) and Feature Importance from ensembles of decision tree methods.
\item One-hot encoding: CSV rows encoded in binary vectors (feature vectors). Each vector represents the summary of 15-minutes log contents.
\item Labelling: operation specific for StoRM log files done manually discriminating between normal and critical period based on help-desk tickets. 
\end{enumerate}
Feature vectors obtained in (iii) and labeled datasets built in (iv) were used to train several ML algorithms and to test their accuracy. Figure \ref{fig2} depicts the results of tests performed on the following algorithms: LogisticRegression (LR), LinearDiscriminantAnalysis (LDA), KNeighborsClassifier (KNN), GaussianNB (GNB), DecisionTreeClassifier (CART), BaggingClassifier (BgDT), RandomForestClassifier (RF), ExtraTreesClassifier (ET), AdaBoostClassifier (AB), GradientBoostingClassifier (GB), XGBoostClassifier (XGB), MultiLayerPerceptronClassifier (MLP).

\begin{center}
\begin{figure}[h]
\includegraphics[width=20pc]{MLalgorithms.png}\hspace{2pc}
\begin{minipage}[b]{14pc}\caption{\label{fig2}Machine Learning Algorithms Comparison (scorer=accuracy).}
\end{minipage}
\end{figure}
\end{center}

\section{Metrics data adding}
This activity was mainly focused on collecting metric data from InfluxDB in order to put them in relation with StoRM logs obtained with activities explained in previous sections and extract new insights. 
Key components of log files were identified, parsed and structured in a CSV file with the following columns: timestamp, metric, message, descriptive keys and separators. All timestamps were converted in UNIX epoch time in order to be comparable. On one side, InfluxDB stores information with different granularity depending on the age of data collected and on the other side, StoRM front-end and back-end logs are produced with different frequencies (one line each minute for heartbeat logs, multiple lines every minute for metrics logs, one line every five minutes for InfluxDB more recent data, and so on). Therefore, some concatenation rules have been implemented in order to correctly put in relation all data sources based on the time of occurrence of the event: backend metrics are split by type, timestamp is rounded off to one‐minute precision, in case of overlap the more recent is kept and every CSV file is concatenated and ordered by timestamp.

%This requires a site-wide effort in collecting, cleaning and structuring all possibly useful data
%coming from log files of the various Tier-1 services and systems, as a necessary step prior to
%designing machine learning based approaches for predictive maintenance.
%Among the Tier-1 services, efficient storage systems are one of the key ingredients of Tier-1
%operations. CNAF uses the StoRM service as a Grid Storage Resource Manager solution: its
%operations are logged in a very complex manner, as the log content is deeply unstructured and
%hard to be exploited for analytics purposes. Despite such difficulty, the StoRM logs are a precious
%source of information for operators (e. g. real-time monitoring and anomaly detection), for
%developers (e. g. debugging, service stability, code improvements) and for site managers (service
%optimization, storage usage efficiency, time and money saving ways to spot and prevent unwanted
%behaviors).
%Based on previous experiences on Big Data Analytics and Machine/Deep learning in the CMS
%experiment, this work describes how the StoRM logs can be handled and parsed to extract the
%relevant information, how such log handling can be designed to work automatically, how to define
%and implement metrics to tag critical states of the service, how to correlate StoRM events with
%external services events, and ultimately how to contribute to the future CNAF-wide predictive
%maintenance system.
%Initial results in this activity are presented and discussed. Furthermore, a mention to ongoing
%complementary work at the CNAF center is also mentioned.
%
%
%
%\section{Preparing your paper}
%\verb"jpconf" requires \LaTeXe\ and  can be used with other package files such
%as those loading the AMS extension fonts
%\verb"msam" and \verb"msbm" (these fonts provide the
%blackboard bold alphabet and various extra maths symbols as well as
%symbols useful in figure captions); an extra style file \verb"iopams.sty" is
%provided to load these packages and provide extra definitions for bold Greek letters.
%\subsection{Headers, footers and page numbers}
%Authors should {\it not} add headers, footers or page numbers to the pages of their article---they will
%be added by \iopp\ as part of the production process.
%
%\subsection{{\cls\ }package options}
%The \cls\ class file has two options `a4paper' and `letterpaper':
%\begin{verbatim}
%\documentclass[a4paper]{jpconf}
%\end{verbatim}
%
%or \begin{verbatim}
%\documentclass[letterpaper]{jpconf}
%\end{verbatim}
%
%\begin{center}
%\begin{table}[h]
%\caption{\label{opt}\cls\ class file options.}
%%\footnotesize\rm
%\centering
%\begin{tabular}{@{}*{7}{l}}
%\br
%Option&Description\\
%\mr
%\verb"a4paper"&Set the paper size and margins for A4 paper.\\
%\verb"letterpaper"&Set the paper size and margins for US letter paper.\\
%\br
%\end{tabular}
%\end{table}
%\end{center}
%
%The default paper size is A4 (i.e., the default option is {\tt a4paper}) but this can be changed to Letter by
%using \verb"\documentclass[letterpaper]{jpconf}". It is essential that you do not put macros into the text which alter the page dimensions.
%
%\section{The title, authors, addresses and abstract}
%The code for setting the title page information is slightly different from
%the normal default in \LaTeX\ but please follow these instructions as carefully as possible so all articles within a conference have the same style to the title page.
%The title is set in bold unjustified type using the command
%\verb"\title{#1}", where \verb"#1" is the title of the article. The
%first letter of the title should be capitalized with the rest in lower case.
%The next information required is the list of all authors' names followed by
%the affiliations. For the authors' names type \verb"\author{#1}",
%where \verb"#1" is the
%list of all authors' names. The style for the names is initials then
%surname, with a comma after all but the last
%two names, which are separated by `and'. Initials should {\it not} have
%full stops. First names may be used if desired. The command \verb"\maketitle" is not
%required.
%
%The addresses of the authors' affiliations follow the list of authors.
%Each address should be set by using
%\verb"\address{#1}" with the address as the single parameter in braces.
%If there is more
%than one address then a superscripted number, followed by a space, should come at the start of
%each address. In this case each author should also have a superscripted number or numbers following their name to indicate which address is the appropriate one for them.
%
%Please also provide e-mail addresses for any or all of the authors using an \verb"\ead{#1}" command after the last address. \verb"\ead{#1}" provides the text Email: so \verb"#1" is just the e-mail address or a list of emails.
%
%The abstract follows the addresses and
%should give readers concise information about the content
%of the article and should not normally exceed 200
%words. {\bf All articles must include an abstract}. To indicate the start
%of the abstract type \verb"\begin{abstract}" followed by the text of the
%abstract.  The abstract should normally be restricted
%to a single paragraph and is terminated by the command
%\verb"\end{abstract}"
%
%\subsection{Sample coding for the start of an article}
%\label{startsample}
%The code for the start of a title page of a typical paper might read:
%\begin{verbatim}
%\title{The anomalous magnetic moment of the
%neutrino and its relation to the solar neutrino problem}
%
%\author{P J Smith$^1$, T M Collins$^2$,
%R J Jones$^{3,}$\footnote[4]{Present address:
%Department of Physics, University of Bristol, Tyndalls Park Road,
%Bristol BS8 1TS, UK.} and Janet Williams$^3$}
%
%\address{$^1$ Mathematics Faculty, Open University,
%Milton Keynes MK7~6AA, UK}
%\address{$^2$ Department of Mathematics,
%Imperial College, Prince Consort Road, London SW7~2BZ, UK}
%\address{$^3$ Department of Computer Science,
%University College London, Gower Street, London WC1E~6BT, UK}
%
%\ead{williams@ucl.ac.uk}
%
%\begin{abstract}
%The abstract appears here.
%\end{abstract}
%\end{verbatim}
%
%\section{The text}
%The text of the article should should be produced using standard \LaTeX\ formatting. Articles may be divided into sections and subsections, but the length limit provided by the \corg\ should be adhered to.
%
%\subsection{Acknowledgments}
%Authors wishing to acknowledge assistance or encouragement from
%colleagues, special work by technical staff or financial support from
%organizations should do so in an unnumbered Acknowledgments section
%immediately following the last numbered section of the paper. The
%command \verb"\ack" sets the acknowledgments heading as an unnumbered
%section.
%
%\subsection{Appendices}
%Technical detail that it is necessary to include, but that interrupts
%the flow of the article, may be consigned to an appendix.
%Any appendices should be included at the end of the main text of the paper, after the acknowledgments section (if any) but before the reference list.
%If there are two or more appendices they will be called Appendix A, Appendix B, etc.
%Numbered equations will be in the form (A.1), (A.2), etc,
%figures will appear as figure A1, figure B1, etc and tables as table A1,
%table B1, etc.
%
%The command \verb"\appendix" is used to signify the start of the
%appendixes. Thereafter \verb"\section", \verb"\subsection", etc, will
%give headings appropriate for an appendix. To obtain a simple heading of
%`Appendix' use the code \verb"\section*{Appendix}". If it contains
%numbered equations, figures or tables the command \verb"\appendix" should
%precede it and \verb"\setcounter{section}{1}" must follow it.
%
%\section{References}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%In the online version of \jpcs\ references will be linked to their original source or to the article within a secondary service such as INSPEC or ChemPort wherever possible. To facilitate this linking extra care should be taken when preparing reference lists.
%
%Two different styles of referencing are in common use: the Harvard alphabetical system and the Vancouver numerical system.  For \jpcs, the Vancouver numerical system is preferred but authors should use the Harvard alphabetical system if they wish to do so. In the numerical system references are numbered sequentially throughout the text within square brackets, like this [2], and one number can be used to designate several references.
%
%\subsection{Using \BibTeX}
%We highly recommend the {\ttfamily\textbf\selectfont iopart-num} \BibTeX\ package by Mark~A~Caprio \cite{iopartnum}, which is included with this documentation.
%
%\subsection{Reference lists}
%A complete reference should provide the reader with enough information to locate the article concerned, whether published in print or electronic form, and should, depending on the type of reference, consist of:
%
%\begin{itemize}
%\item name(s) and initials;
%\item date published;
%\item title of journal, book or other publication;
%\item titles of journal articles may also be included (optional);
%\item volume number;
%\item editors, if any;
%\item town of publication and publisher in parentheses for {\it books};
%\item the page numbers.
%\end{itemize}
%
%Up to ten authors may be given in a particular reference; where
%there are more than ten only the first should be given followed by
%`{\it et al}'. If an author is unsure of a particular journal's abbreviated title it is best to leave the title in
%full. The terms {\it loc.\ cit.\ }and {\it ibid.\ }should not be used.
%Unpublished conferences and reports should generally not be included
%in the reference list and articles in the course of publication should
%be entered only if the journal of publication is known.
%A thesis submitted for a higher degree may be included
%in the reference list if it has not been superseded by a published
%paper and is available through a library; sufficient information
%should be given for it to be traced readily.
%
%\subsection{Formatting reference lists}
%Numeric reference lists should contain the references within an unnumbered section (such as \verb"\section*{References}"). The
%reference list itself is started by the code
%\verb"\begin{thebibliography}{<num>}", where \verb"<num>" is the largest
%number in the reference list and is completed by
%\verb"\end{thebibliography}".
%Each reference starts with \verb"\bibitem{<label>}", where `label' is the label used for cross-referencing. Each \verb"\bibitem" should only contain a reference to a single article (or a single article and a preprint reference to the same article).  When one number actually covers a group of two or more references to different articles, \verb"\nonum"
%should replace \verb"\bibitem{<label>}" at
%the start of each reference in the group after the first.
%
%For an alphabetic reference list use \verb"\begin{thereferences}" ... \verb"\end{thereferences}" instead of the
%`thebibliography' environment and each reference can be start with just \verb"\item" instead of \verb"\bibitem{label}"
%as cross referencing is less useful for alphabetic references.
%
%\subsection {References to printed journal articles}
%A normal reference to a journal article contains three changes of font (see table \ref{jfonts}) and is constructed as follows:
%
%\begin{itemize}
%\item the authors should be in the form surname (with only the first letter capitalized) followed by the initials with no periods after the initials. Authors should be separated by a comma except for the last two which should be separated by `and' with no comma preceding it;
%\item the article title (if given) should be in lower case letters, except for an initial capital, and should follow the date;
%\item the journal title is in italic and is abbreviated. If a journal has several parts denoted by different letters the part letter should be inserted after the journal in Roman type, e.g. {\it Phys. Rev.} A;
%\item the volume number should be in bold type;
%\item both the initial and final page numbers should be given where possible. The final page number should be in the shortest possible form and separated from the initial page number by an en rule `-- ', e.g. 1203--14, i.e. the numbers `12' are not repeated.
%\end{itemize}
%
%A typical (numerical) reference list might begin
%
%\medskip
%\begin{thebibliography}{9}
%\item Strite S and Morkoc H 1992 {\it J. Vac. Sci. Technol.} B {\bf 10} 1237
%\item Jain S C, Willander M, Narayan J and van Overstraeten R 2000
%{\it J. Appl. Phys}. {\bf 87} 965
%\item Nakamura S, Senoh M, Nagahama S, Iwase N, Yamada T, Matsushita T, Kiyoku H
%and 	Sugimoto Y 1996 {\it Japan. J. Appl. Phys.} {\bf 35} L74
%\item Akasaki I, Sota S, Sakai H, Tanaka T, Koike M and Amano H 1996
%{\it Electron. Lett.} {\bf 32} 1105
%\item O'Leary S K, Foutz B E, Shur M S, Bhapkar U V and Eastman L F 1998
%{\it J. Appl. Phys.} {\bf 83} 826
%\item Jenkins D W and Dow J D 1989 {\it Phys. Rev.} B {\bf 39} 3317
%\end{thebibliography}
%\smallskip
%
%\noindent which would be obtained by typing
%
%\begin{verbatim}
%\begin{\thebibliography}{9}
%\item Strite S and Morkoc H 1992 {\it J. Vac. Sci. Technol.} B {\bf 10} 1237
%\item Jain S C, Willander M, Narayan J and van Overstraeten R 2000
%{\it J. Appl. Phys}. {\bf 87} 965
%\item Nakamura S, Senoh M, Nagahama S, Iwase N, Yamada T, Matsushita T, Kiyoku H
%and 	Sugimoto Y 1996 {\it Japan. J. Appl. Phys.} {\bf 35} L74
%\item Akasaki I, Sota S, Sakai H, Tanaka T, Koike M and Amano H 1996
%{\it Electron. Lett.} {\bf 32} 1105
%\item O'Leary S K, Foutz B E, Shur M S, Bhapkar U V and Eastman L F 1998
%{\it J. Appl. Phys.} {\bf 83} 826
%\item Jenkins D W and Dow J D 1989 {\it Phys. Rev.} B {\bf 39} 3317
%\end{\thebibliography}
%\end{verbatim}
%
%\begin{center}
%\begin{table}[h]
%\centering
%\caption{\label{jfonts}Font styles for a reference to a journal article.}
%\begin{tabular}{@{}l*{15}{l}}
%\br
%Element&Style\\
%\mr
%Authors&Roman type\\
%Date&Roman type\\
%Article title (optional)&Roman type\\
%Journal title&Italic type\\
%Volume number&Bold type\\
%Page numbers&Roman type\\
%\br
%\end{tabular}
%\end{table}
%\end{center}
%
%\subsection{References to \jpcs\ articles}
%Each conference proceeding published in \jpcs\ will be a separate {\it volume};
%references should follow the style for conventional printed journals. For example:\vspace{6pt}
%\numrefs{1}
%\item Douglas G 2004 \textit{J. Phys.: Conf. Series} \textbf{1} 23--36
%\endnumrefs
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\subsection{References to preprints}
%For preprints there are two distinct cases:
%\renewcommand{\theenumi}{\arabic{enumi}}
%\begin{enumerate}
%\item Where the article has been published in a journal and the preprint is supplementary reference information. In this case it should be presented as:
%\medskip
%\numrefs{1}
%\item Kunze K 2003 T-duality and Penrose limits of spatially homogeneous and inhomogeneous cosmologies {\it Phys. Rev.} D {\bf 68} 063517 ({\it Preprint} gr-qc/0303038)
%\endnumrefs
%\item Where the only reference available is the preprint. In this case it should be presented as
%\medskip
%\numrefs{1}
%\item Milson R, Coley A, Pravda V and Pravdova A 2004 Alignment and algebraically special tensors {\it Preprint} gr-qc/0401010
%\endnumrefs
%\end{enumerate}
%
%\subsection{References to electronic-only journals}
%In general article numbers are given, and no page ranges, as most electronic-only journals start each article on page 1.
%
%\begin{itemize}
%\item For {\it New Journal of Physics} (article number may have from one to three digits)
%\numrefs{1}
%\item Fischer R 2004 Bayesian group analysis of plasma-enhanced chemical vapour deposition data {\it New. J. Phys.} {\bf 6} 25
%\endnumrefs
%\item For SISSA journals the volume is divided into monthly issues and these form part of the article number
%
%\numrefs{2}
%\item Horowitz G T and Maldacena J 2004 The black hole final state {\it J. High Energy Phys.}  	JHEP02(2004)008
%\item Bentivegna E, Bonanno A and Reuter M 2004 Confronting the IR fixed point cosmology 	with 	high-redshift observations {\it J. Cosmol. Astropart. Phys.} JCAP01(2004)001
%\endnumrefs
%\end{itemize}
%
%\subsection{References to books, conference proceedings and reports}
%References to books, proceedings and reports are similar to journal references, but have
%only two changes of font (see table~\ref{book}).
%
%\begin{table}
%\centering
%\caption{\label{book}Font styles for references to books, conference proceedings and reports.}
%\begin{tabular}{@{}l*{15}{l}}
%\br
%Element&Style\\
%\mr
%Authors&Roman type\\
%Date&Roman type\\
%Book title (optional)&Italic type\\
%Editors&Roman type\\
%Place (city, town etc) of publication&Roman type\\
%Publisher&Roman type\\
%Volume&Roman type\\
%Page numbers&Roman type\\
%\br
%\end{tabular}
%\end{table}
%
%Points to note are:
%\medskip
%\begin{itemize}
%\item Book titles are in italic and should be spelt out in full with initial capital letters for all except minor words. Words such as Proceedings, Symposium, International, Conference, Second, etc should be abbreviated to {\it Proc.}, {\it Symp.}, {\it Int.}, {\it Conf.}, {\it 2nd}, respectively, but the rest of the title should be given in full, followed by the date of the conference and the town or city where the conference was held. For Laboratory Reports the Laboratory should be spelt out wherever possible, e.g. {\it Argonne National Laboratory Report}.
%\item The volume number, for example vol 2, should be followed by the editors, if any, in a form such as `ed A J Smith and P R Jones'. Use {\it et al} if there are more than two editors. Next comes the town of publication and publisher, within brackets and separated by a colon, and finally the page numbers preceded by p if only one number is given or pp if both the initial and final numbers are given.
%\end{itemize}
%
%Examples taken from published papers:
%\medskip
%
%\numrefs{99}
%\item Kurata M 1982 {\it Numerical Analysis for Semiconductor Devices} (Lexington, MA: Heath)
%\item Selberherr S 1984 {\it Analysis and Simulation of Semiconductor Devices} (Berlin: Springer)
%\item Sze S M 1969 {\it Physics of Semiconductor Devices} (New York: Wiley-Interscience)
%\item Dorman L I 1975 {\it Variations of Galactic Cosmic Rays} (Moscow: Moscow State University Press) p 103
%\item Caplar R and Kulisic P 1973 {\it Proc. Int. Conf. on Nuclear Physics (Munich)} vol 1 (Amsterdam: 	North-Holland/American Elsevier) p 517
%\item Cheng G X 2001 {\it Raman and Brillouin Scattering-Principles and Applications} (Beijing: Scientific)
%\item Szytula A and Leciejewicz J 1989 {\it Handbook on the Physics and Chemistry of Rare Earths} vol 12, ed K A Gschneidner Jr and L Erwin (Amsterdam: Elsevier) p 133
%\item Kuhn T 1998 {\it Density matrix theory of coherent ultrafast dynamics Theory of Transport Properties of Semiconductor Nanostructures} (Electronic Materials vol 4) ed E Sch\"oll (London: Chapman and Hall) chapter 6 pp 173--214
%\endnumrefs
%
%\section{Tables and table captions}
%Tables should be numbered serially and referred to in the text
%by number (table 1, etc, {\bf rather than} tab. 1). Each table should be a float and be positioned within the text at the most convenient place near to where it is first mentioned in the text. It should have an
%explanatory caption which should be as concise as possible.
%
%\subsection{The basic table format}
%The standard form for a table is:
%\begin{verbatim}
%\begin{table}
%\caption{\label{label}Table caption.}
%\begin{center}
%\begin{tabular}{llll}
%\br
%Head 1&Head 2&Head 3&Head 4\\
%\mr
%1.1&1.2&1.3&1.4\\
%2.1&2.2&2.3&2.4\\
%\br
%\end{tabular}
%\end{center}
%\end{table}
%\end{verbatim}
%
%The above code produces table~\ref{ex}.
%
%\begin{table}[h]
%\caption{\label{ex}Table caption.}
%\begin{center}
%\begin{tabular}{llll}
%\br
%Head 1&Head 2&Head 3&Head 4\\
%\mr
%1.1&1.2&1.3&1.4\\
%2.1&2.2&2.3&2.4\\
%\br
%\end{tabular}
%\end{center}
%\end{table}
%
%Points to note are:
%\medskip
%\begin{enumerate}
%\item The caption comes before the table.
%\item The normal style is for tables to be centred in the same way as
%equations. This is accomplished
%by using \verb"\begin{center}" \dots\ \verb"\end{center}".
%
%\item The default alignment of columns should be aligned left.
%
%\item Tables should have only horizontal rules and no vertical ones. The rules at
%the top and bottom are thicker than internal rules and are set with
%\verb"\br" (bold rule).
%The rule separating the headings from the entries is set with
%\verb"\mr" (medium rule). These commands do not need a following double backslash.
%
%\item Numbers in columns should be aligned as appropriate, usually on the decimal point;
%to help do this a control sequence \verb"\lineup" has been defined
%which sets \verb"\0" equal to a space the size of a digit, \verb"\m"
%to be a space the width of a minus sign, and \verb"\-" to be a left
%overlapping minus sign. \verb"\-" is for use in text mode while the other
%two commands may be used in maths or text.
%(\verb"\lineup" should only be used within a table
%environment after the caption so that \verb"\-" has its normal meaning
%elsewhere.) See table~\ref{tabone} for an example of a table where
%\verb"\lineup" has been used.
%\end{enumerate}
%
%\begin{table}[h]
%\caption{\label{tabone}A simple example produced using the standard table commands
%and $\backslash${\tt lineup} to assist in aligning columns on the
%decimal point. The width of the
%table and rules is set automatically by the
%preamble.}
%
%\begin{center}
%\lineup
%\begin{tabular}{*{7}{l}}
%\br
%$\0\0A$&$B$&$C$&\m$D$&\m$E$&$F$&$\0G$\cr
%\mr
%\0\023.5&60  &0.53&$-20.2$&$-0.22$ &\01.7&\014.5\cr
%\0\039.7&\-60&0.74&$-51.9$&$-0.208$&47.2 &146\cr
%\0123.7 &\00 &0.75&$-57.2$&\m---   &---  &---\cr
%3241.56 &60  &0.60&$-48.1$&$-0.29$ &41   &\015\cr
%\br
%\end{tabular}
%\end{center}
%\end{table}
%
%\section{Figures and figure captions}
%Figures must be included in the source code of an article at the appropriate place in the text not grouped together at the end.
%
%Each figure should have a brief caption describing it and, if
%necessary, interpreting the various lines and symbols on the figure.
%As much lettering as possible should be removed from the figure itself and
%included in the caption. If a figure has parts, these should be
%labelled ($a$), ($b$), ($c$), etc.
%\Tref{blobs} gives the definitions for describing symbols and lines often
%used within figure captions (more symbols are available
%when using the optional packages loading the AMS extension fonts).
%
%\begin{table}[h]
%\caption{\label{blobs}Control sequences to describe lines and symbols in figure
%captions.}
%\begin{center}
%\begin{tabular}{lllll}
%\br
%Control sequence&Output&&Control sequence&Output\\
%\mr
%\verb"\dotted"&\dotted        &&\verb"\opencircle"&\opencircle\\
%\verb"\dashed"&\dashed        &&\verb"\opentriangle"&\opentriangle\\
%\verb"\broken"&\broken&&\verb"\opentriangledown"&\opentriangledown\\
%\verb"\longbroken"&\longbroken&&\verb"\fullsquare"&\fullsquare\\
%\verb"\chain"&\chain          &&\verb"\opensquare"&\opensquare\\
%\verb"\dashddot"&\dashddot    &&\verb"\fullcircle"&\fullcircle\\
%\verb"\full"&\full            &&\verb"\opendiamond"&\opendiamond\\
%\br
%\end{tabular}
%\end{center}
%\end{table}
%
%
%Authors should try and use the space allocated to them as economically as possible. At times it may be convenient to put two figures side by side or the caption at the side of a figure. To put figures side by side, within a figure environment, put each figure and its caption into a minipage with an appropriate width (e.g. 3in or 18pc if the figures are of equal size) and then separate the figures slightly by adding some horizontal space between the two minipages (e.g. \verb"\hspace{.2in}" or \verb"\hspace{1.5pc}". To get the caption at the side of the figure add the small horizontal space after the \verb"\includegraphics" command and then put the \verb"\caption" within a minipage of the appropriate width aligned bottom, i.e. \verb"\begin{minipage}[b]{3in}" etc (see code in this file used to generate figures 1--3).
%
%Note that it may be necessary to adjust the size of the figures (using optional arguments to \verb"\includegraphics", for instance \verb"[width=3in]") to get you article to fit within your page allowance or to obtain good page breaks.
%
%%\begin{figure}[h]
%%\begin{minipage}{14pc}
%%\includegraphics[width=14pc]{name.eps}
%%\caption{\label{label}Figure caption for first of two sided figures.}
%%\end{minipage}\hspace{2pc}%
%%\begin{minipage}{14pc}
%%\includegraphics[width=14pc]{name.eps}
%%\caption{\label{label}Figure caption for second of two sided figures.}
%%end{minipage}
%%end{figure}
%
%%\begin{figure}[h]
%%\includegraphics[width=14pc]{name.eps}\hspace{2pc}%
%%\begin{minipage}[b]{14pc}\caption{\label{label}Figure caption for a narrow figure where the caption is %ut at the side of the figure.}
%%\end{minipage}
%%\end{figure}
%
%Using the graphicx package figures can be included using code such as:
%%\begin{verbatim}
%%\begin{figure}
%%\begin{center}
%%\includegraphics{file.eps}
%%\end{center}
%%caption{\label{label}Figure caption}
%%\end{figure}
%%\end{verbatim}

\section{Conclusion}
This experience is a good example of mutually beneficial collaboration between university students and INFN CNAF. The outcome has allowed master students (i.e. Diotalevi T. and Giommi L.) to publish papers at international conferences \cite{diotalevi, giommi20191}, to win Giulia Vita Finzi's award \cite{giommi20192}, and to start their PhD courses with success. Furthermore, the undergraduate student (i.e. Rossi Tisbeni R) will hold a master degree in Physics in July 2019. On the other hand, the INFN CNAF data center managers has decided to continue exploiting predictive and preventive maintenance to establish where and when to use it to keep services running optimally.

\section*{References}
\begin{thebibliography}{9}
\bibitem{seminario} https://agenda.infn.it/event/17430/, site visited on June 2019.
\bibitem{bovina2015} Bovina S, Michelotto D, Misurelli G, \emph{CNAF Annual Report}, pp. 111--114, 2015.
\bibitem{bovina2017} Bovina S, Michelotto D, In Proc of CHEP 2017.
\bibitem{fattibene2018} Fattibene E, Dal Pra S, Falabella A, De Cristofaro T, Cincinelli G, Ruini M, In Proc of CHEP 2018.
\bibitem{diotalevi} Diotalevi T, Bonacorsi D, Michelotto D, Falabella A, In Proc of International Symposium on Grids \& Clouds (ISGC), Taipei, Taiwan, 2019 (under review).
\bibitem{giommi20191} Giommi L, Bonacorsi D, Diotalevi T, Rossi Tisbeni S, Rinaldi L, Morganti L, Falabella A, Ronchieri E, Ceccanti A, Martelli B, In Proc of International Symposium on Grids \& Clouds (ISGC), Taipei, Taiwan, 2019 (under review).
\bibitem{giommi20192} Giommi L, In INFN CCR Workshop, La Biodola, 3-7 June 2019.
\bibitem{elk}https://www.elastic.co/, site visited on June 2019.
\end{thebibliography}

\end{document}