AMS-report-2019.tex

\documentclass[a4paper]{jpconf}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{color}

\begin{document}
\title{AMS-02 data processing and analysis at CNAF}

\author{B. Bertucci$^{1,2}$, M. Duranti$^2$, V. Formato$^{2,\ast}$, D. Spiga$^{2}$}

\address{$^1$ Universit\`a di Perugia, Perugia, IT}
\address{$^2$ INFN Sezione di Perugia, Perugia, IT}

\address{AMS experiment \url{http://ams.cern.ch}, \url{http://www.ams02.org}, \url{http://www.pg.infn.it/ams/}}

\ead{* valerio.formato@infn.it}


\begin{abstract}
AMS is a large acceptance instrument conceived to search for anti-particles (positrons, anti-protons, anti-deuterons) coming from dark matter
annihilation, primordial anti-matter (anti-He or light anti nuclei) and to
perform accurate measurements in space of the cosmic radiation in the GeV-TeV
energy range.
Installed on the International Space Station (ISS) in mid-May 2011, it is
operating continuously since then, with a collected statistics of $\sim$ 130
billion events up to the end of 2018.
CNAF is one of the repositories of the full AMS data set and contributes to the
data production and Monte Carlo simulation of the international collaboration.
It represents the central computing resource for the data analysis performed by
Italian collaboration.
In the following, the AMS computing framework, the role of the CNAF computing
center and the use of the CNAF resources in 2018 will be given.
\end{abstract}

\section{Introduction}
AMS is a large acceptance instrument conceived to search for anti-particles
(positrons, anti-protons, anti-deuterons) coming from dark matter annihilation,
primordial anti-matter (anti-He or light anti nuclei) and to perform accurate
measurements in space of the cosmic radiation in the GeV-TeV energy range.

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.49\textwidth]{AMS_nuovo.pdf}
\end{center}
\caption{\label{fig:ams_layout} AMS-02 detector consists of nine planes of
precision silicon tracker, a transition radiation detector (TRD), four planes
of time of flight counters (TOF), a permanent magnet, an array of
anticoincidence counters (ACC), surrounding the inner tracker, a ring imaging
Cherenkov detector (RICH), and an electromagnetic calorimeter (ECAL).}
\end{figure}
The layout of the AMS-02 detector is shown in Fig. \ref{fig:ams_layout}.
A large spectrometer is the core of the instrument: a magnetic field of 0.14
T generated by a permanent magnet deflects in opposite directions positive and
negative particles whose trajectories are accurately measured up to TeV
energies by means of 9 layers of double side silicon micro-strip detectors -
the Tracker -  with a spatial resolution of $\sim 10 \mu m$ in the single point
measurement along the track. Redundant measurements of the particle
characteristics, as velocity, absolute charge magnitude ($Z$), rigidity and
energy are performed by a Time of Flight system, the tracker, a RICH detector
and a 3D imaging calorimeter with a 17 $X_0$ depth. A transition radiation
detector provides an independent e/p separation with a rejection power of
$\sim 10^3$ around 100 GeV.

AMS has been installed on the International Space Station (ISS) in mid-May 2011
and it is operating continuously since then, with a collected statistics of
$\sim$ 130 billion events up to the end of 2018.
The signals from the $\sim$ 300.000 electronic channels of the detector and its
monitoring system (thermal and pressure sensors) are reduced on board to match
the average bandwidth of $\sim$10 Mbit/s for the data transmission from space
to ground, for a $\sim$ 100 GB/day of raw data produced by the experiment.

Due to the rapidly changing environmental conditions along the $\sim$ 90 minutes
orbit of the ISS at 390 Km of altitude, continuous monitoring and adjustments of
the data taking conditions are performed in the Payload and Operation Control
Center (POCC) located at CERN and a careful calibration of the detector response
is needed to process the raw data and reconstruct physics quantities for data analysis.

CNAF is one of the repositories of the full AMS data set, both raw and processed
data are stored at CNAF which represents the central computing resource for the
data analysis performed by Italian collaboration and contributes as well to the
data production and Monte Carlo simulation of the international collaboration.


\section{AMS-02 Computing Model and Computing Facilities}

As a payload on the ISS, AMS has to be compliant to all of the standard
communication protocols used by NASA to communicate with ISS, and its data have
to be transmitted through the NASA communication network.
On the ground, data are finally stored at the AMS Payload Operation Control
Center (POCC) at CERN.
Data are continuously collected, 24 hours per day, 365 days per year.
The data reconstruction pipeline is mainly composed by two logical step:
\begin{itemize}
\item[1)]{
    the {\bf First Production} runs continuously over incoming data doing an
    initial validation and indexing. It produces the so called "standard" (STD)
    reconstructed data stream, ready within two hours after data are received at
    CERN, that is used to calibrate different sub-detectors as well as to monitor
    off-line the detector performances. In this stage Data Summary Files are
    produced for fast event selections.
}
\item[2)]{
    Data from the First Production are reprocessed applying all of sub-detector
    calibrations, alignments, ancillary data from ISS and slow control data to
    produce reconstructed data for the physics analysis.
    This {\bf Second production} step is usually applied in an incremental way
    to the STD data sample, every 6 months, the time needed to produce and
    certify the calibrations. A full reprocessing of all AMS data is carried
    out periodically in case of major software major updates, providing the so
    called "pass" production. Up to 2018 there were 7 full data reproductions
    done. The last published measurements were based on the pass6 data set, but all the analyses being carried out for the next publications are based on the pass7 ones.
}
\end{itemize}

The First Production is processed at CERN on a dedicated farm of about 200
cores, whereas Monte Carlo productions, ISS data reprocessing and user data
analysis are supported by a network of computing centers (see fig.
\ref{fig:contributors}).

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.5\textwidth]{contributors.pdf}
\end{center}
\caption{AMS-02 Major Contributors to Computing Resources.}
\label{fig:contributors}
\end{figure}

Usually China and Taiwan centers are mostly devoted to Monte Carlo production,
while CERN, CNAF and FZJ Julich are the main centers for data reprocessing.
A light-weight production platform has been realized to run on different
computing centers, using different platforms. Based on perl, python and sqlite3,
it is easily deployable and allows to have a fully automated production cycle,
from job submission to monitoring, validation, transferring.


\section{CNAF contribution}
CNAF is the main computing resource for data analysis of the AMS Italian collaboration with $\sim$ 20000 HS06, $\sim$ 2 PB of storage on disk and 1 PB on tape, allocated in 2018.
A full copy of the AMS raw data is preserved on tape, while, usually, the latest production and part of the Monte Carlo sample are available on disk.
More then 30 users are routinely performing the bulk of their analysis at CNAF, transferring to local sites (i.e. their small local computing farm or their laptop) just reduced data sets or histograms.

As described in the following, during 2018, the possibility of a XRootD endpoint at CNAF has been explored. The goal is to federate, through XRootD, the $\sim$ 5 PB available for the AMS Collaboration at CERN, with the $\sim$ 2 PB at CNAF. In this picture, CNAF will be the second data center to share its disk space togheter with the one available for the collaboration, large-scale optimizing it.

\section{Data processing strategy at CNAF}

Two local queues are available for the AMS users: the default running is the
{\it ams} queue, with a CPU limit of 11520 minutes  (it allows to run 8 core
multi-thread jobs for 1 day) and a maximum of 1500 job running simultaneously,
where as for test runs the {\it ams$\_$short} queue, with high priority but a
CPU limit of 360 minutes and 100 jobs as running limit.
For data reprocessing or MC production the AMS production queue {\it ams$\_$prod},
with a CPU limit of 5760 minutes and no jobs limit, is available and accessible
only to the data production team of the international collaboration and few
experts users of the Italian team.
In fact, the {\it ams$\_$prod} queue is used within the data analysis process
to produce data streams of pre-selected events and lightweight data files with a
custom format \cite{dst} on the full AMS data statistics.
In such a way, the final analysis can easily process the reduced data set
avoiding the access to the large AMS data sample.
The data-stream and custom data files productions are usually repeated few times a year.\\

At CNAF, on the local filesystem, usually, the last data production (at the time of writing is still the pass6 but the full transfer of the pass7 is proceeding) and a sub-sample of Monte Carlo production are available.\\
%Due to the high I/O throughput from disk of AMS jobs, $\sim$ 1MB/s (1000 running
%jobs may generate $\sim$ 1GB/s of I/O throughput from disk), the AMS Collaboration verified the possibility to remotely access, in a batch job run ning , all the files of the AMS production at CERN via XRootD.
%To efficiently use all of the resources allocated at CNAF, in terms of disk and
%CPU time, AMS decided to adopt a double strategy to process AMS data files:
%lsf jobs using data stored on local filesystem and lsf jobs accessing data files
%at CERN via xrootd protocol.
%{\color{red}
%BOH PURE TUTTA STA PARTE QUA \\
%We limit the number of jobs running on local filesystem to $\sim$ 800 and use all the available computing resources to process files via xrootd, not saturating the access to local disk. Typical use of CNAF resources is shown in figure \ref{fig:input_output_net}: the number of pending (green) and running (blue) AMS jobs in the top plot and the input/output (green/blue) network traffic rate, in the bottom plot, as a function of time are displayed.
%From figure \ref{fig:input_output_net}, one can observe the increasing in the
%network traffic correlated to the number of running jobs making use of xrootd.
%}
%Currently, AMS server disk can support I/O throughput up to 40 Gb/s and we are verifying our strategy, increasing the number of lsf running on local disk and accessing external files via xrootd only when they are not locally available or if the local filesystem is going to be saturated.
In the past years, the AMS-Italy collaboration, explored the possibility to run part of the batch (LSF) jobs running at CNAF, on data accessed remotely, through XRootD, from the AMS dataset at CERN. This was completely proven to run smoothly and since a couple of years, once a dataset is not present at CNAF, due to the lack of space, the analysis is performed simply accessing these files from CERN. While usually the last `pass` is fully avalaible at CNAF, we don't have enough space to host also the corresponding Monte Carlo data samples.\\
On the contrary there are some data samples (e.g. reduced {\it streams} or reduced ntuples produced with a custom and lightest data format) tailored for the analyses being carried by the AMS-Italy members, that are not present at CERN and so, essentially, available only directly from CNAF.\\
To facilitate data analysis for users on small remote sites (i.e. the computing farms present in the various INFN branches and in the ASI headquarters) and to seamlessly integrate extra available resources (e.g. cloud resources), in 2018 the possibility of a XRootD endpoint at CNAF has been explored and is currently under test. This will allow users to read/write data on the GPFS area and it will serve as a starting point for an eventual federation with the CERN AMS XRootD endpoint.\\
Having a XRootD federation with CERN will automatically provide all the functionalities described above:
\begin{itemize}
    \item CNAF users will ``see'' a single {\it namespace} with all the AMS data files. They will access all the data files ignoring if these are present or not at CNAF: if a file is not locally avalaible the XRootD server will transparently redirect to the corresponding version at CERN;
    \item the online disk space will be optimized between CERN and CNAF: most used data sets (e.g. pass7) will be present in both the sites while less accessed ones will be present just in one of the two sites. Custom data sets, instead, will be present just at CNAF;
    \item the smaller computing centers and cloud resources, essentially without disk space, will access the complete AMS data set from the XRootD federation and will have the possibility to write their output directly at CNAF.
\end{itemize}
The potentiality of the integration of external resources (AMS-ASI computing center and cloud resources) into a single batch system will be described in more detailes in Sec.\ref{ReD}.


%\begin{figure}[h]
%\begin{center}
%\includegraphics[width=20pc]{input_output.jpg}
%\end{center}
%\caption{Number of pending (green) and running (blue) AMS jobs, in the top plot, and input (green)/output(blue) network traffic rate, on the lower plot,  as a function of time}.
%\label{fig:input_output_net}
%\end{figure}

\section{Activities in 2018}

AMS activities at CNAF in 2018 have been related to data reprocessing, Monte
Carlo production and data analysis. Those activities have produced four publications reporting the measurement of the primary and secondary component of cosmic rays from Lithium to Oxygen \cite{Aguilar:2018keu,Aguilar:2018njt} and of the fine time-structures of electron, positron, proton and Helium fluxes \cite{Aguilar:2018wmi,Aguilar:2018ons} perfomed by AMS.

%\subsection*{Monte Carlo production}

%As part of the network AMS computing centers, CNAF has been involved in the Monte Carlo campaign devoted to the study of proton, helium and light nuclei ions for AMS publications. To support Monte Carlo campaign, special LSF profile has been implemented to allow AMS users to submit multi-thread simulation jobs. The AMS collaboration in 2016 used $\sim$11000 CPU-years for MC production. In particular in 2016 the collaboration started to face one of the main physics objectives of the experiment: the search for primordial anti-matter, i.e. anti-Helium. Being the anti-Helium more rare than 1 particle over 10 million Helium particles, and so the signal/background so tiny, a large effort to produce a MC sample of the background (i.e. Helium) is needed in order to have a statistical meaning sample to search Helium particles being mis-identified as anti-Helium. A large MC Helium production, with 35 billion simulated events, corresponding to $\sim$ 6000 CPU-years, has been conducted. This effort has been shared among the various AMS collaboration production sites, including CNAF, as shown in Fig.\ref{fig:He-MC}.
%\begin{figure}[h]
%\begin{center}
%\includegraphics[trim = 145pt 270pt 145pt 270pt, clip, width=0.35\textwidth]{He-MC.pdf}
%\end{center}
%\caption{Sharing among the various production sites of the $\sim$ 6000 CPU-years needed for the anti-Helium analysis.}.
%\label{fig:He-MC}
%\end{figure}

\subsection*{Data analysis}

Different analysis are carried on by the Italian collaboration. In 2018, the CNAF resources for user analysis have been devoted to several different topic: the update, with more statistics, of the electron and positron analyses (they resulted in two PRL publications in 2019 \cite{Aguilar:2019pos,Aguilar:2019ele}), the measurement of the light nuclei abundances (that resulted in the PRL publications \cite{Aguilar:2018keu,Aguilar:2018njt}) and the study of their time variation as  well as the study of the proton and helium flux as a function of time, the deuteron abundance measurement and the antideuteron search analysis.
%The disk resources pledged in 2018, $\sim$ 2 PB, were mostly devoted to the pass7 data sample ($\sim$ 1 PB), MC data sample ($\sim$ 400 TB), selected data streams ($\sim$ 100 TB of pre-selected data used for common electron/positron, antiproton, antideuteron, proton and ion analysis) and scratch area for users.

\subsection*{Research and Development}
\label{ReD}
As mentioned above, during 2017 AMS started evaluating the technical feasibility of integrating also cloud resources (possibly seamlessly) in order to primarily benefit of external computing resources, meant as opportunistic resources. The architectural model foreseen is that all AMS data are and will be hosted at CNAF. Possible cloud compute resources should be able to remotely access data (might be caching locally for the sake of the I/O optimization) and produced data (namely output files) should be moved into the CNAF storage.\\
AMS work-flow has been successfully integrated in DODAS (Dynamic On Demand Analysis Service, a thematic service funded by the EOSC-hub European project) and the work-flow has been validated and consolidated during 2018. The success of the validation tests performed over HelixNebula Science Cloud provided resources and over Google Cloud INFN grant motivate further exploitation as well as evolution of the strategy. In total in 2018 the Italian collaboration benefited of more than 4\textit{\,k\,HS06\,yr} of opportunistic resources, that represent $\sim$ 20\% of the ones obtained from CNAF.\\
More in detail during the 2019 the plan is to consolidate the usage of the INFN on-premises cloud providers, namely Cloud@ReCaS Bari and Cloud@CNAF in the context of DODAS. Consolidation by means of improvement in managing I/O by using emerging solution for data caching as well as starting exploiting geographically distributed clusters.\\ 
The latter is about exploiting DODAS based solutions to create a single logical cluster running over any available resource provider. The desired solution is to allow user submitting jobs from e.g. CNAF provided User Interface to a single queue and allow dynamic clusters to fetch payloads in a secure and transparent (to the end user) way.\\
From a technical perspective the distributed cluster implementation will be based on HTCondor technology which is a important strategic aspect because of we expect this will allow, later on, a completely seamless integration within the batch system of the CNAF Tier 1. 
As a note, the two above mentioned activities are strictly related and the optimization of the I/O strategy will be key to the success of the distributed cluster implementation.\\ 
Another initiative started during 2017 is the geographically extension of the OpenStack to a remote site. More in detail this activity is about a geographically distributed cloud system (based on OpenStack) aiming to share and manage computing and storage resources, owned by heterogeneous cooperating entities.\\ 
The prototype has been already developed and tested. During the 2019 the plan is to complete the integration of the whole available hardware hosted at  ASI-SSDC (Space Science Data Center at the Italian Space Agency) located in Rome and starting exploiting it through DODAS. This will be one of the provider we expect to include in the geo-distributed setup above mentioned.
The goal by the end of 2019 is to bring the ASI-SSDC hosted computing resources to production.

\section*{References}

\begin{thebibliography}{9}

\bibitem{Aguilar:2018keu}
  M.~Aguilar {\it et al.} [AMS Collaboration],
  %``Precision Measurement of Cosmic-Ray Nitrogen and its Primary and Secondary Components with the Alpha Magnetic Spectrometer on the International Space Station,''
  Phys.\ Rev.\ Lett.\  {\bf 121} (2018) no.5,  051103. DOI:\url{10.1103/PhysRevLett.121.051103}
  %%CITATION = doi:10.1103/PhysRevLett.121.051103;%%
  %6 citations counted in INSPIRE as of 24 Apr 2019
  
\bibitem{Aguilar:2018wmi}
  M.~Aguilar {\it et al.} [AMS Collaboration],
  %``Observation of Fine Time Structures in the Cosmic Proton and Helium Fluxes with the Alpha Magnetic Spectrometer on the International Space Station,''
  Phys.\ Rev.\ Lett.\  {\bf 121} (2018) no.5,  051101. doi:\url{10.1103/PhysRevLett.121.051101}
  %%CITATION = doi:10.1103/PhysRevLett.121.051101;%%
  %8 citations counted in INSPIRE as of 24 Apr 2019
  
\bibitem{Aguilar:2018ons}
  M.~Aguilar {\it et al.} [AMS Collaboration],
  %``Observation of Complex Time Structures in the Cosmic-Ray Electron and Positron Fluxes with the Alpha Magnetic Spectrometer on the International Space Station,''
  Phys.\ Rev.\ Lett.\  {\bf 121} (2018) no.5,  051102. doi:\url{10.1103/PhysRevLett.121.051102}
  %%CITATION = doi:10.1103/PhysRevLett.121.051102;%%
  %10 citations counted in INSPIRE as of 24 Apr 2019
  
\bibitem{Aguilar:2018njt}
  M.~Aguilar {\it et al.} [AMS Collaboration],
  %``Observation of New Properties of Secondary Cosmic Rays Lithium, Beryllium, and Boron by the Alpha Magnetic Spectrometer on the International Space Station,''
  Phys.\ Rev.\ Lett.\  {\bf 120} (2018) no.2,  021101. doi:\url{10.1103/PhysRevLett.120.021101}
  %%CITATION = doi:10.1103/PhysRevLett.120.021101;%%
  %34 citations counted in INSPIRE as of 24 Apr 2019
  
  \bibitem{Aguilar:2019pos}
  M.~Aguilar {\it et al.} [AMS Collaboration],
  % Towards Understanding the Origin of Cosmic-Ray Positrons
  Phys.\ Rev.\ Lett.\ {\bf 122} (2019) no.4,   041102.
  doi:\url{10.1103/PhysRevLett.122.041102}
  
  \bibitem{Aguilar:2019ele}
  M.~Aguilar {\it et al.} [AMS Collaboration],
  % Towards Understanding the Origin of Cosmic-Ray Electrons
  Phys.\ Rev.\ Lett.\ {\bf 122} (2019) no.10,  101101.
  doi:\url{10.1103/PhysRevLett.122.101101}

\bibitem{dst} D. D'Urso \& M. Duranti, Journal of Physics: Conference Series, 664 (2015), 072016

%\bibitem{xrootd} http://xrootd.org.

\end{thebibliography}

\end{document}