Skip to content
Snippets Groups Projects
Commit 3b0bedc3 authored by Doina Cristina Duma's avatar Doina Cristina Duma
Browse files

Merge branch 'cloud_c' into 'master' 

Cloud@CNAF contribution

See merge request !1
parents 622fbc2b fa40c735
No related branches found
No related tags found
1 merge request!1Cloud@CNAF contribution
Pipeline #22568 passed
Stage: build
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 1366 additions and 130 deletions
build.sh
+ 1
− 1
View file @ 3b0bedc3
......@@ -131,7 +131,7 @@ build_from_source virgo AdV_computing_CNAF.tex
#build_from_source st StatMet.tex sm2017.bib
build_from_source ds_eoscpilot ds_eoscpilot.tex
build_from_source ds_eoschub ds_eoschub.tex
build_from_source ds_cloud_c ds_cloud_c.tex
build_from_source ds_cloud_c ds_cloud_c.tex *.png
build_from_source ds_infn_cc ds_infn_cc.tex
build_from_source ds_devops_pe ds_devops_pe.tex
#build_from_source cloud_b cloud_b.tex *.png *.jpg
......
cnaf-annual-report-2018.tex
+ 1
− 1
View file @ 3b0bedc3
......@@ -213,7 +213,7 @@ Introducing the sixth annual report of CNAF...
\ia{Common software lifecycle management in external projects: Placeholder}{ds_devops_pe}
\ia{EOSC-hub: Placeholder}{ds_eoschub}
\ia{EOSCpilot - Interoperability aspects and results}{ds_eoscpilot}
\ia{Cloud@CNAF Management and Evolution: Placeholder}{ds_cloud_c}
\ia{Cloud@CNAF Management and Evolution: The flooding}{ds_cloud_c}
\ia{INFN CorporateCloud: Placeholder}{ds_infn_cc}
\ia{eXtreme DataCloud project: Advanced data management services for distributed e-infrastructures}{sdds-xdc}
\ia{DEEP-HybridDataCloud project: Hybrid services for distributed e-infrastructures}{sdds-deep}
......
contributions/ds_cloud_c/Artifact/ds_cloud_c.pdf 0 → 100644
+ 0
− 0
View file @ 3b0bedc3
File added
contributions/ds_cloud_c/Artifact/iopams.sty 0 → 100644
+ 87
− 0
View file @ 3b0bedc3
%%
%% This is file `iopams.sty'
%% File to include AMS fonts and extra definitions for bold greek
%% characters for use with iopart.cls
%%
\NeedsTeXFormat{LaTeX2e}
\ProvidesPackage{iopams}[1997/02/13 v1.0]
\RequirePackage{amsgen}[1995/01/01]
\RequirePackage{amsfonts}[1995/01/01]
\RequirePackage{amssymb}[1995/01/01]
\RequirePackage{amsbsy}[1995/01/01]
%
\iopamstrue % \newif\ifiopams in iopart.cls & iopbk2e.cls
% % allows optional text to be in author guidelines
%
% Bold lower case Greek letters
%
\newcommand{\balpha}{\boldsymbol{\alpha}}
\newcommand{\bbeta}{\boldsymbol{\beta}}
\newcommand{\bgamma}{\boldsymbol{\gamma}}
\newcommand{\bdelta}{\boldsymbol{\delta}}
\newcommand{\bepsilon}{\boldsymbol{\epsilon}}
\newcommand{\bzeta}{\boldsymbol{\zeta}}
\newcommand{\bfeta}{\boldsymbol{\eta}}
\newcommand{\btheta}{\boldsymbol{\theta}}
\newcommand{\biota}{\boldsymbol{\iota}}
\newcommand{\bkappa}{\boldsymbol{\kappa}}
\newcommand{\blambda}{\boldsymbol{\lambda}}
\newcommand{\bmu}{\boldsymbol{\mu}}
\newcommand{\bnu}{\boldsymbol{\nu}}
\newcommand{\bxi}{\boldsymbol{\xi}}
\newcommand{\bpi}{\boldsymbol{\pi}}
\newcommand{\brho}{\boldsymbol{\rho}}
\newcommand{\bsigma}{\boldsymbol{\sigma}}
\newcommand{\btau}{\boldsymbol{\tau}}
\newcommand{\bupsilon}{\boldsymbol{\upsilon}}
\newcommand{\bphi}{\boldsymbol{\phi}}
\newcommand{\bchi}{\boldsymbol{\chi}}
\newcommand{\bpsi}{\boldsymbol{\psi}}
\newcommand{\bomega}{\boldsymbol{\omega}}
\newcommand{\bvarepsilon}{\boldsymbol{\varepsilon}}
\newcommand{\bvartheta}{\boldsymbol{\vartheta}}
\newcommand{\bvaromega}{\boldsymbol{\varomega}}
\newcommand{\bvarrho}{\boldsymbol{\varrho}}
\newcommand{\bvarzeta}{\boldsymbol{\varsigma}} %NB really sigma
\newcommand{\bvarsigma}{\boldsymbol{\varsigma}}
\newcommand{\bvarphi}{\boldsymbol{\varphi}}
%
% Bold upright capital Greek letters
%
\newcommand{\bGamma}{\boldsymbol{\Gamma}}
\newcommand{\bDelta}{\boldsymbol{\Delta}}
\newcommand{\bTheta}{\boldsymbol{\Theta}}
\newcommand{\bLambda}{\boldsymbol{\Lambda}}
\newcommand{\bXi}{\boldsymbol{\Xi}}
\newcommand{\bPi}{\boldsymbol{\Pi}}
\newcommand{\bSigma}{\boldsymbol{\Sigma}}
\newcommand{\bUpsilon}{\boldsymbol{\Upsilon}}
\newcommand{\bPhi}{\boldsymbol{\Phi}}
\newcommand{\bPsi}{\boldsymbol{\Psi}}
\newcommand{\bOmega}{\boldsymbol{\Omega}}
%
% Bold versions of miscellaneous symbols
%
\newcommand{\bpartial}{\boldsymbol{\partial}}
\newcommand{\bell}{\boldsymbol{\ell}}
\newcommand{\bimath}{\boldsymbol{\imath}}
\newcommand{\bjmath}{\boldsymbol{\jmath}}
\newcommand{\binfty}{\boldsymbol{\infty}}
\newcommand{\bnabla}{\boldsymbol{\nabla}}
\newcommand{\bdot}{\boldsymbol{\cdot}}
%
% Symbols for caption
%
\renewcommand{\opensquare}{\mbox{$\square$}}
\renewcommand{\opentriangle}{\mbox{$\vartriangle$}}
\renewcommand{\opentriangledown}{\mbox{$\triangledown$}}
\renewcommand{\opendiamond}{\mbox{$\lozenge$}}
\renewcommand{\fullsquare}{\mbox{$\blacksquare$}}
\newcommand{\fulldiamond}{\mbox{$\blacklozenge$}}
\newcommand{\fullstar}{\mbox{$\bigstar$}}
\newcommand{\fulltriangle}{\mbox{$\blacktriangle$}}
\newcommand{\fulltriangledown}{\mbox{$\blacktriangledown$}}
\endinput
%%
%% End of file `iopams.sty'.
contributions/ds_cloud_c/Artifact/jpconf.cls 0 → 100644
+ 957
− 0
View file @ 3b0bedc3
This diff is collapsed.
contributions/ds_cloud_c/Artifact/jpconf11.clo 0 → 100644
+ 141
− 0
View file @ 3b0bedc3
%%
%% This is file `jpconf11.clo'
%%
%% This file is distributed in the hope that it will be useful,
%% but WITHOUT ANY WARRANTY; without even the implied warranty of
%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
%%
%% \CharacterTable
%% {Upper-case \A\B\C\D\E\F\G\H\I\J\K\L\M\N\O\P\Q\R\S\T\U\V\W\X\Y\Z
%% Lower-case \a\b\c\d\e\f\g\h\i\j\k\l\m\n\o\p\q\r\s\t\u\v\w\x\y\z
%% Digits \0\1\2\3\4\5\6\7\8\9
%% Exclamation \! Double quote \" Hash (number) \#
%% Dollar \$ Percent \% Ampersand \&
%% Acute accent \' Left paren \( Right paren \)
%% Asterisk \* Plus \+ Comma \,
%% Minus \- Point \. Solidus \/
%% Colon \: Semicolon \; Less than \<
%% Equals \= Greater than \> Question mark \?
%% Commercial at \@ Left bracket \[ Backslash \\
%% Right bracket \] Circumflex \^ Underscore \_
%% Grave accent \` Left brace \{ Vertical bar \|
%% Right brace \} Tilde \~}
\ProvidesFile{jpconf11.clo}[2005/05/04 v1.0 LaTeX2e file (size option)]
\renewcommand\normalsize{%
\@setfontsize\normalsize\@xipt{13}%
\abovedisplayskip 12\p@ \@plus3\p@ \@minus7\p@
\abovedisplayshortskip \z@ \@plus3\p@
\belowdisplayshortskip 6.5\p@ \@plus3.5\p@ \@minus3\p@
\belowdisplayskip \abovedisplayskip
\let\@listi\@listI}
\normalsize
\newcommand\small{%
\@setfontsize\small\@xpt{12}%
\abovedisplayskip 11\p@ \@plus3\p@ \@minus6\p@
\abovedisplayshortskip \z@ \@plus3\p@
\belowdisplayshortskip 6.5\p@ \@plus3.5\p@ \@minus3\p@
\def\@listi{\leftmargin\leftmargini
\topsep 9\p@ \@plus3\p@ \@minus5\p@
\parsep 4.5\p@ \@plus2\p@ \@minus\p@
\itemsep \parsep}%
\belowdisplayskip \abovedisplayskip}
\newcommand\footnotesize{%
% \@setfontsize\footnotesize\@xpt\@xiipt
\@setfontsize\footnotesize\@ixpt{11}%
\abovedisplayskip 10\p@ \@plus2\p@ \@minus5\p@
\abovedisplayshortskip \z@ \@plus3\p@
\belowdisplayshortskip 6\p@ \@plus3\p@ \@minus3\p@
\def\@listi{\leftmargin\leftmargini
\topsep 6\p@ \@plus2\p@ \@minus2\p@
\parsep 3\p@ \@plus2\p@ \@minus\p@
\itemsep \parsep}%
\belowdisplayskip \abovedisplayskip
}
\newcommand\scriptsize{\@setfontsize\scriptsize\@viiipt{9.5}}
\newcommand\tiny{\@setfontsize\tiny\@vipt\@viipt}
\newcommand\large{\@setfontsize\large\@xivpt{18}}
\newcommand\Large{\@setfontsize\Large\@xviipt{22}}
\newcommand\LARGE{\@setfontsize\LARGE\@xxpt{25}}
\newcommand\huge{\@setfontsize\huge\@xxvpt{30}}
\let\Huge=\huge
\if@twocolumn
\setlength\parindent{14\p@}
\else
\setlength\parindent{18\p@}
\fi
\if@letterpaper%
%\input{letmarg.tex}%
\setlength{\hoffset}{0mm}
\setlength{\marginparsep}{0mm}
\setlength{\marginparwidth}{0mm}
\setlength{\textwidth}{160mm}
\setlength{\oddsidemargin}{-0.4mm}
\setlength{\evensidemargin}{-0.4mm}
\setlength{\voffset}{0mm}
\setlength{\headheight}{8mm}
\setlength{\headsep}{5mm}
\setlength{\footskip}{0mm}
\setlength{\textheight}{230mm}
\setlength{\topmargin}{1.6mm}
\else
%\input{a4marg.tex}%
\setlength{\hoffset}{0mm}
\setlength{\marginparsep}{0mm}
\setlength{\marginparwidth}{0mm}
\setlength{\textwidth}{160mm}
\setlength{\oddsidemargin}{-0.4mm}
\setlength{\evensidemargin}{-0.4mm}
\setlength{\voffset}{0mm}
\setlength{\headheight}{8mm}
\setlength{\headsep}{5mm}
\setlength{\footskip}{0mm}
\setlength{\textheight}{230mm}
\setlength{\topmargin}{1.6mm}
\fi
\setlength\maxdepth{.5\topskip}
\setlength\@maxdepth\maxdepth
\setlength\footnotesep{8.4\p@}
\setlength{\skip\footins} {10.8\p@ \@plus 4\p@ \@minus 2\p@}
\setlength\floatsep {14\p@ \@plus 2\p@ \@minus 4\p@}
\setlength\textfloatsep {24\p@ \@plus 2\p@ \@minus 4\p@}
\setlength\intextsep {16\p@ \@plus 4\p@ \@minus 4\p@}
\setlength\dblfloatsep {16\p@ \@plus 2\p@ \@minus 4\p@}
\setlength\dbltextfloatsep{24\p@ \@plus 2\p@ \@minus 4\p@}
\setlength\@fptop{0\p@}
\setlength\@fpsep{10\p@ \@plus 1fil}
\setlength\@fpbot{0\p@}
\setlength\@dblfptop{0\p@}
\setlength\@dblfpsep{10\p@ \@plus 1fil}
\setlength\@dblfpbot{0\p@}
\setlength\partopsep{3\p@ \@plus 2\p@ \@minus 2\p@}
\def\@listI{\leftmargin\leftmargini
\parsep=\z@
\topsep=6\p@ \@plus3\p@ \@minus3\p@
\itemsep=3\p@ \@plus2\p@ \@minus1\p@}
\let\@listi\@listI
\@listi
\def\@listii {\leftmargin\leftmarginii
\labelwidth\leftmarginii
\advance\labelwidth-\labelsep
\topsep=3\p@ \@plus2\p@ \@minus\p@
\parsep=\z@
\itemsep=\parsep}
\def\@listiii{\leftmargin\leftmarginiii
\labelwidth\leftmarginiii
\advance\labelwidth-\labelsep
\topsep=\z@
\parsep=\z@
\partopsep=\z@
\itemsep=\z@}
\def\@listiv {\leftmargin\leftmarginiv
\labelwidth\leftmarginiv
\advance\labelwidth-\labelsep}
\def\@listv{\leftmargin\leftmarginv
\labelwidth\leftmarginv
\advance\labelwidth-\labelsep}
\def\@listvi {\leftmargin\leftmarginvi
\labelwidth\leftmarginvi
\advance\labelwidth-\labelsep}
\endinput
%%
%% End of file `iopart12.clo'.
contributions/ds_cloud_c/catc_monitoring.png 0 → 100644
+ 0
− 0
View file @ 3b0bedc3
contributions/ds_cloud_c/catc_monitoring.png

31.7 KiB

contributions/ds_cloud_c/ds_cloud_c.tex
+ 179
− 128
View file @ 3b0bedc3
\documentclass[a4paper]{jpconf}
\usepackage{graphicx}
\begin{document}
\title{Cloud@CNAF Management and Evolution: placeholder}
\title{Cloud@CNAF Management and Evolution}
\author{C. Duma$^1$, A. Costantini$^1$, D. Michelotto$^1$ and D. Salomoni$^1$}
\address{$^1$INFN Division CNAF, Bologna, Italy}
%\address{$^2$IFCA, Consejo Superior de Investigaciones Cientificas-CSIC, Santander, Spain}
\ead{ds@cnaf.infn.it}
\begin{abstract}
This paper describes the achievements of the H2020 project INDIGO-DataCloud in the field of software lifecycle management, the Continuous Integration and Delivery systems setup to manage the new releases, as a first step towards the implementation of a DevOps approach.
Cloud@CNAF is the cloud infrastructure hosted at CNAF, based on open source solutions aiming
to serve different use cases present here. The infrastructure is the result of
the collaboration of a transversal group of people from all CNAF
functional units: networking, storage, farming, national services, distributed systems.
If 2016 was for the Cloud@CNAF IaaS (Infrastructure as a Service) based on OpenStack,
a period of consolidation and improvement, 2017 was an year of consolidation and
operation ended with an extreme event - the flooding of the DataCenter, when an
aqueduct pipe located in the street nearby CNAF went broke. This event caused
down of the entire DataCenter, including the Cloud@CNAF infrastructure.This paper
presents the activities carried out throughout 2018 to ensure the functioning
of the center cloud infrastructure, that saw the its migration from CNAF to INFN-Ferrara,
starting to the re-design of the entire to cope with the limited availability of
space and weigth imposed by the new location, to the physical migration of the
racks and remote management and operation of infrastructure in order to continue
to provide high-quality services for our users and communities.
\end{abstract}
\section{Introduction}
Text
\section{First section}
\label{sec:release}
TOCHANGE
The software development lifecycle (SDL) process (Figure~\ref{fig:1}) in INDIGO has been supported by a continuous
software improvement process that regarded the software quality assurance, software maintenance,
including release management, support services, and the management of pilot infrastructures
needed for software integration and acceptance testing.
%\begin{figure}
% \centering
% \includegraphics[width=\textwidth]{Figure5.pdf}
% \caption{Software development lifecycle implementation}
% \label{fig:1}
%\end{figure}
Preview releases are made available for evaluation by user communities and
resource providers through the pilot infrastructures. Release
candidates are subjected to integration testing, which may include the
%\subsection{Software development lifecycle management}
Software lifecycle management is performed mostly via automated actions orchestrated.
In Figure we depict the project's software lifecycle management services and
activities and their interdependencies:
The main goal of Cloud@CNAF \cite{catc} project is to provide a production quality
Cloud Infrastructure for CNAF internal activities as well as national and
international projects hosted at CNAF:
\begin{itemize}
\item Version Control System (VCS) - Source Code is made available through public VCS
repositories, hosted externally in GitHub repositories, guaranteeing in this
way the software openness and visibility, simplifying the exploitation beyond the
project lifetime. The INDIGO-DataCloud software is released under the Apache 2.0
software license and can be deployed on both public and private Cloud infrastructures.
\item Software quality assurence criteria and control activities and services to enable them:
\begin{itemize}
\item Continuous Integration service using {\bf Jenkins}: Service to automate the building,
packaging (where applicable) and execution of unit and functional tests of software components.
\item Code review service using GitHub: Code review of software source code is one integral part of the SQA\@. This service facilitates the code review proces. It records the
comments and allows the reviewer to verify the software modification.
\item Code metrics services using {\bf Grimoire}: To collect and visualize several metrics about the software components.
\end{itemize}
\item Software release and maintenance activities, services and supporting infrastructures
\begin{itemize}
\item A project management service using {\bf openproject.org} is made available by the
project: It provides tools such as an issue tracker, wiki, a placeholder for documents and a project management timeline.
\item Artifacts repositories for RPM and Debian packages, and Docker Hub for containers:
In INDIGO-DataCloud there are two types of artifacts, packaged software and virtual images.
The software can be downloaded from our public repository\footnote{http://repo.indigo-datacloud.eu}.
\item Release notes, installation and configuration guides, user and development manuals are made
available on {\bf GitBook}\footnote{https://indigo-dc.gitbooks.io/indigo-datacloud-releases}.
\item Bug trackers using GitHub issues tracker: Services to track issues and bugs of INDIGO-DataCloud software components.
\item Integration infrastructure: this infrastructure is composed of computing resources to support directly
the Continuous Integration service. It's the place where building and packaging of software
occurs as well as the execution of unit and functional tests. These resources are provided by INDIGO partners.
\item Testing infrastructure: this infrastructure aims to provide several types of environment. A stable environment
for users where they can preview the software and services developed by INDIGO-DataCloud, prior to its public release.
\item Preview infrastructure: where the released artifacts are deployed and made available for testing and validation by the use-cases.
\item Internal activities
\begin{itemize}
\item Provisioning VM for CNAF departments and staff members
\item Tutorial and courses
\end{itemize}
\item National and international projects
\begin{itemize}
\item Providing VMs for experiments hosted at CNAF, like CMS, ATLAS, EEE and FAZIA
\item testbeds for testing the services developed by projects like the INDIGO-DataCloud, eXtreme-DataCLoud and DEEP-HybridDataCloud
\end{itemize}
\end{itemize}
\end{itemize}
The first INDIGO-DataCloud major release (codename {\tt MidnightBlue}) was released 1st of August 2016 (see table~\ref{tab:1} for the fact sheet). The
second INDIGO-DataCloud major release (codename {\tt ElectricIndigo}) was made publicly available on April 14th 2017 (see table~\ref{tab:2} for the fact sheet).
\section{DevOps approach in INDIGO}
Progressive levels of automation were adopted throughout the different phases of
the INDIGO-DataCloud project software development and delivery processes.
\subsection{Services for continuous integration and SQA}
The INDIGO-DataCloud CI process is schematically shown
in Figure~\ref{fig:3}. The process, in its different steps, reflects some of
the main and important achievements of the software integration team, such as:
The infrastructure made available is based on OpenStack \cite{openstack}, version Mitaka, with all the
services deployed using a High-Availability (HA) setup or in a
clustered manner (for ex. for the DBs used). During 2016 the infrastructure has been
enhanced, by adding new resources, compute and network, and its operation has been improved and guaranteed by
adding the monitoring part, improving the support, automating the maintenance activities.
Thanks to this enhancement, Cloud@CNAF was able to offer high reliable services to the users and communities who rely on such infrastructure.
At the end of 2017, on November 9th early at morning, an aqueduct pipe located in the street nearby CNAF, broke as documented in Ref. \cite{flood}.
As a result, a river of water and mud flowed towards the Tier1 data center. The level of the water did not exceeded the
threshold of safety of the waterproof doors but, due to the porosity of the external walls and the floor, it could find a way
into the data center. Both electric lines failed at about 7.10AM CET. Access to the data center was possible only
in the afternoon, after all the water had been pumped out.
As a result, the entire Tier1 data center went down, included the Cloud@CNAF infrastructure.
\section{The resource migration}
Some weeks after the flooding, has been decided to move the Cloud@CNAF core services in a different location
in order to recover the services provided for communities and experiments.
Thanks to a strong relationship, both University of Parma/INFN-Parma and INFN-Ferrara proposed to host our
core machinery and related services.
Due to the geographical proximity and the presence of Point of Presence (PoP) GARR, the
Cloud@CNAF core machinery was moved to the INFN-Ferrara location.
Unfortunately, we were not able to move all the Cloud@CNAF resources due to a limited power and weight availability in the new location.
For the above mentioned reason, the re-design of the new infrastructure has been considered.
As a first step, the services and the related machinery to move to the new - temporary - location have been selected in order to
fit the maximum power consumption and weight estimated for each of the two rooms devoted to host Cloud@CNAF services (see Table \ref{table:1} for details).
\begin{table} [ht]
\centering
\begin{tabular}{ l|c|c|c||c||c| }
\cline{2-6}
& \multicolumn{3}{c||}{Room1} & Room2 & Tot \\
\cline{2-5}
& Rack1 & Rack2 & Tot & Rack3 & \\
\hline
Power consumption (kW) & 8,88 & 4,91 & 13,79 (15) & 5,8 (7)& 19,59\\
Weight (Kg) & 201 & 151 & 352 (400Kg/mq) & 92 (400Kg/mq) & 444 \\
Occupancy (U) & 9 & 12 & 21 & 10 & 31 \\
\hline
\end{tabular}
\caption{Power consumption weight and occupancy for each Rack. In brackets, the maximum value admitted for the Room.}
\label{table:1}
\end{table}
\section{Re-design the new infrastructure}
Due to the limitations described in Table\ref{table:1} only three racks have been used to host Cloud@CNAF core service.
Among this three racks, the first hosts the storage resources, the second hosts the Openstack controller, the network
services and the GPFS cluster. The third hosts Ovirt and Openstack compute nodes, together with
some other ancillary services (see Table \ref{table:2} for details).
Rack1 and Rack2 have been connected by 2x40Gbps through our Brocade VDX switches and Rack1 and Rack3 have been connected
by 2x10Gbps through PowerConnect switches.
\begin{table} [ht]
\centering
\begin{tabular}{ c|l|l|l| }
\cline{2-4}
& \multicolumn{1}{|c|}{Rack1} & \multicolumn{1}{|c|}{Rack2} & \multicolumn{1}{|c|}{Rack3}\\
\hline
& VDX & VDX & PowerConnect x2 \\
Resources & EqualLogic & Cloud controllers & Ovirt nodes\\
and & Powervault & Cloud networks & Compute nodes\\
Services & & Gridstore & DBs nodes\\
& & Other services & Cloud UI\\
\hline
\end{tabular}
\caption{List of resources and services hosted per Rack}
\label{table:2}
\end{table}
Moreover, Rack1 is connected to PoP GARR with 1x1Gbps fiber connection to guarantee external connectivity.
A complete overview of the new infrastructure and related resource location is shown in Figure \ref{new_c_at_c}.
As depicted from the Figure \ref{new_c_at_c} and taking into account the limitations described in Table \ref{table:1}) the power consumption
has been limited up to 13,79kW in respect to Room1 (limit 15kW) and up to 5.8kW (limit 7kW) in respect to Room2.
The whole migration process (from the design to the reconfiguration of the new infrastructure) took just a business week
and after that the Cloud@CNAF infrastructure and related services where up and running, able to serve again different projects and communities.
Thanks to the experience and documentation gathered, in June 2018 - after the Tier1 returned in its production status,
Cloud@CNAF has been migrated back in less than three business days.
\section{Cloud@CNAF evolution}
Starting from the activity carried out in 2016 related to the improvements done at the infrastructure level \cite{catc}, in
2018 (after the return of the core infrastructure services due to the flooding)
the increase of the computing resources, in terms of quality and quantity, continued in order to enhance both the
services and the performance offered to users.
Thanks to such activity, during the last year the Cloud@CNAF saw a growth on the number of users and use cases
implemented in the infrastructure, in particular the number of projects increased up to 87 using approximately
1035 virtual CPUS, 1.766TB of RAM, with a total of 267 virtual machines (see Figure \ref{catc_monitor} for more details).
Among others, some of the projects that used the cloud infrastructure are:
\begin{itemize}
\item New features are developed independently from the
production version in \textit{feature branches}. The creation of
a pull request for a specific feature branch marks the start of
the automated validation process through the execution of the
SQA jobs.
\item The SQA jobs perform the code style verification and calculate unit
and functional test coverage.
\begin{itemize}
\item The tools necessary for tackling these tests are packaged in
Docker images, available in DockerHub.
\item Each test then initiates a new container that provides a
clean environment for its execution.
\item This is an innovative approach that provides the flexibility
needed to cope with the INDIGO-DataCloud software diversity.
\end{itemize}
\item The results of the several SQA jobs are made available in the Jenkins
service which notifies back to GitHub their exit status.
\begin{itemize}
\item Only if the tests have succeeded, the source code is
validated and is ready to be merged into the production branch.
\end{itemize}
\item The last step in the workflow is the code review, where a human
review of the change is performed. After code review the source code
can be merged and becomes ready for integration and later release.
\item HARMONY - Proof-of-concept under the TTLab coordination, is a project aimed at finding resourceful medicines offensive against neoplasms in hematology,
\item EEE - Extreme Energy Events - Science inside Schools (EEE), is a special research activity about the origin of cosmic rays carried out with the essential contribution of students and teachers of high schools,
\item CHNET-DHLab - Cultural heritage network of INFN for the development of virtual laboratories services,
\item USER Support - for the development of experiments dashboard and the hosting of the production instance of the dashboard, displayed on the monitor present on the CNAF hallway,
\item EOSC-hub DODAS - Temaic service for Elastic Extension of Computing Centre batch resources on external clouds,
\item Services devoted to EU projects like DEEP-HDC \cite{deep}, XDC \cite{xdc} and EOSC-pilot \cite{pilot}.
\end{itemize}
As a general rule, the described CI process must be followed by all the PTs
contributing code to INDIGO-DataCloud. However there are exceptions to this rule that fall into two main categories:
\subsection{Continuous delivery}
Continuous delivery adds, on top of the software development chain, a seamless
manufacturing of software packages ready to be deployed into production
services. Therefore, fast, frequent and small releases can be taken over thus
promoting the reliability of the software.
\subsection{DevOps adoption from user communities}
The experience gathered throughout the project with regards to the adoption of different DevOps
practices is not only useful and suitable for the software related to the core services in the
INDIGO-DataCloud solution, but also applicable to the development and distribution of the applications coming from the user communities.
\section{Conclusions}
Thanks to the new common solutions developed by the INDIGO project, teams of first-line
researchers in Europe are using public and private Cloud resources to get new results in Physics, Biology, Astronomy, Medicine, Humanities and other disciplines.
%\section*{Acknowledgments}
%eXtreme-DataCloud has been funded by the European Commision H2020 research and innovation program under grant agreement RIA XXXXXXX.
\section{Conclusions and future work}
Due to a damage in the aqueduct pipe located in the street nearby CNAF, a river of water and mud flowed towards the Tier1 data center causing the
shutdown of the entire data center. For such reason, the services and related resources hosted by Cloud@CNAF went down.
To cope with this problem, the decision to temporary migrate the core resources and services of Clud@CNAF to INFN-Ferrara has been taken and adopted.
In order to do this, a complete re-design of the entire infrastructure was needed to tackle the limitations in terms of power consumption and
weight imposed by the new location.
The joint effort and expertise of all the CNAF people and the INFN-Ferrara colleagues made possible to re-design, migrate and make operational
the Cloud@CNAF infrastructure and related hosted services in less than a business week.
Thanks to the experience and the documentation gathered, in June 2018 - after the Tier1 returned in its production status, Cloud@CNAF
has been migrated back in less than three business days.
Even with the above described problems, the Cloud@CNAF infrastructure has been maintained and evolved, giving the possibility
to the users to carry on their activities and obtain their desidered results.
For the next year new and challenging activities are planned, in particular the migration to the OpenStack Rocky version and the deployment of a new architecture distributed between
differnet functional units, Data Center and SDDS.
\begin{figure}[h]
\centering
\includegraphics[width=15cm,clip]{infn-fe23.png}
\caption{The new architecture of the Cloud@CNAF developed to cope the limitations at INFN-Ferrara.}
\label{new_c_at_c}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=12cm,clip]{catc_monitoring.png}
\caption{Cloud@CNAF monitoring and status}
\label{catc_monitor}
\end{figure}
\section{References}
\begin{thebibliography}{}
\bibitem{catc}
Cloud@CNAF - maintenance and operation, C. Duma, R. Bucchi, A. Costantini, D. Michelotto, M. Panella, D. Salomoni and G. Zizzi, CNAF Annual Report 2016, https://www.cnaf.infn.it/Annual-Report/annual-report-2016.pdf
\bibitem{openstack}
Web site: https://www.openstack.org/
\bibitem{flood}
The flood, L. dell’Agnello, CNAF Annual Report 2017, https://www.cnaf.infn.it/wp-content/uploads/2018/09/cnaf-annual-report-2017.pdf
\bibitem{deep}
Web site: https://deep-hybrid-datacloud.eu/
\bibitem{xdc}
Web site: www.extreme-datacloud.eu
\bibitem{pilot}
Web site: https://eoscpilot.eu
\end{thebibliography}
\end{document}
......
contributions/ds_cloud_c/infn-fe23.png 0 → 100644
+ 0
− 0
View file @ 3b0bedc3
contributions/ds_cloud_c/infn-fe23.png

82.5 KiB

0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment