Skip to content
Snippets Groups Projects
Commit 4fd11f9f authored by Alessandro Costantini's avatar Alessandro Costantini
Browse files

contribution devops: draft

parent d9557f4f
No related branches found
No related tags found
1 merge request!7Devops
Pipeline #22606 passed
Stage: build
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 1309 additions and 107 deletions
build.sh
+ 1
− 1
View file @ 4fd11f9f
......@@ -133,7 +133,7 @@ build_from_source ds_eoscpilot ds_eoscpilot.tex
build_from_source ds_eoschub ds_eoschub.tex
build_from_source ds_cloud_c ds_cloud_c.tex
build_from_source ds_infn_cc ds_infn_cc.tex
build_from_source ds_devops_pe ds_devops_pe.tex
build_from_source ds_devops_pe ds_devops_pe.tex *.png
#build_from_source cloud_b cloud_b.tex *.png *.jpg
#build_from_source cloud_c cloud_c.tex *.png *.pdf
#build_from_source cloud_d cloud_d.tex *.png
......
cnaf-annual-report-2018.tex
+ 1
− 1
View file @ 4fd11f9f
......@@ -210,7 +210,7 @@ Introducing the sixth annual report of CNAF...
%\ia{Summary of a tutorial on statistical methods}{st}
%\ia{Dynfarm: Transition to Production}{dynfarm}
%\ia{Official testing and increased compatibility for Dataclient}{dataclient}
\ia{Common software lifecycle management in external projects: Placeholder}{ds_devops_pe}
\ia{Common software lifecycle management in external projects:}{ds_devops_pe}
\ia{EOSC-hub: Placeholder}{ds_eoschub}
\ia{EOSCpilot - Interoperability aspects and results}{ds_eoscpilot}
\ia{Cloud@CNAF Management and Evolution: Placeholder}{ds_cloud_c}
......
contributions/ds_devops_pe/Artifact/ds_devops_pe.pdf 0 → 100644
+ 0
− 0
View file @ 4fd11f9f
File added
contributions/ds_devops_pe/Artifact/iopams.sty 0 → 100644
+ 87
− 0
View file @ 4fd11f9f
%%
%% This is file `iopams.sty'
%% File to include AMS fonts and extra definitions for bold greek
%% characters for use with iopart.cls
%%
\NeedsTeXFormat{LaTeX2e}
\ProvidesPackage{iopams}[1997/02/13 v1.0]
\RequirePackage{amsgen}[1995/01/01]
\RequirePackage{amsfonts}[1995/01/01]
\RequirePackage{amssymb}[1995/01/01]
\RequirePackage{amsbsy}[1995/01/01]
%
\iopamstrue % \newif\ifiopams in iopart.cls & iopbk2e.cls
% % allows optional text to be in author guidelines
%
% Bold lower case Greek letters
%
\newcommand{\balpha}{\boldsymbol{\alpha}}
\newcommand{\bbeta}{\boldsymbol{\beta}}
\newcommand{\bgamma}{\boldsymbol{\gamma}}
\newcommand{\bdelta}{\boldsymbol{\delta}}
\newcommand{\bepsilon}{\boldsymbol{\epsilon}}
\newcommand{\bzeta}{\boldsymbol{\zeta}}
\newcommand{\bfeta}{\boldsymbol{\eta}}
\newcommand{\btheta}{\boldsymbol{\theta}}
\newcommand{\biota}{\boldsymbol{\iota}}
\newcommand{\bkappa}{\boldsymbol{\kappa}}
\newcommand{\blambda}{\boldsymbol{\lambda}}
\newcommand{\bmu}{\boldsymbol{\mu}}
\newcommand{\bnu}{\boldsymbol{\nu}}
\newcommand{\bxi}{\boldsymbol{\xi}}
\newcommand{\bpi}{\boldsymbol{\pi}}
\newcommand{\brho}{\boldsymbol{\rho}}
\newcommand{\bsigma}{\boldsymbol{\sigma}}
\newcommand{\btau}{\boldsymbol{\tau}}
\newcommand{\bupsilon}{\boldsymbol{\upsilon}}
\newcommand{\bphi}{\boldsymbol{\phi}}
\newcommand{\bchi}{\boldsymbol{\chi}}
\newcommand{\bpsi}{\boldsymbol{\psi}}
\newcommand{\bomega}{\boldsymbol{\omega}}
\newcommand{\bvarepsilon}{\boldsymbol{\varepsilon}}
\newcommand{\bvartheta}{\boldsymbol{\vartheta}}
\newcommand{\bvaromega}{\boldsymbol{\varomega}}
\newcommand{\bvarrho}{\boldsymbol{\varrho}}
\newcommand{\bvarzeta}{\boldsymbol{\varsigma}} %NB really sigma
\newcommand{\bvarsigma}{\boldsymbol{\varsigma}}
\newcommand{\bvarphi}{\boldsymbol{\varphi}}
%
% Bold upright capital Greek letters
%
\newcommand{\bGamma}{\boldsymbol{\Gamma}}
\newcommand{\bDelta}{\boldsymbol{\Delta}}
\newcommand{\bTheta}{\boldsymbol{\Theta}}
\newcommand{\bLambda}{\boldsymbol{\Lambda}}
\newcommand{\bXi}{\boldsymbol{\Xi}}
\newcommand{\bPi}{\boldsymbol{\Pi}}
\newcommand{\bSigma}{\boldsymbol{\Sigma}}
\newcommand{\bUpsilon}{\boldsymbol{\Upsilon}}
\newcommand{\bPhi}{\boldsymbol{\Phi}}
\newcommand{\bPsi}{\boldsymbol{\Psi}}
\newcommand{\bOmega}{\boldsymbol{\Omega}}
%
% Bold versions of miscellaneous symbols
%
\newcommand{\bpartial}{\boldsymbol{\partial}}
\newcommand{\bell}{\boldsymbol{\ell}}
\newcommand{\bimath}{\boldsymbol{\imath}}
\newcommand{\bjmath}{\boldsymbol{\jmath}}
\newcommand{\binfty}{\boldsymbol{\infty}}
\newcommand{\bnabla}{\boldsymbol{\nabla}}
\newcommand{\bdot}{\boldsymbol{\cdot}}
%
% Symbols for caption
%
\renewcommand{\opensquare}{\mbox{$\square$}}
\renewcommand{\opentriangle}{\mbox{$\vartriangle$}}
\renewcommand{\opentriangledown}{\mbox{$\triangledown$}}
\renewcommand{\opendiamond}{\mbox{$\lozenge$}}
\renewcommand{\fullsquare}{\mbox{$\blacksquare$}}
\newcommand{\fulldiamond}{\mbox{$\blacklozenge$}}
\newcommand{\fullstar}{\mbox{$\bigstar$}}
\newcommand{\fulltriangle}{\mbox{$\blacktriangle$}}
\newcommand{\fulltriangledown}{\mbox{$\blacktriangledown$}}
\endinput
%%
%% End of file `iopams.sty'.
contributions/ds_devops_pe/Artifact/jpconf.cls 0 → 100644
+ 957
− 0
View file @ 4fd11f9f
This diff is collapsed.
contributions/ds_devops_pe/Artifact/jpconf11.clo 0 → 100644
+ 141
− 0
View file @ 4fd11f9f
%%
%% This is file `jpconf11.clo'
%%
%% This file is distributed in the hope that it will be useful,
%% but WITHOUT ANY WARRANTY; without even the implied warranty of
%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
%%
%% \CharacterTable
%% {Upper-case \A\B\C\D\E\F\G\H\I\J\K\L\M\N\O\P\Q\R\S\T\U\V\W\X\Y\Z
%% Lower-case \a\b\c\d\e\f\g\h\i\j\k\l\m\n\o\p\q\r\s\t\u\v\w\x\y\z
%% Digits \0\1\2\3\4\5\6\7\8\9
%% Exclamation \! Double quote \" Hash (number) \#
%% Dollar \$ Percent \% Ampersand \&
%% Acute accent \' Left paren \( Right paren \)
%% Asterisk \* Plus \+ Comma \,
%% Minus \- Point \. Solidus \/
%% Colon \: Semicolon \; Less than \<
%% Equals \= Greater than \> Question mark \?
%% Commercial at \@ Left bracket \[ Backslash \\
%% Right bracket \] Circumflex \^ Underscore \_
%% Grave accent \` Left brace \{ Vertical bar \|
%% Right brace \} Tilde \~}
\ProvidesFile{jpconf11.clo}[2005/05/04 v1.0 LaTeX2e file (size option)]
\renewcommand\normalsize{%
\@setfontsize\normalsize\@xipt{13}%
\abovedisplayskip 12\p@ \@plus3\p@ \@minus7\p@
\abovedisplayshortskip \z@ \@plus3\p@
\belowdisplayshortskip 6.5\p@ \@plus3.5\p@ \@minus3\p@
\belowdisplayskip \abovedisplayskip
\let\@listi\@listI}
\normalsize
\newcommand\small{%
\@setfontsize\small\@xpt{12}%
\abovedisplayskip 11\p@ \@plus3\p@ \@minus6\p@
\abovedisplayshortskip \z@ \@plus3\p@
\belowdisplayshortskip 6.5\p@ \@plus3.5\p@ \@minus3\p@
\def\@listi{\leftmargin\leftmargini
\topsep 9\p@ \@plus3\p@ \@minus5\p@
\parsep 4.5\p@ \@plus2\p@ \@minus\p@
\itemsep \parsep}%
\belowdisplayskip \abovedisplayskip}
\newcommand\footnotesize{%
% \@setfontsize\footnotesize\@xpt\@xiipt
\@setfontsize\footnotesize\@ixpt{11}%
\abovedisplayskip 10\p@ \@plus2\p@ \@minus5\p@
\abovedisplayshortskip \z@ \@plus3\p@
\belowdisplayshortskip 6\p@ \@plus3\p@ \@minus3\p@
\def\@listi{\leftmargin\leftmargini
\topsep 6\p@ \@plus2\p@ \@minus2\p@
\parsep 3\p@ \@plus2\p@ \@minus\p@
\itemsep \parsep}%
\belowdisplayskip \abovedisplayskip
}
\newcommand\scriptsize{\@setfontsize\scriptsize\@viiipt{9.5}}
\newcommand\tiny{\@setfontsize\tiny\@vipt\@viipt}
\newcommand\large{\@setfontsize\large\@xivpt{18}}
\newcommand\Large{\@setfontsize\Large\@xviipt{22}}
\newcommand\LARGE{\@setfontsize\LARGE\@xxpt{25}}
\newcommand\huge{\@setfontsize\huge\@xxvpt{30}}
\let\Huge=\huge
\if@twocolumn
\setlength\parindent{14\p@}
\else
\setlength\parindent{18\p@}
\fi
\if@letterpaper%
%\input{letmarg.tex}%
\setlength{\hoffset}{0mm}
\setlength{\marginparsep}{0mm}
\setlength{\marginparwidth}{0mm}
\setlength{\textwidth}{160mm}
\setlength{\oddsidemargin}{-0.4mm}
\setlength{\evensidemargin}{-0.4mm}
\setlength{\voffset}{0mm}
\setlength{\headheight}{8mm}
\setlength{\headsep}{5mm}
\setlength{\footskip}{0mm}
\setlength{\textheight}{230mm}
\setlength{\topmargin}{1.6mm}
\else
%\input{a4marg.tex}%
\setlength{\hoffset}{0mm}
\setlength{\marginparsep}{0mm}
\setlength{\marginparwidth}{0mm}
\setlength{\textwidth}{160mm}
\setlength{\oddsidemargin}{-0.4mm}
\setlength{\evensidemargin}{-0.4mm}
\setlength{\voffset}{0mm}
\setlength{\headheight}{8mm}
\setlength{\headsep}{5mm}
\setlength{\footskip}{0mm}
\setlength{\textheight}{230mm}
\setlength{\topmargin}{1.6mm}
\fi
\setlength\maxdepth{.5\topskip}
\setlength\@maxdepth\maxdepth
\setlength\footnotesep{8.4\p@}
\setlength{\skip\footins} {10.8\p@ \@plus 4\p@ \@minus 2\p@}
\setlength\floatsep {14\p@ \@plus 2\p@ \@minus 4\p@}
\setlength\textfloatsep {24\p@ \@plus 2\p@ \@minus 4\p@}
\setlength\intextsep {16\p@ \@plus 4\p@ \@minus 4\p@}
\setlength\dblfloatsep {16\p@ \@plus 2\p@ \@minus 4\p@}
\setlength\dbltextfloatsep{24\p@ \@plus 2\p@ \@minus 4\p@}
\setlength\@fptop{0\p@}
\setlength\@fpsep{10\p@ \@plus 1fil}
\setlength\@fpbot{0\p@}
\setlength\@dblfptop{0\p@}
\setlength\@dblfpsep{10\p@ \@plus 1fil}
\setlength\@dblfpbot{0\p@}
\setlength\partopsep{3\p@ \@plus 2\p@ \@minus 2\p@}
\def\@listI{\leftmargin\leftmargini
\parsep=\z@
\topsep=6\p@ \@plus3\p@ \@minus3\p@
\itemsep=3\p@ \@plus2\p@ \@minus1\p@}
\let\@listi\@listI
\@listi
\def\@listii {\leftmargin\leftmarginii
\labelwidth\leftmarginii
\advance\labelwidth-\labelsep
\topsep=3\p@ \@plus2\p@ \@minus\p@
\parsep=\z@
\itemsep=\parsep}
\def\@listiii{\leftmargin\leftmarginiii
\labelwidth\leftmarginiii
\advance\labelwidth-\labelsep
\topsep=\z@
\parsep=\z@
\partopsep=\z@
\itemsep=\z@}
\def\@listiv {\leftmargin\leftmarginiv
\labelwidth\leftmarginiv
\advance\labelwidth-\labelsep}
\def\@listv{\leftmargin\leftmarginv
\labelwidth\leftmarginv
\advance\labelwidth-\labelsep}
\def\@listvi {\leftmargin\leftmarginvi
\labelwidth\leftmarginvi
\advance\labelwidth-\labelsep}
\endinput
%%
%% End of file `iopart12.clo'.
contributions/ds_devops_pe/CI-tools.png 0 → 100644
+ 0
− 0
View file @ 4fd11f9f
contributions/ds_devops_pe/CI-tools.png

33.2 KiB

contributions/ds_devops_pe/ds_devops_pe.tex
+ 122
− 105
View file @ 4fd11f9f
\documentclass[a4paper]{jpconf}
\usepackage{graphicx}
\begin{document}
\title{Common software lifecycle management in external projects: placeholder}
\title{Common software lifecycle management in external projects}
\author{C. Duma$^1$, A. Costantini$^1$, D. Michelotto$^1$,
P. Orviz$^2$ and D. Salomoni$^1$}
......@@ -12,139 +12,156 @@
\begin{abstract}
This paper describes the achievements of the H2020 project INDIGO-DataCloud in the field of software lifecycle management, the Continuous Integration and Delivery systems setup to manage the new releases, as a first step towards the implementation of a DevOps approach.
This paper describes the common procedure defined and adopted in the field of software lifecycle management,
the Continuous Integration and Delivery systems setup to manage the new releases, as a first step to ensure
the quality of the provided solutions, services or components, while strengthening the collaboration between
developers and operations teams among different external projects.
In paricular, the papaer analyses the common software lifecycle management procedure adoped in two EC funded
project: eXtreme DataCloud and DEEP Hybrid DataCloud.
\end{abstract}
\section{Introduction}
Text
\section{First section}
\label{sec:release}
TOCHANGE
The software development lifecycle (SDL) process (Figure~\ref{fig:1}) in INDIGO has been supported by a continuous
software improvement process that regarded the software quality assurance, software maintenance,
including release management, support services, and the management of pilot infrastructures
needed for software integration and acceptance testing.
%\begin{figure}
% \centering
% \includegraphics[width=\textwidth]{Figure5.pdf}
% \caption{Software development lifecycle implementation}
% \label{fig:1}
%\end{figure}
A relevant activity in the software-oriented projects is the definition and implementation of the entire Software
Lifecycle Management process. As the software components envisaged by the project have a history of development
in previous successful European projects implementing different types of modern software development techniques,
the natural choice was to complement the previous, individual, Continuous Development and Integration services
with a Continuous Testing, Deployment and Monitoring as part of a DevOps approach:
\begin{itemize}
\item Continuous Testing - the activity of continuously testing the developed software in order to identify issues in
the early phases of the development. For Continuous testing, automation tools will be used. These tools enable
the QA’s for testing multiple code-bases and in parallel, to ensure that there are no flaws in the functionality. In
this activity the use of Docker containers for simulating testing environments on the fly, is also a preferred choice.
Once the code is tested, it is continuously integrated with the existing code.
\item Continuous Deployment - the activity of continuously updating production environment once new code is made
available. Here we ensure that the code is correctly deployed on all the servers. If there is any addition of functionality
or a new feature is introduced, then one should be ready to add resources according to needs. Therefore, it is also
the responsibility of the SysAdmin to scale up the servers. Since the new code is deployed on a continuous basis,
automation tools play an important role for executing tasks quickly and frequently. Puppet, Chef, SaltStack and
Ansible are some popular tools that could be used at this step. This activity represents the Configuration
Management - the process of standardising the resources configurations and enforcing their state across
infrastructures in an automated manner. The extensive use of Containerisation techniques will provide an
entire runtime environment: application/service, all its dependencies, libraries and binaries, and configuration
files needed to run it, bundled in one package - container. T3.1 will also manage the scalability testing, being
able to manage the configurations and do the deployments of any number of nodes automatically.
\item Continuous Monitoring - very crucial activity in the DevOps model of managing software lifecycle, which is
aimed at improving the quality of the software by monitoring its performance. This practice involves the participation
of the Operations team who will monitor the users’ activity to discover bugs or improper behaviour of the system.
This can also be achieved by making use of dedicated monitoring tools, which will continuously monitor the application
performance and highlight issues. Some popular tools useful in this step are Nagios, NewRelic and Sensu. These tools
help to monitor the health of the system proactively and improve productivity and increase the reliability of the
systems, reducing IT support costs. Any major issues found could be reported to the Development team to be
fixed in the continuous development phase.
\end{itemize}
Preview releases are made available for evaluation by user communities and
resource providers through the pilot infrastructures. Release
candidates are subjected to integration testing, which may include the
These DevOps activities are carried out on loop continuously until the desired product quality is achieved.
Automation will play a central role in all the activities in order to achieve a complete release automation, moving the
software from the developers through build and quality assurance checks, to deployment into integration testbeds
and finally to production sites part of the Pilot Infrastructures.
%\subsection{Software development lifecycle management}
Software lifecycle management is performed mostly via automated actions orchestrated.
\section{Software Quality Assurance and Control}
In Figure we depict the project's software lifecycle management services and
activities and their interdependencies:
Software Quality Assurance (SQA) covers the set of software engineering processes
that foster the quality and reliability in the software produced. The activities involved in this task are mainly focused on:
\begin{itemize}
\item Version Control System (VCS) - Source Code is made available through public VCS
repositories, hosted externally in GitHub repositories, guaranteeing in this
way the software openness and visibility, simplifying the exploitation beyond the
project lifetime. The INDIGO-DataCloud software is released under the Apache 2.0
software license and can be deployed on both public and private Cloud infrastructures.
\item Software quality assurence criteria and control activities and services to enable them:
\begin{itemize}
\item Continuous Integration service using {\bf Jenkins}: Service to automate the building,
packaging (where applicable) and execution of unit and functional tests of software components.
\item Code review service using GitHub: Code review of software source code is one integral part of the SQA\@. This service facilitates the code review proces. It records the
comments and allows the reviewer to verify the software modification.
\item Code metrics services using {\bf Grimoire}: To collect and visualize several metrics about the software components.
\end{itemize}
\item Software release and maintenance activities, services and supporting infrastructures
\begin{itemize}
\item A project management service using {\bf openproject.org} is made available by the
project: It provides tools such as an issue tracker, wiki, a placeholder for documents and a project management timeline.
\item Artifacts repositories for RPM and Debian packages, and Docker Hub for containers:
In INDIGO-DataCloud there are two types of artifacts, packaged software and virtual images.
The software can be downloaded from our public repository\footnote{http://repo.indigo-datacloud.eu}.
\item Release notes, installation and configuration guides, user and development manuals are made
available on {\bf GitBook}\footnote{https://indigo-dc.gitbooks.io/indigo-datacloud-releases}.
\item Bug trackers using GitHub issues tracker: Services to track issues and bugs of INDIGO-DataCloud software components.
\item Integration infrastructure: this infrastructure is composed of computing resources to support directly
the Continuous Integration service. It's the place where building and packaging of software
occurs as well as the execution of unit and functional tests. These resources are provided by INDIGO partners.
\item Testing infrastructure: this infrastructure aims to provide several types of environment. A stable environment
for users where they can preview the software and services developed by INDIGO-DataCloud, prior to its public release.
\item Preview infrastructure: where the released artifacts are deployed and made available for testing and validation by the use-cases.
\item Defining and maintaining a common SQA procedure to guide the software development efforts throughout its life cycle.
\item Formulating a representative set of metrics for the software quality control to follow up on the behaviour of the
software produced, aiming to detect and fix early deviations in the software produced.
\item Enabling a continuous integration process, eventually complemented by a continuous delivery scenario, promoting
the automation adoption for the testing, building, deployment and release activities.
\end{itemize}
In order to define the SQA process, the specific context of the software developed in the project has to be taken into account.
The following particularities characterise the corresponding development teams:
\begin{itemize}
\item Heterogeneous developer profiles: different backgrounds and different degrees of expertise.
\item Geographically distributed.
\item Different home institutes which implies different cultures, different development technologies, process and methods.
\item High turnover due to the limited duration of the projects where the grid software has been developed so far.
\item More focus on development activities, with limited resources, if any, available for quality assurance activities.
\end{itemize}
The Quality Assurance process has to take all these factors into account to define the Software Quality Assurance Plan (SQAP).
A set of "QA Policies” have also to be defined to guide the development teams towards uniform practices and processes.
These QA Policies define the main activities of the software lifecycle, such as releasing, tracking, packaging and documenting
the software carried out by the project. This is done in collaboration with development teams, making sure they are flexible
enough to co-exist as much as possible with current development methods. The SQA activities have to be monitored and controlled
to track their evolution and put in place corrective countermeasures in case of deviations.
The first INDIGO-DataCloud major release (codename {\tt MidnightBlue}) was released 1st of August 2016 (see table~\ref{tab:1} for the fact sheet). The
second INDIGO-DataCloud major release (codename {\tt ElectricIndigo}) was made publicly available on April 14th 2017 (see table~\ref{tab:2} for the fact sheet).
Finally, in order to measure and evaluate the quality of the software, a quality model have to be defined
to help in evaluating the software products and process quality. It helps to set quality goals for software products and processes.
The Quality Model has to follow the ISO/IEC 25010:2011 “Systems and software engineering - Systems and software
Quality Requirements and Evaluation (SQuaRE) - System and software quality models” [R18] to identify a set of characteristics
that need to be present in software products and processes to be able to meet the quality requirements.
\section{DevOps approach in INDIGO}
Progressive levels of automation were adopted throughout the different phases of
the INDIGO-DataCloud project software development and delivery processes.
\section{Software Maintenance and Support}
Regarding the software maintenance and support area of the software lifecycle management
the main objectives covered by the project are:
\begin{itemize}
\item To increase the quality levels of the software by contributing to the implementation and automation
of the Quality Assurance (QA) and Control procedures defined by the project.
\item To boost the software delivery process, relying on automation.
\item To emphasize the communication and feedback with/from end users, in order to guarantee adequate
requirements gathering and support.
\item To guarantee the stability of services already deployed in production and the increase of their readiness
levels, where needed.
\end{itemize}
Moreover the common practices deal with the definition of all processes and procedures regarding the software maintenance and
support, and their continuous execution:
\begin{itemize}
\item Software Maintenance - regarding software preparation \& transition from the developers to production
repositories and final users.
\item Problem Management - providing the analysis \& documentation of problems.
\item Change Management - control code, configuration changes, retirement calendars.
\item Coordination the provisioning of adequate support to released software.
\item Responsible for the release management and coordination and the maintenance of the artefacts
repositories, defining policies and release cycles.
\end{itemize}
The plan regarding the software maintenance and support management have to follow the guidelines of the
ISO/IEC 14764:2006 standard [R30], and includes a set of organizational roles and administrative roles to handle
maintenance implementation, change management and validation, software release, migration and retirement, support
and helpdesk activities.
Component releases are classified in major, minor, revision and emergency, based on the impact of the changes on the
component interface and behaviour. Requests for Change (RfC) are managed adopting a priority-driven approach,
so that the risk of compromising the stability of the software deployed in a production environment is minimized.
The User Support activity deals with the coordination of the support, to users of the software components developed
within the project and included in the main project software distributions.
\section{Services for continuous integration and SQA}
A set of tools and services are needed to support the PTs, the Software Quality Assurance, the
Continuous Integration and the software release and maintenance.
The choice of using publicly available cloud services has three main reasons:
\begin{itemize}
\item Higher public visibility and in line with project objectives for open source software
\item Provides a path to further development, support and exploitation beyond the end of the project.
\item Smaller effort needed inside the project to operate and manage those services.
\end{itemize}
The list of services needed is given below with a small description for each service and the Web link.
\subsection{Services for continuous integration and SQA}
\begin{figure}[h]
\centering
\includegraphics[width=10cm,clip]{CI-tools.png}
%\caption{The list of services.}
\label{citools}
\end{figure}
The INDIGO-DataCloud CI process is schematically shown
in Figure~\ref{fig:3}. The process, in its different steps, reflects some of
the main and important achievements of the software integration team, such as:
\begin{itemize}
\item New features are developed independently from the
production version in \textit{feature branches}. The creation of
a pull request for a specific feature branch marks the start of
the automated validation process through the execution of the
SQA jobs.
\item The SQA jobs perform the code style verification and calculate unit
and functional test coverage.
\begin{itemize}
\item The tools necessary for tackling these tests are packaged in
Docker images, available in DockerHub.
\item Each test then initiates a new container that provides a
clean environment for its execution.
\item This is an innovative approach that provides the flexibility
needed to cope with the INDIGO-DataCloud software diversity.
\end{itemize}
\item The results of the several SQA jobs are made available in the Jenkins
service which notifies back to GitHub their exit status.
\begin{itemize}
\item Only if the tests have succeeded, the source code is
validated and is ready to be merged into the production branch.
\end{itemize}
\item The last step in the workflow is the code review, where a human
review of the change is performed. After code review the source code
can be merged and becomes ready for integration and later release.
\end{itemize}
As a general rule, the described CI process must be followed by all the PTs
contributing code to INDIGO-DataCloud. However there are exceptions to this rule that fall into two main categories:
\section{DevOps adoption from user communities}
\subsection{Continuous delivery}
Continuous delivery adds, on top of the software development chain, a seamless
manufacturing of software packages ready to be deployed into production
services. Therefore, fast, frequent and small releases can be taken over thus
promoting the reliability of the software.
\subsection{DevOps adoption from user communities}
\section{Conclusions}
The experience gathered throughout the project with regards to the adoption of different DevOps
practices is not only useful and suitable for the software related to the core services in the
INDIGO-DataCloud solution, but also applicable to the development and distribution of the applications coming from the user communities.
\section{Conclusions}
Thanks to the new common solutions developed by the INDIGO project, teams of first-line
researchers in Europe are using public and private Cloud resources to get new results in Physics, Biology, Astronomy, Medicine, Humanities and other disciplines.
\section*{Acknowledgments}
DEEP-HybridDataCloud has been funded by the European Commision H2020 research and innovation program under grant agreement RIA XXXXXXX.
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment