diff --git a/build.sh b/build.sh index 145a5229eb809ee80ad52dd247f0f9072ccb335f..8f4f42b7e3ae02bc1019c2c93c668c7e4e08e32c 100755 --- a/build.sh +++ b/build.sh @@ -78,7 +78,7 @@ build_from_source atlas atlas.tex build_from_source cms report-cms-feb-2019.tex tier1-jobs-2018.pdf tier1-readiness-2018.pdf link_pdf belle Cnaf-2019-5.0.pdf #build_from_source cosa cosa.tex biblio.bib beegfs.PNG -#build_from_source cnprov cnprov.tex +build_from_source cnprov cnprov.tex build_from_source cta CTA_annualreport_2018_v1.tex *.eps #build_from_source cuore cnaf_cuore.tex cnaf_cuore.bib build_from_source cupid main.tex cupid-biblio.bib diff --git a/cnaf-annual-report-2018.tex b/cnaf-annual-report-2018.tex index 070fc3148565753b4ceb42122ec4f28d73550358..33135c86c53c2b59c69cdbb0d3da6cc590886376 100644 --- a/cnaf-annual-report-2018.tex +++ b/cnaf-annual-report-2018.tex @@ -191,7 +191,7 @@ Introducing the sixth annual report of CNAF... %\ia{National ICT Services Infrastructure and Services}{ssnn1} %\ia{National ICT Services hardware and software infrastructures for Central Services}{ssnn2} \ia{The INFN Information System}{sysinfo} -%\ia{CNAF Provisioning system: On the way to Puppet 5}{cnprov} +\ia{CNAF Provisioning system: Puppet 5 upgrade}{cnprov} \cleardoublepage diff --git a/contributions/cnprov/cnprov.tex b/contributions/cnprov/cnprov.tex new file mode 100644 index 0000000000000000000000000000000000000000..cfb780a2648b13cb8f1b2c327f7707dd0b28ed18 --- /dev/null +++ b/contributions/cnprov/cnprov.tex @@ -0,0 +1,82 @@ +\documentclass[a4paper]{jpconf} +\usepackage{graphicx} +\usepackage{hyperref} +\usepackage{eurosym} + +\begin{document} +\title{CNAF Provisioning system: Puppet 5 upgrade} + +\author{ + Stefano Bovina$^1$, + Diego Michelotto$^1$, + Enrico Fattibene$^1$, + Antonio Falabella$^1$, + Andrea Chierici$^1$ +} + +\address{$^1$ INFN CNAF, Viale Berti Pichat 6/2, 40126, Bologna, Italy} + +\ead{ + stefano.bovina@cnaf.infn.it, + diego.michelotto@cnaf.infn.it, + enrico.fattibene@cnaf.infn.it, + antonio.falabella@cnaf.infn.it, + andrea.chierici@cnaf.infn.it +} + +\begin{abstract} +Since 2015 CNAF departments can take advantage of a common provisioning system based on Foreman/Puppet to install and configure heterogeneous sets of physical and virtual machines. +During 2017 and 2018, the CNAF provisioning system, preciously based on Puppet~\cite{ref:puppet} version 3, has been upgraded, since that version has reached "End-of-life" at 31/12/2016. +Due to other higher priority tasks, the start of this activity was postponed to 2017. +In this report we are going to describe activities that have been carried on in order to finalize the migration from Puppet 3 to Puppet 5. +\end{abstract} + +\section{Provisioning at CNAF} +The installation and configuration activity, in a big computing centre like CNAF, must take into account the size of the resources +(roughly a thousand nodes to manage), the heterogeneity of the systems (virtual vs physical nodes, computing nodes and different type of servers) +and the different working group in charge for their management. +To meet this challenge CNAF implemented a unique solution, adopted by all the departments, +based on two well known open source technologies: Foreman~\cite{ref:foreman} for the initial installation, and Puppet for the configuration. +\newline + +Due to the importance of this infrastructure, it is crucial to upgrade it trying to minimize the impact on production systems, so that +preventing service disruptions or broken configurations. +To achieve this, we have worked on the Puppet test suite based on RSpec~\cite{ref:rspec} and rspec-puppet~\cite{ref:rspec-puppet} tools. + + +\section{Puppet 5 upgrade: finalization} +Going from Puppet 3 to Puppet 5 implies a major software upgrade, with lots of configurations and functionality changes. +For this reason, the main activity in 2017 has been the setup of automated tests and the development of specific utilities for Puppet modules in order to prepare the migration. +During 2017 we prepared and documented a detailed procedure for the upgrade of all resources administered by the different CNAF departments and several tests were performed +in order to minimize the risks of issues in production. +At the beginning of 2018 we started the upgrade of the production environment which consisted in the following steps: + +\begin{itemize} +\item Upgrade Foreman to version 1.16.0 in order to support Puppet 5; +\item Ensuring that every client configuration contains the ca\_server entry; +\item Ensuring that every client is at version 3.8.7 in order to support Puppet 5; +\item Deploy a Puppet 5 CA as replacement of the Puppet 3 CA; +\item Deploy Puppet 5 workers (3x Puppetserver) in addition to the existing ones; +\item Upgrade the server configuration entry (from Puppet 3 to 5) on every client while keeping client at version 3.8.7; +\item Upgrade Puppet client to version 5; +\item Remove old Puppet 3 infrastructure; +\item Upgrade Puppetforge modules to a newer version (not Puppet 3 compatible). +\end{itemize} + +By the end of 2018 all the ~1500 hosts administered through the CNAF provisioning system were made able to exploit the upgraded infrastructure. + +\section{Future works} +Currently, we are finalizing the Puppetforge modules upgrade. Once all modules are updated, Foreman will be updated to the latest version and tests for Puppet 6 migration will begin immediately after. + + +\section{References} +\begin{thebibliography}{1} + \bibitem{ref:puppet} Puppet webpage: https://puppet.com/ + \bibitem{ref:foreman} The Foreman webpage: https://theforeman.org/ + \bibitem{ref:rspec} The RSpec webpage: http://rspec.info/ + \bibitem{ref:rspec-puppet} The rspec-puppet webpage: http://rspec-puppet.com/ +\end{thebibliography} + + +\end{document} + diff --git a/contributions/cnprov/cnprov.tex.bak b/contributions/cnprov/cnprov.tex.bak new file mode 100644 index 0000000000000000000000000000000000000000..833307d0ec0e9e96849f831a4742048c6def67db --- /dev/null +++ b/contributions/cnprov/cnprov.tex.bak @@ -0,0 +1,82 @@ +\documentclass[a4paper]{jpconf} +\usepackage{graphicx} +\usepackage{hyperref} +\usepackage{eurosym} + +\begin{document} +\title{CNAF Provisioning system: Puppet 5 upgrade} + +\author{ + Stefano Bovina$^1$, + Diego Michelotto$^1$, + Enrico Fattibene$^1$, + Antonio Falabella$^1$, + Andrea Chierici$^1$ +} + +\address{$^1$ INFN CNAF, Viale Berti Pichat 6/2, 40126, Bologna, Italy} + +\ead{ + stefano.bovina@cnaf.infn.it, + diego.michelotto@cnaf.infn.it, + enrico.fattibene@cnaf.infn.it, + antonio.falabella@cnaf.infn.it, + andrea.chierici@cnaf.infn.it +} + +\begin{abstract} +Since 2015 CNAF departments can take advantage of a common provisioning system based on Foreman/Puppet to install and configure heterogeneous sets of physical and virtual machines. +During 2017 and 2018, the CNAF provisioning system, priviously based on Puppet~\cite{ref:puppet} version 3, has been upgraded, since that version has reached "End-of-life" at 31/12/2016. +Due to other higher priority tasks, the start of this activity was postponed to 2017. +In this report we are going to describe activities that have been carried on in order to finalize the migration from Puppet 3 to Puppet 5. +\end{abstract} + +\section{Provisioning at CNAF} +The installation and configuration activity, in a big computing centre like CNAF, must take into account the size of the resources +(roughly a thousand nodes to manage), the heterogeneity of the systems (virtual vs physical nodes, computing nodes and different type of servers) +and the different working group in charge for their management. +To meet this challenge CNAF implemented a unique solution, adopted by all the departments, +based on two well known open source technologies: Foreman~\cite{ref:foreman} for the initial installation, and Puppet for the configuration. +\newline + +Due to the importance of this infrastructure, it is crucial to upgrade it trying to minimize the impact on production systems, so that +preventing service disruptions or broken configurations. +To achieve this, we have worked on the Puppet test suite based on RSpec~\cite{ref:rspec} and rspec-puppet~\cite{ref:rspec-puppet} tools. + + +\section{Puppet 5 upgrade: finalization} +Going from Puppet 3 to Puppet 5 implies a major software upgrade, with lots of configurations and functionality changes. +For this reason, the main activity in 2017 has been the setup of automated tests and the development of specific utilities for Puppet modules in order to prepare the migration. +During 2017 we prepared and documented a detailed procedure for the upgrade of all resources administred by the different CNAF departments and several tests were performed +in order to minimize the risks of issues in production. +At the beginning of 2018 we started the upgrade of the production environment which consisted in the following steps: + +\begin{itemize} +\item Upgrade Foreman to version 1.16.0 in order to support Puppet 5; +\item Ensuring that every client configuration contains the ca\_server entry; +\item Ensuring that every client is at version 3.8.7 in order to support Puppet 5; +\item Deploy a Puppet 5 CA as replacement of the Puppet 3 CA; +\item Deploy Puppet 5 workers (3x Puppetserver) in addition to the existing ones; +\item Upgrade the server config entry (from Puppet 3 to 5) on every client while keeping client at version 3.8.7; +\item Upgrade Puppet client to version 5; +\item Remove old Puppet 3 infrastructure; +\item Upgrade Puppetforge modules to a newer version (not Puppet 3 compatible). +\end{itemize} + +By the end of 2018 all the ~1500 hosts administered through the CNAF provisioning system were made able to exploit the upgraded infrastructure. + +\section{Future works} +Currently, we are finalizing the Puppetforge modules upgrade. Once all modules are updated, Foreman will be updated to the latest version and tests for Puppet 6 migration will begin immediately after. + + +\section{References} +\begin{thebibliography}{1} + \bibitem{ref:puppet} Puppet webpage: https://puppet.com/ + \bibitem{ref:foreman} The Foreman webpage: https://theforeman.org/ + \bibitem{ref:rspec} The RSpec webpage: http://rspec.info/ + \bibitem{ref:rspec-puppet} The rspec-puppet webpage: http://rspec-puppet.com/ +\end{thebibliography} + + +\end{document} +