Skip to content
Snippets Groups Projects

Ops accounts

Merged Carmelo Pellegrino requested to merge ops_accounts into site_admin
1 unresolved thread
1 file
+ 278
281
Compare changes
  • Side-by-side
  • Inline
Requirements for OpenStack based cloud infrastructures
Requirements for OpenStack-based cloud infrastructures
======================================================
This chapter discusses about the federation of resource providers
using OpenStack as cloud middleware framework.
This chapter discusses the federation of resource providers using OpenStack as
cloud middleware framework.
Required OpenStack services
---------------------------
Sites using OpenStack need to operate at least the following services:
Sites using OpenStack need to operate at least the following services:
* Keystone (authentication and authorization)
* Glance (images)
* Nova (computing)
* Cinder (block storage)
* Neutron (networking)
At the moment there are no strict requirements on the Openstack version
that can be used, but we recommend to use a fully supported version.
* Keystone (authentication and authorization)
* Glance (images)
* Nova (computing)
* Cinder (block storage)
* Neutron (networking)
At the moment there are no strict requirements on the OpenStack version that
can be used, but we recommend to use a fully supported version.
General requirements
--------------------
To enable the integration of an OpenStack site into the INFN Cloud federation,
To enable the integration of an OpenStack site into the INFN-Cloud federation,
it is necessary:
* to enable the INFN Cloud IAM instance as one of the Cloud identity provider
* to enable the `INFN-Cloud IAM instance<https://iam.cloud.infn.it>`_ as one of
the Cloud identity provider
* to enable the desired federated users
* to support the creation of INFN Cloud service instances on the local
* to support the creation of INFN-Cloud service instances on the local
resources
* to configure the needed accounting services
Access to resources
-------------------
The instantiation of services using the core services of the INFN-Cloud PaaS
is the blessed mechanism that must be supported by the resource provider.
Unless other specific agreements, the resource provider is not bound to
provide INFN-Cloud users also with
direct access to its resources at the IaaS level.
This basically means that it is not mandatory to provide direct access to the
resources to
the INFN-Cloud users through the OpenStack dashboard or through the
OpenStack APIs.
Supporting the creation of INFN Cloud service instances
The instantiation of services using the core services of the INFN-Cloud PaaS is
the blessed mechanism that must be supported by the resource provider. Unless
other specific agreements, the resource provider is not bound to provide
INFN-Cloud users also with direct access to its resources at the IaaS level.
This basically means that it is not mandatory to provide direct access to the
resources to the INFN-Cloud users through the OpenStack dashboard or through
the OpenStack APIs.
Supporting the creation of INFN-Cloud service instances
-------------------------------------------------------
INFN Cloud services can be grouped in two categories:
The INFN-Cloud services can be grouped in two categories:
* services instantiated on public networks
* services instantiated on private networks.
The service instances of the first group are exposed on the Internet and
are therefore directly accessible by their users.
The service instances of the first group are exposed on the Internet and are
therefore directly accessible by their users.
Service instances on private networks can instead be accessed by their users
Service instances on private networks can instead be accessed by their users
only through a VPN.
A site can support both types of services, or just the services of one of the
A site can support both types of services, or just the services of one of the
two categories.
Details about the configurations needed to support the two types of
services are reported below.
Details about the configurations needed to support the two types of services
are reported below.
Enabling the INFN Cloud IAM instance
Enabling the INFN-Cloud IAM instance
------------------------------------
The procedure to enable the INFN Cloud IAM is detailed in
The procedure to enable the INFN-Cloud IAM is detailed in
`this presentation <https://agenda.infn.it/event/23774/contributions/122401/attachments/76818/98889/Keystone_Auth_Federazione.pdf>`__.
Managing users
--------------
Once the INFN Cloud IAM instance is enabled, the site admin must enable
the desired federated users.
Once the INFN-Cloud IAM instance is enabled, the site admin must enable the
desired federated users.
Users in the INFN Cloud IAM are organized in groups:
The users in the INFN-Cloud IAM are organized in groups:
* Groups ``/admins/<VO>`` refer to users that can create instances on public networks
* Groups ``/priv-admins/<VO>`` refer to users that can create instances on private networks
* groups in the form of ``/admins/<VO>`` refer to users that can create
instances on public networks;
* groups in the form of ``/priv-admins/<VO>`` refer to users that can create
instances on private networks.
The site admin can enable the desired IAM groups, mapping them to local
The site admin can enable the desired IAM groups, mapping them to local
OpenStack projects, considering that:
* different IAM groups must be mapped to **different** Openstack local
projects
* different IAM groups must be mapped to **different** OpenStack local
projects
* proper isolation must be implemented between users (e.g. a user must be
prevented from deleting an instance or a volume created by another user
of the same project)
The procedure to map a IAM group into a Openstack local project is described
in
`this presentation <https://agenda.infn.it/event/23774/contributions/122401/attachments/76818/98889/Keystone_Auth_Federazione.pdf>`__.
The procedure to map a IAM group into a OpenStack local project is described in
`this presentation <https://agenda.infn.it/event/23774/contributions/122401/attachments/76818/98889/Keystone_Auth_Federazione.pdf>`__.
Service accounts
^^^^^^^^^^^^^^^^
It is mandatory to enable the ``ops`` IAM user group, which
must be mapped to a local Openstack project.
This is used to test the proper functionality
of the site, by instantiating new virtual machines, by creating
volumes, etc. Such temporary resources are deleted once the test is completed.
It is mandatory to enable the ``ops`` IAM user, which must be mapped to a local
OpenStack project. This is used to test the proper functionality of the site,
by instantiating new virtual machines, by creating volumes, etc. Such temporary
resources are deleted once the test is completed.
It is not necessary to assign floating IPs to the local project mapped to
the ``ops`` IAM user group.
It is necessary to assign **at least two floating IPs** to the local project mapped
to the ``ops`` IAM user.
The ``ops`` IAM user does also perform monitoring and accounting measurements.
Please also note that each IAM user group includes a special user
called ``monitoring`` which is used for other monitoring purposes.
In particular it is needed:
In particular it is needed by:
* the Cloud Information Provider (CIP);
* the INFN-Cloud security-scan central service, which is used to detect
possible vulnerabilities in the services deployed on INFN-Cloud resources.
* for the Cloud Information Provider;
* by the INFN Cloud security scan central service, which is used to detect possible vulnerabilities in the services deployed on INFN Cloud resources.
This user simply retrieves the list of images and flavors available
in all federated projects (for the cloud information provider) and the list of instances
(for the security scan service).
No special actions are needed by the site to enable such functionality.
This user simply retrieves the list of images and flavors available in all
federated projects (for the cloud information provider) and the list of
instances (for the security scan service). No special actions are needed by the
site to enable such functionality.
The ``ops`` user **must not** have admin privileges on any OpenStack project.
Configuring projects for services on public networks
----------------------------------------------------
A project mapped to a ``/admins/<VO>`` will be used to create
service instances on the public network.
Such project must therefore be given the needed quota of public floating
IP addresses: in general one floating IP is needed for each service instance.
To support the deployment of the high-level services and to allow their
access by the users,
the resource provider firewall has to be configured in a way to allow inbound
access
to the following ports belonging to the public IP addresses allocated for
such services:
* port 22
* port 80
* port 443
* any port greater than 1024 except for the following ones, which must be kept closed:
* 1080 (socks proxy)
* 1191 (gpfs) (udp+tcp)
* 2049, 4045, 4046, 4049, 20048, 20049 (nfs) (udp+tcp)
* 3260 (iscsi)
* 3389 (rdp)
* 5900 (vnc)
* 5800 (jvr)
* 10000 (webmin)
* 6000 to 6023 (X11)
Any other port must be kept closed, unless properly documented and authorized.
If a resource provider for some reasons needs to have other ports open, it must contact the INFN-Cloud security team (WP4)
providing all the details related with the request. This must be done using the INFN-Cloud service desk (https://servicedesk.cloud.infn.it/)
and then selecting “Technical support”. All the motivations and details related with the request must be specified in the ticket.
The INFN-Cloud security team will then start a discussion with the proponent to determine the best solution.
The public IP addresses provided by the resource center (needed for the
services instantiated by the INFN Cloud users)
will be monitored by the INFN Cloud security incident team, in order to
promptly detect possible vulnerabilities.
If a problem is found, both the user who instantiated the service and the
admin of the site where the service was instantiated are notified.
Such project must therefore be given the needed quota of public floating IP
addresses: in general one floating IP is needed for each service instance.
To support the deployment of the high-level services and to allow their access
by the users, the resource-provider firewall has to be configured to allow
inbound access to the following ports belonging to the public IP addresses
allocated for such services:
* port 22
* port 80
* port 443
* any port greater than 1024 except for the following ones, that must be kept
closed:
* 1080 (socks proxy)
* 1191 (gpfs) (udp+tcp)
* 2049, 4045, 4046, 4049, 20048, 20049 (nfs) (udp+tcp)
* 3260 (iscsi)
* 3389 (rdp)
* 5900 (vnc)
* 5800 (jvr)
* 10000 (webmin)
* 6000 to 6023 (X11)
Any other port below 1025 must be kept closed, unless properly documented and
authorized. If a resource provider for some reasons needs to have other ports
open, it must contact the INFN-Cloud security team (WP4) providing all the
details related to the request. This must be done using the INFN-Cloud
service desk (https://servicedesk.cloud.infn.it/) and then selecting "Technical
support". All the motivations and details related with the request must be
specified in the ticket. The INFN-Cloud security team will then start a
discussion with the proponent to determine the best solution.
The public IP addresses provided by the resource center (needed for the
services instantiated by the INFN-Cloud users) will be monitored by the
INFN-Cloud security incident team, in order to promptly detect possible
vulnerabilities.
If a problem is found, both the user who instantiated the service and the admin
of the site where the service was instantiated are notified.
Configuring projects for services on private networks
@@ -173,51 +166,54 @@ Configuring projects for services on private networks
A project mapped to a ``/priv-admins/<VO>`` will be used to create
service instances on the private network.
Such project doesn't need public floating IP addresses but
a special virtual machine (usually to be created on the same project)
must be configured by the site administrator.
This special instance must implement two functionality:
Such project doesn't need public floating IP addresses except one for a special
virtual machine (usually to be created on the same project) must be configured
by the site administrator.
This special instance must implement two functionalities:
* **proxy-host**, to allow the configuration of services by
the INFN Cloud PaaS layer services.
* **VPN server**, to allow users to access their services.
* **proxy-host**, to allow the configuration of services by the INFN-Cloud PaaS
layer services;
* **VPN server**, to allow users to access their services.
The VPN server must be integrated with the INFN Cloud IAM, so
that only the members of the IAM group mapped to this project are authorized
to use this VPN server.
The VPN server must be integrated with the INFN-Cloud IAM, so that only the
members of the IAM group mapped to this project are authorized to use this VPN
server.
This means that:
This means that this special virtual machine:
* this special virtual machine must have a public IP address
* this special virtual machine must have port 22 open to the following networks:
* 90.147.174.0/24
* must have a public IP address
* must have port 22 open to the following networks:
* 90.147.174.0/24
* 192.135.24.0/24
* 90.147.176.0/24
* this special virtual machine must be able to access port 22 of the virtual machines
created on this project
* this special virtual machine must have port 1194 (the one used by VPN) open.
To install and configure this special virtual machine, first of all please create a Ubuntu
20.04 instance. The flavor of this virtual machine depends on how many users
it needs to support. A VM with 2 VCPUs, 4 GB of RAM and 20 GB of root disk should be able to
support most use cases.
This machine must have a public floating IP and its network must be
configured as stated above.
Then you need to register a client in IAM. Client registration using the command line is described at `this page <https://indigo-iam.github.io/v/v1.7.2/docs/reference/api/oidc-client-registration/>`__. We detail bellow the registration using the IAM GUI.
Go to https://iam.cloud.infn.it/login and, after the login, select **MitreID Dashboard**.
Then select **Self-service client registration** to register a new client.
You need to specify the following parameters:
* **Main** --> **Client name**: write here a string to identity the client (e.g. "VPN on <server ip>")
* **Main** --> **Redirect URI(s)**: this field is mandatory but it is not used in this case (you can set it e.g. to "https://<server ip>")
* **Access** --> **Scope**: you must enable 'openid' and 'profile'
* **Access** --> **Grant Types**: you must enable 'urn:ietf:params:oauth:grant-type:device_code'
* 90.147.176.0/24
* must be able to access port 22 of all the virtual machines created in its
project
* must have port 1194 (the one used by the VPN) open.
To install and configure this special virtual machine, first of all please
create an Ubuntu 20.04 instance. The flavor of this virtual machine depends on
how many users it needs to support. A VM with 2 VCPUs, 4 GB of RAM and 20 GB of
root disk should be able to support most use cases.
This machine must have a public floating IP and its network must be configured
as stated above.
Then the site admin needs to register a client in IAM. Client registration
using the command line is described at `this page
<https://indigo-iam.github.io/v/v1.7.2/docs/reference/api/oidc-client-registration/>`_.
Details on how to perform the IAM client registration using the IAM GUI follow.
Go to https://iam.cloud.infn.it/login and, after the login, select **MitreID
Dashboard**. Then select **Self-service client registration** to register a
new client.
Specify the following parameters:
* **Main** --> **Client name**: write here a string to identity the client
(e.g. "VPN on <server ip>")
* **Main** --> **Redirect URI(s)**: this field is mandatory but it is not used
in this case (you can set it e.g. to "https://<server ip>")
* **Access** --> **Scope**: you must enable 'openid' and 'profile'
* **Access** --> **Grant Types**: you must enable
``urn:ietf:params:oauth:grant-type:device_code``
Then select **Save** to register the new client.
Please be sure to copy the following information (that are needed later):
@@ -226,11 +222,11 @@ Please be sure to copy the following information (that are needed later):
* Client Secret
* Registration Access Token
You then need to configure the VPN server (integrated with IAM). These are the relevant instructions
(also available at
`this page <https://confluence.infn.it/display/INFNCLOUD/PAM+module+for+OAuth2.0+device+flow>`__):
Configure the VPN server (integrated with IAM). These are the relevant
instructions (also available at `this page
<https://confluence.infn.it/display/INFNCLOUD/PAM+module+for+OAuth2.0+device+flow>`_):
First of all you need to install the needed packages:
First of all the site admin must install the needed packages:
::
@@ -240,11 +236,11 @@ First of all you need to install the needed packages:
sudo echo "deb http://build.openvpn.net/debian/openvpn/release/2.5 focal main" > /etc/apt/sources.list.d/openvpn-aptrepo.list
sudo apt update
Then you need to configure the PAM module for OAuth 2.0 Device flow
properly setting the file /etc/pam_oauth2_device/config.json file.
An example is provided here:
Then the site admin has to properly configure the PAM module for OAuth 2.0 Device flow by
setting the ``/etc/pam_oauth2_device/config.json`` file.
An example is provided here below:
::
::
{
"oauth": {
@@ -253,13 +249,13 @@ An example is provided here:
"secret": "CJKSDVCHyy8Sj5H2jjdpaDS4E6HLCYr3MWMG1NigFU__vmVGnW7xVFKHN6MwhoMVW3dt-pHVYJPLyalLbPekrKQ"
},
"scope": "openid profile",
"device_endpoint":"https://iam.cloud.infn.it/devicecode",
"device_endpoint":"https://iam.cloud.infn.it/devicecode",
"token_endpoint": "https://iam.cloud.infn.it/token",
"userinfo_endpoint": "https://iam.cloud.infn.it/userinfo",
"username_attribute": "preferred_username",
"groups": ["developers"]
},
"enable_email" : true,
"send_mail" : {
"smtp_server_url": "smtp://smtp-cc.infn.it:587",
@@ -276,30 +272,26 @@ An example is provided here:
}
}
You will need to modify the following fields:
The site admin has to modify the following fields:
* **id**: write here the id of the IAM client you registered
* **secret**: write here the secret of the IAM client you registered
* **secret**: write here the secret of the IAM client you registered
* **groups**: write here the name of the IAM group you are authorizing
* **smtp_username**: ask the INFN Cloud WP1 team for this information
* **smtp_password**: ask the INFN Cloud WP1 team for this information
* **smtp_username**: ask the INFN-Cloud WP1 team for this information
* **smtp_password**: ask the INFN-Cloud WP1 team for this information
* **from_address**: this must be ``<smtp_username>@infn.it``
* **from_username**: this can be set to ``noreply@cloud.infn.it``
Then to configure the OpenVPN server:
Then, to configure the OpenVPN server:
::
wget https://baltig.infn.it/infn-cloud/site-admins/-/raw/main/openvpn-infncloud-install.sh
chmod +x /openvpn-infncloud-install.sh
./openvpn-infncloud-install.sh
chmod +x /openvpn-infncloud-install.sh
./openvpn-infncloud-install.sh
Then create the file /etc/pam.d/openvpn with this content:
Then, create the ``/etc/pam.d/openvpn`` file with this content:
::
@@ -307,7 +299,8 @@ Then create the file /etc/pam.d/openvpn with this content:
account sufficient pam_oauth2_device.so
Finally edit the /etc/openvpn/server/server.conf file, adding the following lines:
Edit the ``/etc/openvpn/server/server.conf`` file, by adding the following
lines:
::
@@ -316,7 +309,7 @@ Finally edit the /etc/openvpn/server/server.conf file, adding the following line
where "172.30.6.0/24" is the private network used by the instances
An example of the /etc/openvpn/server/server.conf file is provided here:
An example of the ``/etc/openvpn/server/server.conf`` file is provided here:
::
@@ -352,17 +345,17 @@ An example of the /etc/openvpn/server/server.conf file is provided here:
hand-window 300
username-as-common-name
Finally enable and restart the OpenVPN server:
Finally, enable and restart the OpenVPN server:
::
::
systemctl restart openvpn-server@server.service
systemctl enable openvpn-server@server.service
The script openvpn-infncloud-install.sh will also create in your home directory the VPN client file (client.ovpn).
This file must be uploaded in https://baltig.infn.it/infn-cloud/vpnconfiles. Its name must
have the following format:
The ``openvpn-infncloud-install.sh`` script will also create in your home
directory the VPN client file (client.ovpn). This file must be uploaded in
https://baltig.infn.it/infn-cloud/vpnconfiles. Its name must have the following
format:
::
@@ -370,48 +363,45 @@ have the following format:
vpn-<IP address of the VPN server>-client.ovpn
Then to configure this special virtual machine also as proxy-host, you
need to create a ``im`` account, and for such account you simply need to authorize
`such SSH key <https://baltig.infn.it/infn-cloud/site-admins/-/raw/main/ssh_im_pub.key>`__.
We recommend to configure the SSH service to allow authentication only through SSH keys.
To configure this special virtual machine also as proxy-host, the site admin
needs to create a ``im`` account, that must be configured to authorize access
with `this SSH key
<https://baltig.infn.it/infn-cloud/site-admins/-/raw/main/ssh_im_pub.key>`_.
It is recommend to configure the SSH service to allow authentication only
through SSH keys.
Other network requirements
--------------------------
Firewall configuration for the Openstack services
Firewall configuration for the OpenStack services
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The endpoints exposing the OpenStack services
must be reachable at least by the core services of the INFN-Cloud PaaS and by
the
INFN-Cloud monitoring servers.
The endpoints exposing the OpenStack services must be reachable at least by the
core services of the INFN-Cloud PaaS and by the INFN-Cloud monitoring servers.
This means that they must be reachable by the following networks:
* 90.147.174.0/24
* 90.147.174.0/24
* 192.135.24.0/24
* 90.147.176.0/24
* 90.147.176.0/24
OpenStack network service quota
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Each time a service is deployed through the INFN-CLOUD services, multiple
Each time a service is deployed through the INFN-CLOUD services, multiple
security groups
are created (they are then deleted when the service is canceled).
This means that, for each federated project,
the quota values for security groups and security rules need to be increased
the quota values for security groups and security rules need to be increased
since the default values are usually
pretty low.
We suggest setting a default value of 300 security groups and 1000 security
We suggest setting a default value of 300 security groups and 1000 security
rules for the federated
projects.
Images
------
The site must provide all projects federated to INFN Cloud with
The site must provide all projects federated to INFN-Cloud with
images at least for the following operating systems:
* Ubuntu 20.04
@@ -421,9 +411,9 @@ images at least for the following operating systems:
.. IMPORTANT ::
These images must be configured so that
These images must be configured so that
password authentication is disabled: SSH access should be possible
only using key.
only using key.
Moreover instances created using these images should log the fingerprint
of the used SSH keys.
@@ -436,18 +426,16 @@ uses some weak SSH ciphers and KexAlgorithms that should be disabled.
There are at least two possible ways to address this issue:
* Providing an image where the weak protocols are disabled
* Relying on the official CentOS 7 image, but using the ``vendordata``
mechanism to properly patch the
* Relying on the official CentOS 7 image, but using the ``vendordata``
mechanism to properly patch the
sshd configuration file. This option
is discussed in :ref:`Disabling weak SSH ciphers and KexAlgorithms on CentOS7 images
is discussed in :ref:`Disabling weak SSH ciphers and KexAlgorithms on CentOS7 images
using the vendordata mechanis<sshvendordata>`.
There are no requirements on the names of the images, but
each image exposed to INFN Cloud users must be tagged with the ``infn-cloud`` tag.
Moreover for such image it is necessary to set the ``os_distro`` and ``os_version``
metadata, according to the following table:
There are no requirements on the names of the images, but each image exposed to
INFN-Cloud users must be tagged with the ``infn-cloud`` tag. Moreover for such
image it is necessary to set the ``os_distro`` and ``os_version`` metadata,
according to the following table:
+------------------+------------+------------+
@@ -479,8 +467,8 @@ they must be updated whenever there is a vulnerability to be addressed.
When an image is updated the ``infn-cloud`` tag should be added to the new
image, and removed from the old image. To remove the ``infn-cloud`` from an
When an image is updated the ``infn-cloud`` tag should be added to the new
image, and removed from the old image. To remove the ``infn-cloud`` from an
image, you can use the following command:
::
@@ -488,8 +476,9 @@ image, you can use the following command:
openstack image unset --tag infn-cloud <image-id>
The old image can then be disabled, so it won't be used in new
instances. The safest way to do that is to hide the image using one of the following commands (the first one could not work in old versions of OpenStack):
The old image can then be disabled, so it won't be used in new instances. The
safest way to do that is to hide the image using one of the following commands
(the first one could not work in old versions of OpenStack):
::
@@ -504,7 +493,7 @@ or:
.. WARNING ::
Please don't delete or deactivate an image if there are virtual machines instantiated
Please don't delete or deactivate an image if there are virtual machines instantiated
with such image, otherwise there could be problems in performing certain operations (e.g. rebuild or resize)
on the relevant instances.
@@ -515,11 +504,9 @@ Disabling weak SSH ciphers and KexAlgorithms on CentOS7 images using the vendord
.. _sshvendordata:
The idea is to
use the vendordata mechanism to execute a script,
downloaded from a web server, once a new
instance starts.
The idea is to use the vendordata mechanism to execute a script, downloaded
from a web server, once a new instance starts.
This is achieved creating the file /etc/nova/vendor-data.json
with this content:
@@ -528,33 +515,30 @@ with this content:
{"cloud-init": "#cloud-boothook\n#!/bin/bash\n/usr/bin/curl -m20 -s http://<fqdn>/cloud-vendordata-script.sh | /bin/bash\n/usr/bin/wget --timeout 10 --tries 2 -O - http://<fqdn>/cloud-vendordata-script.sh | /bin/bash\n"}
and modifying the nova.conf file of the nova api server(s) adding
these two lines in the "[api]" section:
and modifying the nova.conf file of the nova api server(s) adding these two
lines in the "[api]" section:
::
vendordata_providers = StaticJSON
vendordata_jsonfile_path = /etc/nova/vendor-data.json
"<fqdn>" in the /etc/nova/vendor-data.json file should be replaced
with the fully qualified domain name of your web server hosting the
"<fqdn>" in the /etc/nova/vendor-data.json file should be replaced with the
fully qualified domain name of your web server hosting the
"cloud-vendordata-script.sh" script.
Once a new instance starts, the script published as
http://<fqdn>/cloud-vendordata-script.sh is downloaded via curl or wget
(the hypothesis we make is that at least one of the two commands is provided)
and executed.
Once a new instance starts, the script published as
http://<fqdn>/cloud-vendordata-script.sh is downloaded via curl or wget (the
hypothesis we make is that at least one of the two commands is provided) and
executed.
The implementation of the cloud-vendordata-script.sh script, to disable weak
ssh ciphers and KexAlgorithms, can be the following one:
::
#!/bin/bash \
export SSH_CONFIG=/etc/ssh/sshd_config
export CIPHERS="Ciphers aes128-ctr,aes128-gcm@openssh.com,aes192-ctr,aes256-ctr,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com"
@@ -591,13 +575,12 @@ ssh ciphers and KexAlgorithms, can be the following one:
.. WARNING ::
This script doesn't work with RHEL 6 (or its derivates)
This script doesn't work with RHEL 6 (or its derivates)
images.
The vendordata mechanism can be used also to perform other actions, e.g.
to enable the instances to use a central log server, as
described in :ref:`Logging<logging>`.
The vendordata mechanism can be used also to perform other actions, e.g. to
enable the instances to use a central log server, as described in
:ref:`Logging<logging>`.
Logging
@@ -607,21 +590,21 @@ Logging
OpenStack logs must be kept for at least 90 days and for a maximum of 1 year.
It is also mandatory to collect and centralize the logs
of the virtual machines hosting the INFN Cloud service instances. They must
be kept at least for 6 months.
It is also mandatory to collect and centralize the logs of the virtual machines
hosting the INFN-Cloud service instances. They must be kept at least for 6
months.
This can be implemented e.g. relying on the ``vendordata`` mechanism already
introduced in section :ref:`Disabling weak SSH ciphers and KexAlgorithms on CentOS7 images
using the vendordata mechanis<sshvendordata>`.
introduced in section :ref:`Disabling weak SSH ciphers and KexAlgorithms on
CentOS7 images using the vendordata mechanis<sshvendordata>`.
The implementation of the vendordata script, to send log instances
to a central log server, can be something like:
The implementation of the vendordata script, to send log instances to a central
log server, can be something like:
::
#!/bin/bash
#!/bin/bash
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
export instance_id="`cat /var/lib/cloud/data/instance-id`"
export LOGSERVER=fdqn.log.server
@@ -637,26 +620,26 @@ to a central log server, can be something like:
Ubuntu|Debian)
echo "\$template CloudFormat, \"%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\"" > /etc/rsyslog.d/99-cloudveneto.conf
echo "*.* @${LOGSERVER};CloudFormat" >> /etc/rsyslog.d/99-cloudveneto.conf
# Restart rsyslog if needed
# Restart rsyslog if needed
pidof rsyslogd && service rsyslog restart
logger "Vendor data injected on $LINUX_DISTRIBUTION host"
;;
Scientific)
# New format. Doesn't work for CentOS6
# echo "template(name=\"CloudFormat\" type=\"string\" string= \"%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\")" > /etc/rsyslog.d/99-cloudveneto.conf
# Legacy (old) format
# New format. Doesn't work for CentOS6
# echo "template(name=\"CloudFormat\" type=\"string\" string= \"%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\")" > /etc/rsyslog.d/99-cloudveneto.conf
# Legacy (old) format
echo "\$template CloudFormat, \"%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\"" > /etc/rsyslog.d/99-cloudveneto.conf
echo "*.* @${LOGSERVER};CloudFormat" >> /etc/rsyslog.d/99-cloudveneto.conf
# Restart rsyslog if needed
# Restart rsyslog if needed
pidof rsyslogd && service rsyslog restart
;;
RedHat)
# New format. Doesn't work for CentOS6
# echo "template(name=\"CloudFormat\" type=\"string\" string= \"%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\")" > /etc/rsyslog.d/99-cloudveneto.conf
# Legacy (old) format
# New format. Doesn't work for CentOS6
# echo "template(name=\"CloudFormat\" type=\"string\" string= \"%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\")" > /etc/rsyslog.d/99-cloudveneto.conf
# Legacy (old) format
echo "\$template CloudFormat, \"%TIMESTAMP% $instance_id %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\"" > /etc/rsyslog.d/99-cloudveneto.conf
echo "*.* @${LOGSERVER};CloudFormat" >> /etc/rsyslog.d/99-cloudveneto.conf
# Restart rsyslog if needed
# Restart rsyslog if needed
pidof rsyslogd && service rsyslog restart
logger "Vendor data injected on $LINUX_DISTRIBUTION host"
;;
@@ -670,21 +653,26 @@ to a central log server, can be something like:
Debugging/Troubleshooting
-------------------------
In this section we report the instructions to be applied for some
relevant troubleshooting use cases.
In this section we report the instructions to be applied for some relevant
troubleshooting use cases.
Find the instance that connected to an external server at a given time
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Let's suppose that you need to find the instance that connected to an external server at a given time.
You will need to get this information in particular in case of a security incident.
Let's suppose that you need to find the instance that connected to an external
server at a given time. You will need to get this information in particular in
case of a security incident.
If the relevant instance had a floating IP, then please see :ref:`this section<findVMassociatedtoaFIP>`.
If the relevant instance had a floating IP, then please see :ref:`this
section<findVMassociatedtoaFIP>`.
If instead the instance had a private fixed IP, then it is not trivial to find this information, since the instance is NAT-ted (usually it uses the address
of the OpenStack virtual router, and in general multiple instances refer to the same OpenStack virtual router).
If instead the instance had a private fixed IP, then it is not trivial to find
this information, since the instance is NAT-ted (usually it uses the address of
the OpenStack virtual router, and in general multiple instances refer to the
same OpenStack virtual router).
One possible solution is to use the 'conntrack' tool to log the connections initiated from the Cloud instances. For each virtual router you will need
One possible solution is to use the 'conntrack' tool to log the connections
initiated from the Cloud instances. For each virtual router you will need
to use a command such as this one:
::
@@ -694,23 +682,26 @@ to use a command such as this one:
where:
* 1450abf3-5513-4916-aa96-254c66f96888 is the ID of Openstack router
* 1450abf3-5513-4916-aa96-254c66f96888 is the ID of OpenStack router
* 10.63.0.0/16 represents the instances that need to be monitored
Once you have found the fixed IP of the 'guilty' VM, please see :ref:`this section<findVMassociatedtoafixedIP>` to find this VM.
Once you have found the fixed IP of the 'guilty' VM, please see :ref:`this
section<findVMassociatedtoafixedIP>` to find this VM.
Find the instance associated to a certain floating IP at a given time
Find the instance associated to a certain floating IP at a given time
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. _findVMassociatedtoaFIP:
Let's suppose that we want to find the instance that was using a certain floating IP (e.g. 90.147.77.102) at a certain time.
Let's suppose that we want to find the instance that was using a certain
floating IP (e.g. 90.147.77.102) at a certain time.
Floating IP associations and disassociations are logged in the server.log file of the neutron node:
Floating IP associations and disassociations are logged in the server.log file
of the neutron node:
::
@@ -720,7 +711,8 @@ Floating IP associations and disassociations are logged in the server.log file o
2022-04-28 13:08:11.401 5895 INFO neutron.db.l3_db [req-ad02529b-3ea0-40ca-80f8-0558492314fa e237e43716fb490db5bda4b777835669 b38a0dab349e42bdbb469274b20a91b4 - default default] Floating IP 28bf4ef6-6be1-4ac8-9a7e-dab0c52396ea associated. External IP: 90.147.77.102, port: d7b764fc-5f74-4d8c-9571-de3221505483.
Now we need to find the instance associated to the specific port (e.g. d7b764fc-5f74-4d8c-9571-de322150548).
Now we need to find the instance associated to the specific port (e.g.
d7b764fc-5f74-4d8c-9571-de322150548).
If the relevant VM is still alive, you can simply query the neutron database:
::
@@ -734,18 +726,19 @@ If the relevant VM is still alive, you can simply query the neutron database:
1 row in set (0.00 sec)
If the VM was deleted in the meantime, the information can be found in the dhcp-agent.log, e.g.:
If the VM was deleted in the meantime, the information can be found in the
dhcp-agent.log, e.g.:
::
# grep d7b764fc-5f74-4d8c-9571-de3221505483 /var/log/neutron/dhcp-agent.log
2022-04-28 13:07:11.178 4263 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=, binding:profile=, binding:vif_details=, binding:vif_type=unbound, binding:vnic_type=normal, created_at=2022-04-28T11:07:10Z, description=, device_id=2303896a-61ec-4128-ace4-63672ffff8ea, device_owner=, extra_dhcp_opts=[], fixed_ips=[{'subnet_id': '1f32039f-f426-42f7-8d1a-88cd2862af11', 'ip_address': '10.63.15.74'}], id=d7b764fc-5f74-4d8c-9571-de3221505483, mac_address=fa:16:3e:c7:de:9b, name=, network=admin_state_up=True, availability_zone_hints=[], availability_zones=['nova'], description=None, id=0a1b566c-5479-44dc-8cd8-0e396605b806, ipv4_address_scope=None, ipv6_address_scope=None, mtu=1458, name=AdminTesting-wan, project_id=b38a0dab349e42bdbb469274b20a91b4, provider:network_type=gre, pro...
Another method to see which instance was using a certain floating IP is modifying the neutron database creating some triggers, as described in https://github.com/FranceGrilles/openstack-triggers.
Another method to see which instance was using a certain floating IP is
modifying the neutron database creating some triggers, as described in
https://github.com/FranceGrilles/openstack-triggers.
Then you can simply query the floatingip_actions table of the neutron database:
::
@@ -770,17 +763,17 @@ Then you can simply query the floatingip_actions table of the neutron database:
12 rows in set (0.01 sec)
Find the instance associated to a certain fixed (private) IP at a given time
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. _findVMassociatedtoafixedIP:
Le'ts suppose you want to find the VM which was using a certain private IP (e.g. 10.63.31.231) at a certain time (that instance could have been deleted in the meantime).
Le'ts suppose you want to find the VM which was using a certain private IP
(e.g. 10.63.31.231) at a certain time (that instance could have been deleted in
the meantime).
First of all, grep for that IP on the syslog of the network node for the relevant period, looking for the DHCP requests:
First of all, grep for that IP on the syslog of the network node for the
relevant period, looking for the DHCP requests:
::
@@ -832,14 +825,17 @@ Querying the nova DB, you can find the UUID of the relevant instance:
1 row in set (1.36 sec)
In this example, 9ad1947d-b7e1-43c4-8131-305cfee3ee7c is the UUID of the instance using the IP '10.63.31.231' in the considered time period.
In this example, 9ad1947d-b7e1-43c4-8131-305cfee3ee7c is the UUID of the
instance using the IP '10.63.31.231' in the considered time period.
Find the information associated to a VM
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you need to find information about a VM (even if this was cancelled in the meantime) you can simply query the 'instances' table of the nova database, e.g.:
If you need to find information about a VM (even if this was cancelled in the
meantime) you can simply query the 'instances' table of the nova database,
e.g.:
::
@@ -852,4 +848,5 @@ If you need to find information about a VM (even if this was cancelled in the me
1 row in set (0.00 sec)
If you use the 'nova-manage db archive_deleted_rows' to archive the records of the deleted instances, you will need to query the shadow_instances table.
If you use the 'nova-manage db archive_deleted_rows' to archive the records of
the deleted instances, you will need to query the shadow_instances table.
Loading