Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result
Blame Permalink
manual_installation.md 14.27 KiB

Manual Installation

Official documentation

ssh keys and hostnames

Ensure that all the nodes in your cluster can ssh to each other passwordless. Add the hostnames list to the /etc/hosts file.

Repos and software

Create two repository:

ceph-<version>-noarch.repo
ceph-<version>.repo

with the following content respectively:

[ceph-<version>-noarch]
name=Ceph noarch
baseurl=http://download.ceph.com/rpm-<version>/<os-version>/noarch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
priority=10

and

[ceph-<version>]
name=Ceph <version>
baseurl=http://download.ceph.com/rpm-<version>/<os-version>/$basearch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
priority=10

And then issue

yum install ceph

on all the nodes of the cluster. On a clean installation the following packages will be installed:

cryptsetup-libs
device-mapper
device-mapper-libs
pciutils-libs
platform-python-pip
platform-python-setuptools
binutils
ceph-base
ceph-common
ceph-mds
ceph-mgr
ceph-mgr-modules-core
ceph-mon
ceph-osd
ceph-selinux
cryptsetup
device-mapper-event
device-mapper-event-libs
device-mapper-persistent-data
fmt
gperftools-libs
leveldb
libaio
libbabeltrace
libcephfs2
libconfig
libibverbs
liboath
librabbitmq
librados2
libradosstriper1
librbd1
librdkafka
librdmacm
librgw2
libstoragemgmt
libunwind
libxslt
lttng-ust
lvm2
lvm2-libs
pciutils
python3-bcrypt
python3-beautifulsoup4
python3-ceph-argparse
python3-ceph-common
python3-cephfs
python3-cheroot
python3-cherrypy
python3-jaraco
python3-jaraco-functools
python3-libstoragemgmt
python3-libstoragemgmt-clibs
python3-logutils
python3-lxml
python3-mako
python3-more-itertools
python3-pecan
python3-pip
python3-portend
python3-rados
python3-rbd
python3-rgw
python3-setuptools
python3-simplegeneric
python3-singledispatch
python3-tempora
python3-trustme
python3-waitress
python3-webencodings
python3-webob
python3-webtest
python3-werkzeug
python3-zc-lockfile
python36
rdma-core
userspace-rcu
python3-cssselect
python3-html5lib
python36

This guide consider an installation where user ceph is used and it has sudo privileges.

ceph.conf

Create a cluster id with the following command

/usr/bin/uuidgen
4f0be998-bcbe-4267-a866-a8f0fe74c444
  • Firts node Login into the first node and ensure you have the folder
/etc/ceph

create a ceph.conf file

[global]
fsid = <cluster id>
mon_initial_members = <hostname1, hostname2 ...>
mon_host = <ip1 , ip2 ...>
cluster_network = <network CIDR notation>
public_network = <network CIDR notation>
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
auth_supported = cephx

Where you put the fsid previously generated, the initial monitor member as well as its ip address and the cluster network in CIDR notation. If you have an additional network to be used as public add also this one. This file can contain many other configuration parameters that can be added afterwards. This basic one is sufficient for the first cluster deloyment.

cephx Keys creation

For a reference on the user management: User Management Monitor key creation

sudo ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'

Create admin key

sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'

(Note that the name client.admin is the key for the 'client acting with admin privileges')

Generate a bootstrap-osd keyring and a client.bootstrap-osd user and add the user to the keyring

sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'

(basically the bootstrap roles are used to bootstrap services and add the keys) Add the keys to the mon keyring

sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring

you can check and verify that now /tmp/ceph.mon.keyring contains the monitor key and the admin key appended to it as well as the bootstrap key. Change the ownership

sudo chown ceph:ceph /tmp/ceph.mon.keyring

Create the monitor map

monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap

This command produce an output like this:

monmaptool: monmap file /tmp/monmap
monmaptool: set fsid to a729979a-da01-406e-8097-11dca4c6783f
monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)

Create the directory for the monitor (it important that you do this as ceph user)

sudo -u ceph mkdir /var/lib/ceph/mon/{cluster-name}-{hostname}

Populate the monitor daemon(s) with the monitor map and keyring

sudo -u ceph ceph-mon [--cluster {cluster-name}] --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

Start the monitor (note that the systemd units are installed during package installation)

sudo systemctl start ceph-mon@node1

At this point you can issue

ceph -s

to check the status of cluster. If the status is WARNING as in this example:

  cluster:
    id:     a729979a-da01-406e-8097-11dca4c6783f
    health: HEALTH_WARN
            1 monitors have not enabled msgr2
 
  services:
    mon: 1 daemons, quorum falabella-cloud-1 (age 9s)
    mgr: no daemons active
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

you can enable the msgr2 protocol with the following command:

ceph mon enable-msgr2

The status at this point should be OK.

Create the OSDs

Note that if the new OSD are on different node you have to copy the configuration file the admin key from the first node

ls -l /etc/ceph/
ceph.client.admin.keyring
ceph.conf

Bluestore

Prepare and activate

ceph-volume lvm create --data {data-path}
ceph-volume lvm activate {ID} {FSID}

For example

ceph-volume lvm prepare --bluestore --cluster-fsid 959f6ec8-6e8c-4492-a396-7525a5108a8f --data 26-2EH87DSV-HGST-HUH728080AL4200/sdad_data --block.wal cs-001_journal/sdad_wal --block.db cs-001_journal/sdad_db
ceph-volume lvm activate --bluestore 4 f9c9e764-6646-41ee-b773-24a11252dda5

Bluestore alternative (manual disk preparation)

Considering using two disks (/dev/sdb,/dev/sdc). The first one to be used for data the second one for wal and db. Create a physical volume on the first one:

pvcreate /dev/sdb

Create the volume group

vgcreate disk1_data /dev/sdb
  Volume group "disk1_data" successfully created
vgdisplay 
  --- Volume group ---
  VG Name               disk1_data
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <1,82 TiB
  PE Size               4,00 MiB
  Total PE              476598
  Alloc PE / Size       0 / 0   
  Free  PE / Size       476598 / <1,82 TiB
  VG UUID               JfdKeK-35Ck-wsBF-1pvw-Uj6a-FEdf-LzDPtQ

Finally create the logical volume

lvcreate -l100%FREE -n sdb_data disk1_data
  Logical volume "sdb_data" created.
[root@ds-303 manifests]# lvdisplay 
  --- Logical volume ---
  LV Path                /dev/disk1_data/sdb_data
  LV Name                sdb_data
  VG Name                disk1_data
  LV UUID                gFZQDt-gZ3F-w2If-Us54-ijSA-qzWT-7Uc4jE
  LV Write Access        read/write
  LV Creation host, time ds-303.cr.cnaf.infn.it, 2020-09-30 12:22:19 +0200
  LV Status              available
  # open                 0
  LV Size                <1,82 TiB
  Current LE             476598
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

Now prepare the wal and db partitions on the second disk.

pvcreate /dev/sdc
vgcreate disk2_journal /dev/sdc
lvcreate -L1G -n sdb_wal disk2_journal
lvcreate -L10G -n sdb_db disk2_journal

or

lvcreate -l 100%FREE -n sdb_db disk2_journal

to just use all the remaining space left. Then to create the osd:

ceph-volume lvm create --bluestore --data disk1_data/vdb_data --block.db disk2_journal/vdc_db --block.waldisk2_journal/vdc_wal

Note that if you created the partitions with lvm the format for the volumes is <physical volume name>/<logical volume name>. The command produce the following output

Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b7c58830-cf6b-453c-b144-556b35678cff
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -h ceph:ceph /dev/disk1_data/vdb_data
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0
Running command: /usr/bin/ln -s /dev/disk1_data/vdb_data /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
 stderr: 2021-02-04T10:53:08.381+0000 7f3831e99700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-02-04T10:53:08.381+0000 7f3831e99700 -1 AuthRegistry(0x7f382c0593e0) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
 stderr: got monmap epoch 2
Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQAT0htgylx8HxAAfXiWoAKb73I3iiurLXKeZA==
 stdout: creating /var/lib/ceph/osd/ceph-0/keyring
added entity osd.0 auth(key=AQAT0htgylx8HxAAfXiWoAKb73I3iiurLXKeZA==)
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/chown -h ceph:ceph /dev/disk2_journal/vdc_wal
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/chown -h ceph:ceph /dev/disk2_journal/vdc_db
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path /dev/disk2_journal/vdc_wal --bluestore-block-db-path /dev/disk2_journal/vdc_db --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid b7c58830-cf6b-453c-b144-556b35678cff --setuser ceph --setgroup ceph
 stderr: 2021-02-04T10:53:08.606+0000 7faafc022f40 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
 stderr: 2021-02-04T10:53:08.938+0000 7faafc022f40 -1 freelist read_size_meta_from_db missing size meta in DB
--> ceph-volume lvm prepare successful for: disk1_data/vdb_data
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/disk1_data/vdb_data --path /var/lib/ceph/osd/ceph-0 --no-mon-config
Running command: /usr/bin/ln -snf /dev/disk1_data/vdb_data /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/ln -snf /dev/disk2_journal/vdc_db /var/lib/ceph/osd/ceph-0/block.db
Running command: /usr/bin/chown -h ceph:ceph /dev/disk2_journal/vdc_db
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block.db
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
Running command: /usr/bin/ln -snf /dev/disk2_journal/vdc_wal /var/lib/ceph/osd/ceph-0/block.wal
Running command: /usr/bin/chown -h ceph:ceph /dev/disk2_journal/vdc_wal
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block.wal
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-b7c58830-cf6b-453c-b144-556b35678cff
 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-b7c58830-cf6b-453c-b144-556b35678cff.service → /usr/lib/systemd/system/ceph-volume@.service.
Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service → /usr/lib/systemd/system/ceph-osd@.service.
Running command: /usr/bin/systemctl start ceph-osd@0
--> ceph-volume lvm activate successful for osd ID: 0
--> ceph-volume lvm create successful for: disk1_data/vdb_data

Note that the command creates the proper fs structure under /var/lib/ceph/osd/ceph-0/

ls -l /var/lib/ceph/osd/ceph-0
total 52
-rw-r--r--. 1 ceph ceph 227 Feb  4 10:53 activate.monmap
lrwxrwxrwx. 1 ceph ceph  24 Feb  4 10:53 block -> /dev/disk1_data/vdb_data
lrwxrwxrwx. 1 ceph ceph  25 Feb  4 10:53 block.db -> /dev/disk2_journal/vdc_db
lrwxrwxrwx. 1 ceph ceph  26 Feb  4 10:53 block.wal -> /dev/disk2_journal/vdc_wal
-rw-------. 1 ceph ceph   2 Feb  4 10:53 bluefs
-rw-------. 1 ceph ceph  37 Feb  4 10:53 ceph_fsid
-rw-r--r--. 1 ceph ceph  37 Feb  4 10:53 fsid
-rw-------. 1 ceph ceph  55 Feb  4 10:53 keyring
-rw-------. 1 ceph ceph   8 Feb  4 10:53 kv_backend
-rw-------. 1 ceph ceph  21 Feb  4 10:53 magic
-rw-------. 1 ceph ceph   4 Feb  4 10:53 mkfs_done
-rw-------. 1 ceph ceph  41 Feb  4 10:53 osd_key
-rw-------. 1 ceph ceph   6 Feb  4 10:53 ready
-rw-------. 1 ceph ceph   3 Feb  4 10:53 require_osd_release
-rw-------. 1 ceph ceph  10 Feb  4 10:53 type
-rw-------. 1 ceph ceph   2 Feb  4 10:53 whoami

This folder is tmpfs filesystem containing symbolic links to the volumes and other files for bluestore device.

Managers

Metadata servers