Newer
Older
[Official documentation](https://docs.ceph.com/en/latest/install/manual-deployment/)
## ssh keys and hostnames
Ensure that all the nodes in your cluster can ssh to each other passwordless.
Add the hostnames list to the `/etc/hosts` file.
ceph-<version>-noarch.repo
ceph-<version>.repo
with the following content respectively:
```
[ceph-<version>-noarch]
name=Ceph noarch
baseurl=http://download.ceph.com/rpm-<version>/<os-version>/noarch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
priority=10
```
and
```
[ceph-<version>]
name=Ceph <version>
baseurl=http://download.ceph.com/rpm-<version>/<os-version>/$basearch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
priority=10
```
And then issue
```
yum install ceph
```
on all the nodes of the cluster.
On a clean installation the following packages will be installed:
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
cryptsetup-libs
device-mapper
device-mapper-libs
pciutils-libs
platform-python-pip
platform-python-setuptools
binutils
ceph-base
ceph-common
ceph-mds
ceph-mgr
ceph-mgr-modules-core
ceph-mon
ceph-osd
ceph-selinux
cryptsetup
device-mapper-event
device-mapper-event-libs
device-mapper-persistent-data
fmt
gperftools-libs
leveldb
libaio
libbabeltrace
libcephfs2
libconfig
libibverbs
liboath
librabbitmq
librados2
libradosstriper1
librbd1
librdkafka
librdmacm
librgw2
libstoragemgmt
libunwind
libxslt
lttng-ust
lvm2
lvm2-libs
pciutils
python3-bcrypt
python3-beautifulsoup4
python3-ceph-argparse
python3-ceph-common
python3-cephfs
python3-cheroot
python3-cherrypy
python3-jaraco
python3-jaraco-functools
python3-libstoragemgmt
python3-libstoragemgmt-clibs
python3-logutils
python3-lxml
python3-mako
python3-more-itertools
python3-pecan
python3-pip
python3-portend
python3-rados
python3-rbd
python3-rgw
python3-setuptools
python3-simplegeneric
python3-singledispatch
python3-tempora
python3-trustme
python3-waitress
python3-webencodings
python3-webob
python3-webtest
python3-werkzeug
python3-zc-lockfile
python36
rdma-core
userspace-rcu
python3-cssselect
python3-html5lib
python36
```
This guide consider an installation where user `ceph` is used and it has `sudo` privileges.
## ceph.conf
Create a cluster id with the following command
```
/usr/bin/uuidgen
4f0be998-bcbe-4267-a866-a8f0fe74c444
```
* Firts node
Login into the first node and ensure you have the folder
```
/etc/ceph
```
create a `ceph.conf` file
```
[global]
fsid = <cluster id>
mon_initial_members = <hostname1, hostname2 ...>
mon_host = <ip1 , ip2 ...>
cluster_network = <network CIDR notation>
public_network = <network CIDR notation>
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
auth_supported = cephx
```
Where you put the `fsid` previously generated, the initial monitor member as well as its ip address and the cluster network in CIDR notation. If you have an additional network to be used as public add also this one.
This file can contain many other configuration parameters that can be added afterwards. This basic one is sufficient for the first cluster deloyment.
## cephx Keys creation
[User Management](https://docs.ceph.com/en/latest/rados/operations/user-management/)
Monitor key creation
```
sudo ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
```
Create admin key
```
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
```
(Note that the name client.admin is the key for the 'client acting with admin privileges')
Generate a bootstrap-osd keyring and a client.bootstrap-osd user and add the user to the keyring
```
sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
```
(basically the bootstrap roles are used to bootstrap services and add the keys)
Add the keys to the mon keyring
```
sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
```
you can check and verify that now `/tmp/ceph.mon.keyring` contains the monitor key and the admin key appended to it as well as the bootstrap key.
Change the ownership
```
sudo chown ceph:ceph /tmp/ceph.mon.keyring
```
Create the monitor map
```
monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap
```
This command produce an output like this:
```
monmaptool: monmap file /tmp/monmap
monmaptool: set fsid to a729979a-da01-406e-8097-11dca4c6783f
monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
```
Create the directory for the monitor (it important that you do this as `ceph` user)
sudo -u ceph mkdir /var/lib/ceph/mon/{cluster-name}-{hostname}
```
Populate the monitor daemon(s) with the monitor map and keyring
```
sudo -u ceph ceph-mon [--cluster {cluster-name}] --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
```
Start the monitor (note that the systemd units are installed during package installation)
```
sudo systemctl start ceph-mon@node1
```
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
At this point you can issue
```
ceph -s
```
to check the status of cluster. If the status is `WARNING` as in this example:
```
cluster:
id: a729979a-da01-406e-8097-11dca4c6783f
health: HEALTH_WARN
1 monitors have not enabled msgr2
services:
mon: 1 daemons, quorum falabella-cloud-1 (age 9s)
mgr: no daemons active
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
```
you can enable the `msgr2` protocol with the following command:
```
ceph mon enable-msgr2
```
The status at this point should be `OK`.
## Create the OSDs
# Bluestore
Prepare and activate
```
ceph-volume lvm create --data {data-path}
ceph-volume lvm activate {ID} {FSID}
```
For example
```
ceph-volume lvm prepare --bluestore --cluster-fsid 959f6ec8-6e8c-4492-a396-7525a5108a8f --data 26-2EH87DSV-HGST-HUH728080AL4200/sdad_data --block.wal cs-001_journal/sdad_wal --block.db cs-001_journal/sdad_db
ceph-volume lvm activate --bluestore 4 f9c9e764-6646-41ee-b773-24a11252dda5
# Bluestore alternative (manual disk preparation)
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
Considering using two disks (/dev/sdb,/dev/sdc). The first one to be used for data the second one for `wal` and `db`.
Create a physical volume on the first one:
```
pvcreate /dev/sdb
```
Create the volume group
```
vgcreate disk1_data /dev/sdb
Volume group "disk1_data" successfully created
vgdisplay
--- Volume group ---
VG Name disk1_data
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size <1,82 TiB
PE Size 4,00 MiB
Total PE 476598
Alloc PE / Size 0 / 0
Free PE / Size 476598 / <1,82 TiB
VG UUID JfdKeK-35Ck-wsBF-1pvw-Uj6a-FEdf-LzDPtQ
```
Finally create the logical volume
```
lvcreate -l100%FREE -n sdb_data disk1_data
Logical volume "sdb_data" created.
[root@ds-303 manifests]# lvdisplay
--- Logical volume ---
LV Path /dev/disk1_data/sdb_data
LV Name sdb_data
VG Name disk1_data
LV UUID gFZQDt-gZ3F-w2If-Us54-ijSA-qzWT-7Uc4jE
LV Write Access read/write
LV Creation host, time ds-303.cr.cnaf.infn.it, 2020-09-30 12:22:19 +0200
LV Status available
# open 0
LV Size <1,82 TiB
Current LE 476598
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
```
Now prepare the `wal` and `db` partitions on the second disk.
```
pvcreate /dev/sdc
vgcreate disk2_journal /dev/sdc
lvcreate -L1G -n sdb_wal disk2_journal
lvcreate -L10G -n sdb_db disk2_journal
```
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
or
```
lvcreate -l 100%FREE -n sdb_db disk2_journal
```
to just use all the remaining space left.
Then to create the osd:
```
ceph-volume lvm create --bluestore --data disk1_data/vdb_data --block.db disk2_journal/vdc_db --block.waldisk2_journal/vdc_wal
```
Note that if you created the partitions with lvm the format for the volumes is `<physical volume name>/<logical volume name>`.
The command produce the following output
```
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b7c58830-cf6b-453c-b144-556b35678cff
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -h ceph:ceph /dev/disk1_data/vdb_data
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0
Running command: /usr/bin/ln -s /dev/disk1_data/vdb_data /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
stderr: 2021-02-04T10:53:08.381+0000 7f3831e99700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-02-04T10:53:08.381+0000 7f3831e99700 -1 AuthRegistry(0x7f382c0593e0) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
stderr: got monmap epoch 2
Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQAT0htgylx8HxAAfXiWoAKb73I3iiurLXKeZA==
stdout: creating /var/lib/ceph/osd/ceph-0/keyring
added entity osd.0 auth(key=AQAT0htgylx8HxAAfXiWoAKb73I3iiurLXKeZA==)
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/chown -h ceph:ceph /dev/disk2_journal/vdc_wal
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/chown -h ceph:ceph /dev/disk2_journal/vdc_db
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path /dev/disk2_journal/vdc_wal --bluestore-block-db-path /dev/disk2_journal/vdc_db --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid b7c58830-cf6b-453c-b144-556b35678cff --setuser ceph --setgroup ceph
stderr: 2021-02-04T10:53:08.606+0000 7faafc022f40 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: 2021-02-04T10:53:08.938+0000 7faafc022f40 -1 freelist read_size_meta_from_db missing size meta in DB
--> ceph-volume lvm prepare successful for: disk1_data/vdb_data
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/disk1_data/vdb_data --path /var/lib/ceph/osd/ceph-0 --no-mon-config
Running command: /usr/bin/ln -snf /dev/disk1_data/vdb_data /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/ln -snf /dev/disk2_journal/vdc_db /var/lib/ceph/osd/ceph-0/block.db
Running command: /usr/bin/chown -h ceph:ceph /dev/disk2_journal/vdc_db
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block.db
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
Running command: /usr/bin/ln -snf /dev/disk2_journal/vdc_wal /var/lib/ceph/osd/ceph-0/block.wal
Running command: /usr/bin/chown -h ceph:ceph /dev/disk2_journal/vdc_wal
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block.wal
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-b7c58830-cf6b-453c-b144-556b35678cff
stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-b7c58830-cf6b-453c-b144-556b35678cff.service → /usr/lib/systemd/system/ceph-volume@.service.
Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service → /usr/lib/systemd/system/ceph-osd@.service.
Running command: /usr/bin/systemctl start ceph-osd@0
--> ceph-volume lvm activate successful for osd ID: 0
--> ceph-volume lvm create successful for: disk1_data/vdb_data
```
Note that the command creates the proper fs structure under `/var/lib/ceph/osd/ceph-0/`
```
ls -l /var/lib/ceph/osd/ceph-0
total 52
-rw-r--r--. 1 ceph ceph 227 Feb 4 10:53 activate.monmap
lrwxrwxrwx. 1 ceph ceph 24 Feb 4 10:53 block -> /dev/disk1_data/vdb_data
lrwxrwxrwx. 1 ceph ceph 25 Feb 4 10:53 block.db -> /dev/disk2_journal/vdc_db
lrwxrwxrwx. 1 ceph ceph 26 Feb 4 10:53 block.wal -> /dev/disk2_journal/vdc_wal
-rw-------. 1 ceph ceph 2 Feb 4 10:53 bluefs
-rw-------. 1 ceph ceph 37 Feb 4 10:53 ceph_fsid
-rw-r--r--. 1 ceph ceph 37 Feb 4 10:53 fsid
-rw-------. 1 ceph ceph 55 Feb 4 10:53 keyring
-rw-------. 1 ceph ceph 8 Feb 4 10:53 kv_backend
-rw-------. 1 ceph ceph 21 Feb 4 10:53 magic
-rw-------. 1 ceph ceph 4 Feb 4 10:53 mkfs_done
-rw-------. 1 ceph ceph 41 Feb 4 10:53 osd_key
-rw-------. 1 ceph ceph 6 Feb 4 10:53 ready
-rw-------. 1 ceph ceph 3 Feb 4 10:53 require_osd_release
-rw-------. 1 ceph ceph 10 Feb 4 10:53 type
-rw-------. 1 ceph ceph 2 Feb 4 10:53 whoami
```
This folder is `tmpfs` filesystem containing symbolic links to the volumes and other files for bluestore device.
## Managers
## Metadata servers