configuration.md

Configuration

OSD memory Configuration

Changing paramenter of a running service

ceph daemon /var/run/ceph/ceph-osd.22.asok config show | grep memory
ceph daemon  osd.22 config set osd_memory_target 2147483648

or you can inject parameter with

ceph tell mon.\* injectargs '--mon-allow-pool-delete=true'

Changing paramenters permanently

Change the parameters on the admin ceph.conf file

ceph-deploy --overwrite-conf  config push <node>

Crush map

change the device class of an OSD if needed

ceph osd crush rm-device-class osd.1
ceph osd crush set-device-class nvme osd.1

Add bucket type to hierarchy:

ceph osd crush add-bucket group1 host
ceph osd crush set 0 1.0 host=group1

Define the crush map rule

ceph osd crush rule create-replicated nvme_meta default host nvme
ceph osd crush rule create-replicated hdd_data default host hdd

Manual edit crush crush map

ceph osd getcrushmap -o crush.map
crushtool -d crush.map -o decompiled.map
crushtool -c decompiled.map -o manual.map
ceph osd setcrushmap -i manual.map
ceph osd crush rule create-replicated data_custom default custom_group hdd

Pools

ceph osd pool create ceph_data  1024  replicated hdd_data
ceph osd pool set ceph_data size 2
ceph osd pool create ceph_metadata  2  replicated nvme_meta  
ceph osd pool set ceph_metadata size 2
ceph osd force-create-pg 2.0 --yes-i-really-mean-it

To change the number of placement groups in a pool

ceph osd pool set <pool name> pg_num <num>

Pool autotune

ceph osd pool autoscale-status
ceph osd pool set foo pg_autoscale_mode on

Erasure coding

ceph osd erasure-code-profile set ec-31-profile k=3 m=1 crush-failure-domain=osd crush-device-class=hdd
ceph osd pool create ec31 2000 erasure ec-31-profile
ceph osd pool set ec31 allow_ec_overwrites true
ceph osd pool application enable ec31 cephfs

ceph osd erasure-code-profile set ec-22-profile k=2 m=2 crush-failure-domain=host crush-device-class=hdd
ceph osd pool create ec22 1024 erasure ec-22-profile
ceph osd pool set ec22 allow_ec_overwrites true
ceph osd pool application enable ec22 cephfs

Filesystems

ceph fs new cephfs ceph_metadata ceph_data

Authentication

List the current authentication keys

ceph auth list

Adding a new key

ceph auth get-or-create client.1 mon 'allow r' mds 'allow rw' osd 'allow rw pool=ceph_data'

to change ACL you must enable the client to write also on the metadata (note the p on the mds)

ceph auth get-or-create client.2 mon 'allow rw' mds 'allow rwp' osd 'allow rwx pool=ceph_data, allow rwx pool=ceph_metadata'

Mount the filesystem

Copy the content of the key to client that is going to mount the fs

[client.1]
	key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXx==

Create a file /etc/ceph/client.admin with content

XXXXXXXXXXXXXXXXXXXXXXXXXXXXx==

and then issue

mount -v -t ceph 131.154.130.69:6789:/ /mnt/ceph -o name=1,secretfile=/etc/ceph/client.admin

Logging

If your OS disk is relatively full, you can accelerate log rotation by modifying the Ceph log rotation file at /etc/logrotate.d/ceph. Add a size setting after the rotation frequency to accelerate log rotation (via cronjob) if your logs exceed the size setting. For example, the default setting looks like this:

rotate 7
weekly
size 500M
compress
sharedscripts

And add a cron job:

crontab -e

like this

30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1