To perform maintenance on Ceph that brings Ceph down, all VMs must be stopped beforehand and restarted after maintenance has been finished.
Prerequisite
Have the following command line tools available:
bash
curl
jq
- lightweight and flexible command-line JSON processorxargs
As an admin user, create an API key and store it to ADMIN_APIKEY
environment variable. E.g. export ADMIN_APIKEY=x4g...
Store the API hostname to API_HOST environment variable. E.g. export API_HOST=api.pilw.io
Step by Step Guide
Mark all hypervisors as not accepting workloads so no new resources can be created.
curl -sS -X GET -H "apikey: $ADMIN_APIKEY" https://$API_HOST/v1/base-operator/host/list | \ jq -r '.[]|.uuid' | \ xargs -n 1 -I {} curl -X PUT -H "apikey: $ADMIN_APIKEY" https://$API_HOST/v1/base-operator/admin/host_flags -d uuid={} -d is_accepting_workloads=0
Get a list of all currently running VMs, store the result to a file.
curl -sS -H "apikey: $ADMIN_APIKEY" -X GET https://$API_HOST/v1/user-resource/vm/all?status=running > running_vms.json
Stop all running VMs. This can take time. Not all VMs might agree to stop. These must be stopped forcefully.
cat running_vms.json | jq -r '.[]|.uuid' | \ xargs -n 1 -I {} curl -X POST -H "apikey: $ADMIN_APIKEY" https://$API_HOST/v1/.../stop -d uuid={}
Check the responses, not all VMs might agree to stop. These must be stopped forcefully.
cat running_vms.json | jq -r '.[]|.uuid' | \ xargs -n 1 -I {} curl -X POST -H "apikey: $ADMIN_APIKEY" https://$API_HOST/v1/.../stop -d uuid={} -d force=True
It is now safe to perform maintenance and bring Ceph offline.
Maintenance-maintenance-maintenance-...
Once Ceph is available again, mark hosts as accepting workloads.
curl -sS -X GET -H "apikey: $ADMIN_APIKEY" https://$API_HOST/v1/base-operator/host/list | \ jq -r '.[]|.uuid' | \ xargs -n 1 -I {} curl -X PUT -H "apikey: $ADMIN_APIKEY" https://$API_HOST/v1/base-operator/admin/host_flags -d uuid={} -d is_accepting_workloads=1
Finally, start all VMs that were running before.
cat running_vms.json | jq -r '.[]|.uuid' | \ xargs -n 1 -I {} curl -X POST -H "apikey: $ADMIN_APIKEY" https://$API_HOST/v1/.../start -d uuid={}