This guide will explain how to make backups of OpenStack and Ceph configurations.

Background Information

Our Private Cloud Core deployments make use of Kolla Ansible and Ceph Ansible to deploy the OpenStack services that comprise your cloud and the storage backend, Ceph.

OpenStack service configurations are kept within MariaDB databases as well as within the filesystem. An OpenStack service would be nova, cinder, or glance, to name a few examples.

Backup OpenStack services using Kolla Ansible

Kolla Ansible allows you to back up the MariaDB databases associated with OpenStack services by using MariaDB’s mariabackup function.

NOTE! – A good backup policy is to not store backups in the same location where production data lives, but rather to store backups offsite.

For an authoritative source of information on Kolla Ansible and using mariabackup see these documentation links:

The MariaDB database backup and restore guide goes into full detail on how to enable backups, create them, and restore the backups.


Using Kolla Ansible

To start using Kolla Ansible, an environment needs to be created. This section explains the steps needed to create that environment.

To get a quick idea of what is required, here is a high-level overview of the steps:

# Copy Kolla Ansible configuration from FM-Deploy Docker container
$ docker cp fm-deploy:/opt/kolla-ansible /opt/kolla-ansible

# Navigate to /opt/kolla-ansible
$ cd /opt/kolla-ansible

# Initialize a Python virtual environment
$ virtualenv .venv

# Activate the virtual environment
$ source .venv/bin/activate

# Install kolla-ansible using requirements.txt
$ pip install -r requirements.txt

# Set the SSH private key so kolla-ansible can connect to each host
$ export EXTRA_OPTS="--private-key /root/.ssh/fm-deploy"

The above takes care of preparing the Kolla Ansible environment.

These steps are explained in more detail below.


Prepare and use Kolla Ansible

Follow these steps to learn how to prepare and use Kolla Ansible.

Relevant files

  • Globals file: /etc/kolla/globals.yml
  • Iventory file: /etc/fm-deploy/kolla-ansible-inventory

Note! — The above files may not be present on each host. Inspect each host until you find the above, and perform the Kolla Ansible run from this host.


Step 1 – Prepare environment

From the Docker container called fm-deploy, copy /opt/kolla-ansible to the local filesystem:

$ docker cp fm-deploy:/opt/kolla-ansible /opt/kolla-ansible


Step 2 — Prepare Python virtual environment and install requirements

Create a Python virtual environment from which Kolla Ansible will be used:

$ cd /opt/kolla-ansible
$ virtualenv .venv
$ source .venv/bin/activate
(.venv) $ pip install -r requirements.txt


Step 3 — Export the private key

In order for Kolla Ansible to make changes to each host, the SSH private key path needs to be set as an environment variable:

$ export EXTRA_OPTS="--private-key /root/.ssh/fm-deploy"


Backup Kolla Ansible Files

There are important files related to Kolla Ansible that should be backed up.

These files are:

  • /etc/kolla/globals.yml
  • /etc/kolla/passwords.yml

Should they be lost, it will not be possible to manage the OpenStack cloud.

Create copies of these files and store them in an offsite backup location.

NOTE! — Due to the sensitive nature of these files, ensure the files are transported using an encrypted connection and are stored in a secure, private location.


Create OpenStack Database Backups

With Kolla Ansible prepared, you can create a full backup of all OpenStack service databases using mariabackup.

To create backups of the OpenStack databases, use:

$ kolla-ansible -i /etc/fm-deploy/kolla-ansible-inventory <subcommand>

Note that <subcommand> is a placeholder for the Kolla-ansible subcommand to run and in this case is mariadb_backup.

The full command to create database backups is:

$ kolla-ansible -i /etc/fm-deploy/kolla-ansible-inventory mariadb_backup

This will create a Docker volume called mariadb_backup that can be used to restore OpenStack databases.

The output of the Kolla Ansible run will let you know to what host the backups were sent.

To see the Docker volume, SSH into the host where the backups were made, and run:

# docker volume ls | grep mariadb_backup
local               mariadb_backup


Restore OpenStack Database Backups

This procedure outlines how to perform a full database restore. An incremental restoration can also be completed.

See MariaDB’s documentation for information on how to use mariabackup to make incremental backups.

The previous Kolla Ansible run created a Docker volume called mariadb_backup. This volume can be used to restore the OpenStack service databases.

Step 1 — Create Docker container

To restore this backup, create a new Docker container using the previously created volume:

# docker run --rm -it --volumes-from mariadb --name dbrestore 
--volume mariadb_backup:/backup kolla/centos-binary-mariadb:train-centos8 


Step 2 — Perform full backup of databases

Once in that container, these series of commands can be run to perform a full backup of all OpenStack service databases:

$ cd /backup/
$ rm -rf /backup/restore/
$ mkdir -p /backup/restore/full
$ gunzip mysqlbackup-10-08-2020-1597091449.qp.xbc.xbs.gz
$ mbstream -x -C /backup/restore/full < mysqlbackup-10-08-2020-1597091449.qp.xbc.xbs
$ mariabackup --prepare --target-dir /backup/restore/full


Step 3 — Stop MariaDB

Once the above commands are successfully run, stop the MariaDB Docker instance using:

# docker stop mariadb


Step 4 — Restore database backups

Navigate back into the container with the mariabackup volume mounted and either move or remove the contents of /var/lib/mysql and then move the contents of /backup/restore/full into /var/lib/mysql:

$ mkdir /backup/mariadb_original/
$ mv -v /var/lib/mysql/{*,.[^.]*} /backup/mariadb_original/
$ mv -v /backup/restore/full/{*,.[^.]*} /var/lib/mysql/


Step 5 — Start MariaDB

Finally, start the MariaDB Docker container:

# docker start mariadb


OpenStack Configuration Backups

In addition to backing up the MariaDB OpenStack databases, it is imperative the OpenStack service files are backed up. Services include nova, swift, and glance, to name a few.

All service configuration files are stored on the hardware nodes within the folder /etc/kolla/.

To backup the service’s configuration files, copy /etc/kolla to an offsite backup location.


Ceph Configuration Backups

The configuration files for Ceph are stored on each hardware node within the folder /etc/ceph. Backing up this folder is sufficient for backing up Ceph’s configuration.

To backup Ceph’s configuration, copy /etc/ceph to an offsite backup location.