Backup and restore¶

spot-cli backup saves what SPOT needs to come back after a failed upgrade or on a new host, and spot-cli restore puts it back. Both run on the host, from the deploy directory (the default is /opt/spot/deploy), with your own Docker client. They read the configuration volume with the spot-cli image, which they pull when it is missing: the cli profile is not needed.

What a backup holds¶

A backup is one directory:

It does not hold:

• Redis: the plugin catalog, a cache that a refresh rebuilds from the sources of spot.yaml, and the records of queued jobs.
• RabbitMQ: the queued messages, and the parked ones of the dead-letter queues.
• The Ollama models: ollama-init pulls them again.
• The certificates: Traefik asks Let's Encrypt again.

Take a backup¶

Postgres must run; the other services may be stopped. The deploy directory must be the deploy/ directory of a git checkout of the spot repository, as the Quick Start clones it: a backup records the release's commit, which the restore checks out.

cd /opt/spot/deploy
scripts/spot-cli backup

It writes backups/<release>-<UTC time>/, or a directory under --dir DIR. The directory is the only line it prints on standard output; its messages go to standard error, so a script can keep it with dir=$(scripts/spot-cli backup).

The dump is checked before it is kept: it is restored into a scratch database, which must come back at the same migration, then dropped. The check is usually longer than the dump: a restore rebuilds every index. It needs free disk for the database twice, for a moment: the scratch copy in the Postgres volume, and the dump in the backup directory before it is compressed. A backup that fails or is interrupted leaves nothing behind: neither its directory nor its scratch database.

Every night¶

A cron line, as root or as a user of the docker group who owns the deploy directory, that keeps 14 days of backups:

# /etc/cron.d/spot-backup
30 2 * * * root cd /opt/spot/deploy && scripts/spot-cli backup >/dev/null && find backups -mindepth 1 -maxdepth 1 -mtime +14 -exec rm -r {} +

cron mails the messages of each run. Old backups are removed only after a backup succeeds.

Restore¶

cd /opt/spot/deploy
scripts/spot-cli restore backups/<release>-<time>

It checks the backup against its checksums, then asks before it changes anything. The question names the release and the time of the backup, what the restore replaces, and the plugin containers it removes; --yes answers it without a terminal. Everything written since the backup is lost. Then it:

1. Backs up the live database and configuration volume, beside the backup it restores. The last line of the restore names this safety backup. When it cannot be taken, the restore replaces nothing: see When the live state cannot be backed up. After a restore that did not finish, it takes none: see When a restore fails.
2. Stops every SPOT container and every plugin container, and removes the containers of the plugins the backup's spot.yaml does not name.
3. Checks out the backup's deploy release, and puts back its .env and override file. The ones it replaces stay beside them, as .env.before-restore-<time> and docker-compose.override.yml.before-restore-<time>.
4. Pulls the release's images, replaces the database and the configuration volume, starts SPOT and waits for every health check.

At its start, the api-gateway starts the plugins the restored spot.yaml names, and re-creates any whose container differs from it, a plugin token included.

Redis and RabbitMQ are kept. A job queued before the restore may name an email the restored database does not hold: it fails, and is parked (see Messages in a Dead-Letter Queue). The Knowledge Store's query cache may serve a document the restore removed, for 5 minutes at most.

To check SPOT end to end after a restore, sign in and submit the sample message:

scripts/spot-cli auth login
scripts/spot-cli analyze submit -f - < samples/test-phishing.eml

After a failed upgrade¶

Restore the backup taken before the upgrade: spot-cli upgrade prints the command when it fails. After an upgrade by hand, restore the spot-cli backup you took first. The restore reads no compose file before the backup's .env is in place, so it works even when the new release refuses your .env, such as an upgrade stopped on a new required variable.

On a new host¶

1. Install Docker and clone the spot repository, as in the Quick Start, without running setup or creating .env.
2. Copy the backup directory to the host, and keep it readable by its owner only.

3. Restore it:

   cd /opt/spot/deploy
   scripts/spot-cli restore <backup directory>
   

4. Check DOCKER_GID in the restored .env against the group of the Docker socket, stat -c %g /var/run/docker.sock. When they differ, set it in .env, then docker compose up -d.

When a restore fails¶

Once its safety backup is taken, a restore that fails or is interrupted leaves a mix: the backup's release and .env, beside a database and a configuration volume replaced or not. A database left half loaded can even pass a backup's checks, since its alembic_version table loads first: never keep this mix as a backup. The restore says so, and leaves .restore-unfinished beside .env: it names the run's spot-cli image and its safety backup (none when there was no live state). Run scripts/spot-cli restore again: it finds the file, keeps that image and that safety backup, takes no safety backup of the mix, and its last line names the first run's, which holds the state from before the restore. The question names the time the unfinished restore started. A restore that finishes removes the file.

If you repair the stack another way instead, remove .restore-unfinished: while it is there, a restore takes no safety backup of what is live, and spot-cli backup refuses to run, so a nightly backup or an upgrade never keeps the mix (cron mails the refusal each night). A restore refuses to start when the safety backup the file names is gone (pruned, for example): remove the file to restore anyway, backing up what is live.

When the live state cannot be backed up¶

When Postgres does not start, when a configuration volume is left without its database, or when a first start whose migration failed left no alembic_version table, the safety backup fails and the restore replaces nothing. To restore anyway, discarding the live database and configuration on purpose, remove them, then restore as on a new host. These commands read no compose file:

docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=spot)
docker volume rm spot_postgres_data spot_spot-config
scripts/spot-cli restore <backup directory>

Passwords kept by the data volumes¶

Postgres sets POSTGRES_PASSWORD, and RabbitMQ RABBITMQ_DEFAULT_PASS, only when its data volume is first created. A restore on the same host keeps both volumes, so when the restored .env holds another password than the one a volume was created with (a password changed since the backup), the migration and the services cannot log in, and the restore stops waiting for them. Set the volume's password to the restored .env's, then start the services:

docker compose exec postgres psql -U spot -c '\password spot'
docker compose exec rabbitmq rabbitmqctl change_password spot
docker compose up -d

Each command asks for the new value, so it stays off the command line. Use your POSTGRES_USER and RABBITMQ_DEFAULT_USER if you changed them. Then remove .restore-unfinished, which the restore left when it stopped: only the passwords were left to set.

A backup taken by hand¶

A backup taken by hand before spot-cli backup existed, a dump and a copy of spot.yaml without manifest or SHA256SUMS, is restored with the steps of A backup taken by hand in the upgrade guide.