Upgrading SPOT¶

SPOT does not apply updates for you. The dashboard's Updates page (admin-only) reads the version tags of the spot repository on Codeberg, at most once an hour, and surfaces a banner with the upgrade command. The operator runs the command ; that is the only supported path.

Standard upgrade¶

From the deploy directory of your install (the default is /opt/spot/deploy), with the version of the release (the dashboard's banner names the command):

cd /opt/spot/deploy
scripts/spot-cli upgrade <version>

It fetches the release's tag from origin, checks that .env holds every setting the release requires, and asks before it changes anything. Then it takes a backup, checks the release out, pulls its images and starts it, and waits for the migration, the one-shot jobs and every health check. Read the release's section below first: a release that asks for a step during the upgrade, such as draining the queues, is upgraded by hand.

• A setting the release requires is missing: it stops before the backup, and Compose names the variable. Add it to .env, as the release's section says, and run the command again.
• Something fails after the backup: it stops, and prints the command that restores the backup. It never goes back by itself: see Rollback.
• Another remote: --from fetches from another git remote or URL, such as --from https://codeberg.org/SPOT_Project/spot.git. Behind an egress proxy, give git the proxy once: git config http.proxy http://proxy.example.com:3128.
• Without a terminal: --yes answers the question, for scripts.

Upgrades go forward only: to go back to an earlier release, restore its backup. The checkout is left on the release's tag, detached from any branch, so git pull no longer applies.

The release's docker-compose.yml brings the infrastructure version bumps (Traefik, Postgres, Redis, RabbitMQ) and any new services or environment variables. Database migrations run at its first start, in the migration service.

If you installed by docker pull only (no git), spot-cli upgrade does not apply: docker compose pull && docker compose up -d refreshes the images, but new infrastructure pins or new services in a release will not land. Clone the spot repository if you want them.

Upgrade by hand¶

For a release whose section asks for a step between the commands, upgrade by hand, with the backup first:

cd /opt/spot/deploy
scripts/spot-cli backup
git fetch --tags && git checkout --detach v<version>
docker compose pull && docker compose up -d

Always back up first¶

Updates can fail. spot-cli upgrade takes a backup itself. Before an upgrade by hand, take one while the old version still runs: a backup taken after the upgrade holds the migrated schema and cannot take you back. See Backup and restore.

scripts/spot-cli backup

Local customisations: docker-compose.override.yml¶

docker-compose.yml is part of the release and is replaced by every upgrade. Editing it means your edits will be lost on every upgrade, and spot-cli upgrade refuses to run while a file the repository tracks differs from the release (git status names it).

The supported way to customise the stack is docker-compose.override.yml next to docker-compose.yml. Compose merges the two automatically (docker-compose.override.yml is loaded by default, no flag required), and the spot repository does not track it: spot-cli upgrade checks the tracked files only, so it survives every upgrade, and a backup holds it.

Example: bind an extra volume into the api-gateway container.

# docker-compose.override.yml
services:
  api-gateway:
    volumes:
      - /opt/spot/extra:/extra

Apply with docker compose up -d. The merged config is visible with docker compose config.

The dashboard runs an automatic check: it SHA-compares the local docker-compose.yml against the upstream copy at the running version. If they differ, the System Updates page shows a red banner reminding you to move the customisation into docker-compose.override.yml before pulling.

Pin a specific version¶

To stay on a specific version instead of upgrading, set the pins in your .env:

CORE_VERSION=2.0.0-beta.8
DASHBOARD_VERSION=2.0.0-beta.8

These override the defaults from docker-compose.yml. An upgrade still runs cleanly, except across 2.0.0-beta.8 (below); it just changes the defaults under your overrides.

A pin cannot reach across 2.0.0-beta.8. The migration service of the later deployment files imports the shared library as spot_shared, which the images of 2.0.0-beta.8 and earlier do not have, and the deployment files of 2.0.0-beta.8 and earlier import it under its old name, which the later images do not have: either mix stops the migration service, and the api-gateway never starts. Before upgrading from 2.0.0-beta.8 or earlier to a later release, raise CORE_VERSION and DASHBOARD_VERSION to that release, or remove them from .env: the deployment files and the images must both be from before that change, or both from after it.

Rollback¶

Restore the backup taken before the upgrade, the one spot-cli upgrade printed:

scripts/spot-cli restore backups/<release>-<time>

It puts back the earlier release, its .env, its database and its configuration, and starts it: everything written since the backup is lost (see Restore). A release's section below may name more to undo, such as queues to delete.

When no migration ran, you may instead check out the previous deploy release and start it, which keeps what was written since:

git checkout --detach v2.0.0-beta.7   # the release you ran before
docker compose up -d --remove-orphans

--remove-orphans removes the containers the later release added and the earlier one does not know, such as spot-egress-gateway.

A backup taken by hand¶

A backup taken by hand before spot-cli backup existed, a dump and a copy of spot.yaml, is restored with the steps below: a migration's downgrade can lose data, as migration 010 loses the results of 2.0. Use your POSTGRES_USER and POSTGRES_DB if you changed them.

1. Stop the stack, then start Postgres alone:

   docker compose down
   docker compose up -d --wait postgres
   

2. Drop and recreate the database, then restore the dump into it. A dump restored over the migrated database mixes the two schemas.

   docker compose exec -T postgres dropdb -U spot spot
   docker compose exec -T postgres createdb -U spot spot
   docker compose exec -T postgres psql -U spot -d spot -v ON_ERROR_STOP=1 -q < spot-backup-<date>.sql
   

3. Check out the previous deploy release, and write the spot.yaml copy into the config volume before anything starts, so the old release starts from it. run --no-deps writes it from a one-off api-gateway container and starts nothing else:

   git checkout v2.0.0-beta.4   # the release you ran before
   docker compose run --rm --no-deps -T api-gateway sh -c \
     'cat > /app/config/.spot.yaml.new && mv /app/config/.spot.yaml.new /app/config/spot.yaml' \
     < spot-yaml-backup-<date>.yaml
   

4. Start the release. When it starts, the api-gateway (2.0.0-beta.4 or later) recreates, in the background, every plugin container that differs from spot.yaml, so each plugin goes back to the image the copy names:

   docker compose up -d
   

5. Give the recreation a minute, then check that spot.yaml and the plugin containers name the images of the copy, the 1.x releases after a rollback from 2.0.0-beta.5:

   grep 'image:' spot-yaml-backup-<date>.yaml
   docker compose exec -T api-gateway grep 'image:' /app/config/spot.yaml
   docker ps --filter label=spot.managed=true --format '{{.Names}} {{.Image}}'
   

   A plugin still on its newer image was not recreated: docker compose logs api-gateway says why. A container of a plugin the copy does not name was installed after the backup: remove it with docker rm -f <name>.

To an earlier release than 2.0.0-beta.4, follow its own section below as well, such as Rolling back from 2.0.0-beta.4.

Disabling the update notification¶

The dashboard banner and the System -> Updates page can be turned off:

# /opt/spot/deploy/.env
SPOT_UPDATE_CHECK_ENABLED=false

Then docker compose up -d api-gateway. The endpoint disappears and no Codeberg traffic is generated.

From 1.x to 2.0 (deploy 2.0.0-beta.2)¶

Deploy versions now follow the core version they ship: deploy 2.0.0-beta.2 runs core 2.0.0-beta.2 by default. The schema is unchanged since 1.3.2, so the standard upgrade applies. Check these first:

1. CPU. The knowledge service loads NumPy 2.4, which needs an x86-64-v2 CPU. On the host, /lib64/ld-linux-x86-64.so.2 --help | grep x86-64-v2 must show (supported, searched). On a VM with a generic CPU model, set the CPU type to host or x86-64-v2-AES.
2. RabbitMQ. The mail-orchestrator now requires it (REQUIRE_BROKER defaults to true). The stack already runs it; only installs that disabled the broker need to change.
3. Tokens. Refresh tokens no longer work as access tokens. The dashboard and the CLI are unaffected; custom API clients must send the access token.
4. OAuth clients. Every active oauth_clients row allowed client_credentials becomes a working analyst-role client. Audit that table before upgrading.

From the mail-smtp content filter to the mail-milter profile¶

The retriever-smtp service and its mail-smtp profile are gone. Its replacement, retriever-milter, is a milter: Postfix talks to it from smtpd instead of relaying every message through a content filter. Move over in this order:

1. In .env, replace mail-smtp with mail-milter in COMPOSE_PROFILES and drop the RETRIEVER_SMTP_* variables. The new ones are RETRIEVER_MILTER_* (see .env.example).
2. Run docker compose up -d. The milter listens on 127.0.0.1:9900 and starts in observe mode, which only adds X-SPOT-* headers and always accepts mail.

3. Hook the milter on Postfix's inbound smtp service and reload, as described in Mail integration. Do not use the global smtpd_milters: it would also analyse mail sent through submission.

4. Remove the old plumbing: the -o content_filter=smtp:[127.0.0.1]:10025 line on the smtp service and the 127.0.0.1:10026 pickup listener in master.cf, then postfix reload.

5. Watch the verdicts in observe mode, then switch to enforce once you trust them.

The verdict headers changed: X-SPOT-Verdict, X-SPOT-Action and X-SPOT-Mode replace X-SPOT-Status and X-SPOT-Is-Phishing. Update any Sieve or Procmail rules that matched the old ones.

Proxy settings and plugin containers (core 2.0.0-beta.2)¶

Core 2.0.0-beta.2 forwards the operator's egress proxy settings to plugin containers (see Running behind an egress proxy). Behind a proxy, after upgrading:

1. If the proxies of ~/.docker/config.json changed, recreate the compose containers first: docker compose up -d --force-recreate.
2. Plugin containers created before the upgrade keep their old environment. Recreate each one with POST /api/v1/plugins/<id>/restart.

To 2.0.0-beta.4¶

Core 2.0.0-beta.4 isolates the plugins. Each plugin holds its own token, runs on its own network and receives its secrets as files. An internal plugin-manager runs the plugin containers, and nothing internet-facing mounts the Docker socket. Take the backup, check the points below, then run the standard upgrade.

Before you upgrade¶

1. DOCKER_GID. Two socket proxies now hold the Docker socket, unprivileged, with only the group that owns it: plugin-docker-proxy for the plugin-manager, and the read-only traefik-docker-proxy for Traefik. Set DOCKER_GID in .env to that group, stat -c %g /var/run/docker.sock. With a wrong value, Traefik routes nothing and plugins cannot be installed, updated or restarted. The api-gateway no longer mounts the socket.

2. ENCRYPTION_KEY. It now also protects the plugin secrets sealed in spot.yaml and the plugin tokens. Back it up together with the database and with spot.yaml, which lives in the spot-config volume:

   docker compose cp api-gateway:/app/config/spot.yaml spot-yaml-backup-$(date +%Y%m%d).yaml
   

   A spot.yaml restored under another key means entering every plugin secret again. Without the same key, installed plugins also get new tokens at the next start, and plugins you run yourself need a new one.

3. docker-compose.override.yml. Update an override that:

   • attaches a container to spot_spot-network to run a plugin: plugins belong on the external network spot-plugins (see Plugins you run yourself);
   • mounts the Docker socket into the api-gateway or Traefik, or gives the api-gateway a group_add: remove it, the proxies replace them;
   • reaches Postgres, Redis or RabbitMQ over spot_spot-network (a backup job, an exporter): they are now only on the internal network spot_spot-db-network;
   • publishes a host port of Postgres, Redis or RabbitMQ: Docker cannot publish a port from an internal network, so remove it.
4. Docker Compose 2.6 or later (docker compose version). The compose file now declares the milter's token as a secret read from an environment variable, which older versions refuse.
5. Let the running analyses finish. Postgres, Redis and RabbitMQ change network, so docker compose up -d recreates their containers; their named volumes are kept. RabbitMQ has no fixed hostname: the new container gets a new node name and starts with empty queues, so queued messages are dropped. Wait until docker compose exec rabbitmq rabbitmqctl list_queues shows no message waiting, and expect a short outage. Logins, the job history and the volumes survive.

What the first start does¶

• The api-gateway recreates every installed plugin container once, in the background. Each one gets its own token in /run/secrets/spot_plugin_token, no SPOT_INTERNAL_API_KEY, the spot-plugins network, and memory and CPU limits (1024 MB and 1 CPU unless the plugin asks for more). Capabilities are dropped and no-new-privileges is set, so a plugin image that needs a Linux capability no longer works.
• It seals the plain secret settings that 2.0.0-beta.3 stored in spot.yaml, once. They move under secrets:, encrypted with ENCRYPTION_KEY, and reach the plugin as files named in uppercase, such as /run/secrets/MISP_TOKEN, no longer as environment variables. A plugin the catalog does not list keeps its settings, and the api-gateway logs Plugin <id> is not in the catalog, so its settings stay as they are.

After the first start¶

1. Isolation and memory. Every plugin container must be on spot-plugins only, without SPOT_INTERNAL_API_KEY, and not killed for lack of memory:

   for c in $(docker ps -aq --filter label=spot.managed=true); do
     docker inspect -f '{{.Name}} oom={{.State.OOMKilled}} networks={{range $name, $net := .NetworkSettings.Networks}}{{$name}} {{end}}' "$c"
     docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$c" | grep -q '^SPOT_INTERNAL_API_KEY=' && echo "  still holds SPOT_INTERNAL_API_KEY"
   done
   

   Each line must read oom=false networks=spot-plugins. A plugin still on spot_spot-network with the internal key was not recreated: the reconcile failed, usually because of a wrong DOCKER_GID. Fix it in .env, run docker compose up -d, then docker compose restart api-gateway to run the reconcile again.

   The 1024 MB default may be too little for a plugin, since no plugin image asks for more yet: analyzer-nlp (torch, transformers) may exceed it. A plugin with oom=true needs a higher limit, set with resources: on its entry in spot.yaml (see Plugin memory and CPU).

2. No credential left in plain text.

   docker compose exec api-gateway cat /app/config/spot.yaml
   docker compose logs api-gateway | grep "not in the catalog"
   

   If a credential is still under a plugin's settings:, or the log names a plugin, refresh the catalog (POST /api/v1/plugin-catalog/refresh, or the refresh button of the dashboard's plugin catalog), then reload the configuration with POST /api/v1/config/reload: the reload seals them too. provider-employee-dir's LDAP bind password stays under settings: until you seal it yourself (below).

Plugins that stop working until their contract-2 release¶

• Plugins that call SPOT now authenticate with their token. Their SDK 1 releases send the internal API key instead and get 401 until their contract-2 release (2.0.0-beta.5): context providers such as provider-employee-dir, whose knowledge sync fails with 401 in its log, and mail retrievers such as retriever-milter (see the milter). An analyzer that queries the Knowledge Store gets 401 too. Analyzers that do not call SPOT keep working.
• analyzer-misp 1.x does not start. Its MISP token is required and read from the environment, but secrets now reach plugins only as files (/run/secrets/MISP_TOKEN). Disable it (spot-cli plugin disable analyzer-misp), or take it out of your workflows, until its contract-2 release.

• provider-employee-dir's LDAP bind password (LDAP_BIND_PASSWORD) is not marked secret by its current image, so it stays a plain setting, in spot.yaml and in the plugin's environment. Seal it without changing the plugin's version:

  spot-cli plugin settings provider-employee-dir --secret LDAP_BIND_PASSWORD="$(cat bind.pw)"
  

  Read the value from a file, as above: a value typed on the command line stays in your shell history. Do not use plugin update --secret, which also moves the plugin to the catalog's latest version. The plugin gets the change at its next restart (the dashboard's Restart, or POST /api/v1/plugins/{plugin_id}/restart). The password then reaches it only as a file, which the SDK 1 image does not read; the plugin gets 401 on its knowledge sync until its contract-2 release either way.

The milter¶

The retriever-milter compose service now runs on spot-plugins and ingests through the api-gateway (SPOT_GATEWAY_URL) with its token. It no longer gets SPOT_INTERNAL_API_KEY or SPOT_MAIL_ORCHESTRATOR_URL.

• Simplest: turn the milter off until its contract-2 release. Milter 0.1.0 cannot ingest on 2.0.0-beta.4 (below). Before upgrading, unhook it from Postfix, remove mail-milter from COMPOSE_PROFILES and remove its container with docker compose rm -sf retriever-milter: compose leaves the container of a disabled profile running. If you leave Postfix hooked, it applies the milter's default_action while the milter is down.
• Milter 0.1.0 can no longer ingest. It does not read the token and no longer reaches the mail-orchestrator. Every message gets X-SPOT-Verdict: error and the milter applies RETRIEVER_MILTER_ON_ERROR: accept (the default) delivers the mail, tempfail defers every inbound message. If you keep it running, keep accept.

• The token. With the mail-milter profile on and RETRIEVER_MILTER_TOKEN unset, docker compose up -d starts every other service, then stops at the milter with environment variable "RETRIEVER_MILTER_TOKEN" required by secret ... is not set. Until the token is set, the milter is down and Postfix applies its milter default_action. Once the api-gateway runs, issue the token, set it in .env and start the milter:

  spot-cli plugin token retriever-milter --kind mail_retriever
  # .env: RETRIEVER_MILTER_TOKEN=<the printed token>
  docker compose up -d
  

  The token reaches the milter as /run/secrets/spot_plugin_token, through a compose secret. To rotate it, run the spot-cli command again (the old token stops working at once), set the new value and run docker compose up -d --force-recreate retriever-milter: a plain up -d does not recreate a running milter.

• RETRIEVER_MILTER_ALLOWED_CLIENTS. A Postfix on the Docker host now reaches the milter from the gateway of spot-plugins, no longer of spot_spot-network. If you narrowed the list to the old network, allow that gateway as a /32: docker network inspect spot-plugins --format '{{range .IPAM.Config}}{{.Gateway}}{{end}}'. The milter shares spot-plugins with every plugin, so the /32 also keeps plugins from opening milter sessions (see Mail integration).

Plugins you run yourself¶

A plugin you run with your own compose file or docker run on spot_spot-network keeps working, but is not isolated: it still reaches the mail-orchestrator and the plugin-manager. Move it to the external network spot-plugins and remove SPOT_INTERNAL_API_KEY from it: the knowledge service, the plugin-manager and the mail-orchestrator treat whoever sends that key as core. Give it its token instead:

spot-cli plugin token provider-employee-dir --kind context_provider > spot_plugin_token

services:
  provider-employee-dir:
    environment:
      SPOT_PLUGIN_ID: provider-employee-dir
    secrets:
      - spot_plugin_token
    networks:
      - spot-plugins
secrets:
  spot_plugin_token:
    file: ./spot_plugin_token
networks:
  spot-plugins:
    external: true

--kind adds the plugin to spot.yaml as a manual entry when it is not there yet. The file is a secret: never commit it. Compose mounts it with its owner and mode on the host, so the user the plugin's container runs as must be able to read it. The plugin must be on its contract-2 release to use the token. SPOT_PLUGIN_ID is its id in spot.yaml, which the installer sets on the plugins it runs: a context provider writes its documents under it, and without it every sync fails.

Other changes¶

• Postgres, Redis and RabbitMQ are on the internal network spot_spot-db-network only. Deploy never published them on the host; reach them with docker compose exec postgres psql -U spot spot, docker compose exec redis redis-cli (with -a <password> when REDIS_PASSWORD is set) and docker compose exec rabbitmq rabbitmqctl list_queues.
• The analyzer-orchestrator API requires the internal API key on every route except /health and /version. The compose file gives it SPOT_INTERNAL_API_KEY; without it, the dashboard's workflow pages fail with 503. Its open /health now reports only whether each dependency is connected and required; the details are behind the key (/readiness, /metrics). Update monitoring that read them from /health.
• API changes for scripts:
  • PUT /api/v1/plugins/{plugin_id}/settings takes {"settings": {...}, "secrets": {...}} instead of the bare settings, and refuses any other key with 422.
  • DELETE /api/v1/workflows/default-workflow answers 409 instead of 403.
  • DELETE /api/v1/plugins/{plugin_id} no longer takes the force query parameter.
• Traefik no longer routes /internal/* on api.DOMAIN. Those are the plugin routes of the api-gateway, reachable from spot-plugins only.
• The api-gateway no longer accepts the internal API key. It made any holder a synthetic admin. A script that sends X-Internal-API-Key to api.DOMAIN now gets 401: use a user account or an OAuth client instead.
• Platform and proxy variables override a plugin setting of the same name. A per-plugin NO_PROXY or HTTPS_PROXY setting is ignored when a proxy is configured. Move the internal hosts a plugin reaches without the proxy, such as a MISP host, into the Docker client's noProxy (see Running behind an egress proxy).

The rest is in the 2.0.0-beta.4 section of the CHANGELOG.

Rolling back¶

After the upgrade, the plugins sit on spot-plugins, their secrets are sealed in spot.yaml, and migration 009 (plugin tokens) has run. Rolling back to 2.0.0-beta.3 needs the database dump and the spot.yaml copy taken before the upgrade:

1. Restore the database and check out 2.0.0-beta.3, as in Rollback. Start it with docker compose up -d --remove-orphans, which also removes the plugin-manager and the two socket proxies.

2. Write back the spot.yaml copy, under a temporary name renamed into place, and restart the api-gateway:

   docker compose exec -T api-gateway sh -c \
     'cat > /app/config/.spot.yaml.new && mv /app/config/.spot.yaml.new /app/config/spot.yaml' \
     < spot-yaml-backup-<date>.yaml
   docker compose restart api-gateway
   

3. Remove the plugin containers 2.0.0-beta.4 created (docker rm -f spot-<plugin-id>) and recreate each with POST /api/v1/plugins/<plugin-id>/restart, which puts it back on spot_spot-network.

To 2.0.0-beta.5¶

Core 2.0.0-beta.5 speaks plugin contract 2. It takes each message raw and parses it once, and a workflow's verdict policy decides the verdict: malicious, suspicious, spam, benign or unknown, with a score from 0 to 100. Every plugin must move to its contract-2 release: an earlier one stops working (core still calls a 1.x analyzer, but its answer counts as unknown), and the catalog refuses them. Take the backup of the database, and of spot.yaml as for 2.0.0-beta.4, check the points below, then run the standard upgrade. The first start writes spot.yaml again, without its comments, when a stored workflow is still in contract 1 (see What the first start does): the backup's copy keeps the original.

Before upgrading¶

1. ENCRYPTION_KEY reaches the migration. Migration 010 converts the stored analyses and encrypts what it keeps of them, so the migration service now gets ENCRYPTION_KEY. Keep the key you have. Without it the migration stops before changing anything (ENCRYPTION_KEY must be set to convert the stored contract 1 analyses), and the api-gateway does not start.
2. Let the queues drain, right before docker compose up -d. Analysis messages still in RabbitMQ at the upgrade are refused ("invalid analysis job message"). After git pull and docker compose pull, pause what submits analyses: your scripts, the mail retrievers, and the milter (docker compose stop retriever-milter; Postfix applies its milter default_action meanwhile, so tempfail defers the mail rather than delivering it unanalysed). Wait until docker compose exec rabbitmq rabbitmqctl list_queues shows no message waiting, then run docker compose up -d, which starts the milter again.

3. Retention. EMAIL_RETENTION_DAYS is gone: platform.data_retention_days of spot.yaml replaces it, 365 days by default, from 1 to 36500. The first start purges the expired emails, those stored before migration 002 included. To keep emails longer than 365 days, set the retention before the first start, in docker-compose.override.yml:

   services:
     api-gateway:
       environment:
         DATA_RETENTION_DAYS: "730"
   

   The environment overrides spot.yaml. Keep the override, or later set the value with PUT /api/v1/config/platform (see Retention) and remove it.

4. Memory. The api-gateway now parses each message in worker processes, and the compose file limits it to 3328M, what the default intake settings need (see Intake and api-gateway memory). Make sure the host has it.

5. Redis 7.0 or later. The rate limiter sets each key's expiry with EXPIRE ... NX. The compose file runs Redis 7.2. If you bring your own Redis, it needs 7.0 or later: on an older one rate limiting does not apply, and the api-gateway's log warns about it.
6. .env. Remove EMAIL_RETENTION_DAYS, and any RETRIEVER_MILTER_VERSION pin to 0.1.0.

What the first start does¶

• Stored verdicts are converted (migration 010). An analysis of 1.x keeps a verdict: is_phishing false is benign; true is malicious with a high or critical threat level, suspicious otherwise; a failed or undecided one is unknown. Its score is the confidence in percent (100 minus it for benign), and its decided_by is v1. Its indicators become signals legacy.<type> of weight 0, and the rest of what 1.x recorded is kept encrypted.

• Workflows are rewritten. A workflow that spot.yaml holds in contract 1 is converted when read and written back once, as contract 2. The whole file is written again, without its comments:

  • final_stage_name becomes a verdict_policy of method most_confident on that stage (the last stage when none was named), which keeps the 1.x verdicts; an analyzer of weight 0 in that stage gets weight 1;
  • <stage>.weighted_confidence > X becomes <stage>.score > 100X, and its meaning changes: a score measures how malicious the email is, not how confident the analyzers were;
  • failure_strategy: fallback becomes skip;
  • a stage's aggregation_method, continue_on_failure and min_successful_analyzers are dropped.

  A workflow already in contract 2 that holds a stage's continue_on_failure or min_successful_analyzers, or max_parallel_analyzers, which the orchestrator never read, loses them the same way. A workflow that already has a verdict_policy keeps it. Only a spot.yaml that an earlier api-gateway saved holds workflows; otherwise the new shipped defaults apply. - The shipped defaults now come from the image (/app/defaults/spot.yaml). The defaults/ directory an earlier release left in the spot-config volume is not read.

After the first start¶

1. Workflows. Check what the conversion did:

   docker compose logs api-gateway | grep -i workflow
   

   • Workflow <id>: <change>, one line for each change made;
   • Rewrote workflow '<id>' of /app/config/spot.yaml as v2, once for each workflow written back;
   • Workflow '<id>' is left out: <reason>: that workflow cannot run, and its analyses fail; when it is the default workflow, so do the analyses that name none. Fix it in spot.yaml, edited as in Editing spot.yaml by hand, then docker compose restart api-gateway analyzer-orchestrator.

   Check every converted condition on a score. An analysis that names no workflow (the milter's, a retriever's) runs default_workflow_id: default-workflow in the shipped defaults. A spot.yaml with workflows of its own and no default_workflow_id now runs default-workflow rather than its only workflow: set default_workflow_id to one of its workflows.

2. Refresh the plugin catalog (POST /api/v1/plugin-catalog/refresh, or the refresh button of the dashboard's catalog). Until then, the entries cached by 2.0.0-beta.4 are listed as refused ("listed by an earlier SPOT: refresh the catalog"), and nothing can be installed or updated from them. A refresh of every source scans the enabled ones only: refresh a disabled source on its own, or remove it.

3. Update every plugin to its contract-2 release (below). Until then, an analyzer of 1.x counts as unknown, the milter's or a retriever's ingest is refused, and a context provider cannot sync.
4. Authentication results. mail.trusted_authserv_ids is empty, so no SPF, DKIM or DMARC result counts, in the email's view, the forensics and analyzer-rules, until you set your MTAs' authserv-ids (see Authentication results). Each of these MTAs must remove the incoming Authentication-Results headers that carry its id.

Plugins on contract 2¶

Update each installed plugin, after refreshing the catalog:

spot-cli plugin update analyzer-nlp --version 2.0.0b1
spot-cli plugin update analyzer-ml --version 2.0.0b1
spot-cli plugin update analyzer-rules --version 2.0.0b2
spot-cli plugin update analyzer-misp --version 2.0.0b2
spot-cli plugin update provider-employee-dir --version 2.0.0b2
spot-cli plugin update analyzer-llm --version 2.0.0b1 --consent-content

• Consent. Installing or updating to a plugin of egress class content needs an admin's consent, --consent-content ("consent_to_content": true in the API), or it answers 409: analyzer-llm sends the email's content to its model server, and the milter sends every message whole to SPOT. A consent is recorded once. The compose file's milter is a manual entry: nothing asks for its consent. analyzer-misp sends indicators only; the other plugins send nothing (see Egress and consent).
• Settings. The renamed settings below must be set again under their new name (--set NAME=value on the update, or spot-cli plugin settings <id> --set NAME=value); the old names are no longer read. Saving a plugin's settings restarts it. A value out of its range, or a URL the plugin refuses, stops it at start, and its log names the setting, never the value.
• Plugins you added by hand need a token, or SPOT does not call them: spot-cli plugin token <id> --kind <kind> (see Plugins you run yourself).

analyzer-llm 2.0.0b1 speaks the OpenAI chat completions API, Ollama's included, instead of Ollama's own:

• OLLAMA_HOST is LLM_URL, the API root, ending in /v1: http://ollama:11434 becomes http://ollama:11434/v1. It may hold no credentials (@), query or fragment.
• OLLAMA_MODEL is LLM_MODEL, OLLAMA_TIMEOUT is LLM_TIMEOUT, MAX_TOKENS is LLM_MAX_TOKENS and TEMPERATURE is LLM_TEMPERATURE.
• OLLAMA_NUM_CTX and CONFIDENCE_THRESHOLD are removed. New: LLM_API_KEY (a secret, for a server that needs one) and LLM_MAX_BYTES.
• It no longer pulls its model: pull it on the server (docker exec spot-ollama ollama pull llama3.1 for the bundled side-car, or add it to OLLAMA_EXTRA_MODELS).
• A request takes up to about 11300 tokens with the default settings, and the server's context must hold it. The bundled side-car moves from Ollama 0.5.7, whose context is 2048 tokens, to 0.13.5, and serves every model with OLLAMA_CONTEXT_LENGTH tokens, 16384 by default. That costs llama3.1 about 2 GiB more memory while it is loaded, an estimate; The side-car and analyzer-llm says how to lower it with LLM_MAX_BYTES. On an Ollama of your own, set OLLAMA_CONTEXT_LENGTH to at least 12288. An Ollama that ignores it, such as 0.5.7, keeps a context of 2048 tokens, which no LLM_MAX_BYTES fits: upgrade that Ollama.

analyzer-misp 2.0.0b2:

• MISP_HOST is MISP_URL: an http(s) URL with no credentials (@), query or fragment.
• MISP_TIME_RELEVANCY now limits the search, and defaults to 90d: only the events published in the last 90 days match. 1.x declared it but never sent it, and matched every event. Set a longer time, such as 3650d, to search further back. The unit is d, h, m (minutes) or s, in lower case: 3M stops the plugin.
• MISP_SPEAR_PHISHING_TAGS is MISP_TAGS, a comma-separated string instead of a JSON list: MISP_TAGS=phishing,tlp:red.
• MISP_TOKEN stays a secret, read from /run/secrets/MISP_TOKEN. It must be printable ASCII without spaces.
• Redirects are no longer followed: point MISP_URL at MISP's final address, or the search fails with "MISP answered HTTP 30x".
• MISP_ATTRIBUTE_TYPES and CONFIDENCE_THRESHOLD are removed. New: MISP_VERIFY_TLS, MISP_TIMEOUT and MISP_CA_CERT, which takes a PEM of several lines.

provider-employee-dir 2.0.0b2:

• Each sync replaces the provider's documents with the directory as it is: an employee who left is deleted. A read that gives no employee fails the sync and writes nothing.
• Keep the plugin id provider-employee-dir: the documents 0.2.x wrote carry it, and while they exist core refuses every sync under another id (403).
• The image ships no employee file: the yaml backend without a file mounted at EMPLOYEES_FILE is unhealthy (no employee file). Use the ldap backend.
• LDAP_BIND_PASSWORD is a secret: core seals a password stored as a plain setting when the plugin is updated. LDAP_URL may hold no credentials, path or query, and only ldap:// or ldaps://. LDAP_CA_CERT takes a PEM of several lines.

analyzer-nlp 2.0.0b1 and analyzer-ml 2.0.0b1:

• MODEL_PATH (analyzer-nlp) and MODELS_DIR (analyzer-ml) may hold no credentials: a value with an @, wherever it is, stops the plugin, and so does a URL that is not http(s) or whose host holds a %. Give access to a private archive with a presigned URL.
• CONFIDENCE_THRESHOLD is removed: the workflow's verdict policy decides.
• analyzer-nlp: weights without a trained classifier, such as a base model, no longer load. analyzer-ml: models must be saved with scikit-learn 1.7.2.

analyzer-rules 2.0.0b2:

• MIN_INDICATORS_FOR_PHISHING and CONFIDENCE_PER_INDICATOR are removed: the score is the sum of the signals' weights, and the verdict policy decides. BUSINESS_HOURS_START and BUSINESS_HOURS_END must be hours of the day (0-23).
• It reads SPF, DKIM and DMARC from the Authentication-Results of mail.trusted_authserv_ids only (see above).

The milter 2.0.0b1¶

The compose file now runs retriever-milter 2.0.0b1. It sends SPOT the raw message and its SMTP envelope. If you turned the milter off for 2.0.0-beta.4, turn it back on as in Mail integration: its token is the one spot-cli plugin token retriever-milter --kind mail_retriever issues.

• In enforce mode, observe first. The verdict classes, the meaning of the score and the actions (ACTION_SPAM tags spam by default) all change at once. Set RETRIEVER_MILTER_MODE=observe for the upgrade, check the new verdicts and the X-SPOT-Action that enforce would apply, then set enforce again.

• The headers change. X-SPOT-Score (0 to 100) replaces X-SPOT-Threat-Level and X-SPOT-Confidence, and X-SPOT-Verdict also takes spam and unknown. Update the Sieve or Procmail rules that matched the old headers (see What the headers mean).

• New actions. RETRIEVER_MILTER_ACTION_SPAM (default tag) and RETRIEVER_MILTER_ACTION_UNKNOWN (default accept; tempfail defers a message SPOT keeps judging unknown until the sender gives up, typically after 5 days).
• ON_TIMEOUT is narrower. A workflow that does not finish within ANALYSIS_TIMEOUT_MS is now unknown, and goes to ACTION_UNKNOWN. ON_TIMEOUT applies only when SPOT does not answer within ANALYSIS_TIMEOUT_MS + 5 s.
• Size. MAX_MESSAGE_BYTES counts the headers too: a message over it is accepted with X-SPOT-Verdict: skipped.
• Memory. MAX_BYTES_IN_FLIGHT (75 MiB) bounds the messages the milter holds at once; past it, mail waits in Postfix's queue, and once the milter's content_timeout passes, Postfix applies its default_action to the waiting message. The milter needs at most about 64 + 2 x MAX_CONNECTIONS + 9 x MAX_BYTES_IN_FLIGHT MiB, about 940 MiB with the defaults, and the compose file now limits it to 1024M, what its image declares. If you raised RETRIEVER_MILTER_MAX_CONNECTIONS or MAX_BYTES_IN_FLIGHT, raise the limit with them (see Mail integration).
• Removed. SPOT_MAIL_ORCHESTRATOR_URL, SPOT_INTERNAL_API_KEY and RETRIEVER_ID are no longer read: remove them from a docker-compose.override.yml that sets them. Port 8000 serves /health and /manifest; /version and /settings/schema are gone.

Other changes¶

• Viewers lose email content. GET /api/v1/emails lists summaries (subject, sender, source, size and dates). An email's detail, forensics, view and raw message are for analysts and admins: viewers get 403.
• Emails stored before 2.0 still list and open, but have no raw copy or view (/view, /raw and the parts answer 409), and cannot be analysed again. Analysis jobs of 1.x in Redis show no verdict.
• API changes for scripts:
  • POST /api/v1/analyze takes the raw message only: a message/rfc822 body, or JSON with the message base64-encoded in raw.
  • Results carry verdict, score, decided_by and signals instead of is_phishing, threat_level and confidence. The threat_level and is_phishing filters of GET /api/v1/analyze answer 422, and the analysis statistics count by_verdict.
  • The workflow API refuses a contract 1 workflow with 400, naming its contract 1 field.
  • A message larger than 25 MiB is refused with 413, one whose parse runs past a limit with 422 (parse_limit), and the api-gateway answers 503 when it already takes as many messages as it can.
• spot-cli. analyze get becomes analyze status, and analyze submit takes an RFC 5322 file with -f (-f - reads standard input) and no longer has --priority. plugin install and plugin update take --consent-content. scripts/spot-cli-completion.bash completes analyze status.
• Benchmarks. Normalize the benchmark datasets and run the benchmarks again with this spot-cli (spot-cli benchmark normalize, then benchmark run): it refuses the datasets, runs and metrics of an earlier one, and writes nothing over them. benchmark run no longer turns the platform's rate limiting off: it waits when the api-gateway answers 429, so a large run is bound by rate_limit_requests. The report counts a malicious or suspicious verdict as phishing, leaves unknown out of the confusion matrix, and shows the mean score instead of the confidence.

Web dashboard¶

The dashboard 2.0.0-beta.5 needs core 2.0.0-beta.5.

• Viewers see summaries and verdicts only. Opening an email, its download and its forensics are for analysts and admins.
• An email stored before 2.0 has no parsed view. Its page says so, keeps its analyses and Delete, and cannot be analysed again.
• A re-analysis (Retry, Analyze, "Use another workflow") runs on the stored email, and runs the workflow's responders again.
• The settings form follows each plugin's schema. Values are typed, a multi-line setting such as a PEM is a text box, and a setting left unset keeps the plugin's default.
• Saving settings restarts the plugin.
• Installing or updating a content plugin asks for your consent.
• The workflow editor edits the verdict policy and on_verdict.

The rest is in the 2.0.0-beta.5 section of the CHANGELOG and in each plugin's CHANGELOG.

Rolling back to 2.0.0-beta.4¶

Migration 010 has a downgrade, but it loses the results of 2.0. Follow the steps of Rollback instead, with the database dump and the spot.yaml copy taken before the upgrade: they bring each plugin back to its 1.x image, since plugins of contract 2 do not work on 2.0.0-beta.4.

2.0.0-beta.4's compose file runs the milter 0.1.0, which cannot ingest there: every message gets X-SPOT-Verdict: error and RETRIEVER_MILTER_ON_ERROR applies, so tempfail defers all inbound mail. Before starting it, turn the milter off or keep accept, as in The milter.

To 2.0.0-beta.6¶

Core 2.0.0-beta.6 has no migration and changes no plugin: run the standard upgrade.

• Plugin health. Every view shows each plugin's own state beside its container's: healthy, starting, unhealthy or unknown, with the plugin's reason and the version it runs. A plugin whose container is healthy can show unhealthy, such as provider-employee-dir without its employee file: its reason says why. The dashboard 2.0.0-beta.6 needs core 2.0.0-beta.6.
• API changes for scripts: GET /api/v1/config/plugins/status and the plugins of GET /api/v1/system/health give each plugin's status, reason and version, instead of url and error. In the workflow readiness, an analyzer that was timeout or unreachable is unknown.
• HEALTH_CHECK_TIMEOUT_SECONDS is no longer read: remove it from .env if you set it.
• A workflow with a stage of another type than parallel, sequential or conditional is left out at start, with the reason in the api-gateway's log; it used to fail every job. Correct the stage's type.
• Rate limits. A change of rate_limit_requests, rate_limit_period or plugin_rate_limit_requests applies to the next request, without a restart.

To roll back to 2.0.0-beta.5, check out v2.0.0-beta.5 and start it, as in Rollback: no migration ran.

To 2.0.0-beta.7¶

Core 2.0.0-beta.7 parks a message that keeps failing instead of retrying it forever, gives the retrievers that wait for their verdict a queue of their own, and cancels analyses. Every plugin now has one kind, and RabbitMQ keeps its queues when its container is recreated. Take the backup, check the points below, then run the standard upgrade. The first start writes spot.yaml again, without its comments: the backup's copy keeps the original.

Before upgrading¶

1. A plugin of several kinds stops the api-gateway. Run the check of Plugins of one kind while the old release runs.
2. Let the queues drain, right before docker compose up -d. RabbitMQ now has a fixed hostname, rabbitmq (spot/core#84). This upgrade changes its node name once more, so it starts with empty queues and the messages still queued are lost. After git pull and docker compose pull, pause what submits analyses: your scripts, the mail retrievers, and the milter (docker compose stop retriever-milter; Postfix applies its milter default_action meanwhile, and docker compose up -d starts the milter again). Wait until docker compose exec rabbitmq rabbitmqctl list_queues shows no message waiting.
3. Concurrency. The analyzer-orchestrator runs up to 14 analyses at once, up from 10: 10 from the bulk queue (MAX_CONCURRENT_JOBS) and 4 for the retrievers that wait for their verdict (MAX_CONCURRENT_SYNC_JOBS). If your analyzers cannot take more, lower MAX_CONCURRENT_JOBS in .env (see Analysis concurrency).
4. Settings no longer read. Remove from .env or docker-compose.override.yml the analyzer-orchestrator variables you set among TEST_WORKFLOW_TIMEOUT_SECONDS, JOB_TIMEOUT, MAX_WORKFLOW_TIMEOUT, ANALYZER_TIMEOUT_MS, WORKFLOW_TIMEOUT_MS, MAX_RETRY_ATTEMPTS, RETRY_BACKOFF_MS, MAX_RETRY_BACKOFF_MS and JOB_PROCESSOR_SLEEP_SECONDS: a workflow's run is bounded by its own timeout_ms. platform.rpc_timeout of spot.yaml is ignored, and dropped at its next save: on an install coming from 2.0.0-beta.6, that is the first start, which writes spot.yaml again (below).

What the first start does¶

• Migrations 011 to 013. 011 indexes analysis_results.analyzed_at for the daily statistics. 012 adds knowledge_documents.content_hash, so a provider's re-sync skips the documents that did not change; the first re-sync writes each document once more. 013 adds emails.dry_run. The migration service logs only Migrations complete; check that the database is at 013 (use POSTGRES_USER and POSTGRES_DB of .env if you changed them):

  docker compose exec -T postgres psql -U spot -d spot -Atc 'select version_num from alembic_version'   # 013_dry_run_emails
  

• The queues the analyzer-orchestrator consumes become quorum queues: orchestrator.analysis, orchestrator.analysis.sync and orchestrator.dispatch, with the dead-letter queues orchestrator.analysis.dead-letter and orchestrator.dispatch.dead-letter. The first core service to start empties the classic queues of earlier releases into them and deletes them; with RabbitMQ starting empty, there is nothing to move.

• spot.yaml. The api-gateway drops the kinds: that 2.0.0-beta.6 and earlier wrote on every plugin entry, and writes the file again. It logs Dropped kinds from plugin <section>.<id> of /app/config/spot.yaml: a plugin has one kind for each entry. A plugin's token gets the scopes of its entry's kind alone, the token unchanged. Only the token of a plugin of several kinds held more: when you cut its entry down to one kind (see Plugins of one kind), its token is narrowed to that kind, and the log says Gave the token of <id> the scopes of its kind.

After the first start¶

1. Refresh the plugin catalog (POST /api/v1/plugin-catalog/refresh, or the refresh button of the dashboard's catalog). Core now refuses a plugin setting out of the minimum, maximum or pattern its image declares, and reads them from the catalog: the entries cached by 2.0.0-beta.6 hold none.
2. The queues. The dashboard's Queues page lists the five queues above. A count on a dead-letter queue means messages are parked: see Messages in a dead-letter queue.

Plugins of one kind¶

A plugin's kind is the spot.yaml section that stores it: analyzers holds analyzers, and so on. An entry whose kinds names another kind than its section's, or several, stops the api-gateway. None of SPOT's plugins has several kinds, so only an entry written by hand can.

Before upgrading, while the old release runs, check spot.yaml in the spot-config volume. The file is block-style YAML, so a grep cannot tell one kind from several: this parses it, and prints each entry the new release refuses, nothing when the upgrade can go ahead.

docker compose exec -T api-gateway python - <<'EOF'
import yaml
plugins = (yaml.safe_load(open("/app/config/spot.yaml")) or {}).get("plugins") or {}
for section, entries in plugins.items():
    for plugin_id, entry in (entries or {}).items():
        if isinstance(entry, dict) and entry.get("kinds", []) not in ([], [section[:-1]]):
            print(f"plugins.{section}.{plugin_id}: kinds {entry['kinds']}")
EOF

For each entry it prints, either split it into one entry per kind, each with its own id, under its kind's section and without kinds:, or keep only the kind the plugin is used for. Edit the file as in the recovery below, while the old release runs, and make no other change until the upgrade: the running api-gateway's next save, such as an install or a settings change, writes its own copy over your edit.

If the new api-gateway refuses to start, docker compose logs api-gateway names the entry (plugins.<section>.<id>: kinds must be [] or [<kind>]). The api-gateway restarts in a loop and cannot be execed into, so copy the file out of the volume with a one-off container of its image, without network or log driver (what it prints would otherwise reach Docker's logs), split the entry, write the file back under a temporary name renamed into place, and start again:

docker run --rm --log-driver none --network none -v spot_spot-config:/config:ro \
  --entrypoint cat "$(docker inspect -f '{{.Config.Image}}' spot-api-gateway)" /config/spot.yaml > spot.yaml
# split the entry in spot.yaml
docker compose run --rm --no-deps -T api-gateway sh -c \
  'cat > /app/config/.spot.yaml.new && mv /app/config/.spot.yaml.new /app/config/spot.yaml' \
  < spot.yaml
docker compose up -d

Other changes¶

• Parked messages. A message whose handling keeps failing is retried with a growing wait. After 11 deliveries, about 13.5 minutes, the bulk and dispatch queues park it in their own dead-letter queue, and the milter's queue moves it into the bulk queue; one that is not JSON or names no job is parked at once. Before, it was retried at once, forever. Once the cause is fixed, an admin replays the parked messages from the Queues page or with spot-cli queues replay <queue>, one first: see Messages in a dead-letter queue.
• The milter's own queue. A retriever that waits for its verdict, such as the milter, has a queue and slots of its own, so a backlog of bulk analyses no longer holds it up. An ingest not finished within the milter's ANALYSIS_TIMEOUT_MS gets X-SPOT-Verdict: unknown and ACTION_UNKNOWN, and one not started by then is analysed later with the bulk analyses (see Mail integration).
• Cancel works. An analysis that is queued or running can be cancelled from its page in the dashboard, with spot-cli analyze cancel <job_id>, or with DELETE /api/v1/analyze/{job_id}: it stops at once and no responder runs. Before, the cancel answered "Cancel request sent" and cancelled nothing. A cancelled job counts only as cancelled in the statistics, not in their totals or verdicts.
• Dry runs (POST /api/v1/workflows/{workflow_id}/test, the dashboard's workflow test) store their message, encrypted, for the run, so analyzers fetch its raw copy and its parts as during an analysis. It is deleted when the run ends, and no list or statistic shows it. The answer no longer has note.
• Plugin settings out of their bounds (minimum, maximum, pattern, or a number that is not finite) are refused with 422 on install, update and save, naming the setting and the bound, never the value. Before, core stored them and the plugin stopped at its start.
• HTTPS. Core's services and spot-cli verify HTTPS servers against the image's system trust store (ca-certificates) instead of certifi's bundle. SSL_CERT_FILE and SSL_CERT_DIR, set on a service in docker-compose.override.yml with the file mounted, still apply, in two ways:
  • the calls to plugin registries and forges, the update check and spot-cli's calls trust only the CAs these variables name, so name a bundle that holds the public CAs too;
  • the calls between SPOT's services and to plugins, an external Ollama included, trust the system store plus the CAs that one of these variables names, or only those two when both are set. Before, these calls ignored the variables: an https plugin or an Ollama behind an internal CA now works once its CA is named.
• Logs. The HTTP client is httpx2: each request is logged under httpx2 (and httpcore2.*) instead of httpx. Update a log filter that matches httpx.
• API changes for scripts:
  • the plugin answers (/api/v1/config/plugin/..., /api/v1/plugin-catalog, /api/v1/plugins/...) have kind, no kinds, and GET /api/v1/plugin-catalog?kind= lists the plugins of that kind;
  • spot-cli plugin --kind is the plugin's section: given another, info, schema, enable, disable and token stop and name the plugin's kind;
  • DELETE /api/v1/analyze/{job_id} answers {job_id, status: "cancelled"}, 409 for a completed or failed job;
  • the daily statistics (GET /api/v1/analyze/stats/daily) count cancelled.

Web dashboard¶

Upgrade the dashboard with core: the dashboard 2.0.0-beta.7 needs core 2.0.0-beta.7.

• Cancel. The page of a queued or running analysis has a "Cancel" button for analysts and admins, and the lists and statistics show the cancelled analyses.
• Queues. The Queues page lists the analysis queues first. Each dead-letter queue says "Parked after failing", and admins have "Replay one" and "Replay all".
• Workflow tests run as an analysis does: analyzers fetch the tested message.
• Plugins show one kind each.

The rest is in the 2.0.0-beta.7 section of the CHANGELOG.

Rolling back to 2.0.0-beta.6¶

2.0.0-beta.6 does not start on a database migrated past it: its migration service stops with Can't locate revision identified by '013_dry_run_emails'. Migrations 011 to 013 have downgrades that lose nothing: 013's deletes the emails of dry runs, which are kept only for their run. Let the analyses finish first: the messages still queued are lost (below). Then downgrade the database with this release's image, before going back:

docker compose stop
docker compose run --rm -w /app/shared/database migration alembic downgrade 010_contract_v2
git checkout v2.0.0-beta.6   # or pin the versions, below
docker compose up -d

spot.yaml needs nothing: 2.0.0-beta.6 reads entries without kinds: as plugins of one kind.

RabbitMQ under 2.0.0-beta.6's compose file has no fixed hostname: it starts as a new node, without the new queues, and the messages still queued are lost. If you keep this deploy release and pin CORE_VERSION and DASHBOARD_VERSION to 2.0.0-beta.6 instead, RabbitMQ keeps the new queues, which stay bound and would collect every analysis nobody reads. After starting the older release, look at the two dead-letter queues (see Messages in a dead-letter queue), then delete the five queues:

for queue in orchestrator.analysis orchestrator.analysis.sync orchestrator.dispatch \
    orchestrator.analysis.dead-letter orchestrator.dispatch.dead-letter; do
  docker compose exec rabbitmq rabbitmqctl delete_queue "$queue"
done

To 2.0.0-beta.8¶

Core 2.0.0-beta.8 keeps the history of spot.yaml, chooses each email's workflow by routes, runs shadow workflows beside production, and sets each mail connection's mode, observe or enforce, in SPOT. It also writes JSON logs that carry each request's correlation ID, serves Prometheus metrics, and keeps an audit log (Monitoring). Migrations 014 to 017 change the database, and 017's downgrade can refuse (Rolling back to 2.0.0-beta.7). 2.0.0-beta.7 has neither spot-cli backup nor spot-cli upgrade: check the points below, then back up and upgrade by hand.

Before upgrading¶

1. Upgrade the milter with core. Core 2.0.0-beta.8 refuses the ingests of a milter older than 2.0.0b4, which name their workflow (422): that milter answers error on every message, and its ON_ERROR applies (accept by default delivers the mail unanalysed). The release's compose file runs the milter 2.0.0b4: if you pinned RETRIEVER_MILTER_VERSION in .env, set it to 2.0.0b4 or remove it. A milter installed from the catalog instead: update it right after the upgrade, spot-cli plugin update retriever-milter --version 2.0.0b4.
2. Note the milter's mode and workflow. A milter run by this compose file had them as RETRIEVER_MILTER_MODE and RETRIEVER_MILTER_WORKFLOW_ID in .env. The new compose file no longer passes them, and the milter 2.0.0b4 ignores them: note their values, which you set in SPOT after the first start (The milter 2.0.0b4).
3. Responders observe until their connection enforces. Every mail connection is in observe until switched, whatever its workflows' response_mode said: the first start drops response_mode. Responders act only for the jobs of connections in enforce, on routes in enforce, and never for submissions. A responder that acts today stops acting until you switch its connection (below).
4. workflows.yaml is no longer read, nor SPOT_WORKFLOWS_FILE. Under 2.0.0-beta.7, the analyzer-orchestrator ran the workflows of workflows.yaml whenever the file existed, those of spot.yaml aside: its list is the one your jobs run, and the one to keep. If the spot-config volume holds one (docker compose exec api-gateway ls /app/config), or you set SPOT_WORKFLOWS_FILE in docker-compose.override.yml, copy its workflows: list into spot.yaml, in place of the workflows: there, edited as in Editing spot.yaml by hand, while the old release runs; then reload (POST /api/v1/config/reload) so that the api-gateway's next save keeps them.
5. Logs are JSON lines by default. The core services log one JSON object per line (LOG_FORMAT, json by default). A log pipeline that parsed the old text lines needs a JSON parser, or LOG_FORMAT=text in .env. LOG_FORMAT=simple and detailed stop the services at start: remove them from .env or docker-compose.override.yml.
6. Add SPOT_METRICS_TOKEN to .env (openssl rand -hex 32), see .env.example. Until it is set, every /metrics answers 503. Scrape it from inside spot-network only, as in A Prometheus beside SPOT: Traefik does not route the api-gateway's /metrics. The analyzer-orchestrator's JSON /metrics, which earlier notes pointed monitoring at, is gone: it is the Prometheus endpoint now, and GET /api/v1/analyze/stats gives the totals.
7. ENABLE_AUDIT_LOGGING and ANONYMIZE_PII are no longer read; they did nothing. Remove them from .env or docker-compose.override.yml if you set them.
8. Set REDIS_PASSWORD in .env (openssl rand -hex 32). Redis now always asks for it, and docker compose up stops without it, naming it. Letters and digits only: it goes into the services' redis:// URLs (Redis). Every service is recreated with it.
9. The egress proxy reaches only the plugins of egress class indicators or content, and never with its user and password in their environment. Behind an authenticated proxy, update analyzer-llm to 2.0.0b4 and analyzer-misp to 2.0.0b5, which read them from /run/secrets/spot_proxy_credentials. Proxy variables with different credentials, or credentials on some only, are refused. A plugin of class none gets no proxy: analyzer-nlp's MODEL_PATH as a URL or a Hugging Face id needs direct egress (Running behind an egress proxy).

The upgrade by hand¶

2.0.0-beta.7 has no spot-cli backup or spot-cli upgrade, so this upgrade is the last one by hand. While 2.0.0-beta.7 runs, back up the database, spot.yaml and .env into a directory of their own, readable by you only, and check the dump: it ends with its completion line, holds migration 013, and restores into a scratch database. Use POSTGRES_USER and POSTGRES_DB of .env if you changed them.

cd /opt/spot
backup=backups/2.0.0-beta.7-$(date -u +%Y%m%d-%H%M%S)
(umask 077 && mkdir -p "$backup" && cp .env "$backup/env")
docker compose exec -T postgres pg_dump -U spot spot > "$backup/spot-backup.sql"
docker compose cp api-gateway:/app/config/spot.yaml "$backup/spot-yaml-backup.yaml"
grep -c 'PostgreSQL database dump complete' "$backup/spot-backup.sql"          # 1
grep -A1 '^COPY public.alembic_version' "$backup/spot-backup.sql" | tail -n 1  # 013_dry_run_emails
docker compose exec -T postgres createdb -U spot spot_check
docker compose exec -T postgres psql -U spot -d spot_check -v ON_ERROR_STOP=1 -q < "$backup/spot-backup.sql"
docker compose exec -T postgres dropdb -U spot spot_check

Then check the release out, set REDIS_PASSWORD and SPOT_METRICS_TOKEN in .env (above), and start it:

git fetch --tags && git checkout --detach v2.0.0-beta.8
docker compose config --quiet   # names a setting the release requires that .env lacks
docker compose pull && docker compose up -d

From 2.0.0-beta.8 on, upgrade with scripts/spot-cli upgrade <version> (Standard upgrade): it checks .env against the release, and backs up first. A backup taken by hand, such as this one, has no manifest: spot-cli restore refuses it, and A backup taken by hand restores it.

What the first start does¶

• Migrations 014 to 017. 014 adds analysis_results.route and the shadow_differences table, 015 analysis_results.mode and the analysis_actions table, 016 recreates audit_logs, which nothing had written, in the audit log's shape, and 017 keys knowledge documents by their source and id, and deletes those without a source, written before 2.0.0-beta.4, logging how many. Check that the database is at 017 (use POSTGRES_USER and POSTGRES_DB of .env if you changed them):

  docker compose exec -T postgres psql -U spot -d spot -Atc 'select version_num from alembic_version'   # 017_knowledge_ids_per_source
  

• The history of spot.yaml starts, in history.git beside it on the spot-config volume. Its first commit, spot.yaml as found at start, by SPOT, holds the file once the plain secret settings of catalog plugins are sealed, the conversions below included. See History of spot.yaml.

• Retrievers' settings. An installed retriever's WORKFLOW_ID setting becomes a route of its own, {sources: [<id>]} to that workflow, appended to the routes and named after the retriever: Mail retriever <id>: its WORKFLOW_ID <workflow> is now route <name>. A workflow that is not loaded is dropped, with a warning: that retriever's mail then runs the other routes or the default workflow. Its MODE setting becomes its connection's mode, when it is observe or enforce: Mail retriever <id>: its MODE setting is now its mode, <mode>; another value is dropped, with a warning. Both settings leave the retriever's settings.
• Workflows lose their response_mode, each change logged, and are written back once.
• spot.yaml is written again when one of these changes it, with its recipient_groups, routing and shadow sections, empty ones included, and without its comments: the backup's copy keeps the original.
• The shadow's queue, orchestrator.shadow, and its dead-letter queue are declared and bound to the results. They hold at most 1 000 messages each, dropping the oldest. The dashboard's Queues page lists them.
• Every service gets a memory limit (Memory limits): a container that passes it is killed, and Docker restarts it. Raise one in docker-compose.override.yml, as that section shows. The bundled Ollama's is OLLAMA_MEMORY_LIMIT in .env, 12G by default, what llama3.1 8B needs: raise it for a larger model, or after an OOM kill (Ollama's limit).
• Every plugin container is re-created once, to rotate its logs: at most five files of 10 MB. What a plugin holds in its container outside a volume, such as a file copied in by hand, is gone: copy it in again.
• The audit log starts recording: sign-ins, plugin lifecycle actions, user accounts, plugin sources, deletions of stored data and replays of parked messages. An entry is kept for audit_retention_days, 365 days by default (Retention).

After the first start¶

1. Switch the connections you want enforced. Watch what each connection would do, recorded as each job's actions, then switch it: spot-cli plugin mode <id> enforce, or from its page in the dashboard. Nothing restarts. A retriever whose MODE setting was enforce is in enforce already.
2. The milter run by compose: see The milter 2.0.0b4.
3. Check the routes the first start made: spot-cli routing show, or the dashboard's Routing page. A route on sender domains needs mail.trusted_authserv_ids (see Authentication results): SPOT refuses to save one while the list is empty.
4. The orchestrator's old counters. The Redis hash spot:orchestrator:metrics, which the JSON /metrics read, is no longer written: delete it with docker compose exec redis redis-cli DEL spot:orchestrator:metrics.
5. Audit entries of the dashboard. An action taken in the dashboard is recorded with the dashboard's address and User-Agent (spot-web-dashboard/<version>), not the browser's (Audit log).
6. Plugins log JSON lines with the correlation ID once they pin spot-sdk-python 2.0.0b11 or later: retriever-milter 2.0.0b4, the release's default (RETRIEVER_MILTER_VERSION), analyzer-llm 2.0.0b4 and analyzer-misp 2.0.0b5 do; analyzer-nlp, analyzer-ml, analyzer-rules and provider-employee-dir follow in their next releases.
7. The wrapper. scripts/spot-cli runs the CLI as the calling user and without a log driver, and loads scripts/ops/common.sh: it must stay in the deploy directory, or be a symlink to it. Replace a copy installed on PATH with the symlink: sudo ln -sf /opt/spot/scripts/spot-cli /usr/local/bin/spot-cli.

The milter 2.0.0b4¶

The milter takes its mode from SPOT, in each answer, and names no workflow: SPOT's routes choose it. For a milter run by this compose file, set in SPOT what .env held:

• RETRIEVER_MILTER_MODE=enforce: switch its connection once its actions in observe look right, spot-cli plugin mode retriever-milter enforce. Until then it observes.
• RETRIEVER_MILTER_WORKFLOW_ID=<workflow>: add a route {sources: [retriever-milter]} to that workflow, as in The milter's workflow. Until then its mail runs the default workflow.

Then remove both variables from .env. A milter installed from the catalog had them as its MODE and WORKFLOW_ID settings, which the first start moved (above). The milter's log line at start no longer shows a mode: Milter listening on 0.0.0.0:9900.

Other changes¶

• Rollback of the configuration. Admins read every change of spot.yaml and roll it back, every section but plugins:: spot-cli config history, config diff <commit> and config rollback <commit>, or the History of the dashboard's Configuration page. A workflow has its own versions: spot-cli workflow history <id> and workflow restore <id> <commit>. The history is never pruned, and spot-cli backup holds it with spot.yaml.
• Shadow workflows run again on the emails their match takes, after production has decided, and SPOT keeps where their verdict differs (see Routing). Each adds one workflow run per email it takes, which competes with production's for the analyzers.
• A workflow that a route or the shadow names cannot be deleted (409): change the routing first.
• Settings no longer read from the environment. DEFAULT_WORKFLOW_ID, MAIL, INTAKE, PLUGINS and WORKFLOWS, which an override file could set, no longer apply: set them in spot.yaml.
• API changes for scripts:
  • POST /api/v1/workflows/clone is gone: POST /api/v1/workflows/{workflow_id}/clone (spot-cli workflow clone <id> <new-id>) copies a workflow;
  • a workflow save is numbered by SPOT, and the body's version and created_by are ignored;
  • a workflow written with response_mode is refused (400);
  • a mail retriever's WORKFLOW_ID and MODE settings are refused (422), on the config routes, an install, an update and the settings route: a route on its source chooses its workflow (spot-cli routing apply), and spot-cli plugin mode switches its mode. One written in spot.yaml by hand leaves the settings at the api-gateway's next start, a WORKFLOW_ID as a route;
  • POST /internal/ingest refuses a request that names a workflow (422);
  • an analysis has route and mode, and its status lists its actions;
  • the analyzer-orchestrator's own workflow write routes are gone: the api-gateway writes the workflows, into spot.yaml;
  • GET and DELETE /api/v1/knowledge/documents/{id} are now /api/v1/knowledge/sources/{source}/documents/{id}.
• spot-cli. New: audit, routing show and routing apply, plugin mode, plugin restart and plugin resources, profile list, profile requirements, profile env and profile apply, config history, config diff and config rollback, workflow history, workflow restore and workflow clone; scripts/spot-cli-completion.bash completes them. config validate has no --kind any more. The CLI runs in a container that sees no host file: routing apply and config validate read a copy from standard input (spot-cli routing apply /dev/stdin < routing.yaml).

Web dashboard¶

Upgrade the dashboard with core: the dashboard 2.0.0-beta.8 needs core 2.0.0-beta.8.

• Routing, under Configure, shows the routes, the recipient groups and the shadow workflows to analysts and admins; admins edit them. A mail retriever's page lists the routes that name it, and its mode, which admins switch; its "Workflow binding" is gone.
• History. The Configuration page lists the changes of spot.yaml and rolls one back, and a workflow's page lists its versions and restores one.
• Analyses show their route, their mode and their actions.
• Audit log, under Administration, shows admins who did what, newest first, filtered by family, actor, outcome and day.
• Knowledge documents are addressed by their source and their id, at /knowledge/<source>/<id>: a link to /knowledge/<id> is not found.

The rest is in the 2.0.0-beta.8 section of the CHANGELOG.

Rolling back to 2.0.0-beta.7¶

Either way back, the milter must run a release before 2.0.0b4, such as 2.0.0b3: the milter 2.0.0b4 takes its mode from SPOT's answers, which 2.0.0-beta.7 does not give, so it observes for good, whatever its MODE says. Then delete the shadow's queues (below): neither way touches RabbitMQ.

Restore the backup taken before the upgrade, as in A backup taken by hand, and put its .env copy back before starting 2.0.0-beta.7 (its step 4): cp -p <backup>/env .env. The database, .env and spot.yaml go back to what they were before the upgrade, the milter's mode and workflow included, and what SPOT did since is lost. A milter installed from the catalog runs the version the restored spot.yaml names, and one run by compose the version its .env pins, else 2.0.0b1, the default of 2.0.0-beta.7's compose file. If the restored .env pins RETRIEVER_MILTER_VERSION=2.0.0b4, as "Before upgrading" offered, set it to 2.0.0b3, then run docker compose up -d.

Or go back by hand, to keep the analyses made since. 2.0.0-beta.7 does not start on a database migrated past it: downgrade it with this release's image first. The downgrade drops the routes and modes the analyses recorded, their actions, the shadow's differences and the audit log; the knowledge documents 017 deleted stay deleted. 017's downgrade refuses once two sources hold the same document id ("restore the backup taken before the upgrade instead"): roll back with the backup. For a milter run by compose, set in .env, before docker compose up -d: RETRIEVER_MILTER_VERSION=2.0.0b3, and RETRIEVER_MILTER_MODE and RETRIEVER_MILTER_WORKFLOW_ID as they were.

docker compose stop
docker compose run --rm -w /app/shared/database migration alembic downgrade 013_dry_run_emails
git checkout v2.0.0-beta.7
docker compose up -d

• A milter installed from the catalog stays on the version spot.yaml names. Take it back, then set its two settings again, as they were; they merge into the others, and the milter restarts once more:

  spot-cli plugin update retriever-milter --version 2.0.0b3
  spot-cli plugin settings retriever-milter --set MODE=<mode> --set WORKFLOW_ID=<workflow>
  

  Not plugin update --set: the settings it is given replace all the others. - spot.yaml. 2.0.0-beta.7 ignores the routing, the shadow and the connections' modes, and its next save drops them. The WORKFLOW_ID and MODE settings that the first start moved are not restored: until you set them again, the retrievers' mail runs the default workflow, and the retrievers observe. The workflows have no response_mode any more: set response_mode: enforce again on those whose responders should act. - history.git stays on the volume, unread. Upgrading again records the file as found at that start.

The shadow's queues, either way, stay bound to the results, and collect up to 1 000 of them each that nobody reads. Delete them:

for queue in orchestrator.shadow orchestrator.shadow.dead-letter; do
  docker compose exec rabbitmq rabbitmqctl delete_queue "$queue"
done

To 2.0.0-beta.9: the spot repository¶

From this release SPOT's deployment files are in the spot repository, in its deploy/ directory, next to the platform's code and the web dashboard's: one repository, one version, with the history of the former deploy repository. The images, their tags, the ports, the environment variables and the volumes do not change: name: spot in the compose file keeps the volume names whichever directory Compose runs from. What changes is where you run it. Move once, with the old version still running:

cd /opt/spot && scripts/spot-cli backup        # the backup's directory is printed
docker compose down                            # without -v: the volumes stay
cd /opt && mv spot spot-deploy                 # the old checkout: keep it, see below
git clone https://codeberg.org/SPOT_Project/spot.git spot
cd spot && git checkout --detach v2.0.0-beta.9
cp ../spot-deploy/.env deploy/.env
cp ../spot-deploy/docker-compose.override.yml deploy/ 2>/dev/null || true
cd deploy && docker compose pull && docker compose up -d
sudo ln -sf /opt/spot/deploy/scripts/spot-cli /usr/local/bin/spot-cli

Behind an egress proxy, give git the proxy for the clone (git -c http.proxy=http://proxy.example.com:3128 clone ...), as for any upgrade. With plugins installed, docker compose down ends with an error about the spot-plugins network having active endpoints: plugin containers are not Compose services and keep running. It is harmless, up reuses the network.

Then point everything that names /opt/spot at the new paths. Each is easy to miss, and none fails loudly:

• The nightly backup. If you installed the cron line of Backup and restore, change it to cd /opt/spot/deploy && scripts/spot-cli backup .... Check with grep -r /opt/spot /etc/cron.d /etc/crontab and crontab -l, and by running the line by hand: a new directory appears in /opt/spot/deploy/backups/. Until then the old line fails every night with "not found" and the backups stop.
• The completion. If you sourced /opt/spot/scripts/spot-cli-completion.bash (in ~/.bashrc, ~/.zshrc or /etc/bash_completion.d/), source /opt/spot/deploy/scripts/spot-cli-completion.bash instead. Check with grep -r spot-cli-completion ~/.bashrc ~/.zshrc /etc/bash_completion.d.
• The wrapper's link. The ln -sf above replaces /usr/local/bin/spot-cli; check with readlink -f /usr/local/bin/spot-cli, which must name /opt/spot/deploy/scripts/spot-cli. A link elsewhere (~/.local/bin/spot-cli) is replaced the same way.
• Files your override mounts. Open deploy/docker-compose.override.yml: a volume such as /opt/spot/extra:/extra now names a path of the new clone, where Docker creates an empty directory and starts the service on it. Move the files from /opt/spot-deploy/extra to /opt/spot/deploy/extra and change the path, or point it at /opt/spot-deploy/... for as long as you keep that directory. Check with grep -n /opt/spot deploy/docker-compose.override.yml, then docker compose config from deploy/, which prints the paths it will mount.

What stays true:

• Compose is run from deploy/. Later upgrades are scripts/spot-cli upgrade <version>, from /opt/spot/deploy.
• New backups go to /opt/spot/deploy/backups/.
• Backups taken before the move are restored from the old checkout, and only there. A backup names the release commit it was taken at, and the commits of the former deploy repository are not in spot: they were rewritten when the history was imported. So /opt/spot-deploy and its backups/ stay where they are, and are not copied into the new checkout.
• To go back to the release before the move, take a backup in the new checkout first, then restore the old one from the old checkout:

cd /opt/spot/deploy && scripts/spot-cli backup
cd /opt/spot-deploy && scripts/spot-cli restore backups/<the backup taken before the move>

Do not run docker compose down before the restore: it removes the Postgres container, and the restore then cannot take its safety backup and refuses. It stops the stack itself. It is the same spot project, with the same volumes. The restore's safety backup is taken by the old checkout, so it records the old release beside the new database: no checkout can restore it. The way back to the new release is the backup you just took in /opt/spot/deploy, restored from there. - Remove /opt/spot-deploy only once a rollback to before the move is no longer wanted. - The dashboard's System -> Updates page and spot-cli upgrade read the version tags of the spot repository from now on.

To 2.0.0-beta.10¶

Check the workflow ids first, as below says, then run scripts/spot-cli upgrade 2.0.0-beta.10 from deploy/. No migration runs. The services restart once, as their network moves onto a fixed subnet, and every installed plugin is re-created once.

The network gets a fixed subnet¶

spot_spot-network now has the fixed subnet DOCKER_NETWORK_SUBNET (192.168.229.0/24 by default). The api-gateway trusts X-Forwarded-For from it only, so it sees the address of each caller behind Traefik and the dashboard, no longer Traefik's or the dashboard's.

Every install before this release has a network without a subnet. docker compose up -d brings it to the subnet by itself: compose stops the containers, removes the network, creates it again with the subnet and starts them again. The containers are the same, and no volume is touched. The network itself needs no step, but the services restart once. Checked with Docker Compose 5.0.2 and 5.5.1 on two containers started before the subnet was added. A container you attached to spot_spot-network yourself with docker run or another compose file stops the removal of the network: disconnect it first, or move it to spot-plugins.

If .env sets the dashboard's FORWARDED_ALLOW_IPS to the network's former subnet, the dashboard stops trusting Traefik once the network moves: it sees Traefik's address, and passes it on to the api-gateway for every sign-in. Set it to DOCKER_NETWORK_SUBNET, or remove it: its default, *, is safe while Traefik is the only way in.

Check the subnet an install has:

docker network inspect spot_spot-network --format '{{json .IPAM.Config}}'

If the subnet collides with a network of the host, compose fails with Pool overlaps with other one on this address space. Compose has already stopped the services and removed the network by then, so the services stay stopped until the next up -d. Set DOCKER_NETWORK_SUBNET in .env to a free one (docker network ls and docker network inspect list the used ones), then run docker compose up -d again.

Plugins of egress class none lose their route out¶

A plugin of egress class none now runs on spot-plugins-isolated, an internal network with no route out, which reaches only the api-gateway, the knowledge service, the analyzer-orchestrator and the bundled Ollama: an Ollama outside the stack, set by OLLAMA_URL, is out of its reach. At the first start, the api-gateway re-creates every installed plugin once, through the plugin-manager, since a plugin's container now records whether it is isolated: each keeps its settings and secrets and restarts, and those of class none move onto spot-plugins-isolated. A plugin with no recorded class, such as a manual entry, stays on spot-plugins.

A plugin of class none that fetched something by itself, a model by URL or by Hugging Face id for example, can no longer: give it the model in its image or on a volume. docker compose up -d creates the network.

Workflow ids follow a pattern¶

A workflow's id is a letter or a digit, then up to 127 letters, digits, dots, underscores or hyphens. A stored workflow with another id is left out at start, with a warning naming the rule. An analysis that names it fails, and a route that names it is skipped: its mail goes on to the next route, or to the default workflow in its connection's mode. A route in observe on a connection in enforce then no longer holds its mail in observe: that mail is enforced. SPOT renames no id, since a rename would break the routes and the history that name it.

Before upgrading, check the workflow ids in spot.yaml on the config volume (grep -n '^ *- id:' spot.yaml) or with spot-cli workflow list, and rename any outside the rule, routes included.

Rolling back to 2.0.0-beta.9¶

Restore the backup spot-cli upgrade took, as Rollback says. When 2.0.0-beta.9 starts, compose moves spot_spot-network back to a subnet Docker picks, restarting the services once, and the api-gateway re-creates every installed plugin once, those of egress class none on spot-plugins. The spot-plugins-isolated network stays, unused: remove it with docker network rm spot-plugins-isolated once docker network inspect spot-plugins-isolated --format '{{len .Containers}}' says 0.

To 2.0.0-rc.1¶

No migration runs: scripts/spot-cli upgrade 2.0.0-rc.1 from deploy/ restarts the services, and the plugins are not re-created. Then update each installed plugin to its 2.0.0 release, on spot-sdk-python 2.0.0:

spot-cli plugin list
spot-cli plugin update analyzer-nlp --version 2.0.0   # and so on, for each plugin listed

A consent recorded for a plugin of egress class content, such as analyzer-llm, covers its new version; an entry without one, written by hand, needs --consent-content. Each plugin's update re-creates its container, with its settings and secrets. The milter run by the mail-milter profile moves to 2.0.0 with the upgrade, unless .env pins RETRIEVER_MILTER_VERSION.

To go back to 2.0.0-beta.10, restore the backup spot-cli upgrade took, as Rollback says: its spot.yaml names the plugins' earlier versions, and the api-gateway re-creates them so. Since no migration ran, checking out 2.0.0-beta.10 and starting it instead keeps what was written since, and the plugins' 2.0.0 releases, which run on 2.0.0-beta.10 too.

To 2.0.0¶

From 2.0.0-rc.1, scripts/spot-cli upgrade 2.0.0 from deploy/: no migration runs, and the plugins are not re-created.

From 1.x or an earlier 2.0 pre-release, back up first, then follow each section of this page in order, from the one after the release you run (From 1.x to 2.0 for 1.x), down to this one. Up to 2.0.0-beta.8, which brought spot-cli backup and spot-cli upgrade, the checkout has neither: back up and upgrade each release by hand, as The upgrade by hand of 2.0.0-beta.8 does (a dump, spot.yaml and .env, then git fetch --tags && git checkout --detach v<version> and docker compose pull && docker compose up -d). The install moves from the deploy repository to the spot repository at 2.0.0-beta.9. Once on 2.0.0, update every installed plugin to its 2.0.0 release, as To 2.0.0-rc.1 says.

To go back to 2.0.0-rc.1, check out v2.0.0-rc.1 and start it, as Rollback says: no migration ran.

To 2.0.1¶

From 2.0.0, check the Docker Engine first (below), then run scripts/spot-cli upgrade 2.0.1 from deploy/: no migration runs, and the plugins are not re-created. To go back to 2.0.0, check out v2.0.0 and start it, as Rollback says: no migration ran.

The bundled Ollama no longer joins spot-plugins-isolated: a plugin of egress class none reaches the api-gateway, the knowledge service and the analyzer-orchestrator only. Ollama has a route out, and its API pulls from and pushes to a registry the caller names, so a class none plugin could send data out through it. docker compose up -d, as spot-cli upgrade runs it, disconnects Ollama from that network. A plugin that needs a model gets embeddings through the knowledge service.

SPOT needs Docker Engine 25.0.4 or later, 24.0 before. On earlier engines the embedded DNS of an internal network still forwards lookups of outside names to the host's resolvers (CVE-2024-29018), so a plugin of egress class none could send data out as DNS queries. spot-cli setup checks the new floor. Check yours with docker version -f '{{.Server.Version}}', and when it is older, upgrade Docker before SPOT.

provider-employee-dir is of egress class none, so since 2.0.0-beta.10 it runs on spot-plugins-isolated, where its ldap backend reaches no directory: each sync fails while /health stays healthy, and the Knowledge Store keeps the employees of the last sync that worked. Use its yaml backend for now. Letting a plugin reach the hosts an admin names is spot/spot#143. This still holds for a plugin upgraded to 2.0.2 until its url settings are recorded, as To 2.0.2 says.

To 2.0.2¶

From 2.0.1, run scripts/spot-cli upgrade 2.0.2 from deploy/. It adds the egress gateway and two networks, spot-plugins-reach and spot-egress. No migration runs, and no plugin is re-created: a plugin gets its reach, and is re-created on the gateway, only once its settings of type url are recorded (below). To go back to 2.0.1, check out v2.0.1 and start it, as Rollback says: no migration ran.

A plugin of egress class none now reaches the hosts its settings of type url name, through the egress gateway: see Reaching configured hosts. An installed plugin gets its reach once its url settings are recorded, which happens at its next update or settings change, from the dashboard or from spot-cli. For provider-employee-dir's ldap backend, set its settings once after the upgrade, which records them and restarts the plugin on the gateway:

spot-cli plugin settings provider-employee-dir --set SOURCE_BACKEND=ldap --set LDAP_URL=ldaps://ldap.corp.example

When 192.168.231.0/24 collides with a network of the host, set PLUGIN_REACH_SUBNET and EGRESS_GATEWAY_ADDRESS in .env before upgrading.

After going back to 2.0.1 and upgrading again, set each plugin's settings once more: 2.0.1 writes spot.yaml back without the url settings it does not know, so every plugin loses its reach until then.

To 2.1.0¶

From 2.0.2 or a 2.1.0 pre-release, run scripts/spot-cli upgrade 2.1.0 from deploy/. From a release before 2.1.0-beta.4, the plugins of egress classes indicators and content are re-created once: read Every plugin reaches only the hosts it is configured for first. The migrations your release has not run yet run, and ask nothing of you: 018 adds two tables, benchmark_runs and benchmark_results; 019 adds two tables and two indexed columns to emails, whose indexes take a while on a large table; 020 adds shadow_tallies, a partial unique index on benchmark_runs that lets one platform run run at a time, benchmark_results.baseline_job_id, and an index on emails.created_at, which also takes a while on a large table. The new queue orchestrator.benchmark, with its dead-letter queue, is declared at the first start and appears on the Queues page. To go back to 2.0.2 or a pre-release, restore the backup spot-cli upgrade took, as Rollback says, rather than checking that release out: it does not start on a database migrated past it.

Before upgrading, rename a plugin whose id is spot, in any section of spot.yaml's plugins:. spot is now the id core's own signals are attributed to, those of the sender lists, and a spot.yaml that holds it stops the api-gateway at its start. An analyzer's signal named sender_list.blocked or sender_list.allowed is now shown and never decisive: only core's lists give those.

A copy of an email promoted into a dataset is exempt from the retention purge until it is removed from its dataset or the dataset is deleted (Retention).

analyzer-rules now observes in the shipped workflows, and an attachment name that hides its real extension behind a bidi control (attachment.bidi_override) is decisive. The change reaches new installs and the defaults only: a spot.yaml that holds the workflows keeps them as they are. To give an existing workflow the same, add the signal to its verdict_policy:

verdict_policy:
  decisive_signals:
    - signal: "attachment.bidi_override"
      verdict: "suspicious"

A workflow that does not run analyzer-rules also needs it in a stage, as an observer, which only its decisive signals count for (a workflow of spot-cli setup already runs it at weight 1):

analyzers:
  - id: "analyzer-rules"
    weight: 0
    timeout_ms: 10000
    required: false
    failure_strategy: "skip"

Edit the workflow from the dashboard or spot-cli workflow update. Where analyzer-rules is not installed, the observer is unknown and the workflow decides as before.

Through the wrapper, analyze submit, workflow create and update, routing apply and config validate now see the directory they run from, read-only, so spot-cli analyze submit -f mail.eml reads ./mail.eml. /dev/stdin and -f - keep working. A .spot-token of spot-cli 1.x in the working directory is no longer moved into the token directory: its user signs in again with spot-cli auth login.

With analyzer-llm 2.1, a request takes up to 12400 tokens with the default settings, where 2.0 took 11300: its margin for the instructions and the answer went from 2300 to 3400 tokens. An Ollama of your own needs OLLAMA_CONTEXT_LENGTH of at least 12400; the bundled side-car's 16384 is enough. With a lower context, keep LLM_MAX_BYTES + LLM_MAX_TOKENS + 3400 within it (The side-car and analyzer-llm).

Every plugin reaches only the hosts it is configured for¶

From 2.1.0-beta.4, a plugin of every egress class runs as a plugin of class none has since 2.0.2: on spot-plugins-isolated, without a route out, reaching the hosts its settings of type url name through the egress gateway, and nothing else (Reaching configured hosts). At the first start, the api-gateway re-creates once each installed plugin of class indicators or content, with its settings and secrets. A plugin of class none is not re-created, unless it gets a reach.

Before upgrading, check each plugin of class indicators or content (analyzer-misp, analyzer-llm, and the plugins you installed):

• each url setting names a host, not an address, or names it for the gateway, as Reaching configured hosts shows. The gateway also refuses a name that resolves to an address inside its own networks, spot-plugins-reach and spot-outbound (EGRESS_SUBNET, 192.168.230.0/24 by default), the bundled Ollama's names excepted for a content plugin. Set EGRESS_SUBNET in .env before upgrading when it collides with a network of the host;
• its url settings name one host per port: two of them naming different hosts on one port are refused, and the plugin is then stopped;
• it calls no host its url settings do not name: it can no longer reach one;
• no url setting is stored as a secret (secrets: in its entry). The API and the dashboard keep a stored secret sealed, so they cannot fix it: move it back to settings: by hand (Editing spot.yaml by hand).

A plugin you run yourself on spot-plugins no longer reaches the bundled Ollama: check that none calls it.

Behind a proxy, the egress gateway goes through it, with CONNECT, for every host noProxy does not list (Running behind an egress proxy). List in noProxy the hosts a plugin reaches directly, those of class none included: provider-employee-dir's LDAP went direct in 2.0.2. A proxy that tunnels port 443 only (Squid's default) refuses a host on another port: list it in noProxy, or let the proxy tunnel that port. The gateway tunnels through an http:// proxy only: an https:// or SOCKS URL in HTTPS_PROXY turns its /health unhealthy, and spot-cli upgrade then fails waiting for it, so check the URL first.

A plugin whose url settings are refused is stopped at the first start, whatever its class. A plugin of class indicators or content, or of no recorded class, that cannot be re-created for any other reason (its secrets cannot be decrypted, the plugin-manager refuses it) is stopped too. It stays enabled, so fixing the cause starts it. The api-gateway's log says Reconcile refused <id>, stopped: ..., naming the setting and the rule, or Reconcile failed <id>, stopped: .... Docker being unavailable is not one of these cases: the api-gateway tries again. A plugin of class none that cannot be re-created for another reason is left running, since its container has had no route out since 2.0.2. An entry that spot.yaml holds but the api-gateway cannot read, such as one with a malformed field edited by hand, is logged and left as it is until the entry is fixed: it is not stopped. Setting the refused value right starts the plugin:

spot-cli plugin settings analyzer-misp --set MISP_URL=https://misp.corp.example

An entry installed before 2.0.2 records no url settings. At the first start, the api-gateway takes them from the catalog: Updated the url settings of analyzer-misp in its log, Record the url settings of analyzer-misp in the history of spot.yaml. When the catalog cannot be read then, the plugin runs isolated without a reach (The url settings of analyzer-misp are not recorded in the log): set its settings once, as for 2.0.2.

docker compose up -d, as spot-cli upgrade runs it, moves the bundled Ollama from spot-plugins to spot-outbound. Only a plugin of class content reaches it.

The upgrade creates spot-outbound, the egress gateway's new way out, and leaves spot-egress as it was, unused, for a rollback. Remove it with docker network rm spot-egress once you will not roll back.

If the upgrade stops with Pool overlaps with other one on this address space, 192.168.230.0/24 (or the EGRESS_SUBNET you set) is taken on this host. Set EGRESS_SUBNET in .env to a free /24 and run the upgrade again.

To go back to the release you upgraded from, restore the backup spot-cli upgrade took. The api-gateway then re-creates the plugins of classes indicators and content on spot-plugins, with the proxy, and compose moves Ollama back. The rollback leaves spot-outbound behind, empty. Before you change EGRESS_SUBNET and upgrade again, remove it with docker network rm spot-outbound: otherwise compose cannot re-create it, and the upgrade stops with container ... is not connected to the network spot-outbound.

The plugin sources move into spot.yaml¶

The plugin sources and their tokens are kept in spot.yaml, under plugins.sources, no longer in Redis. The first start of the api-gateway writes them there, with their tokens sealed as they were, and nothing is to be done. A backup now holds them.

Redis keeps its copy, unread, so that a rollback to the release you upgraded from finds the sources it had; a source added or changed since is not in it. Going back writes spot.yaml without its sources at that release's first start, and upgrading again moves those of Redis once more: add again, with its token, a source added or changed in 2.1, and remove again one removed in 2.1. A start that cannot read Redis moves nothing, and tries again at the next start; until a start has moved them, no source can be added (the API answers 503). A 2.0 source that no longer validates is not moved: the start's log names it, and you add it again.

The catalog lists the versions this SPOT runs¶

The catalog lists, for each plugin, the version an install takes on this SPOT, no longer the latest one scanned: SPOT 2.1.0 lists analyzer-rules 2.0.0, where 2.0 and the 2.1 pre-releases listed 2.1.0b1. The first start of the api-gateway after an upgrade refreshes the catalog once, in the background, as Refresh catalog on the Plugins pages does; until it ends, the catalog shows its earlier listing. A refresh that fails, or that cannot scan one of the sources, is logged, and the next refresh or start tries again. Every later upgrade does the same.

The plugin index¶

The first start adds the signed plugin index, spot-plugin-index, to the plugin sources once, before the Docker Hub source by priority (50, where dockerhub-spot-project has 100), so an install that names no source takes the index's entry (when, and the opt-out). See The signed plugin index.

The plugins installed keep their source: those of dockerhub-spot-project update from it as in 2.0, and no container is re-created. spot-cli plugin update <id> --source spot-plugin-index moves one to the index, which re-creates its container once, on its image by digest; its settings and secrets stay as they are. The Docker Hub source can be removed once no plugin installed from it is left: spot-cli plugin list shows each one's source.

The index's keys change through SPOT releases: a release ships the new key, the index is then signed with both, and the old one is dropped later. An install that does not upgrade stops reading the index once its key no longer signs it, and keeps trusting a key a later release removed: upgrade with each release.

Release notes¶

See the CHANGELOG of the spot repository for release history and migration notes.