Administrator Guide¶

This guide covers operations and maintenance of a deployed SPOT platform. The install procedure itself ; prerequisites, DNS, secrets, admin bootstrap, plugin installation ; lives in a single place:

Deployment guide

This page picks up after the stack is running. For configuration options that the deploy .env does not cover, see the Configuration reference.

Hardening¶

TLS¶

The deployment's Traefik terminates TLS for the dashboard (https://DOMAIN) and the API (https://api.DOMAIN), with certificates from Let's Encrypt, and redirects HTTP to HTTPS. No other SPOT service publishes a port: the api-gateway, the orchestrators, the knowledge service, Postgres, Redis and RabbitMQ are reached over Docker networks only. Traefik does not route the plugin routes (/internal/*) or /metrics.

Firewall¶

Traefik needs port 80 for Let's Encrypt's HTTP challenge and 443 for the dashboard and the API. With the mail-milter profile, open port 9900 to your mail servers only (see Mail integration):

ufw allow 80/tcp
ufw allow 443/tcp
ufw allow from 192.168.1.0/24 to any port 22 proto tcp     # SSH, from management addresses
ufw allow from 192.0.2.25 to any port 9900 proto tcp       # the milter, from the MX only
ufw enable

Docker publishes ports through its own iptables rules, which ufw does not filter: keep RETRIEVER_MILTER_BIND on a private address as well.

Monitoring¶

Health Checks¶

# Overall platform health
curl https://api.spot.example.com/health

# Service status
docker compose ps

A plugin's container can be up while the plugin is not: its /health always answers 200, with the plugin's own state in the body. The plugin views show both: status and health are the container's, and plugin_health is the plugin's own, {"status", "reason", "version"}. Its status is healthy, starting or unhealthy, with the plugin's reason and the version it runs, or unknown when the plugin cannot be reached, does not answer in time or answers something else:

curl -H "Authorization: Bearer $TOKEN" \
  https://api.spot.example.com/api/v1/plugins/provider-employee-dir/status

Logging¶

Every core service, and every plugin built with the SDK, writes one JSON object per line on stdout:

{"timestamp": "2026-10-13T09:06:55.186Z", "level": "INFO", "logger": "src.api.plugin_install", "message": "Installed plugin analyzer-nlp version 2.0.1", "correlation_id": "6d0f3c1e-8a9b-4c1d-9e2f-0a1b2c3d4e5f"}

The errors of one service:

docker compose logs --no-log-prefix api-gateway | jq -cR 'fromjson? | select(.level == "ERROR")'

To follow one request, take the X-Correlation-ID its answer carries, then look for it in the lines of every service and plugin:

curl -si -H "Authorization: Bearer $TOKEN" https://api.spot.example.com/api/v1/... | grep -i x-correlation-id
docker compose logs --no-log-prefix | grep <id>
# the plugins run outside the compose project, as spot-<plugin id>
docker ps -q --filter label=spot.managed=true | xargs -n1 docker logs 2>&1 | grep <id>

A caller may send its own X-Correlation-ID: an ID that another request also sent mixes the two requests' lines.

A line never holds an email's content (a subject, a body, a header value, an address), a secret or a token, a query string, or an exception's text that can quote one of them. A request's line leaves out its query string, and an unhandled exception is logged by its type and frames.

LOG_FORMAT=text writes a readable line instead, for development; LOG_LEVEL sets the level. The deploy rotates the logs.

Metrics¶

Every core service serves its Prometheus metrics at GET /metrics, for a scraper that holds SPOT_METRICS_TOKEN. Monitoring shows how to set the token, run a Prometheus beside SPOT and load its alerts; the Metrics reference lists every metric and what its labels hold.

Audit log¶

The api-gateway records who did what, and how it ended, in the audit log:

• sign-ins of users and OAuth clients, successes and failures, and the refresh tokens refused because they were used again or their session had ended, which end their session;
• plugin lifecycle actions: install, update, uninstall, start, stop, restart, settings, resources, tokens, manual entries, enable and disable, a mail connection's mode, a provider's sync;
• user accounts: created, changed, their role or password changed, deactivated and activated again;
• plugin sources: added, changed, removed, enabled and disabled;
• labels set and removed, and datasets: created, deleted, an email promoted into one and a copy removed;
• benchmark runs started and deleted;
• deletions of stored data by a user or an API client: emails (with their analyses) and knowledge documents;
• reads of a raw message by a user;
• replays of parked messages.

Each entry names the actor (its name then, and an id a rename keeps), the action, its target, its outcome (a failure with the HTTP status), what changed by name, the client's address and User-Agent, and the request's correlation ID (Logging). The API reference lists every action and what it records.

It is not recorded:

• configuration changes: each is a commit of spot.yaml's history, by its user (History of spot.yaml);
• requests refused before their route runs (by their role, or for want of a valid token), and token refreshes, which the HTTP metrics count. A refusal inside a route is recorded as a failure: a viewer changing another user's password, for example;
• reading the audit log;
• a restart of the api-gateway (POST /api/v1/config/restart, the dashboard's Save and Restart), which saves nothing: the settings it applies are in the history;
• refreshes of the plugin catalog, its update checks and scans of a plugin source, which only renew what SPOT knows of the registries;
• analysis submissions and re-analyses, kept as analyses (their job and result, not who asked for them), and workflow dry runs;
• job cancellations, which delete nothing; the daily retention purge; a plugin's own deletions at the knowledge service.

An entry never holds a secret value, a token, a source's URL or a message's content: settings and secrets by name, a deleted email or document by its id. A failed sign-in names the account only when it exists, so a password typed into the username field never reaches the log. A knowledge document's id is its provider's, which can hold an address: provider-employee-dir names its documents employee:<email>, so the entry of such a deletion keeps the employee's address for audit_retention_days.

Addresses. An entry holds the address the gateway sees. Behind the deploy's Traefik, that is Traefik's address for an API client, and the dashboard's for an action taken in the dashboard, which calls the gateway from its own server: the dashboard names itself in the User-Agent (spot-web-dashboard/<version>), so its entries show where they came from, not the browser's address.

Retention. An entry is kept for audit_retention_days (365 by default, 30 to 3650), and purged with the emails, at the gateway's start and then daily (Retention). The audit log is in the database, and so in every backup.

Reading it. Admins read it in the dashboard (Administration, Audit log), with spot-cli audit (CLI guide) and through GET /api/v1/audit, filtered by action or family (plugin.), actor, target, outcome and time.

An entry that cannot be written (the database down) does not stop the action: the gateway logs it, by the action and the error's type, and counts it in spot_audit_write_failures_total, which the SpotAuditWriteFailures alert of Monitoring watches.

Knowledge Store¶

SPOT's RAG layer is a dedicated microservice (spot-knowledge) with its own Postgres table (knowledge_documents, migration 008) and pgvector HNSW index. See the Knowledge Store guide for concepts; this section covers the day-to-day admin surface.

Dashboard¶

Every signed-in user gets a Knowledge top-level nav item; admins also delete documents:

• /knowledge ; total doc count, distinct sources, tag breakdown, semantic search, and a filterable/paginated browse table with inline delete.
• /knowledge/{source}/{id} ; single document inspector.

From any context-provider detail page (/plugins/context_provider/{id}) operators can:

• Trigger a one-shot sync (Sync now button).
• Jump to the Knowledge Store filtered by this provider's documents.
• See the last-run timestamp, doc count, and next scheduled run.

Editing ingestion schedules¶

Go to /config/plugin/context_provider/{id} → Scheduling card. Edit the 5-field cron (or @hourly/@daily/@weekly/@monthly alias) and the timeout (ms), then Save. The running scheduler reconciles immediately ; no process restart required. Leaving the schedule empty makes the provider manual-only.

Invalid cron expressions are rejected by the scheduler and surfaced on the same page ("Invalid cron expression") so they're visible before the next firing.

Required environment¶

Both the knowledge service and the api-gateway need SPOT_INTERNAL_API_KEY set in .env; it authenticates core services to each other. The knowledge service, the plugin-manager and the mail-orchestrator treat whoever sends it as core. Plugins never receive it: each holds its own token (see Plugin tokens). Rotating the key requires docker compose up -d, which re-creates the services with it. The tokens and the plugin secrets are stored encrypted with ENCRYPTION_KEY. When that key changes, installed plugins get a new token at the next api-gateway start; reissue the tokens of the plugins you run yourself with spot-cli plugin token. Plugin secrets cannot be recovered: set them again (see Plugin secrets).

Plugin tokens¶

Each plugin holds an opaque token (spot_pt_...), delivered as the file /run/secrets/spot_plugin_token. The scopes depend on the plugin's kind: mail retrievers ingest, context providers knowledge:write (their own documents only), analyzers knowledge:read and content:read, responders results and content:read. Plugins authenticate with Authorization: Bearer <token> and nothing else, and core calls each plugin with the same token: an analyzer without one is not called (it counts as unknown, or fails the job when its failure strategy is fail), and the scheduler does not call a context provider without one.

• Installed plugins get a token at install; nothing to do.
• Issue or rotate one: spot-cli plugin token <id> prints it (admin only, POST /api/v1/plugins/{plugin_id}/token). The old token stops working and an installed plugin's container is recreated with the new one. The new token is stored before the container is recreated: when the recreate fails (503), the container keeps the revoked token until a plugin restart, or the reconcile at the next api-gateway start, gives it the new one.
• A plugin run by your own compose file: spot-cli plugin token <id> --kind <kind> > spot_plugin_token (adds the manual entry first when needed), then mount the file as the secret spot_plugin_token and join the external Docker network spot-plugins (networks: {spot-plugins: {external: true}}), no longer spot_spot-network. The file is a secret: never commit it. Compose mounts it with its owner and mode on the host, so the user the plugin's container runs as must be able to read it.
• Installing a plugin from the catalog over a manual entry of the same id replaces the manual plugin's token. If the install fails, the manual plugin is left without a token: reissue it with spot-cli plugin token.
• Removing a plugin with DELETE /api/v1/config/plugin/{kind}/{id} revokes its token. A plugin removed from spot.yaml by hand loses its token at the next api-gateway start or config reload (POST /api/v1/config/reload). At the same times every other token gets the scopes of its plugin's kind in spot.yaml; the token itself does not change. When spot.yaml is missing or cannot be parsed, no token changes and the gateway logs a warning.

SPOT runs plugins of contract 2 only, built on SDK 2. The catalog refuses an image of another contract, with the reason, and a plugin built on SDK 1 that is still installed stops working until it is updated to its contract 2 release: an analyzer's answer counts as unknown, a retriever's ingest is refused, a provider cannot sync.

Plugin secrets¶

A plugin setting is secret when the plugin's catalog schema marks it (type: secret or secret: true in its spot.plugin.settings label), or when it is already stored as one. Names are case-insensitive, as in the plugin environment, and secrets are kept under their uppercase name: a misp_token setting is sealed as MISP_TOKEN. spot.yaml keeps secrets apart from the plain settings, encrypted with ENCRYPTION_KEY:

plugins:
  analyzers:
    analyzer-misp:
      settings:
        MISP_HOST: https://misp.example.com
      secrets:
        MISP_TOKEN: gAAAAAB...   # sealed, never plain text

• An installed plugin gets each secret as the file /run/secrets/<NAME>, beside its token, never as an environment variable.
• The install, settings and config-API plugin requests take secrets by name, secrets: {"NAME": "value"}, beside settings, and the CLI takes --secret NAME=VALUE (spot-cli plugin install, plugin settings and plugin update). A setting the schema marks secret, sent in settings, is sealed the same way. To seal a credential the plugin's image does not mark, such as provider-employee-dir's LDAP_BIND_PASSWORD, without touching the plugin's version: spot-cli plugin settings provider-employee-dir --secret LDAP_BIND_PASSWORD=.... The save restarts the plugin, which is unavailable for a few seconds; a restart that fails answers 503, and the secrets stay saved.
• A secret's name is a file name: letters, digits, _, . and -, not . or ... No setting may be named spot_plugin_token, the file of the plugin's token. Either answers 422.
• Plugin responses list the names (secrets: ["MISP_TOKEN"]), never a value; the config views (GET /api/v1/config/...) leave secrets out.
• An empty value ("" or null) keeps the stored secret, so a form can be saved without retyping it. To remove a secret, delete it from secrets: in spot.yaml and reload the config (POST /api/v1/config/reload).
• A secret setting sent through the config options API (PUT /api/v1/config/options/analyzers.<id>.settings...) is refused, even an empty one, since an option shows its value: write secrets through the plugin endpoints.
• The config API needs the plugin catalog to tell secret settings apart; while it cannot be read (Redis down), a write of plugin settings or secrets answers 503. An entry without them is still created.

Upgrade from 2.0.0-beta.3: at its first start, the api-gateway seals the settings each plugin's catalog schema marks secret and removes their plain copies; a plugin the catalog does not list keeps its settings, and the gateway logs a warning naming it. Set such a plugin's secrets through secrets. The same move runs at every start and after a config reload (POST /api/v1/config/reload), so a secret written into settings: by hand is sealed too; it changes nothing when there is nothing to move.

When ENCRYPTION_KEY changes, a plugin's secrets can no longer be decrypted: the plugin keeps its token and its container as they are, and the api-gateway logs an error naming the plugin and its secrets. Set them again: the save restarts the plugin.

A plugin reads its secrets from /run/secrets with the SDK's PluginSettings, which takes a file there over an environment variable of the same name.

Observe, then enforce¶

A new mail connection (a mail retriever) is in observe: SPOT analyses its mail and calls the responders, which only record what they would do, and the retriever lets each message through as it is, recording what it would have done. Watch a connection in observe before it acts:

1. Read what it would have done: GET /api/v1/analyze/actions?days=7 counts the actions of the last days by actor (the retriever or a responder), action, mode and whether each was applied; a job's status, GET /api/v1/analyze/{job_id}, lists its own.
2. Compare them with the verdicts you expect. A route whose workflow you do not trust yet can stay in observe on its own (mode: observe on the route) while the connection enforces the others.
3. Switch the connection: spot-cli plugin mode <id> enforce (it asks first; --yes does not), or PUT /api/v1/config/plugin/mail_retriever/{id}/mode with {"mode": "enforce"}. Nothing restarts: the next message is in enforce, on the routes that are. spot-cli plugin mode <id> observe switches it back.

A switch reaches the jobs queued after it: those already queued keep the mode they were queued in. To stop the responders at once, in an emergency, disable them (spot-cli plugin disable <responder>): the dispatcher skips a disabled responder from its next read of spot.yaml. A re-analysis that names a workflow takes its connection's mode, not a route's observe.

The history of spot.yaml records each switch, by its user ("Set to enforce"). User submissions are always in observe. See Observe and enforce.

Restarting a plugin¶

spot-cli plugin restart <plugin_id> (POST /api/v1/plugins/{plugin_id}/restart, or Restart on the plugin's page) re-creates the plugin's container from spot.yaml when its entry changed, such as after a hand edit and a reload, and restarts it in place otherwise. The plugin is unavailable for a few seconds. A disabled plugin is refused (409): enable it first. A change of settings or resources restarts the plugin by itself.

Plugin resources¶

Each installed plugin's container runs with a memory and a CPU limit, shown as effective_resources in the plugin's answer (GET /api/v1/plugins/installed/{plugin_id}):

• the override an admin set (resources), else
• what the image asks for in its spot.plugin.resources label (requested_resources, recorded at install and at each update), else
• 1024 MB and 1 CPU.

spot-cli plugin resources <plugin_id> shows the override, the image's request and the limits in force. An admin sets the override with spot-cli plugin resources <plugin_id> --memory-mb N --cpus X (PUT /api/v1/plugins/{plugin_id}/resources, a body of memory_mb, at least 128, cpus, above 0 and at most 64, or both), and removes it with --reset (DELETE on the same path), after which the image's request applies again. A value left out is copied from the limits in force: the override holds both values, and a later image's request changes neither until the override is removed. An update keeps the override. An enabled plugin is restarted to take the new limits, and is unavailable for a few seconds; a disabled one takes them when it is started.

Docker refuses limits the host cannot give, such as more CPUs than it has. A plugin that cannot be restarted with its new limits gets its previous ones back: they are saved again and the plugin restarted with them. The change then answers 422 with Docker's reason, or 503 when the restart failed for another reason. A disabled plugin is not restarted, so such a limit stops it at its next start: set limits the host can give.

A plugin that goes over its memory limit is killed by the kernel, and Docker restarts it. To see it happen:

# How often Docker restarted the container
docker inspect -f '{{.RestartCount}}' spot-<plugin_id>

# The kernel's record of each kill, which lasts
sudo journalctl -k | grep -i 'memory cgroup out of memory'

# Each kill Docker still remembers (it keeps its recent events only)
docker events --since 24h --until 0s --filter event=oom --filter container=spot-<plugin_id>

# Its memory used against the limit
docker stats --no-stream spot-<plugin_id>

.State.OOMKilled is true only while the container stays down: once Docker restarts it, the flag reads false.

Backup and Recovery¶

The deployment wrapper, deploy/scripts/spot-cli, has three commands that run on the host, from the deploy directory (the default is /opt/spot/deploy), with your own Docker client: backup, restore and upgrade. The deployment guide's Backup and restore and Upgrading pages give every detail.

Back up¶

cd /opt/spot/deploy
scripts/spot-cli backup

A backup is one directory under backups/ (or under --dir DIR), readable by its owner only. It holds:

• the database, as plain SQL, kept only once a restore into a scratch database comes back at the same migration;
• the spot-config volume: spot.yaml and its history (history.git, every value it ever had), when they exist;
• .env, with ENCRYPTION_KEY: the plugin secrets and the plugin sources' tokens in spot.yaml, the stored emails and the plugin tokens are encrypted with it;
• docker-compose.override.yml, when there is one;
• a manifest (the deploy release and its commit, the migration, the time) and the checksums of the files.

It does not hold Redis (the plugin catalog, which a refresh rebuilds from the sources of spot.yaml, and the records of queued jobs), RabbitMQ's queued and parked messages, the Ollama models or the certificates. A backup holds every secret of .env: keep it like .env, and copy it off the host, encrypted. The deployment guide gives a nightly cron line that keeps 14 days of backups.

Restore¶

scripts/spot-cli restore backups/<release>-<time>

The restore asks before it changes anything (--yes answers for scripts). It backs up the live database and configuration first, beside the backup it restores (after a restore that did not finish, it keeps that run's safety backup instead, and backup refuses until the restore finishes). It then puts back the backup's deploy release, .env, override file, database and configuration volume, removes the plugin containers its spot.yaml does not name, and starts SPOT. The same command restores on a new host. A restore that does not finish says how to go on.

Upgrade¶

scripts/spot-cli upgrade <version>

The upgrade fetches the release, and checks that .env holds every setting it requires. Then it takes a backup, checks the release out, pulls its images, starts it and waits for every health check. When something fails after the backup, it prints the scripts/spot-cli restore command that goes back; it never goes back by itself.

Security¶

SPOT ships with these on: rate limiting (enable_rate_limiting), the audit log, encryption of stored emails and plugin secrets with ENCRYPTION_KEY, a token of its own for each plugin, and plugins on a network of their own. What is left to the operator:

1. Keep .env and every backup readable by their owner only: they hold every secret. Copy backups off the host, encrypted (Backup and restore).
2. After the first sign-in, set SPOT_BOOTSTRAP_ADMIN=false and remove SPOT_ADMIN_PASSWORD from .env (After first start).
3. Give each person an account of their own, with the lowest role that does the job: viewer, analyst or admin.
4. Upgrade when a release comes out: the dashboard's Updates page shows it (Upgrading).
5. Watch the failed sign-ins (spot-cli audit --action auth. --outcome failure, or the SpotSignInFailures alert).
6. Install a plugin of egress class content only when its destination may receive your mail.

Troubleshooting¶

The deployment guide's Troubleshooting page covers the common failures. To check the infrastructure from the host:

docker compose ps
docker compose exec postgres pg_isready
docker compose exec rabbitmq rabbitmq-diagnostics check_running
docker stats --no-stream

Maintenance¶

Updates¶

Upgrade with scripts/spot-cli upgrade <version>, which takes a backup first: see Upgrade above.

Capacity¶

SPOT runs one instance of each service. These settings size it:

• MAX_CONCURRENT_JOBS and MAX_CONCURRENT_SYNC_JOBS: the analyses the analyzer-orchestrator runs at once (Analysis concurrency);
• the intake section of spot.yaml: the messages the api-gateway parses at once (Intake and api-gateway memory);
• each service's memory limit (Memory limits), and each plugin's (Plugin resources).

Operator commands reference¶

Day-to-day operations on a deployed stack go through docker compose from the deploy checkout, plus the operator spot-cli for things that hit the API:

# Stack lifecycle
docker compose up -d              # Start (or recreate) all services
docker compose down               # Stop and remove containers
docker compose restart            # Restart services
docker compose ps                 # Show container status

# Logs
docker compose logs -f                      # All services, follow
docker compose logs -f api-gateway          # One service, follow

# Backup, restore and upgrade (on the host, from the deploy directory)
scripts/spot-cli backup                     # One backup under backups/
scripts/spot-cli restore <dir>              # Put a backup back
scripts/spot-cli upgrade <version>          # Back up, then upgrade to a release

# Operator CLI (auth, plugins, workflows, analyze, config, benchmark)
spot-cli --help

The operator CLI is documented in the CLI user guide.

Support¶

• Documentation: https://spot-project.codeberg.page/documentation/
• Issues: https://codeberg.org/SPOT_Project/spot/issues
• Email: spot@sonn.lu