Monitoring¶
SPOT's core services (api-gateway, analyzer-orchestrator, mail-orchestrator, knowledge, plugin-manager) write JSON logs that carry each request's correlation ID, serve Prometheus metrics, and the api-gateway keeps an audit log of who did what. This page covers the deployment side; the Metrics reference lists every metric, and the API every audit action.
Logs¶
Each core service writes one JSON object per line on stdout:
{"timestamp": "2026-10-13T09:06:55.186Z", "level": "INFO", "logger": "src.api.plugin_install", "message": "Installed plugin analyzer-nlp version 2.0.1", "correlation_id": "6d0f3c1e-8a9b-4c1d-9e2f-0a1b2c3d4e5f"}
timestamp is UTC to the millisecond; logger is uvicorn.access for a request's line; correlation_id is left out when there is none, at startup; a traceback comes as exception and stack. The milter (mail-milter profile) logs the same way. LOG_FORMAT=text in .env writes a readable line instead, for development, in the core services and the milter; any value but json and text stops them at start. Docker rotates the logs (Logging).
Read them with jq. fromjson? skips the lines that are not JSON:
# the errors of one service
docker compose logs --no-log-prefix api-gateway | jq -cR 'fromjson? | select(.level == "ERROR")'
# the warnings and errors of every core service, in the last hour
docker compose logs --no-log-prefix --since 1h | jq -cR 'fromjson? | select(.level == "WARNING" or .level == "ERROR")'
Following one request. Every answer of the api-gateway carries an X-Correlation-ID, a 500 included, and every line written for that request carries it, in each service it reaches: an analysis's queue messages, its analyzers and responders, a retriever's ingest. Take the ID, then look for it:
curl -si -H "Authorization: Bearer $TOKEN" https://api.$DOMAIN/api/v1/... | grep -i x-correlation-id
docker compose logs --no-log-prefix | grep <id>
# the plugins run outside the compose project, as spot-<plugin id>
docker ps -q --filter label=spot.managed=true | xargs -n1 docker logs 2>&1 | grep <id>
A caller may send its own X-Correlation-ID: two requests that sent the same one mix their lines. An invalid ID (anything but 1 to 128 characters of A-Z a-z 0-9 . _ : -) is replaced with a new one. Plugins log with the ID once they run spot-sdk-python 2.0.0b11 or later.
A line never holds an email's content (a subject, a body, a header value, an address), a secret or a token, a query string, or an exception's text: a request's line leaves out its query string, and an unhandled exception is logged by its type and frames.
Metrics¶
Every core service serves its Prometheus metrics at GET /metrics, for a scraper that holds SPOT_METRICS_TOKEN. spot-cli setup generates it with the other secrets. On an install made by hand, set it in .env, then recreate the services so that they read it:
While the token is unset, or still the CHANGE_ME_ placeholder of .env.example, every /metrics answers 503; a scrape without the token answers 401. /metrics is never published: Traefik does not route the api-gateway's /metrics, and the other services have no route at all. Scrape them from inside spot-network.
The Metrics reference lists every metric and what its labels hold: a route template or an id from spot.yaml, never a raw path, an address or a job id.
A Prometheus beside SPOT¶
SPOT bundles no Prometheus. To run one beside it, add it in docker-compose.override.yml, on spot-network, with the token as a file:
# docker-compose.override.yml
services:
prometheus:
# Pinned by digest, as the rest of the stack: a release of 2026-09-25.
image: prom/prometheus:v3.15.0@sha256:efd719c99d83b060d9daefdcf00360461adf279f45ef5391f8d111892118753e
restart: unless-stopped
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.retention.time=30d
volumes:
- ./prometheus:/etc/prometheus:ro
- prometheus_data:/prometheus
secrets:
- spot_metrics_token
networks:
- spot-network
# The web UI on the host only; reach it through an SSH tunnel.
ports:
- "127.0.0.1:9090:9090"
secrets:
spot_metrics_token:
environment: SPOT_METRICS_TOKEN
volumes:
prometheus_data:
prometheus/prometheus.yml, beside it, scrapes the five services with the token:
global:
scrape_interval: 30s
rule_files:
- /etc/prometheus/alerts.yml
scrape_configs:
- job_name: spot
authorization:
type: Bearer
credentials_file: /run/secrets/spot_metrics_token
static_configs:
- targets:
- api-gateway:8000
- analyzer-orchestrator:8000
- mail-orchestrator:8000
- knowledge:8000
- plugin-manager:8000
prometheus/alerts.yml holds the alerts of core's admin guide:
groups:
- name: spot
rules:
# Messages parked in a dead-letter queue. The shadow queue drops its oldest
# work there when the candidates cannot keep up, which is expected: it has
# its own, gentler alert.
- alert: SpotMessagesParked
expr: max by (queue) (spot_queue_messages{queue=~".*dead-letter", queue!="orchestrator.shadow.dead-letter"}) > 0
- alert: SpotShadowBacklog
expr: spot_queue_messages{queue="orchestrator.shadow.dead-letter"} > 500
# RabbitMQ's queues cannot be read.
- alert: SpotRabbitMQUnreadable
expr: spot_rabbitmq_up == 0
# An analyzer failing on real mail.
- alert: SpotAnalyzerFailing
expr: >
sum by (analyzer) (rate(spot_analyzer_calls_total{run="job", status=~"error|timeout"}[15m]))
/ sum by (analyzer) (rate(spot_analyzer_calls_total{run="job", status!="skipped"}[15m])) > 0.2
# The milter waiting for its verdicts.
- alert: SpotIngestWait
expr: histogram_quantile(0.95, sum by (le, retriever) (rate(spot_ingest_wait_seconds_bucket[15m]))) > 20
# A burst of failed sign-ins.
- alert: SpotSignInFailures
expr: sum(increase(spot_logins_total{outcome="failure"}[10m])) > 50
# An audit entry that could not be written.
- alert: SpotAuditWriteFailures
expr: increase(spot_audit_write_failures_total[15m]) > 0
Start it with docker compose up -d prometheus. Its Targets page (http://127.0.0.1:9090/targets on the host) shows the five services UP; a 401 there means the file holds another token than .env, a 503 that the services have none (recreate them). Prometheus evaluates the alerts; an Alertmanager sends them on. spot-cli backup does not hold the prometheus/ directory or its data: keep a copy of the two files.
Audit log¶
The api-gateway records in the audit log who did what, and how it ended: sign-ins (successes and failures), plugin lifecycle actions, user accounts, plugin sources, labels set and removed, datasets and their copies, deletions of stored emails and knowledge documents, reads of a raw message, and replays of parked messages. Each entry names the actor, the action, its target, its outcome, what changed by name, the client's address and User-Agent, and the request's correlation ID. It never holds a secret value, a token or a message's content, and a failed sign-in names the account only when it exists. A deleted knowledge document is named by its id, which can hold an address: provider-employee-dir names its documents employee:<email>, so such an entry keeps the employee's address for audit_retention_days. Configuration changes are not in it: each is a commit of spot.yaml's history, which spot-cli config history lists.
Admins read it in the dashboard (Administration, Audit log), with spot-cli audit, or through GET /api/v1/audit:
scripts/spot-cli audit # the last 50 entries
scripts/spot-cli audit --action plugin. --since 2026-10-01
scripts/spot-cli audit --actor alice --outcome failure --limit 200
--action takes an action (plugin.install) or a family ending in a dot (plugin., auth., user., source., email., dataset., knowledge., queue.).
Addresses behind Traefik. An entry holds the address the api-gateway sees. In this deployment, that is Traefik's address for an API client's action, and the dashboard's for an action taken in the dashboard, which calls the gateway from its own server: those entries show spot-web-dashboard/<version> as their User-Agent, not the browser's address.
Retention. An entry is kept for audit_retention_days of spot.yaml, 365 days by default (30 to 3650), and the api-gateway's daily retention purge deletes the older ones (Retention). The audit log is in the database, so every backup holds it.
An entry that cannot be written (the database down) does not stop its action: the api-gateway logs it, and counts it in spot_audit_write_failures_total (the SpotAuditWriteFailures alert above).