Configuration Reference¶
SPOT reads its configuration from two places: environment variables, for the infrastructure and the secrets, and spot.yaml, for the platform settings, the plugins, the workflows and the routing. This page lists both. Its environment variables are those of the development stack, the repository's own compose files and .env. An install from deploy/ sets its variables in deploy/.env: Deployment configuration lists them.
Quick Start¶
Copy .env.example to .env and customize:
See .env.example for comprehensive documentation of all available variables.
Environment Control¶
See Environment Configuration for details on environment management.
Required Variables¶
These variables MUST be set in production:
| Variable | Description | Example |
|---|---|---|
APP_ENV | Environment mode | prod, dev, or test |
SECRET_KEY | JWT token signing key | Generate with: python -c "import secrets; print(secrets.token_urlsafe(32))" |
SPOT_INTERNAL_API_KEY | Key core services authenticate each other with | Generate like SECRET_KEY |
ENCRYPTION_KEY | Fernet key the stored emails, results, plugin tokens, plugin secrets and plugin-source tokens are encrypted with | python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" |
POSTGRES_DB | PostgreSQL database name | spot |
POSTGRES_USER | PostgreSQL username | spot |
POSTGRES_PASSWORD | PostgreSQL password | secure_password |
RABBITMQ_DEFAULT_USER | RabbitMQ username | guest |
RABBITMQ_DEFAULT_PASS | RabbitMQ password | secure_password |
Core Configuration¶
Docker Compose¶
| Variable | Default | Description |
|---|---|---|
COMPOSE_PROJECT_NAME | spot | Project name for consistent container/network naming |
Application Settings¶
| Variable | Default | Description |
|---|---|---|
LOG_LEVEL | INFO | Logging level (platform.log_level in spot.yaml): DEBUG, INFO, WARNING, ERROR or CRITICAL, in any case; WARN and FATAL are read as WARNING and CRITICAL. Any other value stops the service at start, and is refused with a 422 when saved through the API |
LOG_FORMAT | json | json, or text for development; any other value stops the service at start |
DEBUG | false | Enable debug mode |
TZ | UTC | Timezone |
Security¶
| Variable | Default | Description |
|---|---|---|
SECRET_KEY | (required) | Secret key for JWT token signing - MUST be set |
SPOT_INTERNAL_API_KEY | (required) | Key core services authenticate each other with: the api-gateway sends it to the plugin-manager, to the analyzer-orchestrator (every route but /health and /version), to the knowledge service (Knowledge admin pages) and to the mail-orchestrator (mail-retriever views, retrievers' ingest). Whoever sends it is treated as core, so it never goes to a plugin, which uses its own token |
ENCRYPTION_KEY | (required) | Fernet key that encrypts what SPOT stores of an email: its raw message and view, its analyses' results and their signals' evidence and location; the plugin tokens and secrets; and the auth tokens of the plugin sources. Migration 010 needs it as soon as analyses exist. Keep it with your backups: data encrypted with another key cannot be read |
ALLOWED_HOSTS | localhost,127.0.0.1,api-gateway,testserver | The host names the api-gateway answers, comma-separated or a JSON list; any other Host is refused unless DEBUG is set |
Database (PostgreSQL)¶
| Variable | Default | Description |
|---|---|---|
POSTGRES_DB | spot | Database name |
POSTGRES_USER | spot | Database username |
POSTGRES_PASSWORD | spot123 | Database password |
DATABASE_URL | (auto-constructed) | Full connection URL |
Redis Cache¶
| Variable | Default | Description |
|---|---|---|
REDIS_PASSWORD | (empty) | Redis password (empty = no auth) |
REDIS_URL | (auto-constructed) | Full connection URL |
REDIS_MAXMEMORY | 256mb | Maximum memory |
REDIS_MAXMEMORY_POLICY | allkeys-lru | Eviction policy |
RabbitMQ Message Queue¶
| Variable | Default | Description |
|---|---|---|
RABBITMQ_DEFAULT_USER | guest | RabbitMQ username |
RABBITMQ_DEFAULT_PASS | guest | RabbitMQ password |
RABBITMQ_URL | (auto-constructed) | Full connection URL |
The analyzer-orchestrator consumes five queues:
| Queue | Bound to | Holds | Dead-letter queue |
|---|---|---|---|
orchestrator.analysis | spot.analysis / email.analyze | Analysis jobs | orchestrator.analysis.dead-letter |
orchestrator.analysis.sync | spot.analysis / email.analyze.sync | The ingests of retrievers that wait for their verdict | orchestrator.analysis: an ingest expired, refused, or failing past the limit is handled with the bulk |
orchestrator.dispatch | spot.results / analysis.complete | Verdicts to dispatch to responders | orchestrator.dispatch.dead-letter |
orchestrator.shadow | spot.results / analysis.complete | Results to run the shadow workflows on, at most 1 000 | orchestrator.shadow.dead-letter, at most 1 000 too |
orchestrator.benchmark | spot.analysis / email.benchmark | The emails of a platform benchmark run, one message each, consumed one at a time | orchestrator.benchmark.dead-letter |
The api-gateway, the mail-orchestrator and the analyzer-orchestrator declare them once connected, so nothing is published to a queue not declared yet. The first of them to start after an upgrade empties the classic queues of earlier releases (orchestrator.analysis.requests, orchestrator.results.dispatch) into them and deletes those. A queue the analyzer-orchestrator of the earlier release still consumes is emptied but kept, since that one may hold messages it has not finished: they go back to the queue when it stops, and the new analyzer-orchestrator, which starts after it, empties and deletes the queue. The unused exchange spot.direct is deleted too, unless a queue of an older release is still bound to it.
The queues are quorum queues with a delivery limit of 10 (x-delivery-limit). A message whose handler fails is handed back after a wait that starts at 1 s and doubles up to 5 minutes; once it has failed 10 times, its next failure parks it in its dead-letter queue, 11 deliveries and about 13.5 minutes after it was first received. A message that is not JSON, or that its consumer can never process (an analysis job message that names no job, a message that is no result), is parked at once. A parked message stays in its dead-letter queue until it is purged or replayed. The sync queue's failures, expired or refused messages included, move to orchestrator.analysis and are handled there. Parking is at least once: a message being parked stays in its queue until its dead-letter queue has taken it.
orchestrator.shadow is bounded: it holds at most 1 000 results (x-max-length), and past that its oldest are dropped into its dead-letter queue (x-overflow: drop-head), which holds at most 1 000 and drops its own oldest. So when the shadow workflows cannot keep up, the oldest shadow work goes, never production's, and the drops show as the dead-letter queue's depth on the Queues page. RabbitMQ parks a bounded queue's messages at most once (at least once needs reject-publish overflow): shadow work is best effort.
The bound is part of the queue's declaration, and RabbitMQ refuses to declare the queue again with another one (PRECONDITION_FAILED). A release that changes the bound says so, and its upgrade deletes orchestrator.shadow and orchestrator.shadow.dead-letter (rabbitmqctl delete_queue <name>), losing the shadow work they hold. Until then, the services log that orchestrator.shadow cannot be declared and the shadow does not run; production's queues are declared and consumed all the same.
To replay parked messages once their cause is fixed, an admin calls POST /api/v1/observability/queues/{queue}/replay (?limit=N), runs spot-cli queues replay <queue> [--limit N], or uses the Queues page's "Replay one" and "Replay all" buttons, for orchestrator.analysis, orchestrator.dispatch, orchestrator.shadow or orchestrator.benchmark (what a replay of the last does: Messages in a dead-letter queue):
- Fix the cause first.
- Replay one message (
--limit 1,?limit=1, or "Replay one"), and check that it is handled: it does not come back to the dead-letter queue, and its job completes (for a benchmark message, its run's count moves). - Then replay the rest.
Why one first: a message that fails again holds one of its queue's slots while it waits between its retries, about 13.5 minutes in all, before it is parked again. N such messages hold the analysis queue's 10 slots for about N/10 x 13.5 minutes, and new analyses wait behind them, the synchronous ingests that expire into that queue included.
A replay moves only the messages parked when it starts, at most limit, into the queue alone (through the default exchange, so no other queue bound to the same exchange gets a copy), each with its whole retry budget again. It moves each message at least once: a failure between a message's publish and its ack can leave it in both queues, and two replays at once can move a message twice. That is safe, as for any message delivered twice: an analysis delivered while its job runs waits for that run, one delivered later runs the job again and overwrites its record and result, and each responder acts once per job_id. A broker failure answers 503 and leaves what was not moved parked: replay again.
The Queues page of the dashboard shows how many messages each queue holds, dead-letter queues included. From the host of the stack:
# How many messages each queue holds (the counts refresh every few seconds)
docker compose exec rabbitmq rabbitmqctl list_queues name messages
# Look at up to 10 parked messages, leaving them parked; rabbitmqadmin signs in
# with the user RabbitMQ was created with, read from the container's environment
docker compose exec rabbitmq sh -c 'rabbitmqadmin -u "$RABBITMQ_DEFAULT_USER" \
-p "$RABBITMQ_DEFAULT_PASS" get queue=orchestrator.analysis.dead-letter \
count=10 ackmode=reject_requeue_true'
# Drop the parked messages of a queue
docker compose exec rabbitmq rabbitmqctl purge_queue orchestrator.analysis.dead-letter
# Replay them into their queue, once their cause is fixed, with a one-off shovel:
# it moves what the dead-letter queue holds when it starts, then removes itself
docker compose exec rabbitmq rabbitmq-plugins enable rabbitmq_shovel
docker compose exec rabbitmq rabbitmqctl set_parameter shovel replay-analysis \
'{"src-protocol": "amqp091", "src-uri": "amqp://", "src-queue": "orchestrator.analysis.dead-letter", "src-delete-after": "queue-length", "dest-protocol": "amqp091", "dest-uri": "amqp://", "dest-queue": "orchestrator.analysis"}'
rabbitmqctl, rabbitmq-plugins and the shovel's amqp:// (a direct connection to the node) need no user.
A parked message tells nothing of its content in the logs: the handler logs its job id, the error's type and where it was raised. The shovel, like the replay above, moves what was parked when it starts, and each message gets its whole retry budget again: RabbitMQ counts its deliveries from 0. A shovelled message's first delivery still carries the x-delivery-count of the dead-letter queue (each look counts there), which the consumer ignores on a first delivery; the replay drops it.
How the broker counts deliveries is RabbitMQ 3.13's: every message handed back counts as a failed delivery, whether its consumer nacked it or closed its channel (a crash included). RabbitMQ 4.0 changes the rules and the defaults (quorum queues get a delivery limit of 20). When RabbitMQ is upgraded, run the broker integration tests again (tests/integration/test_broker_flow.py, TestDeadLettering and TestSyncLane): they check that a message that always fails is parked after 11 deliveries, that the queues of earlier releases are emptied into their successors, and that a synchronous ingest that expires (losing its expiration) or keeps failing moves to the bulk queue.
Service Configuration¶
API Gateway¶
| Variable | Default | Description |
|---|---|---|
API_GATEWAY_PORT | 8001 | External API port |
Analyzer Orchestrator¶
| Variable | Default | Description |
|---|---|---|
MAX_CONCURRENT_JOBS | 10 | Analyses run at once from the bulk queue (orchestrator.analysis): user submissions, re-analyses and the ingests of retrievers that do not wait |
MAX_CONCURRENT_SYNC_JOBS | 4 | Analyses run at once from the queue of synchronous ingests (orchestrator.analysis.sync), the retrievers that wait for their verdict, such as the milter |
A synchronous ingest has slots of its own, so a backlog of bulk analyses never holds it up behind them. One not started before its retriever stops waiting expires and moves to the bulk queue, where it is still analysed, later: one not started in time never holds up a fresh synchronous ingest. One whose handling keeps failing (Redis down, say) holds its slot through its retries, as in the bulk queue, before it moves to the bulk queue too. The analyzers stay shared: a synchronous ingest skips the queue, not an analyzer that is saturated, so an analyzer slowed by bulk analyses slows it too.
Plugin Configuration¶
"Plugin" is the umbrella vocabulary for anything pluggable into SPOT. Four kinds exist, each with its section under plugins: in config/spot.yaml:
- analyzers (
plugins.analyzers) answerPOST /internal/analyzewith a verdict, a score and the signals behind it (AnalyzerVerdict); - context providers (
plugins.context_providers) fill the Knowledge Store when core calls theirPOST /internal/sync, on theirsync_schedule; analyzers query the store (Context Providers guide); - mail retrievers (
plugins.mail_retrievers) push raw messages to the api-gateway'sPOST /internal/ingest; each is a mail connection, with amode(see Observe and enforce); - responders (
plugins.responders) act on verdicts onPOST /internal/respond(see Responders).
plugins:
analyzers:
analyzer-nlp:
enabled: true
url: "http://spot-analyzer-nlp:8000"
settings: {}
context_providers:
employee-dir:
enabled: true
url: "http://provider-employee-dir:8000"
sync_schedule: "0 */6 * * *"
settings:
LDAP_URL: ldap://ldap.example.com
mail_retrievers: {}
responders: {}
Each entry needs at minimum url and enabled. Installed plugins also carry image, version, container_id, container_name, installed_at, source_id, source_name, requested_resources, egress, after an update previous_version (the version it replaced, which a roll back installs) and, when the plugin needs one, consent (set by the installer). resources overrides the memory and CPU the image asks for (see Plugin resources). A plugin has one kind, its section's: an entry has no kinds. The api-gateway drops, once at start, the kinds: that 2.0.0-beta.6 and earlier wrote on every entry, and refuses to start on one whose kinds names another kind or several.
Core calls every plugin with the plugin's own token, and a plugin calls SPOT with it (see Plugin tokens). An analyzer that has no token, no url, or is disabled is not called: in the workflows that name it, its failure strategy applies at once, with no retry. It counts as unknown, or fails the job when its strategy is fail (see Analyzer Configuration).
Plugin sources¶
The plugin sources the catalog is read from are kept in spot.yaml, under plugins.sources, each by its id (1 to 50 lowercase letters, digits and hyphens, neither first nor last a hyphen):
plugins:
sources:
spot-plugin-index:
name: "SPOT plugin index"
type: index
url: "https://codeberg.org/SPOT_Project/spot-plugin-index/raw/branch/published/index.json"
enabled: true
priority: 50
corp:
name: "Corporate plugins"
type: gitlab
url: "https://gitlab.example.com"
organization: "plugins"
registry: "gitlab.example.com:5050"
enabled: true
priority: 100
secrets:
AUTH_TOKEN: "gAAAAAB..."
index_offered: true
| Field | Required / default | Meaning |
|---|---|---|
name | Required | The name shown in the dashboard. |
type | Required | index, gitlab, gitea, dockerhub, ghcr, gitlab_registry or gitea_registry. |
url | Required | The forge's, or the registry listing's, base URL; for an index, the URL of its index.json. |
organization | Required, none for an index | The organisation, group or namespace that holds the plugins; a registry source may give image_namespace instead. |
registry | Required, none for an index | The container registry host. |
image_namespace | None | The registry namespace, when it differs from organization. |
enabled | true | A disabled source is not scanned. |
priority | 100 | From 1 to 1000; a lower number comes first. |
secrets.AUTH_TOKEN | None, and none for an index | The source's token, sealed with ENCRYPTION_KEY. |
An index source is read only once a key SPOT ships is found to have signed it (see The signed plugin index). Set a token through the API (/api/v1/plugin-sources) or the dashboard's Plugins → Sources page, which seal it; the API never returns it. A start of the api-gateway that finds no sources key gives spot.yaml the sources 2.0 kept in Redis. Until 2.2, removing the key so brings back the sources 2.0 kept in Redis, those removed since included.
A start adds the signed plugin index, spot-plugin-index, when it finds in plugins.sources, those it moves from Redis included, no source of type index and none of its id, no record that it was offered (plugins.index_offered, which that start writes), and SPOT_BOOTSTRAP_DEFAULT_PLUGIN_SOURCE not set to false. Removed, it stays removed; lifting the opt-out later, or removing an index source of another id, offers it once at the next start.
Plugin settings¶
A plugin's image declares its settings in its spot.plugin.settings label: each a type (string, integer, number, boolean, url or secret), with optionally required, default, description, secret and enum. The dashboard renders its form from it, and GET /api/v1/config/plugin/{kind}/{id}/schema serves it from the catalog, that of the version installed, so it answers while the plugin is down: 404 for a manual entry, 503 when the catalog cannot be read, does not list the plugin at that version or cannot read its entry. An installed plugin's settings and secrets are written against that schema, which tells the secret ones: without it a write answers 503 and saves nothing, until the catalog is refreshed, the plugin's source is re-added or the plugin is updated to a version it lists. A manual entry has no catalog schema: its settings are stored as given.
The values live on the plugin's entry: settings for the plain ones, secrets for the secret ones, encrypted with ENCRYPTION_KEY. The installer starts the container with each setting as an environment variable of its name in upper case, each secret as the file /run/secrets/<NAME>, and SPOT's URLs (SPOT_GATEWAY_URL, SPOT_KNOWLEDGE_URL) and the plugin's id (SPOT_PLUGIN_ID); the SDK's PluginSettings reads them. Set secrets through the API (see Plugin secrets), not by hand.
Saving an installed plugin's settings or secrets restarts it: its container is re-created from spot.yaml, and the plugin is unavailable for a few seconds. This applies to PUT /api/v1/plugins/{id}/settings, to a settings, secrets or url change through PUT /api/v1/config/plugin/{kind}/{id}, and to a settings change through PUT /api/v1/config/options/analyzers.<id>.settings...; their answers carry restarting: true. The save and the restart are one step for the plugin: a stop or an uninstall sent meanwhile waits for both. A disabled plugin is not restarted (it takes the settings when started), nor is a manual entry. Each setting given is read as the plugin reads its text, against the plugin's catalog schema: for its type ("8000" and 8000 are an integer, "true" and true a boolean, any value a string), then its bounds (a number is finite and within its minimum and maximum, a string matches its pattern whole), then its enum when it has one. The value is stored as read. One the plugin cannot read answers 422, naming the setting and the rule. So does a setting named spot_plugin_token, the file of the plugin's token, and a secret whose name is no safe file name (letters, digits, _, . and -, not . or ..). A restart that fails answers 503, and the settings stay saved.
Egress and consent¶
Each plugin image declares in spot.plugin.egress what it sends outside SPOT: none, indicators (URLs, domains, hashes and the like) or content (email content). An image without the label counts as content. The catalog shows the class, and the installer records it on the plugin's entry as egress.
Every plugin, of any class, runs on spot-plugins-isolated, a Docker network with no route out. It reaches the gateway, the knowledge service and the analyzer-orchestrator only (the bundled Ollama is out of its reach unless the plugin is of class content), so a model it would fetch by URL or by Hugging Face id cannot load: ship the model in the image or on a volume. A plugin reaches the hosts its settings of type url name, and nothing else, through the egress gateway: see Reaching configured hosts.
Installing a content plugin needs an admin's consent: "consent_to_content": true in POST /api/v1/plugins/install, else 409. The installer records the consent on the entry:
plugins:
analyzers:
analyzer-llm:
url: "http://spot-analyzer-llm:8000"
image: "registry.example.com/spot/analyzer-llm:2.0.0"
version: "2.0.0"
egress: content
consent:
egress: content
user: admin
at: "2026-10-08T09:30:00Z"
Updating to a version whose egress is content needs the consent again (POST /api/v1/plugins/{id}/update with "consent_to_content": true), unless a consent to content is recorded already. A manual entry (POST /api/v1/config/plugin/{kind}, no image) records no egress and no consent: the admin wires a service of their own, outside the catalog.
The catalog lists an image whose spot.plugin.contract is not 2 (an image without the label is contract 1; a label that is no number is unreadable) with a refusal that says why; installing or updating to it answers 409 with that reason. An entry listed by an earlier SPOT, which did not read the label, is refused until the catalog is refreshed.
A plugin has one kind: the catalog leaves out an image whose spot.plugin.kind names no kind, one core does not know or several, and the scan logs why ("Refused plugin
Responders¶
Responders act on verdicts: quarantine, notify, open a ticket. They are the entries of plugins.responders: core calls a plugin as a responder only when it is an entry there. A mailbox connector that fetches mail and moves a malicious message to quarantine is two plugins, a retriever and a responder, each with its own id:
plugins:
mail_retrievers:
mailbox-connector:
url: "http://spot-mailbox-connector:8000"
responders:
mailbox-quarantine:
url: "http://spot-mailbox-quarantine:8000"
responder-ticket:
url: "http://spot-responder-ticket:8000"
A workflow names the responders of each verdict in on_verdict:
workflows:
- id: default-workflow
on_verdict:
malicious: [mailbox-quarantine, responder-ticket]
unknown: [responder-ticket]
Whether they act is the job's mode, not the workflow's (see Observe and enforce). Once a job has its verdict, the analyzer-orchestrator calls POST /internal/respond on each responder named, at once, each with its own token. The call carries the verdict, the score, decided_by, the signals and the job's mode: a responder acts only in enforce, and in observe only records what it would do. Each responder called is recorded as an action, respond, applied when the job is in enforce and the responder answered. A failed job is unknown and goes to on_verdict.unknown. A call that fails, or gets no answer within 10 s, is retried three times (after 1, 2 and 4 s), then given up and logged. A name that is not a configured, enabled responder is skipped with a warning, so a workflow may name a responder installed later.
When a responder gives up, the result is delivered again once every call has ended, and every responder of its verdict is called again, until the delivery limit parks the result in orchestrator.dispatch.dead-letter (see RabbitMQ Message Queue). Calling the responders that succeeded again is safe: each acts once per job_id.
Delivery is at least once: a responder may be called more than once for the same job_id. A call that timed out may still complete while it is retried, a result delivered again calls every responder of its verdict again, and a restart between the calls and the end of the dispatch calls them again. A responder acts once per job_id.
Observe and enforce¶
Each job is in observe or enforce mode. In observe, SPOT records what would be done with the message; in enforce, it is done. A job is in enforce only when its mail connection and its route both are:
- a mail connection is a mail retriever's entry: its
modeisobserveorenforce. A new retriever is writtenmode: observe, and an entry without one is in observe. Switch it withPUT /api/v1/config/plugin/mail_retriever/{id}/mode(admin, body{"mode": "enforce"}) orspot-cli plugin mode <id> enforce; the history records "Setto enforce", and nothing is restarted: the retriever learns its mode from SPOT's answers to its ingests ( IngestionResult.mode), so a switch reaches it with its next answer. While SPOT does not answer, the milter keeps the mode of the last answer, observe until the first. A plugin of another kind has nomode; - a route's
modeisenforceby default, which follows its connection, orobserve, which keeps its jobs in observe whatever the connection says. An email no route takes is in its connection's mode.
plugins:
mail_retrievers:
retriever-milter:
url: "http://spot-retriever-milter:8000"
mode: enforce
recipient_groups:
finance: ["cfo@example.com", "@finance.example.com"]
routing:
- name: finance
match: {recipient_groups: [finance]}
workflow: strict-workflow
mode: observe # this route's jobs stay in observe
A user's submission (POST /api/v1/analyze) comes through no connection: it is in observe. A re-analysis of a stored email takes the mode of the connection the email came through, and its route's when it names no workflow. One that names a workflow is not routed, so it takes the connection's mode alone: a route's observe holds only the jobs the route chooses, and an analyst's re-analysis naming a workflow, of an email whose connection is in enforce, is in enforce.
A job's mode is decided when it is queued: a switch reaches the jobs queued after it, and those already queued keep theirs. To stop the responders at once, disable them (spot-cli plugin disable <id>): the dispatcher skips a disabled responder from its next read of spot.yaml.
The mode travels with the job: the responders get it as mode, and the retriever in its answer. What was done, or would have been, is recorded as the job's actions, one per job, actor and kind: the retriever reports its own (POST /internal/jobs/{job_id}/action, scope ingest, body {"action": "reject", "applied": true}; applied is refused for a job in observe), and each responder called is one. A job's status (GET /api/v1/analyze/{job_id}) lists them, and GET /api/v1/analyze/actions?days=7 counts those of the last days by actor, kind, action, mode and whether applied, and lists every connection with its current mode (connections).
A retriever's MODE setting (the milter's, before it took its mode from SPOT) is refused by the API in a mail retriever's settings (422, naming the switch above: the plugin and config routes, an install, an update), as WORKFLOW_ID is. One written by hand leaves its settings at the api-gateway's start. It becomes the mode of an entry written before modes, which has none, when it is observe or enforce; any other value, and the setting of an entry that has a mode, is dropped with a warning: only the switch above changes a connection's mode. No plugin may take the id api, the source of user submissions. A workflow's response_mode is gone: a stored one is dropped when read, the change logged, and a workflow written with one is refused.
Mail Orchestrator¶
Mail retrievers are configured in config/spot.yaml under plugins.mail_retrievers.
| Variable | Default | Description |
|---|---|---|
REQUIRE_BROKER | true | RabbitMQ is required: every ingest is published to it, and startup fails if it is unreachable |
VERDICT_POLL_INTERVAL_MS | 100 | How often a synchronous ingest polls Redis for the analysis job record |
Workflow and Analyzer Configuration Files¶
| Variable | Default | Description |
|---|---|---|
SPOT_CONFIG_FILE | config/spot.yaml | The operator's spot.yaml (/app/config/spot.yaml in the containers, on the spot-config volume); it overrides the shipped defaults |
Each image ships the defaults at /app/defaults/spot.yaml (config/defaults/spot.yaml in this repository), outside /app/config: the spot-config volume keeps the files of its first install, and the defaults change with each release. A defaults/ directory an older release left in the volume is not read. The operator's spot.yaml overrides the defaults: platform key by key; plugins, mail, intake, workflows, recipient_groups, routing, shadow, sender_lists and default_workflow_id whole.
default_workflow_id names the workflow an analysis that names none runs when no route takes it (a retriever's, the milter's, a submission's): default-workflow in the defaults. A spot.yaml with workflows of its own sets it to one of them. SPOT writes it as the last key of the file, after the routing sections.
See Workflow YAML Schema below, and Plugin settings above.
Workflow YAML Schema¶
Workflows define which analyzers look at an email, in which order, and how their answers become the email's verdict. They live in the workflows section of spot.yaml, which the API and the dashboard edit. The models are the SDK's (spot_sdk.workflow). Workflows explains the ideas.
Workflow Structure¶
workflows:
- id: "workflow-id" # Required: unique identifier
name: "Human Readable Name" # Required: display name
version: 1 # Integer, at least 1
description: "" # Optional
stages: [...] # Required: at least one stage
timeout_ms: 300000 # The whole run, 1000 to 600000 (default 5 min)
verdict_policy: {...} # How the analyzers' answers become the verdict
on_verdict: {} # Responders to call, by verdict
created_by: "system" # Creator identifier
The analyzer-orchestrator bounds the whole run by timeout_ms: an analyzer still running then, or never reached, is recorded as timeout, and the verdict is decided on what finished.
Stage Configuration¶
stages:
- name: "stage-name" # Required: unique within the workflow
type: "parallel" # parallel | sequential | conditional (default: sequential)
depends_on: [] # Stages that run before this one
condition: null # Run the stage only when this holds
retrieval_limits: null # Caps on Knowledge Store queries (see below)
analyzers: [...] # List of analyzer configs
Stages run in the order their depends_on give; a cycle or an unknown stage fails the job.
Stage Types: - parallel - Run all analyzers at once - sequential - Run the analyzers one after another - conditional - Run the analyzers one after another, each only when its own condition holds; the others are recorded as skipped
A workflow with a stage of any other type is refused, naming the stage.
A stage whose condition does not hold is skipped: its analyzers are recorded as skipped, and a condition on that stage is false.
retrieval_limits caps what the stage's analyzers ask of the Knowledge Store: max_top_k (1 to 1000) and min_score_floor (0 to 1). They only shrink a query, never widen it; the orchestrator sends them in each AnalyzeRequest and the SDK's KnowledgeClient.for_analysis applies them.
Conditions¶
A stage's condition, or an analyzer's in a conditional stage, compares the outcome of an earlier stage: the workflow's policy applied to that stage's analyzers alone.
<stage>.score <operator> <number>: the stage's score, 0 to 100, with>,>=,<,<=,==or!=. It is false when the stage has no score, its verdict beingunknown.<stage>.verdict == <verdict>or!=: one ofmalicious,suspicious,spam,benign,unknown.
A condition must parse and name a stage of the workflow, or the workflow is refused. A condition on a stage that was skipped is false; one on a stage that has not run yet fails the job.
Analyzer Configuration (in Workflow)¶
Each analyzer within a stage:
analyzers:
- id: "analyzer-nlp" # Required: its id under plugins.analyzers
weight: 1.0 # 0 to 10; 0 only observes
timeout_ms: 30000 # 1000 to 300000; the workflow's timeout may cut it short
required: false # If true and unknown, the email is unknown
failure_strategy: "retry" # retry | skip | fail (default: retry)
retry_config: # Used by retry
max_attempts: 3
backoff_ms: 1000
max_backoff_ms: 10000
exponential_backoff: true
condition: null # In a conditional stage: when to run this analyzer
The orchestrator sends each analyzer an AnalyzeRequest with its own token: the job id, the email's view, the outcomes of the stages that ran before (previous, by stage name), the stage's retrieval_limits and the time left (deadline_ms). The answer must be an AnalyzerVerdict.
An analyzer that does not answer one (a timeout, an HTTP error, an answer that is invalid, larger than 1 MiB or encoded, no token, no URL) is handled by its failure strategy:
retry- Try again after the backoff, up tomax_attempts, then count the analyzer asunknownskip- Count it asunknownat oncefail- Fail the job: its verdict isunknown, decided byfailed
An analyzer that answers unknown itself is unknown too, whatever its strategy. An analyzer of weight 0 only observes: it runs, and its decisive signals count, nothing else does.
Verdict Policy¶
verdict_policy:
method: weighted_mean # most_confident | weighted_mean | mean | max | majority
malicious_threshold: 80 # 1 to 100
suspicious_threshold: 50 # 1 to 100, at most malicious_threshold
decisive_signals:
- signal: "misp.*" # a signal name, or a prefix ending in ".*"
verdict: malicious
decision_stages: [decision] # the stages whose analyzers count; unset: every stage
The decision, in order:
- The sender lists. A
decisive_signalsrule that matchessender_list.blockedorsender_list.allowed, the signals core adds to an email whose sender is listed (Sender lists), sets the verdict before any analyzer's: score 100 when the domain is blocked, 0 when it is allowed;decided_byissignal:<name>. The blocked signal is read first. Without such a rule the lists' signals are shown and never counted. - Decisive signals. The first signal that a
decisive_signalsrule matches, emitted by any analyzer that ran in any stage, sets the verdict. An analyzer's signal namedsender_list.blockedorsender_list.allowedis shown, never decisive: only core's lists give those. Signals are read in workflow order: the stages in their order, each stage's analyzers as listed. The score is the highest score of the analyzers that emitted it;decided_byissignal:<name>. - The analyzers that count are those of
decision_stages, less theunknownones and those of weight 0. Arequiredanalyzer of those stages that isunknown(a timeout included, not one skipped by a condition) makes the verdictunknown(required_unknown). With no analyzer left, it isunknowntoo (all_unknown). An unknown verdict has score 0. - The method decides the rest;
decided_byismethod:<name>: most_confident: the analyzer with the highestconfidencegives its own verdict and score; ties go to the higher weight, then the higher score.weighted_mean,mean,max: the score is the mean weighted byweight, the plain mean, or the highest score. It ismaliciousat or abovemalicious_thresholdandsuspiciousat or abovesuspicious_threshold. Below, the verdict isspamwhen at least half of the counting weight saysspam, andbenignotherwise.majority: the verdict with the most weight wins; ties go to the more severe (malicious,suspicious,spam,benign). The score is the weighted mean of the scores of the analyzers that chose it.
Means are rounded half up.
The decision stages must hold an analyzer of weight above 0, or the workflow is refused. A new workflow's policy defaults to weighted_mean over every stage.
Responders of a workflow¶
on_verdict maps a verdict to the responders to call. Whether they act is the job's mode, its connection's and its route's: see Observe and enforce and Responders.
Workflows written in contract 1¶
The workflow API (create, update, validate) refuses a contract 1 workflow with 400, naming where its contract 1 field is. Workflows stored in contract 1, in spot.yaml or a workflows file, are converted when read, each change logged, and the api-gateway writes them back once when spot.yaml is writable:
final_stage_namebecomesverdict_policy: {method: most_confident, decision_stages: [<that stage>]}, which keeps contract 1's verdicts; the last stage when none was named. A workflow that already has averdict_policykeeps it;<stage>.weighted_confidence > Xbecomes<stage>.score > 100X. A score measures how malicious the email is, not how confident the analyzers were: check every converted condition;failure_strategy: fallbackbecomesskip;- a stage's
aggregation_method,continue_on_failureandmin_successful_analyzersare dropped.
A workflow of either contract that still holds continue_on_failure, min_successful_analyzers or max_parallel_analyzers, which SDK 2 removed, loses them the same way, each removal logged. The workflow API accepts a body with them and ignores them.
Complete Workflow Example¶
The default workflow that ships in config/defaults/spot.yaml:
workflows:
- id: "default-workflow"
name: "Default Phishing Detection Workflow"
version: 1
description: "Standard workflow: parallel analysis (NLP, LLM) followed by decision"
stages:
- name: "parallel-analysis"
type: "parallel"
depends_on: []
analyzers:
- id: "analyzer-nlp"
weight: 0.5
timeout_ms: 30000
required: false
failure_strategy: "skip"
retry_config:
max_attempts: 2
backoff_ms: 1000
max_backoff_ms: 5000
exponential_backoff: false
- id: "analyzer-llm"
weight: 0.5
timeout_ms: 45000
required: false
failure_strategy: "skip"
retry_config:
max_attempts: 2
backoff_ms: 2000
max_backoff_ms: 10000
exponential_backoff: true
- name: "decision"
type: "sequential"
depends_on: ["parallel-analysis"]
analyzers:
- id: "analyzer-llm"
weight: 1.0
timeout_ms: 60000
required: true
failure_strategy: "retry"
retry_config:
max_attempts: 3
backoff_ms: 2000
max_backoff_ms: 15000
exponential_backoff: true
timeout_ms: 300000
verdict_policy:
method: "most_confident"
decision_stages: ["decision"]
created_by: "system"
The three workflows of the defaults (default-workflow, nlp-only-workflow and fast-workflow) hold analyzer-rules in their first stage as an observer: weight 0, not required, failure strategy skip. Only its decisive signals count, so nothing in its score moves a verdict, and where the plugin is not installed it is unknown and the workflow decides without it. Every shipped workflow, those three and the three of spot-cli setup, sets suspicious on the signal attachment.bidi_override, an attachment name that hides its real extension behind a bidi control (invoice<U+202E>fdp.exe shows as invoiceexe.pdf). The example above omits the observer and the signal for brevity.
A workflow that weighs every analyzer, overrides on a MISP hit, and sends malicious emails to a responder:
workflows:
- id: "weighted-workflow"
name: "Weighted analysis"
stages:
- name: "analysis"
type: "parallel"
analyzers:
- id: "analyzer-rules"
weight: 1.0
failure_strategy: "skip"
- id: "analyzer-nlp"
weight: 2.0
failure_strategy: "skip"
- id: "analyzer-misp"
weight: 0 # observes: only its decisive signals count
failure_strategy: "skip"
verdict_policy:
method: "weighted_mean"
malicious_threshold: 80
suspicious_threshold: 50
decisive_signals:
- signal: "misp.ioc_match"
verdict: "malicious"
on_verdict:
malicious: ["responder-ticket"]
Retention (spot.yaml)¶
A received email is kept for data_retention_days, and an entry of the audit log for audit_retention_days, platform settings of config/spot.yaml:
| Setting | Default | Description |
|---|---|---|
platform.data_retention_days | 365 | Days a received email is kept, with its raw copy and analyses. DATA_RETENTION_DAYS in the environment overrides it |
platform.audit_retention_days | 365 | Days an entry of the audit log is kept. AUDIT_RETENTION_DAYS in the environment overrides it |
Each email expires data_retention_days after it is stored; an email stored before its expiry was recorded (migration 002) expires data_retention_days after it was created. The api-gateway purges the expired emails when it starts and then once a day, a thousand at a time, each batch a transaction of its own; the database deletes their raw copies, analyses and signals with them. A purge that fails is logged and tried again an hour later. The gateway reads the setting for each message it stores and for each purge: a change (PUT /api/v1/config/platform, or an edit of spot.yaml and a config reload) applies to the emails stored afterwards, and every email keeps the expiry it was stored with. The analysis jobs' copies in Redis, which hold no content, expire on their own within 24 hours.
A dataset's copy of an email is exempt: the purge never deletes it until it is removed from its dataset or the dataset is deleted (Retention).
The same purge then deletes the audit log's entries older than audit_retention_days, read at each purge: a change applies at the next one. A failure of either purge is logged, and both are tried again an hour later.
Routing (spot.yaml)¶
Routes choose the workflow each email runs; the shadow runs candidate workflows beside production, to compare. Three sections of spot.yaml hold them, which GET /api/v1/routing (analysts and admins: the groups list staff addresses) answers and PUT /api/v1/routing (admins) replaces together, as spot-cli routing show and spot-cli routing apply <file> do:
recipient_groups:
finance: ["cfo@example.com", "@finance.example.com"]
routing:
- name: finance
match: {recipient_groups: [finance]}
workflow: strict-workflow
- name: milter
match: {sources: [retriever-milter]}
workflow: fast-workflow
timeout_ms: 15000
- name: vendors
match: {sender_domains: ["vendor.example", "*.vendor.example"]}
workflow: default-workflow
shadow:
- workflow: candidate-workflow
match: {sources: [retriever-milter]}
| Field | Description |
|---|---|
routing[].name | Unique among the routes: lower-case letters, digits, _ and -, starting with a letter or a digit, at most 64 characters |
routing[].match | What an email must be for the route to take it; empty, every email |
routing[].workflow | The workflow the email runs |
routing[].timeout_ms | 1 000 to 300 000: how long a retriever that waits for the verdict waits at most |
match.sources | Retriever ids, or api for the submissions |
match.recipient_groups | Names of recipient_groups: a recipient of the email is in one |
match.sender_domains | example.com, or *.example.com for its subdomains alone: the From domain, when a trusted DMARC result passed for it |
recipient_groups.<name> | The group's entries: addresses (local@domain) or whole domains (@domain, not their subdomains) |
shadow[].workflow | A workflow run in the shadow; one entry per workflow |
shadow[].match | Which emails it runs on, as a route's |
How an email is routed. The first route whose match holds chooses the workflow. Within a match, every criterion given must hold, and within a criterion any value. An email no route takes runs the default workflow (default_workflow_id). A workflow a user names, for a submission or a re-analysis, is not routed, and a dry run is never routed. A re-analysis that names no workflow is routed on the stored email, by today's routes. The analysis records its route (route in GET /api/v1/analyze/{job_id} and the job list; none for a named workflow, the default, or an analysis from before routing). A route's timeout_ms caps a waiting retriever's wait: the retriever's own, or the route's when that is shorter.
- Recipients are the SMTP envelope's when the retriever sent one, else the
ToandCcaddresses; a route matches when any recipient is in a group. They are compared lower-case, their domains in their Unicode form, as the parser gives theFromdomain: an A-label (xn--...) is decoded. Without an envelope (a submission, a retriever that sends none), the recipients are the headers the sender wrote, and as any recipient matches and the first route wins, an address the sender adds can steer a message to an earlier route. Put the stricter routes first, and do not route a group to a laxer workflow for a source that sends no envelope. - Sender domains count only when a DMARC result of one of
mail.trusted_authserv_idspassed for theFromdomain: a forged or untrusted result matches nothing. Whilemail.trusted_authserv_idsis empty, a route or shadow entry withsender_domainsnever matches: saving one through the API is refused (422), and one written by hand loads, each load warning of it once (route <name> matches sender_domains while mail.trusted_authserv_ids is empty: it never matches). - Domains, in group entries and sender domains, are written as mail carries them: letters, digits and hyphens, in their usual form (UTS 46, STD3 rules).
_, full-width forms or a character UTS 46 ignores are refused, as no email could match them. They are stored lower-cased, their A-labels decoded. - A route or shadow entry to a workflow that is not loaded (an edit by hand, a workflow left out at load) is skipped: the email goes on to the next route, or the default workflow. Each load says so once:
route <name> names workflow <id>, which is not loaded: it is skipped. A routing that names a workflow that is not loaded is refused when saved through the API (PUT /api/v1/routing,spot-cli routing apply: 422, naming the route), while a hand edit loads it. A workflow a route or the shadow names cannot be deleted (409): change the routing first.
A routing that breaks a rule (a route named twice, an unknown group, a malformed entry) stops the api-gateway's start and fails a config reload (400), as an invalid mail: section does; POST /api/v1/config/validate reports it. SPOT writes the three sections on every save, empty ones included, before default_workflow_id.
Retrievers that named their workflow. Up to 2.0.0-beta.7, a retriever named its workflow (its WORKFLOW_ID setting); a retriever's ingest no longer names one, and the API refuses the setting in a mail retriever's settings (422: the plugin and config routes, an install, an update). At each start, the api-gateway turns each retriever's WORKFLOW_ID into a route {sources: [<id>]} to that workflow, appended to the routes, and removes the setting; the start's commit of spot.yaml's history records it, and the log names each route. A setting written by hand after a start routes nothing until the next one.
- The route is named after the retriever: its id made a route name (lower-cased, every run of other characters made a
-, leading-and_dropped, at most 64 characters), or the next free name (<name>-2,<name>-3, ...).IMAP.corpbecomesimap-corp. - A setting that names a workflow that is not loaded is removed with a warning: that retriever's mail runs the routes or the default workflow.
- When an earlier route takes all of a retriever's mail (no recipient groups or sender domains, and no sources or that retriever among them), the converted route never applies, and the log says which route takes its mail: move the routes into the order you want.
- Rolling
spot.yamlback to a commit made before the conversion restores that commit's routing, which has no such routes, and keeps today's plugins, which have noWORKFLOW_IDleft: the retrievers' mail then runs the default workflow until the routes are added again.
Shadow¶
Each result of a production job, except a failed one, is run again through each shadow workflow whose match holds for its email, production's own workflow aside, on the stored view, as a run of its own: analyzers are not told. Where its verdict class differs from production's, the difference is stored, with the shadow workflow's result encrypted, and listed newest first at GET /api/v1/workflows/{workflow_id}/shadow (viewers: verdicts and scores only). A redelivered result stores nothing twice.
Every run that completes is also counted for its day (UTC), differing or not, for the rate of the differences the same route gives over the last days days (30 by default, up to 90), counted as Workflows says.
The load. A shadow entry adds one workflow run per matched email. The analyzer-orchestrator runs them one at a time (prefetch 1), after the email's verdict: a shadow run never delays its own email's verdict, nor drops production's work. But its analyzer calls compete with production's, so on a saturated analyzer it can slow other emails' analyses, the next waiting retriever's included. Shadow the routes whose volume the analyzers can carry twice. When the shadow cannot keep up, its queue drops its oldest work past 1 000 results (see RabbitMQ).
Sender lists (spot.yaml)¶
The domains analysts block or allow, from an email's label in the dashboard or through /api/v1/sender-lists (API). Each change is a commit of spot.yaml's history by the analyst. What a list does is a workflow's policy to decide (Sender lists).
| Field | Description |
|---|---|
sender_lists.block | Domains whose mail gets the signal sender_list.blocked: the From domain, authenticated or not |
sender_lists.allow | Domains whose mail gets the signal sender_list.allowed: the From domain, when a DMARC result of mail.trusted_authserv_ids passed for it |
Domains are written as a route's sender_domains are, example.com, or *.example.com for its subdomains, and stored the same way: lower-cased, their A-labels decoded. A domain is on one list at most. The allow list needs mail.trusted_authserv_ids, so a spoofed From never reaches it: while none is trusted, adding a domain to it through the API is refused (422), and one written by hand loads with a warning, as it never matches. An invalid section (a malformed domain, a domain on both lists) stops the api-gateway's start and fails a config reload; the analyzer-orchestrator loads it empty, with a warning that names the rule and its position, never the domain. SPOT writes the section on every save, empty lists included.
Mail and Intake (spot.yaml)¶
The api-gateway parses each received message once, in worker processes, and stores the raw message and its view, both encrypted. A plugin that fetches one decoded part of a stored message (GET /internal/emails/{email_id}/parts/{part_id}) has it parsed again, in the same workers and within the same limits. Two spot.yaml sections tune it:
mail:
# authserv-ids of our own MTAs: only their Authentication-Results headers
# count. The MTAs must remove the incoming headers that carry these ids
# (RFC 8601, section 5), or a sender could forge one.
trusted_authserv_ids: ["mx.example.com"]
intake:
parse_timeout_s: 10 # a parse that runs longer is stopped
parse_memory_mib: 640 # address space of one parse worker
parse_workers: 2 # parses that run at once
max_view_mib: 8 # defence in depth: a view past it is refused
max_in_flight: null # messages taken at once (null: 2 x parse_workers)
max_content_in_flight: 2 # raw messages, parts and views served at once
body_timeout_s: 60 # time a request body has to arrive in full
| Setting | Default | Description |
|---|---|---|
mail.trusted_authserv_ids | [] | Authentication results from any other authserv-id are ignored |
intake.parse_timeout_s | 10 | Time limit of one parse, and of a worker's start; a 25 MiB ordinary message takes about 1 s |
intake.parse_memory_mib | 640 | Address-space limit of a worker; a 25 MiB ordinary message needs about 300 MiB |
intake.parse_workers | 2 | Worker processes, so one slow message stalls no other |
intake.max_view_mib | 8 | A view's JSON may not be larger. Defence in depth: spot_sdk.mime.parse bounds every view to about 6.84 MiB, so none reaches this limit (an ordinary 25 MiB message gives at most about 3 MiB) |
intake.max_in_flight | 2 x parse_workers | Messages the gateway takes at once, each from reading the body to storing the email |
intake.max_content_in_flight | 2 | Raw messages, parts and views the gateway serves at once (GET /internal/emails/{email_id}/raw, .../parts/{part_id}, GET /api/v1/emails/{email_id}, /raw, /view and /forensics), each until its response is sent |
intake.body_timeout_s | 60 | A body still arriving after this is answered 408, so slow uploads cannot hold every slot |
A message larger than 25 MiB is refused with 413. An empty one, or one whose parse runs past a limit, is refused with 422 (parse_limit, naming the time, memory or view limit). Nothing is stored then. A part fetch past the time or memory limit is refused the same way. Past max_in_flight messages taken at once, past max_content_in_flight raw messages, parts and views served at once, or when no parse worker can take one, the gateway answers 503 with Retry-After: retry later.
A raw message, a part or a view served holds its place until its response is sent, so a client that stops reading holds one until its connection closes. That delays other content reads, never intake, which has a bound of its own. A user's read also holds its authentication's database connection until then: the bound keeps those connections within max_content_in_flight.
The gateway reads the intake section when it starts: restart it after a change. It reads mail.trusted_authserv_ids for each message, so a configuration reload is enough.
Memory. One message taken costs the gateway about 256 MiB at its peak (a 25 MiB message, its base64 JSON body, its view and their encryption), and each worker may use up to parse_memory_mib. A raw message, a part or a view served costs at most about 190 MiB (measured for 25 MiB messages: a raw read 175 to 191 MiB, a part fetch 158 to 175 MiB, a view of 7 MiB 83 MiB; decrypting dominates), plus, for a slow reader, up to the body's size in the server's send buffer: count 256 MiB. Size the api-gateway's memory limit as:
512M + max_in_flight x 256 MiB + max_content_in_flight x 256 MiB
+ parse_workers x parse_memory_mib
With the defaults: 512 + 4 x 256 + 2 x 256 + 2 x 640 = 3328 MiB.
Config Reload¶
Configuration can be reloaded without restart:
# Reload config via API
curl -X POST http://localhost:8001/api/v1/config/reload \
-H "Authorization: Bearer $TOKEN"
# Response shows what changed
{
"old_version": "abc123@20251204",
"new_version": "def456@20251204",
"changed": {
"platform": false,
"workflows": true,
"analyzers": ["analyzer-nlp"]
}
}
Reload behavior: - Invalid YAML/schema returns 400, previous config preserved - Version only bumps if content actually changed - Reloads run one at a time, and wait for the start's sealing step when it is still running
Editing spot.yaml by hand¶
Change the configuration through the API, the dashboard or spot-cli when you can. To edit spot.yaml by hand, replace it whole, in one step, then reload:
# Write the new file under a temporary name in the volume, then rename it
# into place: no service ever reads it half copied
docker compose exec -T api-gateway sh -c \
'cat > /app/config/.spot.yaml.new && mv /app/config/.spot.yaml.new /app/config/spot.yaml' \
< spot.yaml
curl -X POST http://localhost:8001/api/v1/config/reload -H "Authorization: Bearer $TOKEN"
A spot.yaml that does not parse or validate stops the api-gateway from starting (it never runs on the shipped defaults in its place), so docker compose exec finds no container to enter. Replace the file from a one-off container of the same image, user and volume, then start the gateway; its start records the file:
docker compose run --rm --no-deps -T api-gateway sh -c \
'cat > /app/config/.spot.yaml.new && mv /app/config/.spot.yaml.new /app/config/spot.yaml' \
< spot.yaml
docker compose up -d api-gateway
- Do not write
spot.yamlwhile a start or a reload runs. For the few seconds the sealing takes, a write is recorded in the load's commit, a plain secret it holds included. - Raise the
versionof a workflow you change past the highest it has had (spot-cli workflow history <id>). SPOT numbers each save past it, so a number names one version; an edit that keeps the number gives it a second definition. - The gateway rewrites the file in its own form at each start and reload: comments, unknown keys and misspelt keys do not survive.
spot-cli config validatedoes not report an unknown key either, so check each key against this reference. - The environment sets no section of
spot.yaml. It sets the infrastructure settings and the platform settings (Configuration Precedence);plugins,workflows,default_workflow_id,mail,intake,recipient_groups,routingandshadowcome from the file alone, and a variable of the same name (MAIL, which login shells set) is ignored.
History of spot.yaml¶
Every change of spot.yaml is a commit of history.git, a git repository beside it: /app/config/history.git on the spot-config volume (config/history.git in development). Each commit holds the whole file. Its author is the user who made the change, or SPOT for its own, and its message says what the change did, which sections and workflows it touched, and the version of each of those workflows the file holds. The api-gateway writes it itself; the images hold no git binary, but git --git-dir history.git log reads it anywhere.
- Changes made outside SPOT (an edit, a copy) are recorded at the gateway's next start, by SPOT, or at its next
POST /api/v1/config/reload, by the reload's user. Either first seals the plain secret settings of catalog plugins, so no commit holds a plain value the catalog marks secret, as long as the catalog can be read then (Redis up): otherwise the settings are recorded as they are. Either then writes the file back in SPOT's form (Editing spot.yaml by hand). - A change saved during a load (the few seconds the sealing takes, longer when a plugin it seals is being installed or updated) is written, and recorded in the load's commit, under the load's author. The log keeps its own author and message:
spot.yaml saved by <author> while its history is paused: <message>. - The history is never pruned, unlike the emails (
data_retention_days). It keeps every value ever saved, each author's email and the recipient groups' staff addresses included: an address removed from a group, such as a departing employee's, stays in every earlier commit. Keep a manual plugin's secret settings in its sealed secrets (PUT /api/v1/config/plugin/{kind}/{id}withsecrets), never insettings: without a catalog entry, SPOT cannot tell that they are secret, and records them as they are. - Backups.
history.gitis on thespot-configvolume withspot.yaml: the deploy wrapper'sspot-cli backuptakes both (see the Admin guide).
Reading and rolling back. Admins read the history through the API (API). A commit's detail shows what it changed, each sealed secret reading (sealed), or (sealed, changed), and previews what a rollback to it would change in the settings in force, as SPOT writes them: a hand edit not yet reloaded shows in neither the preview nor the rollback's sections, and a rollback replaces it. A rollback (POST /api/v1/config/history/{commit}/rollback) restores every section of that commit but plugins:, as a new commit by the admin:
plugins:stays as SPOT holds it: a rollback never touches a plugin's entry, container, token or settings. A hand edit ofspot.yamlnot yet reloaded is replaced, as by any change saved through SPOT. A restored workflow that names a plugin no longer installed loads, and the readiness shows it broken.- A platform setting the environment sets (compose sets
LOG_LEVEL,DEBUGandENABLE_RATE_LIMITING) is written as the commit's file had it, and the environment's value still applies. - The analyzer-orchestrator runs the restored workflows from its next job. The platform settings apply as after a reload: those read only at start need a restart of the api-gateway, for example
log_level,debug, rate limiting,allowed_hosts,allowed_origins,enable_oauth, the token lifetimes and theintakesection. - A commit whose
spot.yamlno longer validates (a rule added since) is refused with 422, naming each field and the rule it breaks; nothing is written.
When a change cannot be recorded. A change that is written but cannot be committed (a full volume, a damaged history.git) stays written: the error log names the change and the cause (spot.yaml was saved but not recorded in its history: ...), and the next commit records it. A start that cannot record the file starts anyway.
- A lock left behind. A crash during a commit can leave
history.git/refs/heads/main.lock. Every commit then fails, the log naming it (HistoryConflictError: the history's branch is locked by .../main.lock). When no gateway is saving, remove the lock:docker compose exec api-gateway rm /app/config/history.git/refs/heads/main.lock. - A damaged history. When an object or a ref of
history.gitis missing or damaged (a copy taken while a commit was written, say), the history routes, a workflow's versions and the workflow saves that number a version answer 503, naming it. Movehistory.gitaside, as below.
When commits keep failing for another reason, move history.git aside: the next change starts a new history, whose first commit holds the whole file, and git still reads the old one. The new history knows only the workflows in force: a workflow deleted before the move is numbered from 1 again when it is created again.
Development Configuration¶
When APP_ENV=dev, additional variables are available:
Source Code Mounting¶
| Variable | Description |
|---|---|
API_GATEWAY_SRC_MOUNT | Path to API Gateway source |
API_GATEWAY_TEST_MOUNT | Path to API Gateway tests |
ANALYZER_ORCHESTRATOR_SRC_MOUNT | Path to Analyzer Orchestrator source |
ANALYZER_ORCHESTRATOR_TEST_MOUNT | Path to Analyzer Orchestrator tests |
MAIL_ORCHESTRATOR_SRC_MOUNT | Path to Mail Orchestrator source |
MAIL_ORCHESTRATOR_TEST_MOUNT | Path to Mail Orchestrator tests |
SHARED_MOUNT | Path to shared modules |
CONFIG_MOUNT | Path to config directory |
MOUNT_MODE | Mount mode (rw or ro) |
Debug Ports¶
| Variable | Default | Description |
|---|---|---|
ANALYZER_ORCHESTRATOR_DEBUG_PORT | (a free port) | Debug port for analyzer orchestrator |
KNOWLEDGE_DEBUG_PORT | (a free port) | Debug port for the knowledge service |
MAIL_ORCHESTRATOR_DEBUG_PORT | 8092 | Debug port for mail orchestrator |
The debug ports listen on 127.0.0.1 only. Left empty, Docker picks a free host port.
Development Tools¶
| Variable | Default | Description |
|---|---|---|
MAILHOG_SMTP_PORT | 1025 | Mailhog SMTP port |
MAILHOG_WEB_PORT | 8025 | Mailhog web UI port |
ADMINER_PORT | 8080 | Adminer database UI port |
HOST_UID | 1000 | Host user ID for devtools container |
HOST_GID | 1000 | Host group ID for devtools container |
Production Configuration¶
When APP_ENV=prod:
Docker Registry¶
| Variable | Default | Description |
|---|---|---|
REGISTRY_PORT | 5000 | Docker registry port |
CI_REGISTRY_IMAGE | (CI only) | Full registry path for platform services (e.g., localhost:5000/spot/platform) |
CI_REGISTRY | (CI only) | Registry host for external analyzers (e.g., localhost:5000) |
VERSION | latest | Image version tag |
BASE_IMAGE | base:latest | Base image name and tag |
Note: CI_REGISTRY_IMAGE and CI_REGISTRY are NOT set for local development. They are only set in .gitlab-ci-local-env for CI context.
CI/CD Configuration¶
GitLab-specific variables (only needed for CI/CD):
| Variable | Description |
|---|---|
GITLAB_HOST | GitLab hostname |
GITLAB_TOKEN | GitLab access token (uses CI_JOB_TOKEN if available) |
CI_REGISTRY | Container registry URL |
GITLAB_GROUP | GitLab group name |
Configuration Precedence¶
Configuration is loaded in this order (highest to lowest priority):
- The environment's variables, command line included
spot.yaml, over the shipped defaults (/app/defaults/spot.yaml)- The
.envfile the settings read, in the service's working directory - Code defaults (in Pydantic Settings classes)
In a compose deployment the project's .env reaches the services through the environment: compose passes its values to the containers as variables, so they rank first.
Example:
# .env file has: LOG_LEVEL=INFO
# Command-line override:
LOG_LEVEL=DEBUG make service:start # Uses DEBUG
The environment sets the infrastructure settings and the platform settings only. A platform setting it sets (compose sets LOG_LEVEL, DEBUG and ENABLE_RATE_LIMITING) outranks spot.yaml, which keeps its own value: SPOT never writes the environment's value into the file. The sections of spot.yaml (plugins, workflows, default_workflow_id, mail, intake, recipient_groups, routing, shadow) are never read from the environment: set the default workflow with POST /api/v1/workflows/{id}/set-default, or in spot.yaml.
Configuration Format¶
Environment Variable Prefixes¶
SPOT uses standard environment variable names without a global prefix:
- Infrastructure:
POSTGRES_*,REDIS_*,RABBITMQ_* - Application:
APP_ENV,LOG_LEVEL,SECRET_KEY - Plugins: analyzers, context providers, mail retrievers and responders are configured under
plugins:inconfig/spot.yaml
Quick Reference Examples¶
Minimal Production .env¶
# Environment
APP_ENV=prod
# Security (REQUIRED - generate secure values)
SECRET_KEY=generate-with-python-secrets-module
SPOT_INTERNAL_API_KEY=generate-with-python-secrets-module
ENCRYPTION_KEY=generate-with-cryptography-fernet
# Database
POSTGRES_DB=spot
POSTGRES_USER=spot
POSTGRES_PASSWORD=secure_db_password
# Redis (optional password)
REDIS_PASSWORD=secure_redis_password
# RabbitMQ
RABBITMQ_DEFAULT_USER=spot
RABBITMQ_DEFAULT_PASS=secure_rabbitmq_password
# The host names the api-gateway answers
ALLOWED_HOSTS=localhost,127.0.0.1,api.spot.example.com
Minimal Development .env¶
# Environment
APP_ENV=dev
# Security
SECRET_KEY=dev-secret-key-for-testing-only
SPOT_INTERNAL_API_KEY=dev-internal-key-for-testing-only
ENCRYPTION_KEY=generate-with-cryptography-fernet
# Database (dev defaults)
POSTGRES_DB=spot
POSTGRES_USER=spot
POSTGRES_PASSWORD=spot123
# Redis (no password in dev)
REDIS_PASSWORD=
# RabbitMQ (dev defaults)
RABBITMQ_DEFAULT_USER=guest
RABBITMQ_DEFAULT_PASS=guest
Full Example with Analyzers¶
.env:
# Environment
APP_ENV=prod
# Security
SECRET_KEY=your-32-character-random-key-here
SPOT_INTERNAL_API_KEY=another-32-character-random-key
ENCRYPTION_KEY=your-fernet-key
ALLOWED_HOSTS=localhost,127.0.0.1,api.spot.example.com
# Infrastructure
POSTGRES_PASSWORD=secure_password
RABBITMQ_DEFAULT_PASS=secure_password
config/spot.yaml:
plugins:
analyzers:
analyzer-nlp:
enabled: true
url: "http://10.0.1.10:8000"
settings: {}
analyzer-llm:
enabled: true
url: "http://10.0.1.11:8000"
settings: {}
Each analyzer needs a token (spot-cli plugin token <id>) before core calls it.
Validation¶
The platform validates configuration at startup:
- Database URL format
- RabbitMQ URL format
- Analyzer URL formats
- Port ranges (1-65535)
Invalid configuration will cause startup to fail with a descriptive error message.
Related Documentation¶
- Environment Management - Environment switching guide
- Admin Guide - Deployment configuration
- Developer Guide - Development setup