Mail integration with SPOT¶

SPOT analyses emails that mail-retriever plugins push into the platform. This document covers the bundled retriever-milter plugin, a Postfix (or Sendmail) milter that holds each message until SPOT returns a verdict. SPOT does not run the MTA itself ; you keep your existing Postfix. The behaviour and every setting of the milter are documented in its own README; this guide covers the deployment side.

How it fits together¶

%%{init: {'flowchart': {'subGraphTitleMargin': {'top': 12, 'bottom': 12}}}}%%
flowchart TB
    sender([External sender])
    subgraph mtasrv[MTA server]
        direction TB
        mta[Postfix smtpd<br/>port 25]
        hold[(Hold queue)]
    end
    retriever[retriever-milter<br/>:9900]
    gateway[spot-api-gateway]
    orch[spot-mail-orchestrator]
    analyzer[spot-analyzer-orchestrator]
    delivery([Final delivery])

    sender -->|1. SMTP| mta
    mta -->|2. milter protocol| retriever
    retriever -->|3. POST /internal/ingest + token| gateway
    gateway -->|4. forward| orch
    orch -->|5. enqueue| analyzer
    analyzer -.->|6. verdict| orch
    orch -.->|7. verdict| gateway
    gateway -.->|8. verdict| retriever
    retriever -.->|9. X-SPOT-* headers / quarantine| mta
    mta -->|10. deliver| delivery
    mta -.->|quarantined| hold

    classDef mail fill:#889b4a,stroke:#5d6f2e,color:#1f1610
    classDef analyzer fill:#e48212,stroke:#a95e09,color:#1f1610

    class retriever,orch mail
    class analyzer analyzer

    style mtasrv fill:#efe9dc,stroke:#a99b78,color:#1f1610

Postfix opens one milter connection per SMTP session and keeps the message waiting until the verdict arrives or ANALYSIS_TIMEOUT_MS expires. The milter sits on the spot-plugins network and ingests through the api-gateway with its own plugin token; it never reaches the mail-orchestrator directly.

A milter's ingest waits for its verdict, so it goes to a queue of its own, orchestrator.analysis.sync, with its own analysis slots (MAX_CONCURRENT_SYNC_JOBS, see Analysis concurrency): a backlog of bulk analyses never holds it up. The analyzers stay shared, so analyzers slowed by bulk analyses slow it too. An ingest not finished within ANALYSIS_TIMEOUT_MS gets X-SPOT-Verdict: unknown and ACTION_UNKNOWN, and its analysis still runs to its end: one not started by then is analysed later, with the bulk analyses. Its verdict then shows in the dashboard, not on the message.

SPOT answers each message with its verdict and its mode. The milter sets neither the mode nor the workflow: both are set in SPOT.

The mode. The milter's connection is in one of two modes, set in SPOT (section 5); a message is in enforce only when its connection and its route both are.
- observe (a new connection's) ; only adds the X-SPOT-* headers. Mail is always accepted. A slow SPOT still delays delivery by up to ANALYSIS_TIMEOUT_MS + 5 s.
- enforce ; also applies the configured action for each verdict: accept, tag (subject prefix), quarantine (Postfix hold queue), reject, or tempfail when SPOT gives no verdict.
With no answer from SPOT (timeout, error, skipped), the milter keeps the mode of SPOT's last answer; until the first answer after it starts, it observes. * The workflow. SPOT's routes choose it: the first route whose match holds, such as a route for the milter's source, else the default workflow. A route's time limit caps the milter's wait (see The milter's workflow).

1. Bring up the SPOT side¶

The milter is opt-in. It authenticates with its own plugin token, which it reads from /run/secrets/spot_plugin_token. Issue the token once the stack runs:

spot-cli plugin token retriever-milter --kind mail_retriever

--kind adds the milter to spot.yaml as a manual entry the first time, in observe. Running the command again rotates the token: the old one stops working at once.

The milter sends every message whole to SPOT, raw, with its SMTP envelope: its egress class is content. SPOT stores the message encrypted for data_retention_days. Installed from the catalog, it would need an admin's consent; run by this compose file, it is a manual entry, and SPOT records no egress class or consent for it.

Add mail-milter to COMPOSE_PROFILES in your .env and set the token; the other RETRIEVER_MILTER_* variables under # MAIL RETRIEVERS in .env.example are optional overrides:

COMPOSE_PROFILES=cli,mail-milter     # the profiles you had, plus mail-milter
RETRIEVER_MILTER_TOKEN=<the printed token>

Then bring the stack up:

docker compose up -d

Compose hands the token to the milter as a secret, which needs Docker Compose 2.6 or later. With the profile on and RETRIEVER_MILTER_TOKEN unset, docker compose up -d stops at the milter with environment variable "RETRIEVER_MILTER_TOKEN" required by secret ... is not set. An empty value is skipped silently: the milter then starts without its token. After changing the token, recreate the milter: docker compose up -d --force-recreate retriever-milter.

spot-retriever-milter exposes:

:8000 (container-internal) ; /health and /manifest.
:9900 (milter) ; published on RETRIEVER_MILTER_BIND (default 127.0.0.1). Set it to a private address only if Postfix runs on a different host; see Postfix on another host.

Verify:

docker compose ps retriever-milter

Message size and memory. MAX_MESSAGE_BYTES (25 MiB, the most SPOT takes) counts the headers too: a larger message is accepted unanalysed, with X-SPOT-Verdict: skipped. MAX_BYTES_IN_FLIGHT (75 MiB) bounds the size of the messages the milter holds at once. Past it, further mail waits in Postfix's queue until a verdict frees room, and the milter logs waits for room. Once the milter's content_timeout passes, Postfix applies the milter's default_action to the waiting message. The milter needs at most about 64 + 2 x MAX_CONNECTIONS + 9 x MAX_BYTES_IN_FLIGHT MiB (the latter counted in MiB): about 940 MiB with the defaults. The compose file limits it to 1024M, what its image declares. A milter short of memory is killed, and Postfix then applies default_action to every open session: when you raise MAX_BYTES_IN_FLIGHT or MAX_CONNECTIONS, raise the limit with them in docker-compose.override.yml. For example, 150 MiB in flight needs 64 + 2 x 100 + 9 x 150 = 1614 MiB:

services:
  retriever-milter:
    environment:
      MAX_BYTES_IN_FLIGHT: "157286400"
    deploy:
      resources:
        limits:
          memory: 1664M

2. Configure Postfix¶

Use Postfix 3.8.6, 3.7.11 or 3.6.15 or later; these fixed header deletion after prepended headers.

Hook the milter on inbound SMTP only. With the global smtpd_milters, mail that users send through submission (587/465) is analysed as if it were inbound and leaves with X-SPOT-* headers. Name your existing milters in their own parameter and add SPOT only on the port-25 smtp service:

# /etc/postfix/main.cf
base_milters = <current smtpd_milters value>
smtpd_milters = $base_milters
spot_milter = { inet:<VM_IP>:9900, default_action=accept, content_timeout=60s }

# /etc/postfix/master.cf (the existing smtp line)
smtp      inet  n       -       y       -       -       smtpd
  -o smtpd_milters=$base_milters,$spot_milter

<VM_IP> is the address the milter port is published on (RETRIEVER_MILTER_BIND). Check the effective value, then apply it:

postconf -P smtp/inet/smtpd_milters
postfix check && postfix reload

To roll back, remove the -o line from master.cf and reload.

Notes:

If smtpd_milters is empty today, use -o smtpd_milters=$spot_milter.
An -o value must not refer to the parameter it overrides: -o smtpd_milters=$smtpd_milters,... recurses.
Keep DKIM, DMARC and SPF milters before SPOT, so it sees their Authentication-Results.
SPOT reads only the Authentication-Results of the authserv-ids in mail.trusted_authserv_ids of spot.yaml, your MTAs' own; the list is empty by default, so no SPF, DKIM or DMARC result counts until you set it (see Authentication results). Each of these MTAs must remove the incoming Authentication-Results headers that carry its id (RFC 8601, section 5), or a sender can forge one.
default_action=accept lets mail flow if the milter is unreachable. Consider tempfail once you enforce.
Do not add the milter to non_smtpd_milters unless you also want locally submitted mail analysed.
The milter sends progress replies while it waits, so PROGRESS_INTERVAL_MS (default 5000) must stay below Postfix's milter content_timeout.
Settings without a RETRIEVER_MILTER_* variable (PROGRESS_INTERVAL_MS, MAX_MESSAGE_BYTES, MAX_BYTES_IN_FLIGHT, SUBJECT_TAG, HEADER_PREFIX, PRIORITY, LOG_LEVEL, LISTEN_HOST) are described in the milter README settings table. Set them in a docker-compose.override.yml.

3. Postfix on another host¶

Follow these steps in order when Postfix (<MX_IP>) runs on a different machine than SPOT (<VM_IP>). Sections 1 and 2 give the background.

Check the SPOT version and workflow. The milter 2.0.0b4 takes its mode from SPOT and names no workflow: it needs core 2.0.0-beta.8 or later, and its token (section 1). Core 2.0.0-beta.8 refuses the ingests of an older milter, which name their workflow: that milter gets error on every message. The workflow the milter's mail runs, its route's or the default, needs its analyzers installed and healthy; otherwise the verdict is unknown.
Configure the SPOT host. In .env:

COMPOSE_PROFILES=mail-milter
RETRIEVER_MILTER_TOKEN=<token from spot-cli plugin token retriever-milter --kind mail_retriever>
RETRIEVER_MILTER_BIND=<VM_IP>
RETRIEVER_MILTER_ALLOWED_CLIENTS=<MX_IP>/32
RETRIEVER_MILTER_MAX_CONNECTIONS=<at least the Postfix smtpd process limit>

Add mail-milter to your existing COMPOSE_PROFILES if it is not empty.
The default bind 127.0.0.1 is unreachable from another host.
A remote Postfix keeps its real IP, so <MX_IP>/32 is enough.
The milter's connection starts in observe; keep it there until you trust the verdicts (section 5).
Start the milter.

docker compose pull && docker compose up -d

Check that the container is healthy and listening:

docker compose ps retriever-milter
docker compose logs retriever-milter

The log shows Milter listening on 0.0.0.0:9900.

Open the port to the mail server only. Allow TCP 9900 from <MX_IP> and from nowhere else. The milter protocol has no authentication and no encryption: use a private network or a tunnel (WireGuard or SSH).
Hook Postfix on the mail server. Apply the inbound-only configuration from section 2 with <VM_IP>:9900.
Behind an egress proxy, check the milter's proxy settings. The milter container is created by compose, so it receives the Docker client proxy settings (proxies in ~/.docker/config.json). Its calls to the api-gateway then go through the proxy unless api-gateway is in that noProxy list. Check:

docker exec spot-retriever-milter env | grep -i proxy

Add api-gateway to noProxy if it is missing, then recreate the container with docker compose up -d --force-recreate retriever-milter.

Verify. Send a message from outside (see the smoke test). It must carry X-SPOT-Verdict, X-SPOT-Score, X-SPOT-Mode: observe and X-SPOT-Job-Id, and the Postfix log must show no milter errors.
Observe. Count the verdicts:

docker compose logs retriever-milter | grep -o 'verdict=[a-z]*' | sort | uniq -c

Each inbound message waits up to ANALYSIS_TIMEOUT_MS (30 s) for its verdict. If the analyzers are slow, lower RETRIEVER_MILTER_ANALYSIS_TIMEOUT_MS; a workflow that does not finish in time then shows up as unknown. timeout means SPOT did not answer at all within ANALYSIS_TIMEOUT_MS + 5 s.

4. Smoke-test the path¶

You can use swaks to test the plugin.

swaks --to user@your-domain.example \
      --from sender@external.example \
      --server postfix.example:25 \
      --header "Subject: SPOT smoke test" \
      --body "Just verifying the milter chain."

Look for:

spot-api-gateway and spot-mail-orchestrator log lines for POST /internal/ingest.
The delivered message containing X-SPOT-Verdict and X-SPOT-Mode.

5. Observe first, then enforce¶

The milter's connection starts in observe. Keep it there until you trust the verdicts: the headers show what enforce would have done (X-SPOT-Action) without touching any mail, and SPOT records it as each job's actions, which the dashboard shows. Then switch the connection in SPOT:

spot-cli plugin mode retriever-milter enforce

or from the milter's page in the dashboard (Plugins). Nothing restarts: the milter takes the new mode from SPOT's next answer, and spot-cli plugin mode retriever-milter observe switches back the same way. spot-cli plugin info retriever-milter shows the connection's mode.

A message is in enforce only when its route is too: a route with mode: observe keeps its messages in observe whatever the connection says (see Routing). While SPOT does not answer, the milter keeps the mode of its last answer: to stop enforcing then, restart it (docker compose restart retriever-milter), and it observes until SPOT answers again.

In enforce, choose:

RETRIEVER_MILTER_ACTION_MALICIOUS (default quarantine), RETRIEVER_MILTER_ACTION_SUSPICIOUS (default tag) and RETRIEVER_MILTER_ACTION_SPAM (default tag): accept, tag, quarantine or reject.
RETRIEVER_MILTER_ACTION_UNKNOWN (default accept), for a message SPOT could not decide, or whose workflow did not finish within ANALYSIS_TIMEOUT_MS or its route's time limit. It also takes tempfail, which replies 451 so the sender retries; a message SPOT keeps judging unknown is then deferred until the sender's MTA gives up, typically after 5 days, and bounces.
RETRIEVER_MILTER_ON_TIMEOUT, when SPOT does not answer within ANALYSIS_TIMEOUT_MS + 5 s, and RETRIEVER_MILTER_ON_ERROR, when SPOT is unreachable, refuses the message or answers with an error: accept (default, fail-open) delivers the message untagged; tempfail replies 451 so the sender retries (fail-closed).
default_action in the Postfix spot_milter value, which applies when the milter itself is unreachable or refuses the connection.

Quarantined mail sits in the Postfix hold queue; the queue id is in the retriever log line (queue_id=...):

postqueue -j                          # list; look for "queue_name": "hold"
postcat -q <queue id>                 # inspect
postsuper -H <queue id>               # release from hold ...
postqueue -i <queue id>               # ... and deliver now
postsuper -d <queue id> hold          # delete

The milter's workflow¶

SPOT routes each message: the first route whose match holds chooses its workflow, and a message no route takes runs the default workflow (default_workflow_id). To run the milter's mail through a workflow of its own, with a shorter wait, give it a route on its source. spot-cli routing apply replaces the whole routing, so print it, add the route to the copy, and apply the copy:

spot-cli routing show > routing.yaml
# add under routing: (or replace `routing: []`)
#   - name: milter
#     match: {sources: [retriever-milter]}
#     workflow: fast-workflow
#     timeout_ms: 15000
spot-cli routing apply routing.yaml
rm routing.yaml

The CLI sees the directory you run it from, read-only, so apply reads ./routing.yaml. SPOT refuses a route to a workflow that is not loaded, naming the route, and saves nothing. The route applies from the next message: no restart. Its timeout_ms (1 000 to 300 000) caps the milter's wait: the shorter of it and ANALYSIS_TIMEOUT_MS applies, and a message whose workflow does not finish in time gets X-SPOT-Verdict: unknown and ACTION_UNKNOWN. Admins also edit the routes on the dashboard's Routing page. Routing gives the rules.

6. What the headers mean¶

Every accepted message carries (prefix set by HEADER_PREFIX):

Header	Meaning
`X-SPOT-Verdict`	`malicious`, `suspicious`, `spam`, `benign`, `unknown`, `timeout`, `error` or `skipped`.
`X-SPOT-Score`	SPOT's score, 0 to 100: how likely the message is malicious.
`X-SPOT-Action`	The action the policy chose. In observe mode it is not applied.
`X-SPOT-Mode`	`observe` or `enforce`: the message's mode in SPOT's answer, or with no answer, its last answer's.
`X-SPOT-Job-Id`	The SPOT analysis job id, for tracing in the dashboard.

X-SPOT-Score appears only when SPOT returned a verdict, and X-SPOT-Job-Id when SPOT answered. Rejected and tempfailed messages get no headers. X-SPOT-Threat-Level and X-SPOT-Confidence are gone since milter 2.0.0b1.

The milter removes inbound headers with the prefix first, so rules downstream can trust these. Examples:

Sieve:

require ["fileinto"];
if header :is "X-SPOT-Verdict" ["malicious", "spam"] {
    fileinto "Junk";
    stop;
}

Procmail: :0H * ^X-SPOT-Verdict: (malicious|spam) -> Spam folder.

7. Security notes¶

The milter protocol has no authentication or encryption. Bind port 9900 to localhost or a private address (RETRIEVER_MILTER_BIND), set RETRIEVER_MILTER_ALLOWED_CLIENTS to the Postfix hosts only, and use WireGuard or stunnel across untrusted networks. Never publish the port to the internet.
Source address behind the port mapping. With Postfix on the Docker host and the default 127.0.0.1:9900 binding, the milter sees the connection coming from the gateway of its network, spot-plugins (for example 172.18.0.1), not 127.0.0.1. Narrowing ALLOWED_CLIENTS to 127.0.0.1/32 would refuse every connection: allow the gateway of spot-plugins instead. A remote Postfix connecting to a private bind address keeps its real IP. The milter logs Refused milter connection from <address>: not in ALLOWED_CLIENTS with the exact address to allow.
Narrow ALLOWED_CLIENTS. The milter shares spot-plugins with every installed plugin, and the default list admits the private ranges, that network included: any plugin could open a milter session. When Postfix runs on the Docker host, allow only the gateway of spot-plugins, as a /32:

docker network inspect spot-plugins --format '{{range .IPAM.Config}}{{.Gateway}}{{end}}'
# .env: RETRIEVER_MILTER_ALLOWED_CLIENTS=<that address>/32

* MAX_CONNECTIONS (default 100) must be at least the number of smtpd processes Postfix may run: default_process_limit, or the smtpd maxproc in master.cf if set. A connection beyond the limit is refused and Postfix applies default_action to that session. * Inbound headers carrying the X-SPOT- prefix are always stripped, so a sender cannot forge a verdict. Inbound Authentication-Results are not: the MTA removes those that carry its authserv-id (see section 2). * The milter never gets the internal API key (SPOT_INTERNAL_API_KEY). Its own token has the ingest scope only: it submits mail and reads its own jobs. On spot-plugins, the only core services it reaches are the api-gateway, the knowledge service and the analyzer-orchestrator. Rotate the token with spot-cli plugin token retriever-milter, set the new RETRIEVER_MILTER_TOKEN and recreate the milter.

8. Troubleshooting¶

Every message is delayed by the analysis timeout: SPOT is slow or unreachable. Check docker compose logs mail-orchestrator, and that the workflow completes inside RETRIEVER_MILTER_ANALYSIS_TIMEOUT_MS.
Postfix logs milter ... can't connect or sessions skip the milter: Port 9900 is bound to 127.0.0.1 while Postfix runs on another host, or the Postfix host is outside RETRIEVER_MILTER_ALLOWED_CLIENTS. Fix the bind address or the allow-list.
Milter log shows Refused milter connection from <address>: not in ALLOWED_CLIENTS: Add that address's network to RETRIEVER_MILTER_ALLOWED_CLIENTS. For Postfix on the Docker host the address is the bridge gateway, not 127.0.0.1; see the security notes.
X-SPOT-Verdict: error on every message: The milter cannot reach the api-gateway or is rejected. Check docker compose logs retriever-milter. A milter older than 2.0.0b4 always ends here: it names its workflow, which SPOT refuses. Check RETRIEVER_MILTER_VERSION in .env.
X-SPOT-Mode: observe while the milter's connection is in enforce: The message's route is in observe (mode: observe, see spot-cli routing show), or SPOT has not answered the milter since it started: it observes until the first answer.
X-SPOT-Verdict: unknown on every message: No analyzer of the workflow answers, or the workflow does not finish within RETRIEVER_MILTER_ANALYSIS_TIMEOUT_MS or its route's timeout_ms. The analyzer-orchestrator logs why for each analyzer: docker compose logs analyzer-orchestrator | grep 'in stage' (no URL configured, no plugin token, a timeout).
The milter stops at start, and Postfix applies default_action to all mail: A setting is out of its range or not one of its values. The milter's log says which: docker compose logs retriever-milter.
401 Unauthorized in api-gateway logs: The milter's token is missing, revoked or rotated. Issue it again with spot-cli plugin token retriever-milter, set RETRIEVER_MILTER_TOKEN in .env and run docker compose up -d --force-recreate retriever-milter.
docker compose up stops with environment variable "RETRIEVER_MILTER_TOKEN" required by secret ... is not set: The mail-milter profile is on without a token. Issue one and set it as in section 1.