Services¶

Detailed description of SPOT Platform services. Each service's HTTP routes are in its API document.

Core Services¶

API Gateway¶

FastAPI-based HTTP server: the only service users, the dashboard, the CLI and plugins call.

Port: 8001

Responsibilities:

• Authentication and authorization: users with JWT and roles, plugins with their own scoped tokens
• Intake: takes raw messages (POST /api/v1/analyze, POST /internal/ingest), parses each once with spot_sdk.mime in worker processes bounded in time and memory, and stores the raw message and its view, encrypted
• Routing: chooses each stored email's workflow by the routes of spot.yaml, and its mode (observe or enforce) from its mail connection and its route
• Job submission to RabbitMQ (spot.analysis exchange); a retriever's through the Mail Orchestrator
• Analysis status, results and statistics, read from PostgreSQL and Redis
• Content: an email's view and raw message for analysts and admins; the raw message, a decoded part and a job's result for plugins
• Configuration API: reads and writes spot.yaml (plugins, workflows, routing, platform settings). The gateway is the only writer of spot.yaml: it validates each workflow written through the API and saves it there, and records every change as a commit of history.git, beside it, which admins read and roll back
• Plugin catalog, sources and installer (through the Plugin Manager), plugin tokens, egress consent
• Knowledge Store: the scheduler that calls context providers' /internal/sync, and the admin proxies to the knowledge service
• Audit log: records sign-ins, plugin lifecycle actions, user accounts, plugin sources, deletions of stored data and replays, and serves it to admins (GET /api/v1/audit)
• Retention: purges expired emails and audit entries at start and then daily
• Health checks

The gateway authenticates users with JWT and plugins with their token. It does not accept the internal API key.

Technology:

• FastAPI, Pydantic
• PostgreSQL for persistence
• Redis for rate limiting, analysis job records and the plugin catalog
• RabbitMQ for the job queue

Analyzer Orchestrator¶

Workflow execution engine that coordinates the analyzers and the responders.

Responsibilities:

• Consumes analysis jobs (orchestrator.analysis) and loads each email's stored view
• DAG-based workflow execution: stages in the order of their depends_on, conditions on earlier stages' score or verdict, the whole run bounded by the workflow's timeout_ms
• Calls each analyzer's POST /internal/analyze with the analyzer's own token; retries and failure strategies; a failure is unknown
• Decides the email's verdict with the workflow's verdict_policy
• Stores the result (PostgreSQL), updates the job record (Redis) and publishes the verdict to spot.results
• Dispatches each verdict to the responders the workflow's on_verdict names (POST /internal/respond), from the queue orchestrator.dispatch, with the job's mode, and records each responder's action
• Runs the shadow workflows on each production result, one at a time, from the queue orchestrator.shadow, and stores where their verdict differs from production's (Shadow workflows)
• Runs the platform's benchmark runs (backtests) one email at a time, from the queue orchestrator.benchmark, and records each email's result in its run (Benchmark runs)
• Reads the workflows, the default workflow and the plugins' URLs from spot.yaml (mounted read-only) in one read, again before a job, a dispatch or an API read whenever the file changed (its inode, modification time or size), and only then: a change saved through the gateway, or a manual edit, runs from the next job, without a restart. A manual edit is recorded in spot.yaml's history at the gateway's next reload or start
• Once it has read them, it keeps the workflows, the default and the plugins' URLs read last, said once in the log for each state of the file and parsed once, when a re-read finds spot.yaml missing, unreadable or invalid, without workflows (empty, or cut before them), with no workflow that loads, without a default_workflow_id key of its own (null, no default, is one), or not ending with a newline. SPOT always writes that key, last, folds no value onto a second line, and ends the file with a newline, so a file SPOT wrote, or a copy of one in SPOT's form, is never read cut short, even while docker compose cp removes it and writes it in pieces. A hand edit refused for a missing final newline or a missing default_workflow_id applies at the gateway's next reload or start, which writes the file in SPOT's form. Removing the workflows section by hand applies at the gateway's next reload, which writes the shipped workflows back into the file. At the start, a missing spot.yaml, or one without workflows, gives the shipped workflows
• Serves the workflow reads the API Gateway proxies (the list, a workflow, the readiness), and the workflow dry run; it writes no workflow

Technology:

• Python asyncio, FastAPI
• RabbitMQ for messaging
• spot-sdk for the contract models
• PostgreSQL for result storage, Redis for job records

Mail Orchestrator¶

Email ingestion routing layer.

Responsibilities:

• Receives, from the API Gateway and with the internal key, the id of each email a retriever ingested, once the gateway has parsed and stored it
• Publishes it to RabbitMQ (spot.analysis exchange, email.analyze routing key) with a job ID
• Synchronous ingest (wait_for_verdict): waits up to timeout_ms for the analysis job record in Redis, then returns analyzed, with the verdict and the score, or timeout. On timeout the analysis still completes and is stored
• A message that cannot be published, or whose analysis fails, is an HTTP error (503, 502), recorded as rejected with the reason
• Records each ingest in Redis for the dashboard's recent ingestion panel

Configuration:

• REQUIRE_BROKER - RabbitMQ is required by default (true); startup fails without it and /health reports degraded while it is down
• Retrievers are configured in spot.yaml under plugins.mail_retrievers

Technology:

• Python asyncio
• RabbitMQ for publishing, Redis for job records

Knowledge Service¶

SPOT's Knowledge Store: context providers write tagged documents, analyzers query them by tag and semantic similarity. See the Knowledge Store guide.

Responsibilities:

• Embeds documents and queries (Ollama, bge-m3 by default)
• Stores documents in PostgreSQL with pgvector (knowledge_documents)
• Holds each provider to its own documents (source = its plugin id)

Technology:

• FastAPI, asyncpg, pgvector
• Redis for embedding and query caches

Plugin Manager¶

Internal service that runs the plugin containers (install, update, remove). The API Gateway calls it over HTTP with the internal API key (core to core).

Who touches Docker:

• The plugin-manager is the only service with Docker access. It reaches the daemon through plugin-docker-proxy, a socket proxy that allows only the calls it needs (inspect, create, start, stop, restart, remove, image pull, copy files in) and refuses the rest, such as listing containers or bind-mounting host directories. The proxy runs unprivileged: with a read-only file system, no capabilities and no-new-privileges, it needs only the group that owns the Docker socket (set DOCKER_GID, stat -c %g /var/run/docker.sock, in .env). It exits at start when it cannot read the socket, and stops when it loses the socket later (checked every 60 s), so compose restarts it.
• The API Gateway no longer mounts the Docker socket.
• The plugin-manager never uses the egress proxy. The Docker CLI injects the proxies of ~/.docker/config.json into every container, and the Docker SDK would send the calls to plugin-docker-proxy through them, so its compose environment sets HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and their lowercase forms to "". A deployment compose file must carry the same lines. Nothing is lost: the Docker daemon pulls the images with its own proxy settings, and a plugin gets no proxy variable and no proxy credentials file: the egress gateway goes through the operator's proxy for it.
• Plugin containers run with all capabilities dropped (cap_drop: ALL), no-new-privileges and a limit of 512 processes. Their logs rotate: at most five files of 10 MB each. They are limited to 1 GiB of memory and 1 CPU, unless the plugin image asks for more with its spot.plugin.resources label or an admin sets resources: on the plugin entry in spot.yaml.
• A plugin's credentials are files, never environment variables. The API Gateway unseals the plugin's secrets, kept encrypted under secrets: in spot.yaml, into the spec it sends; the plugin-manager writes them, with the plugin's token, as /run/secrets/<NAME> before the container starts.
• The plugin-manager's image also runs the egress gateway (egress-gateway), which forwards a plugin of every egress class to the hosts its settings of type url name, and nowhere else. It has no Docker access and no SPOT secret, and reads spot.yaml read only. See Reaching configured hosts.

Plugins¶

Plugins are external HTTP services built on spot-sdk 2 (plugin contract 2), in repositories of their own, each of one kind. The SDK's create_plugin_app serves the routes core calls, as the plugin documents describe, one per kind:

• GET /health and GET /manifest, open
• POST /internal/analyze (analyzers), POST /internal/sync and GET /internal/state (context providers), POST /internal/respond (responders), each checking the plugin's own token

Each kind has its section under plugins: in config/spot.yaml:

plugins:
  analyzers:
    analyzer-nlp:
      enabled: true
      url: "http://spot-analyzer-nlp:8000"
      settings: {}

The official plugins, and the signal names the SDK publishes for them (spot_sdk.signals):

Infrastructure Services¶

PostgreSQL¶

Primary database for persistent storage.

Port: 5432 (internal network only, not published on the host)

Tables (see Database):

• emails - Email records, their view encrypted
• email_raw - Raw messages, encrypted
• analysis_results - Verdicts, scores and results
• analysis_signals - The signals behind each verdict
• knowledge_documents - The Knowledge Store
• users, plugin_tokens, oauth_clients
• audit_logs - The audit log, written by the API Gateway

Technology:

• PostgreSQL 15+ with pgvector

Redis¶

In-memory store.

Port: 6379 (internal network only, not published on the host)

Usage:

• Rate limiting counters
• Analysis job records (spot:job:*, 24 hours) and ingest job records
• The plugin catalog and what each source's last scan found (spot:plugin-*)
• Knowledge scheduler state, embedding and query caches

Technology:

• Redis 7+
• redis-py client

RabbitMQ¶

Message broker for asynchronous job processing. Messages carry ids and verdicts, never an email's content.

Ports (internal network only, not published on the host):

• 5672 - AMQP protocol
• 15672 - Management UI, reached by the api-gateway over the internal network

Exchanges:

• spot.analysis (TOPIC) - Analysis jobs
• spot.results (TOPIC) - Verdicts of finished jobs
• spot.config (TOPIC) - Configuration reloads

Queues:

• orchestrator.analysis - Analysis jobs (bound to spot.analysis, email.analyze)
• orchestrator.analysis.sync - The ingests of retrievers that wait for their verdict (bound to spot.analysis, email.analyze.sync); one that expires or keeps failing moves to orchestrator.analysis
• orchestrator.dispatch - Verdicts to dispatch to responders (bound to spot.results, analysis.complete)
• orchestrator.shadow - Production's results for the shadow workflows (bound to spot.results, analysis.complete), at most 1 000, dropping the oldest
• orchestrator.benchmark - The emails of a platform benchmark run, one message each (bound to spot.analysis, email.benchmark)
• orchestrator.analysis.dead-letter, orchestrator.dispatch.dead-letter, orchestrator.shadow.dead-letter, orchestrator.benchmark.dead-letter - The messages each queue parked, or the shadow dropped

Each consumed queue is a quorum queue with a delivery limit of 10: a message that keeps failing is parked in its dead-letter queue. The sync queue has none of its own: its failures, expired or refused messages included, move to orchestrator.analysis and are handled there. The api-gateway, the mail-orchestrator and the analyzer-orchestrator declare the queues once connected; the first to start empties the queues of earlier releases into them and deletes them, but keeps one still consumed by the analyzer-orchestrator of the earlier release until a later start.

Technology:

• RabbitMQ 3.13 (quorum queues, x-delivery-limit)
• aio-pika (Python async client)
• Quorum queues with dead-letter queues

Development Services¶

Adminer¶

Database management UI (dev mode only).

Port: 8080

MailHog¶

Email testing server (dev mode only).

Port: 8025

Devtools¶

Development utilities container (dev mode only).

Service Communication¶

API Gateway -> Analyzer Orchestrator¶

Communication: RabbitMQ

Flow:

1. API Gateway parses and stores the message
2. API Gateway publishes the email's id to spot.analysis with the email.analyze routing key
3. Analyzer Orchestrator consumes from orchestrator.analysis
4. Orchestrator executes the workflow on the stored view
5. Results stored in PostgreSQL, the verdict published to spot.results

Analyzer Orchestrator -> Analyzers¶

Communication: HTTP

Flow:

1. Orchestrator sends POST /internal/analyze with the analyzer's token and an AnalyzeRequest (the email's view, the earlier stages' outcomes)
2. Analyzer answers an AnalyzerVerdict; it may fetch the raw message or a part from the API Gateway, and query the Knowledge Store
3. The workflow's verdict policy decides the email's verdict

Analyzer Orchestrator -> Responders¶

Communication: HTTP

The dispatcher calls POST /internal/respond on each responder the workflow's on_verdict names, with the responder's token.

Status Queries¶

Communication: PostgreSQL, Redis

The API Gateway reads a finished job from PostgreSQL, and a job not stored yet from its Redis record.

Who can call what¶

The internal key is core to core only: the services that accept it treat whoever sends it as core. A plugin never receives it: it holds its own token, the file /run/secrets/spot_plugin_token, sent as Authorization: Bearer, and core calls the plugin with the same token. A plugin you run yourself gets a token too, never the key. See Context Providers.

Scaling Considerations¶

Stateless Services¶

Can be scaled horizontally:

• API Gateway
• Plugins

Stateful Services¶

Require coordination for scaling:

• Analyzer Orchestrator (single instance recommended)
• Mail Orchestrator (single instance per configuration)

Infrastructure¶

Standard scaling approaches:

• PostgreSQL: Read replicas, connection pooling
• Redis: Redis Cluster for high availability
• RabbitMQ: Clustered deployment

Health Checks¶

All services provide health endpoints:

# API Gateway
curl http://localhost:8001/health

# PostgreSQL
docker compose exec postgres pg_isready

# Redis
docker compose exec redis redis-cli ping

# RabbitMQ
docker compose exec rabbitmq rabbitmq-diagnostics -q ping

Related Documentation¶

• System Overview - High-level architecture
• Data Flow - Request processing
• Admin Guide - Deployment