Troubleshooting¶

Analyses End unknown¶

An analyzer that does not answer counts as unknown, and the analyzer-orchestrator logs why:

docker compose logs analyzer-orchestrator | grep 'in stage'

"Analyzer analyzer-nlp in stage parallel-analysis: no URL configured"¶

Cause: The workflow names an analyzer that is not installed, or whose entry in plugins.analyzers is disabled.

Fix:

1. Install the analyzer plugins your workflows name from the dashboard catalog (see Installing plugins). The default workflow runs analyzer-nlp and analyzer-llm.
2. The orchestrator re-reads spot.yaml per job, so no restart is required after install.

"Analyzer ... no plugin token"¶

Cause: The analyzer's entry was added by hand and has no token: core calls every analyzer with its own token.

Fix: Issue it with spot-cli plugin token <id> --kind analyzer and mount it in the analyzer's container as /run/secrets/spot_plugin_token (see Plugins you run yourself).

Analysis Jobs Show "Failed" Status¶

A job fails when an analyzer whose failure_strategy is fail does not answer, or when its workflow cannot run. Its verdict is unknown, decided by failed.

1. Check the job detail page for the specific error
2. Common causes:
3. An analyzer is not installed or is down (see above)
4. The workflow is left out at start (see below)
5. Network connectivity issues between SPOT and analyzers
6. Check analyzer-orchestrator logs:

   docker compose logs analyzer-orchestrator | grep -i error
   

"Workflow ... is left out"¶

Cause: A workflow stored in spot.yaml cannot run, for example after its conversion from plugin contract 1; the log line says why. Analyses that name it fail, and so do those that name none when it is the default workflow.

Fix: Correct it in spot.yaml, edited as in Editing spot.yaml by hand (spot-cli config validate spot.yaml checks the copy first, run from the directory that holds it: the CLI sees that directory, read-only), then docker compose restart api-gateway analyzer-orchestrator.

Knowledge Store / RAG layer¶

Dashboard shows "Knowledge Store is not fully operational"¶

The /knowledge page lists which component is missing. Match the failing row below.

Embedding backend unreachable¶

Symptom. The banner shows a red cross next to Embedding backend, typically with http://ollama:11434. A sync attempt surfaces:

Sync failed: Server error '502 Bad Gateway' for url '.../internal/sync'

and docker logs spot-knowledge shows httpx2.ConnectError: No address associated with hostname.

Cause. OLLAMA_URL points at a hostname the knowledge container can't resolve, or the Ollama instance isn't running.

Fix. Two options ; pick one:

• Enable the bundled side-car: add ollama to the profiles COMPOSE_PROFILES lists in /opt/spot/deploy/.env, such as COMPOSE_PROFILES=cli,ollama, then:

  docker compose pull
  docker compose up -d
  docker compose logs -f ollama-init  # watch the model pull
  

• Or point at an existing Ollama:

  echo "OLLAMA_URL=http://my-ollama-host:11434" >> /opt/spot/deploy/.env
  docker compose up -d knowledge
  

  Make sure the model is pulled on that host: ollama pull bge-m3.

Embedding model missing¶

The server is up but the model isn't pulled. Ollama answers /api/version but /api/embeddings returns 404 for the model. Fix:

docker exec spot-ollama ollama pull bge-m3
# or, if you pinned a different EMBEDDING_MODEL:
docker exec spot-ollama ollama pull "$EMBEDDING_MODEL"

If the pull itself fails with dial tcp ...:443: i/o timeout or a DNS error, the container has no path to registry.ollama.ai. Two causes: egress blocked, or the host is behind a proxy that isn't being forwarded into the container.

See Running behind an egress proxy for the full list of what noProxy must contain.

Docker does not propagate the host's HTTP_PROXY / HTTPS_PROXY into containers automatically. Configure the Docker client instead so every container on the host gets the proxy env auto-injected at creation time ; no per-service compose edits. Put this in ~/.docker/config.json (for the user that runs docker) or /root/.docker/config.json (if you run sudo docker):

{
  "proxies": {
    "default": {
      "httpProxy":  "http://proxy.example.com:3128",
      "httpsProxy": "http://proxy.example.com:3128",
      "noProxy":    "localhost,127.0.0.1,0.0.0.0,api-gateway,ollama"
    }
  }
}

Then recreate the side-car so it gets the fresh env (ollama-gpu in place of ollama with the gpu profile):

docker compose up -d --force-recreate ollama ollama-init
docker compose logs -f ollama-init   # watch the model pull succeed

Verify the vars reached the container (no -e needed ; Docker injects them from config.json automatically):

docker run --rm alpine env | grep -i proxy

One gotcha worth calling out: the ollama/ollama image sets OLLAMA_HOST=0.0.0.0 so the server listens on all interfaces. The CLI in the same container reads the same env and preflights its own server at http://0.0.0.0:11434/. If 0.0.0.0 is missing from noProxy, Go routes that loopback call through the proxy, Squid returns TCP_DENIED/403 (it has no business connecting to 0.0.0.0), and the CLI aborts with the unhelpful Error: something went wrong, please see the ollama server logs for details ; before the pull ever reaches the server. Always include 0.0.0.0,localhost,127.0.0.1 in noProxy, as the list above does.

The Docker daemon also needs proxy access to pull the ollama/ollama image itself. That's configured separately on the host ; typically /etc/systemd/system/docker.service.d/http-proxy.conf. If the daemon can pull SPOT's own images, this step is already done.

No context providers enabled¶

The RAG layer has nothing feeding it. Install a context_provider plugin from the Plugins page (e.g. provider-employee-dir) and run a sync. The banner turns green once at least one is enabled.

/readiness for scripts¶

GET /api/v1/knowledge/readiness on the api-gateway (any signed-in user) returns the same data the banner uses. Useful for monitoring:

curl -sS -H "Authorization: Bearer $TOKEN" \
    https://api.$DOMAIN/api/v1/knowledge/readiness

{
  "ready": false,
  "components": {
    "knowledge_service": true,
    "embedding_backend": false,
    "context_providers": false
  },
  "embedding": {
    "url": "http://ollama:11434",
    "model": "bge-m3",
    "reachable": false,
    "service_reachable": true
  },
  "context_providers": {"installed": 0, "enabled": []}
}

Services Not Starting¶

Check container status:

docker compose ps -a

Check startup logs:

docker compose logs --tail=50

A core service that stops at once with an error naming LOG_FORMAT has a value other than json or text, such as simple or detailed from an older .env: set LOG_FORMAT=json, or remove the line.

Reading the Logs¶

The core services write one JSON object per line (Logs). Filter them with jq, which skips the lines that are not JSON with fromjson?:

# the errors of one service
docker compose logs --no-log-prefix api-gateway | jq -cR 'fromjson? | select(.level == "ERROR")'
# the requests one service answered, their messages only
docker compose logs --no-log-prefix api-gateway | jq -rR 'fromjson? | select(.logger == "uvicorn.access") | .message'

To follow one request across the services, take the X-Correlation-ID of its answer, then look for it in every service's lines:

docker compose logs --no-log-prefix | grep <id>

Database Connection Failed¶

Verify postgres is healthy:

docker exec spot-postgres pg_isready -U spot

Check postgres logs:

docker compose logs postgres

Submissions Refused¶

POST /api/v1/analyze and the mail retrievers' ingest answer:

• 413: the message is larger than 25 MiB.
• 422 parse_limit: parsing the message took longer than intake.parse_timeout_s or more memory than intake.parse_memory_mib, or its view is larger than intake.max_view_mib; the answer names the limit.
• 503: the api-gateway already takes intake.max_in_flight messages at once, or no parse worker is free; retry later.

See Intake and api-gateway memory.

API Gateway Returning 500¶

A 500 answers with an X-Correlation-ID: its lines, in the api-gateway and in every service the request reached, carry it, the error's type and frames included:

docker compose logs --no-log-prefix | grep <id>

Check that the connection settings are set, without printing their values, which hold passwords:

docker exec spot-api-gateway env | grep -E "^(DATABASE|REDIS|RABBITMQ)" | cut -d= -f1

Port Already in Use¶

The stack publishes Traefik's ports 80 and 443 on every address and, with the mail-milter profile, the milter's port 9900 on RETRIEVER_MILTER_BIND (127.0.0.1 by default). No other service publishes a port.

Find the process using the port:

sudo lsof -i :443

Stop it, or move it: Traefik needs 80 for Let's Encrypt's HTTP challenge and the redirect to HTTPS, and 443 to serve the dashboard and the API. For 9900, set RETRIEVER_MILTER_BIND in .env to an address where the port is free.

Out of Memory¶

Every service has a memory limit (see Memory limits). A container that passes it is killed, and Docker restarts it, which clears its OOMKilled flag. The first sign is its restart count. The kernel log keeps each kill: its time and the process killed, and, on the line just before, the container's id in oom_memcg=, which docker inspect -f '{{.Name}}' <id> names. docker stats shows how close each runs to its limit:

docker inspect -f '{{.Name}} restarts={{.RestartCount}} oomkilled={{.State.OOMKilled}}' $(docker compose ps -aq)
sudo journalctl -k --since -24h | grep -i -B1 'memory cgroup out of memory'
docker stats --no-stream

Within minutes of a kill, docker events --since 10m --until "$(date +%s)" --filter event=oom --format '{{.Actor.Attributes.name}}' also names the container; later it does not, as Docker keeps only its last 256 events and loses them when it restarts.

Raise the limit of a container killed for lack of memory in docker-compose.override.yml, as Memory limits shows, or OLLAMA_MEMORY_LIMIT for Ollama.

The api-gateway is limited to 3328M, what the default intake settings need. If it was killed for lack of memory after you raised them (its restart count grows, and the kernel log names its container), raise its limit too (see Intake and api-gateway memory).

Reduce Redis memory in .env:

REDIS_MAXMEMORY=128mb

Dashboard Not Loading¶

Check dashboard logs:

docker compose logs web-dashboard

Verify API Gateway is healthy:

docker compose exec api-gateway wget -qO- http://localhost:8000/health

RabbitMQ Connection Issues¶

Check RabbitMQ logs:

docker compose logs rabbitmq

RabbitMQ is on an internal network only and publishes no port. Inspect it from its container:

docker compose exec rabbitmq rabbitmq-diagnostics status
docker compose exec rabbitmq rabbitmqctl list_queues

Or through the api-gateway, with any dashboard account: GET /api/v1/observability/queues.

Messages in a Dead-Letter Queue¶

The dashboard's Queues page counts messages in orchestrator.analysis.dead-letter or orchestrator.dispatch.dead-letter (the shadow's and the benchmark's are below). A message whose handling keeps failing is retried with a growing wait. After 11 deliveries, about 13.5 minutes, the bulk queue (orchestrator.analysis) and the dispatch queue park it in their own dead-letter queue; the milter's queue (orchestrator.analysis.sync) moves its failures into the bulk queue, which parks them in turn. One that is not JSON or names no job is parked at once. A parked analysis did not run, and a parked result reached no responder. It stays parked until it is replayed or purged.

orchestrator.shadow.dead-letter holds the shadow's work that was put aside: the results it could not handle, and, while the shadow workflows cannot keep up, the oldest results past the 1 000 its queue holds (it holds at most 1 000 too, dropping its own oldest). Production lost nothing: those emails have their verdicts. Replay them (spot-cli queues replay orchestrator.shadow) when their differences matter, else purge them.

orchestrator.benchmark.dead-letter holds the messages of a platform benchmark run (a backtest) that could not be handled: a message that names no run and email, parked at once, or one whose run or result could not be read or recorded because the database stayed unreachable past the redelivery window. The run then stays running, and no other backtest starts while it does (409). Cancel it (final: its metrics are counted over what it recorded), or replay its messages once the database answers (spot-cli queues replay orchestrator.benchmark --limit 1 first). A replayed message goes through the runner again:

• of a run that has ended, been cancelled, failed or been deleted, it is acknowledged and does nothing, so purging it is the same;
• of a run still running, it records the email's result (a result recorded twice is kept once) and the last one finishes the run, which is how a stuck run completes;
• one that names no run and email is parked again at once.

The queue is consumed one message at a time, so a replayed message that fails again holds the whole benchmark queue through its retries, about 13.5 minutes, before it is parked again.

A backtest whose progress stays frozen while nothing is parked lost the rest of its messages before they were queued: the api-gateway was stopped outright while it queued them, or could not reach the database to end the run. A start that cannot queue its emails, or whose request is cancelled meanwhile, ends its run failed. Cancel the frozen run: its metrics are counted over what it recorded, and the next backtest can start.

The counts lag: a quorum queue's count in rabbitmqctl list_queues, and on the Queues page, which reads RabbitMQ's statistics every 5 seconds, shows a message parked, replayed or purged a few seconds late. Read them again a few seconds after acting before concluding that nothing moved.

Find the cause. The analyzer-orchestrator logs each failure with its job id and the error's type, never the message's content:

docker compose logs analyzer-orchestrator | grep -E 'failed \(delivery|Parked a message|Moved a message'

Look at them, leaving them parked (rabbitmqadmin signs in with the user RabbitMQ was created with):

docker compose exec rabbitmq sh -c 'rabbitmqadmin -u "$RABBITMQ_DEFAULT_USER" \
  -p "$RABBITMQ_DEFAULT_PASS" get queue=orchestrator.analysis.dead-letter \
  count=10 ackmode=reject_requeue_true'

Replay them once the cause is fixed, one first, then the rest:

1. Replay one message: "Replay one" on the Queues page (admins), or spot-cli queues replay orchestrator.analysis --limit 1.
2. Check that it is handled: it does not come back to the dead-letter queue, and its job completes.
3. Replay the rest: "Replay all", or spot-cli queues replay orchestrator.analysis.

Each replayed message gets its whole retry budget again. Replay one first: a message that fails again holds one of the queue's slots (10 for the bulk queue by default) through its retries, about 13.5 minutes, so many of them at once hold up new analyses. Use orchestrator.dispatch for the parked results. Without the dashboard or the CLI, a one-off shovel moves what the dead-letter queue holds when it starts, then removes itself:

docker compose exec rabbitmq rabbitmq-plugins enable rabbitmq_shovel
docker compose exec rabbitmq rabbitmqctl set_parameter shovel replay-analysis \
  '{"src-protocol": "amqp091", "src-uri": "amqp://", "src-queue": "orchestrator.analysis.dead-letter", "src-delete-after": "queue-length", "dest-protocol": "amqp091", "dest-uri": "amqp://", "dest-queue": "orchestrator.analysis"}'

Purge them when they are not worth running again:

docker compose exec rabbitmq rabbitmqctl purge_queue orchestrator.analysis.dead-letter

"ENCRYPTION_KEY is required"¶

The api-gateway, the analyzer-orchestrator and the migration job refuse to start when ENCRYPTION_KEY is unset - email persistence is mandatory and silent fallback is not allowed. Migration 010 stops with ENCRYPTION_KEY must be set to convert the stored contract 1 analyses when analyses exist and the key is missing.

Fix:

1. On a new install, generate an encryption key:

   python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
   

2. Set it in .env:

   ENCRYPTION_KEY=your_generated_key_here
   

3. Recreate the affected services (restart is not enough):

   docker compose up -d --force-recreate api-gateway analyzer-orchestrator
   

4. Verify the services are healthy:

   docker compose ps api-gateway analyzer-orchestrator
   

"Fernet key must be 32 url-safe base64-encoded bytes"¶

This error appears in analyzer-orchestrator logs when ENCRYPTION_KEY is invalid.

Symptoms: - analyzer-orchestrator shows as "unhealthy" - Dashboard is very slow (10+ second timeouts) - "Failed to load dashboard data" error - API returns 503 for /api/v1/workflows

Cause: The ENCRYPTION_KEY in .env is malformed (wrong length, missing padding, or invalid characters).

Fix:

A key that was valid and got damaged (a character lost in an edit) must be restored as it was, from your backup's env file: a new key cannot read the data the old one encrypted. On a new install only:

1. Generate a valid Fernet key (must be exactly 44 characters, ending with =):

   python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
   

2. Update .env with the new key:

   ENCRYPTION_KEY=your_new_44_character_key_here=
   

3. Recreate the analyzer-orchestrator:

   docker compose up -d analyzer-orchestrator --force-recreate
   

4. Verify it's healthy:

   docker compose ps analyzer-orchestrator
   # Should show: Up ... (healthy)
   

"Message broker disconnected"¶

This appears when services lose their RabbitMQ connection (e.g., after RabbitMQ restarts).

Fix:

Restart the affected services:

docker compose restart api-gateway analyzer-orchestrator mail-orchestrator

Or restart all services:

docker compose restart

Verify connections are restored:

docker exec spot-api-gateway wget -qO- http://localhost:8000/health | grep message_broker
# Should show: "connected": true

Reset Everything¶

This deletes every email, analysis, user and setting. Take a backup first (scripts/spot-cli backup) if anything in it may be wanted later.

Stop and remove all containers and volumes:

docker compose down -v

Start fresh:

docker compose up -d

WARNING: This deletes all data including the database.

Get Help¶

• Check logs: docker compose logs -f
• Documentation: https://spot-project.codeberg.page/documentation/
• Issues: https://codeberg.org/SPOT_Project/spot/issues