Skip to content

Troubleshooting

Analysis Jobs Failing

Most Common Issue: No Analyzers Configured

If all analysis jobs fail immediately, you likely haven't deployed any analyzer services.

"No URL configured for analyzer: analyzer-nlp"

Cause: SPOT requires external analyzer services, which must be deployed separately.

Fix:

  1. Install at least one analyzer plugin from the dashboard catalog (see Installing plugins).
  2. The orchestrator re-reads core/config/spot.yaml per job, so no restart is required after install.

"Only 0 analyzers succeeded, minimum required: N"

Cause: The workflow requires one or multiple analyzer(s) but none are installed or all are disabled.

Fix: Same as above ; install the required analyzer plugin(s), or check that existing entries in plugins.analyzers have enabled: true and a reachable url.

Analysis Jobs Show "Failed" Status

Since v1.0.0, failed jobs now appear in the dashboard with their error messages.

  1. Check the job detail page for the specific error
  2. Common causes:
  3. No analyzers configured (see above)
  4. Analyzer service is down
  5. Network connectivity issues between SPOT and analyzers
  6. Check analyzer-orchestrator logs: 
    docker compose logs analyzer-orchestrator | grep -i error

Knowledge Store / RAG layer

Dashboard shows "Knowledge Store is not fully operational"

The /knowledge page lists which component is missing. Match the failing row below.

Embedding backend unreachable

Symptom. The banner shows a red cross next to Embedding backend, typically with http://ollama:11434. A sync attempt surfaces:

Sync failed: Server error '502 Bad Gateway' for url '.../internal/sync'

and docker logs spot-knowledge shows httpx.ConnectError: No address associated with hostname.

Cause. OLLAMA_URL points at a hostname the knowledge container can't resolve, or the Ollama instance isn't running.

Fix. Two options ; pick one:

  • Enable the bundled side-car in /opt/spot/.env: 
    echo "COMPOSE_PROFILES=ollama" >> /opt/spot/.env
docker compose pull
docker compose up -d
docker compose logs -f ollama-init  # watch the model pull
  • Or point at an existing Ollama: 
    echo "OLLAMA_URL=http://my-ollama-host:11434" >> /opt/spot/.env
docker compose up -d knowledge
    Make sure the model is pulled on that host: ollama pull bge-m3.

Embedding model missing

The server is up but the model isn't pulled. Ollama answers /api/version but /api/embeddings returns 404 for the model. Fix: 

docker exec spot-ollama ollama pull bge-m3
# or, if you pinned a different EMBEDDING_MODEL:
docker exec spot-ollama ollama pull "$EMBEDDING_MODEL"

If the pull itself fails with dial tcp ...:443: i/o timeout or a DNS error, the container has no path to registry.ollama.ai. Two causes: egress blocked, or the host is behind a proxy that isn't being forwarded into the container.

Docker does not propagate the host's HTTP_PROXY / HTTPS_PROXY into containers automatically. Configure the Docker client instead so every container on the host gets the proxy env auto-injected at creation time ; no per-service compose edits. Put this in ~/.docker/config.json (for the user that runs docker) or /root/.docker/config.json (if you run sudo docker):

{
  "proxies": {
    "default": {
      "httpProxy":  "http://proxy.example.com:3128",
      "httpsProxy": "http://proxy.example.com:3128",
      "noProxy":    "localhost,127.0.0.1,0.0.0.0,postgres,redis,rabbitmq,ollama,knowledge,api-gateway,analyzer-orchestrator,mail-orchestrator,web-dashboard,traefik"
    }
  }
}

Then recreate the side-car so it gets the fresh env: 

docker compose --profile ollama up -d --force-recreate ollama ollama-init
docker compose logs -f ollama-init   # watch the model pull succeed

Verify the vars reached the container (no -e needed ; Docker injects them from config.json automatically): 

docker run --rm alpine env | grep -i proxy

One gotcha worth calling out: the ollama/ollama image sets OLLAMA_HOST=0.0.0.0 so the server listens on all interfaces. The CLI in the same container reads the same env and preflights its own server at http://0.0.0.0:11434/. If 0.0.0.0 is missing from noProxy, Go routes that loopback call through the proxy, Squid returns TCP_DENIED/403 (it has no business connecting to 0.0.0.0), and the CLI aborts with the unhelpful Error: something went wrong, please see the ollama server logs for details ; before the pull ever reaches the server. Always include 0.0.0.0,localhost,127.0.0.1 in noProxy.

The Docker daemon also needs proxy access to pull the ollama/ollama image itself. That's configured separately on the host ; typically /etc/systemd/system/docker.service.d/http-proxy.conf. If the daemon can pull SPOT's own images, this step is already done.

No context providers enabled

The RAG layer has nothing feeding it. Install a context_provider plugin from the Plugins page (e.g. provider-employee-dir) and run a sync. The banner turns green once at least one is enabled.

/readiness for scripts

GET /api/v1/knowledge/readiness on the api-gateway (admin auth) returns the same data the banner uses. Useful for monitoring:

curl -sS -H "Authorization: Bearer $TOKEN" \
    https://api.$DOMAIN/api/v1/knowledge/readiness

{
  "ready": false,
  "components": {
    "knowledge_service": true,
    "embedding_backend": false,
    "context_providers": false
  },
  "embedding": {
    "url": "http://ollama:11434",
    "model": "bge-m3",
    "reachable": false,
    "service_reachable": true
  },
  "context_providers": {"installed": 0, "enabled": []}
}

Services Not Starting

Check container status: 

docker compose ps -a

Check startup logs: 

docker compose logs --tail=50

Database Connection Failed

Verify postgres is healthy: 

docker exec spot-postgres pg_isready -U spot

Check postgres logs: 

docker compose logs postgres

API Gateway Returning 500

Check API Gateway logs: 

docker compose logs api-gateway

Verify environment: 

docker exec spot-api-gateway env | grep -E "(DATABASE|REDIS|RABBITMQ)"

Port Already in Use

Find process using the port: 

lsof -i :8001

Change port in .env: 

API_GATEWAY_PORT=9001

Out of Memory

Check container memory: 

docker stats --no-stream

Reduce Redis memory in .env: 

REDIS_MAXMEMORY=128mb

Dashboard Not Loading

Check dashboard logs: 

docker compose logs web-dashboard

Verify API Gateway is healthy: 

curl http://localhost:8001/health

RabbitMQ Connection Issues

Check RabbitMQ logs: 

docker compose logs rabbitmq

Access management UI: 

http://localhost:15672
Username: spot (or RABBITMQ_USER)
Password: (RABBITMQ_PASSWORD from .env)

"ENCRYPTION_KEY is required"

The api-gateway and analyzer-orchestrator refuse to start when ENCRYPTION_KEY is unset - email persistence is mandatory and silent fallback is not allowed.

Fix:

  1. Generate an encryption key: 

    python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

  2. Add to .env: 

    ENCRYPTION_KEY=your_generated_key_here

  3. Recreate the affected services (restart is not enough): 

    docker compose up -d --force-recreate api-gateway analyzer-orchestrator

  4. Verify the services are healthy: 

    docker compose ps api-gateway analyzer-orchestrator

"Fernet key must be 32 url-safe base64-encoded bytes"

This error appears in analyzer-orchestrator logs when ENCRYPTION_KEY is invalid.

Symptoms: - analyzer-orchestrator shows as "unhealthy" - Dashboard is very slow (10+ second timeouts) - "Failed to load dashboard data" error - API returns 503 for /api/v1/workflows

Cause: The ENCRYPTION_KEY in .env is malformed (wrong length, missing padding, or invalid characters).

Fix:

  1. Generate a valid Fernet key (must be exactly 44 characters, ending with =): 

    python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

  2. Update .env with the new key: 

    ENCRYPTION_KEY=your_new_44_character_key_here=

  3. Recreate the analyzer-orchestrator: 

    docker compose up -d analyzer-orchestrator --force-recreate

  4. Verify it's healthy: 

    docker compose ps analyzer-orchestrator
# Should show: Up ... (healthy)

"Message broker disconnected"

This appears when services lose their RabbitMQ connection (e.g., after RabbitMQ restarts).

Fix:

Restart the affected services: 

docker compose restart api-gateway analyzer-orchestrator mail-orchestrator

Or restart all services: 

docker compose restart

Verify connections are restored: 

docker exec spot-api-gateway wget -qO- http://localhost:8000/health | grep message_broker
# Should show: "connected": true

Reset Everything

Stop and remove all containers and volumes: 

docker compose down -v

Start fresh: 

docker compose up -d

WARNING: This deletes all data including the database.

Get Help