Skip to content

Testing Guide

Quick reference for testing the SPOT Platform.

Quick Start

# Unit tests (fast, no services required)
make test:unit

# Integration tests (full stack)
make test:integration

# All tests
make test

# Test specific service
make test:unit SERVICE=api-gateway

# Coverage report
make test:coverage

# Shared library tests (tests/unit) with the 100 % coverage gate
make test:shared

Test Types

Unit Tests

  • Location: services/{service}/tests/
  • Purpose: Test individual components in isolation
  • Speed: Fast (< 1 minute)
  • Dependencies: None (run in containers)
make test:unit SERVICE=api-gateway

Integration Tests

  • Location: tests/integration/
  • Purpose: Test service interaction and workflows
  • Speed: Moderate (2-5 minutes)
  • Dependencies: Full stack (postgres, redis, rabbitmq), two mock analyzers and a mock responder (tests/mocks/mock_analyzer.py, plugin contract 2, of the kind PLUGIN_KIND names). tests/mocks/spot.yaml declares them as manual plugin entries with ids of their own (mock-analyzer-nlp, mock-analyzer-llm, mock-responder, never a real plugin's) and the workflow that runs the analyzers and calls the responder on every verdict, mock-workflow, the default while the suite runs. The setup issues each mock its token (POST /api/v1/plugins/{id}/token, through scripts/issue_plugin_token.py) and recreates it with that token, which it checks on every call. The suite reads the stack's database (TEST_DATABASE_URL, which the setup sets).
make test:integration

Test Environment

Tests automatically run in APP_ENV=test; the test targets set it.

The test targets run in the devtools container and first sync its environment to the checkout (uv sync --frozen --all-extras --all-groups), so a change to shared/ or to uv.lock needs no new image; a new SDK release does (Developer Guide). An editor that runs outside the container uses a local environment, synced with uv sync --all-extras --all-groups (uv 0.12.22; the root .python-version selects Python 3.11, the images' Python).

  • Unit tests need no running service: each suite uses SQLite, or the database DATABASE_URL names, and mocks the other services.
  • Integration tests run against a running stack and read its database (TEST_DATABASE_URL, which the setup sets to the stack's DATABASE_URL). In CI the stack runs on .env.test: database spot_test, Redis index 10. Locally, make test:integration runs against your dev stack and its database.

Test Structure

The test infrastructure uses modular fixtures for better organization:

tests/
├── conftest.py           # Pytest configuration
├── fixtures/             # Modular test fixtures
│   ├── config.py        # Test configuration (immutable)
│   ├── clients.py       # HTTP client factories
│   ├── data.py          # Test data generators
│   └── helpers.py       # Test helper functions
└── integration/          # Integration test suites

Fixture Design

Fixtures are organized by purpose:

  • Each fixture module has a focused responsibility
  • Easy to extend without modifying existing code
  • Fixtures are interchangeable and reusable
  • Split by concern (config, clients, data, helpers)
  • Tests depend on fixture abstractions

Writing Tests

Unit Test Example

# services/analyzer-orchestrator/tests/test_engine.py
import pytest
from src.engine.condition import evaluate_condition
from src.engine.errors import ConditionError


def test_a_condition_on_an_unknown_verdict_fails_the_run() -> None:
    with pytest.raises(ConditionError, match="unknown verdict 'bad'"):
        evaluate_condition("triage.verdict == bad", {})

Integration Test Example

# tests/integration/test_analysis_workflow.py
from tests.fixtures.helpers import TestHelpers

async def test_full_analysis_workflow(authenticated_client, sample_emails):
    # Submit the raw message (message/rfc822)
    response = await TestHelpers.submit_message(authenticated_client, sample_emails[0])

    assert response.status_code == 200
    job_id = response.json()["job_id"]

    # Wait for the verdict
    job = await TestHelpers.wait_for_analysis_completion(authenticated_client, job_id)

    assert (job["status"], job["verdict"]) == ("completed", "benign")

Test Coverage

Generate Coverage Report

# Run all tests with coverage
make test:coverage

# View HTML report
open services/api-gateway/htmlcov/index.html

Coverage Files

  • HTML: services/{service}/htmlcov/index.html (detailed)
  • XML: services/{service}/coverage.xml (CI integration)
  • Terminal: Shown after test run (summary)

CI Testing

Local CI Pipeline

Test your changes before pushing:

# Run full CI pipeline locally
make ci:local FULL=1

# Run specific test job
make ci:local JOB="test-services: [api-gateway]"

# Clean up
make dev:clean

See CI-CD.md for complete CI/CD documentation.

Common Issues

Tests Failing with "APP_ENV must be test"

Solution: Use the CLI commands (they set APP_ENV automatically):

# Correct
make test:integration

# Wrong (will fail)
pytest tests/integration/

Database Connection Errors

Solution: Ensure services are running:

make service:status
make test:integration  # Starts services automatically

Test Cleanup Issues

Solution: Clean test resources:

# Remove test containers and volumes
docker compose down -v

# Restart tests
make test:integration

Best Practices

  1. Always use CLI commands for tests - They handle environment setup
  2. Write focused unit tests - Test one thing at a time
  3. Use fixtures for test data - Keep tests DRY
  4. Mock external services - Unit tests should be isolated
  5. Add assertions - Verify expected behavior explicitly
  6. Run tests before committing - Catch issues early

Testing Plugins

A plugin (analyzer, context provider, mail retriever, responder) is tested in its own repository, against the SDK rather than a running SPOT. spot_sdk.testing provides what its suite needs:

  • view_from_eml parses an .eml file as core does, and make_view builds a view field by field;
  • assert_contract(app, eml, token=...) calls the app's /internal/analyze on a message as core does and checks the answer is an AnalyzerVerdict;
  • FakeSpot, FakeKnowledgeClient and FakeLLM stand in for the gateway, the Knowledge Store and a language model.
# In your plugin repository
pytest tests/ -v

What to test:

  • The verdict, the score and the signals on phishing and benign .eml files
  • unknown, with a summary, when the plugin cannot decide
  • The token check of /internal/* (401 without it)
  • Settings and secrets read from the environment and /run/secrets

Integration with Platform

Once your plugin passes its own tests, run it against a SPOT stack:

  1. Register it in config/spot.yaml, under its kind's section:
plugins:
  analyzers:
    my-analyzer:
      enabled: true
      url: "http://my-analyzer:8000"
      settings: {}

  1. Give it a token: spot-cli plugin token my-analyzer --kind analyzer, mounted as /run/secrets/spot_plugin_token (see Context Providers). Without one, core does not call it: the analyzer counts as unknown, or fails the job when its failure strategy is fail.

  2. Name it in a workflow and submit a message (POST /api/v1/analyze), or dry-run the workflow on one (POST /api/v1/workflows/{workflow_id}/test, which keeps the message for the run only).

Plugin Development Resources