Skip to content

System Architecture Overview

SPOT uses a microservices architecture with pluggable analyzers and workflow-based orchestration.

High-Level Architecture

graph TB
    Client[Client/Mail System]
    MO[Mail Orchestrator]
    Gateway[API Gateway<br/>:8001]
    AO[Analyzer Orchestrator]

    A1[NLP Analyzer]
    A2[LLM Analyzer]
    A3[MISP Analyzer]

    DB[(PostgreSQL)]
    Redis[(Redis)]
    RMQ[RabbitMQ]

    Client --> MO
    MO -->|REST API| Gateway
    Gateway -->|Rate Limiting| Redis
    Gateway -->|Publish to spot.analysis| RMQ

    RMQ -->|Consume from orchestrator.analysis.requests| AO

    AO -->|HTTP POST /internal/analyze| A1
    AO -->|HTTP POST /internal/analyze| A2
    AO -->|HTTP POST /internal/analyze| A3

    A1 -->|HTTP Response| AO
    A2 -->|HTTP Response| AO
    A3 -->|HTTP Response| AO

    AO -->|Store Results| DB
    AO -->|Publish to spot.results| RMQ

Component Overview

Component Technology Dependencies Purpose
API Gateway FastAPI RabbitMQ, Redis, PostgreSQL External API, authentication
Analyzer Orchestrator Python asyncio RabbitMQ, PostgreSQL Workflow execution, result aggregation
Mail Orchestrator Python asyncio HTTP to retrievers Email ingestion routing
Analyzers External services HTTP interface Spear-phishing detection

Core Components

API Gateway

  • Port: 8001
  • Technology: FastAPI (Python)
  • Responsibilities:
  • Authentication & authorization (OAuth2/JWT)
  • Request validation
  • Job submission via RabbitMQ
  • Status queries via RabbitMQ RPC

Analyzer Orchestrator

  • Technology: Python asyncio + RabbitMQ
  • Responsibilities:
  • DAG-based workflow execution
  • Analyzer coordination via HTTP
  • Result aggregation
  • Failure handling & retries
  • Threat level determination

Mail Orchestrator

  • Technology: Python asyncio
  • Responsibilities:
  • Routes to configured mail retrievers
  • Email preprocessing
  • Submits emails for analysis via API Gateway

Analyzers (Pluggable)

  • NLP Analyzer: Text analysis, NER, sentiment
  • LLM Analyzer: AI-powered spear-phishing detection (Ollama)
  • MISP Analyzer: Threat intelligence lookup

Infrastructure Services

PostgreSQL

  • Port: 5432
  • Stores: Analysis results, user accounts

Redis

  • Port: 6379
  • Purpose: Rate limiting

RabbitMQ

  • Ports: 5672 (AMQP), 15672 (Management UI)
  • Exchanges:
  • spot.analysis (TOPIC) - Job routing
  • spot.status (TOPIC) - Status queries
  • spot.results (TOPIC) - Result notifications
  • Queues:
  • orchestrator.analysis.requests - Job consumption
  • job.status.requests - Status RPC
  • Dynamic RPC reply queues

Design Principles

1. Separation of Concerns

  • API Gateway: External interface
  • Orchestrators: Business logic
  • Analyzers: Specialized analysis
  • Infrastructure: Data & messaging

2. Asynchronous Processing

  • Non-blocking I/O throughout
  • RabbitMQ for job distribution
  • Polling for status updates

3. Pluggable Architecture

  • Analyzers as independent HTTP services
  • Configured in config/spot.yaml under the analyzers: section

4. Configuration via Environment + YAML

  • Platform settings via environment variables
  • Analyzer registration via config/spot.yaml
  • No database-backed config

Scalability

Horizontal Scaling

  • Analyzers: Scale independently based on workload
  • API Gateway: Behind load balancer
  • Orchestrators: Single instance (workflow consistency)

Vertical Scaling

  • PostgreSQL: SSD storage, more RAM
  • RabbitMQ: Increase connection limits
  • Redis: Memory tuning

Security

Authentication

  • OAuth2 with JWT tokens
  • API key support for external integrations

Network Security

  • Services in private network
  • Only API Gateway exposed
  • TLS for external connections

Next Steps