Skip to content

Metrics

Every core service (api-gateway, analyzer-orchestrator, mail-orchestrator, knowledge, plugin-manager) serves GET /metrics: the Prometheus text format, or OpenMetrics when the scraper asks for it. A scrape needs Authorization: Bearer <SPOT_METRICS_TOKEN>; without it the answer is 401, and while the token is unset, or still a CHANGE_ME_ placeholder, every scrape is refused with 503. /metrics is never published: scrape it from inside spot-network (see Monitoring).

Each service also serves the process_* and python_* metrics of prometheus-client.

Metric Type Labels Service: counted where
spot_http_requests_total counter method, route, status every service: each HTTP request answered
spot_http_request_duration_seconds histogram (default buckets) method, route every service
spot_analyses_total counter workflow, outcome (completed, failed, cancelled) analyzer-orchestrator: when the job's outcome is recorded
spot_verdicts_total counter workflow, verdict analyzer-orchestrator: each completed job
spot_analysis_duration_seconds histogram (0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600) workflow analyzer-orchestrator: each completed or failed job that ran, from its pickup
spot_jobs_running gauge analyzer-orchestrator: jobs being analysed
spot_analyzer_calls_total counter analyzer, status (success, unknown, error, timeout, skipped), run analyzer-orchestrator: every analyzer called, or skipped by its condition; one not reached before the workflow's timeout is not counted
spot_analyzer_call_duration_seconds histogram (0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120) analyzer, run analyzer-orchestrator: every call not skipped
spot_responder_calls_total counter responder, outcome (answered, gave_up, skipped) analyzer-orchestrator: each responder a verdict calls
spot_messages_total counter queue, outcome (acked, retried, parked) every consumer (the analyzer-orchestrator's queues)
spot_ingests_total counter retriever, outcome (accepted, analyzed, timeout, rejected) mail-orchestrator: each message a retriever ingests
spot_ingest_wait_seconds histogram (0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300) retriever mail-orchestrator: synchronous ingests that got a verdict or timed out
spot_logins_total counter method (password, client_credentials), outcome (success, failure) api-gateway: each sign-in
spot_rate_limited_total counter limit (login, client, plugin) api-gateway: the rate limiter's refusals
spot_provider_syncs_total counter provider, outcome (started, failed) api-gateway: each sync trigger of a context provider
spot_queue_messages gauge queue, state (ready, unacked) api-gateway: read from RabbitMQ at each scrape
spot_rabbitmq_up gauge api-gateway: 1 when that read succeeded, else 0
spot_audit_write_failures_total counter api-gateway: each entry of the audit log it could not write
spot_embeddings_total counter outcome (ok, unreachable, model_missing, server_error) knowledge: each text sent to the embedding backend
spot_embedding_duration_seconds histogram (0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30) knowledge: each text, answered or failed

What the labels hold

A label holds a route template or an id from spot.yaml, never a raw path, an address, a subject or a job id.

Label Values
method The HTTP method: GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, or OTHER
route The full template of the route that served the request, the prefixes of the routers that include it among them (/api/v1/emails/{email_id}, /api/v1/knowledge/sources/{source}/documents/{doc_id:path}), or unmatched for a path no route serves, a request the rate limiter refused among them
status The HTTP status answered
workflow A workflow id of spot.yaml, or unknown for a job that names none, or one spot.yaml does not hold
verdict malicious, suspicious, spam, benign or unknown
analyzer, responder, provider, retriever A plugin id of spot.yaml; retriever is unknown for an ingest without one
run job (an analysis job), dry_run (a workflow test), shadow (a shadow run of a workflow beside the live one), benchmark (an email of a platform benchmark run)
queue One of SPOT's queues: orchestrator.analysis, orchestrator.analysis.sync, orchestrator.dispatch, orchestrator.shadow, orchestrator.benchmark, and the dead-letter queues orchestrator.analysis.dead-letter, orchestrator.dispatch.dead-letter, orchestrator.shadow.dead-letter and orchestrator.benchmark.dead-letter
state ready (waiting for a consumer) or unacked (delivered, not acknowledged yet)
limit login (sign-ins, per address), plugin (a plugin's own limit), client (every other request, per address)

What each outcome means:

  • spot_messages_total.
  • acked: handled.
  • retried: handed back, to be delivered again.
  • parked: refused, or failed on the last delivery its queue allows.
  • A message parked on orchestrator.analysis.sync moves to orchestrator.analysis, which handles it, or parks it in its own dead-letter queue.
  • spot_responder_calls_total.
  • gave_up: the responder failed its last retry, or its call could not be made. The dispatcher has the message delivered again, so a responder can count gave_up and then answered for one verdict.
  • skipped: the responder has no usable token.
  • spot_ingests_total.
  • accepted: queued, for a retriever that does not wait.
  • analyzed: the verdict came in time; timeout: it did not.
  • rejected: the message could not be queued or analysed.
  • spot_analyzer_calls_total. It counts calls, whatever the stage's failure strategy: an analyzer whose strategy is fail is counted with its failure (error or timeout) before its job fails, beside the analyzers that finished before it.

Counting

  • Counters start at 0 when a service starts, and so after every restart: read them with rate() or increase().
  • Counters count runs. A message whose acknowledgement was lost is delivered again and runs again, so its job, its verdict and its analyzer calls count twice.
  • Each service process counts its own requests, jobs and messages.
  • The api-gateway reads spot_queue_messages from RabbitMQ's management API at each scrape, so every gateway answers the same depths. A read that fails sets spot_rabbitmq_up to 0 and leaves no depth.