Benchmark Accuracy Testing¶

Test SPOT's phishing detection accuracy against labeled datasets. The benchmark system normalizes various dataset formats into RFC 5322 messages, submits them to SPOT via the standard API, and computes accuracy metrics from the verdicts.

The suite is the spot-cli benchmark command group, implemented in services/cli/src/spot_cli/commands/benchmark/.

Setup¶

spot-cli runs in a Docker container (see CLI installation), so the benchmark directories must be passed in from the host. Export both variables before running any benchmark command; the wrapper bind-mounts them into the container:

export SPOT_DATASETS_DIR=/path/to/spot/datasets      # raw datasets, mounted read-only
export SPOT_BENCHMARK_DIR=/path/to/spot/benchmarks   # normalized datasets and runs, read-write

Without them, the CLI falls back to ./datasets and ./benchmarks inside the container, where neither exists.

benchmark run talks to the api-gateway, so log in first:

spot-cli auth login

Quick Start¶

# 1. Normalize a small dataset (2K emails, fast)
spot-cli benchmark normalize --dataset miltchev

# 2. Run the benchmark
spot-cli benchmark run --dataset miltchev --workflow default-workflow

# 3. View results
spot-cli benchmark report --latest

Commands¶

benchmark datasets¶

List all available datasets and their normalization status.

spot-cli benchmark datasets

benchmark normalize¶

Convert raw datasets into JSONL files of RFC 5322 messages, the form SPOT takes at intake. This is a one-time step per dataset.

# Normalize one dataset
spot-cli benchmark normalize --dataset alhuzali_balanced

# Normalize all datasets
spot-cli benchmark normalize --all

# Normalize first 100 emails per dataset (for quick testing)
spot-cli benchmark normalize --all --sample 100

Raw files are read from $SPOT_DATASETS_DIR/<path>, where <path> is the dataset's entry in the registry (datasets.py). Output is written to $SPOT_BENCHMARK_DIR/normalized/<dataset>/:

• emails.jsonl -- one normalized email per line: its ground-truth label and its RFC 5322 message, base64-encoded (raw)
• manifest.json -- email count, phishing/legitimate split, and up to 50 parse warnings

Normalizing a dataset overwrites its previous output, including a full normalization replaced by a --sample one. A dataset normalized by an earlier spot-cli, whose lines hold an email object instead of raw, is refused by run ("made by an earlier spot-cli: normalize the dataset again"): normalize it again. A JSON file that is not an email of its dataset is a parse warning, and the dataset's other files are normalized.

benchmark run¶

Submit normalized emails to SPOT and collect results. Each email is submitted as its raw message, via the same POST /api/v1/analyze endpoint used by spot-cli analyze submit, and its result is the workflow's verdict, score and decided_by.

# Benchmark a specific dataset with a specific workflow
spot-cli benchmark run --dataset miltchev --workflow nlp-only-workflow

# Benchmark several datasets in one run
spot-cli benchmark run -d champa_nazario5 -d nahmiasd_enron_ham --workflow default-workflow

# Benchmark all normalized datasets
spot-cli benchmark run --all --workflow default-workflow

# Reproducible random sample of 1000 emails across all datasets
spot-cli benchmark run --all --shuffle --seed 42 --max-emails 1000

# Limit to 500 emails with higher concurrency
spot-cli benchmark run --dataset alhuzali_balanced --max-emails 500 --concurrency 50

# Only test phishing detection (skip legitimate emails)
spot-cli benchmark run --dataset champa_enron --label phishing

# Preview without submitting
spot-cli benchmark run --dry-run --dataset miltchev

# Resume an interrupted run (repeat the original options)
spot-cli benchmark run --resume run_2026-04-08_14-30-00 --dataset miltchev --workflow nlp-only-workflow

Selection. Emails are loaded in dataset-name order. Without --shuffle, --max-emails keeps the first N, so a capped --all run only covers the first datasets alphabetically. With --shuffle, the cap is applied after shuffling; add --seed to get the same sample on every run.

Execution. Emails are processed batch by batch: submit the batch, then poll every job in it until it finishes or --timeout expires. The access token is refreshed between batches. The run is aborted if 20 submissions in a batch fail before any succeeds (platform down, unknown workflow, auth error).

Rate limiting. run stays under the platform's rate limit and never changes the platform's configuration. When the platform answers 429, the whole run waits the seconds its Retry-After gives (without one of 1 s or more, 1 s, doubling up to 60 s), printing Rate limited by the platform: waiting <N>s (again, with the wait left, when a later 429 makes the pause 1 s or more longer), then sends the request again: a 429 is never counted as a failure. Every email costs one submission plus its status polls, all counted against the client IP's limit (rate_limit_requests per rate_limit_period, 100 per 60 s by default), so a large run is bound by that limit. A stored run adds one request to start it, one per batch of up to 1000 results, and one to finish it, through the same pacing; no account is exempt from the limit, which counts per client IP. --timeout leaves the waits out: it measures how long the platform takes, so a job that finished is never recorded as timeout because the run was waiting.

Runs of an earlier spot-cli. run --resume, report and compare refuse a run whose results or metrics an earlier spot-cli wrote ("made by an earlier spot-cli: run it again"), and write nothing: its saved metrics are kept. A line that is no JSON, such as one an interrupted write cut short, is refused the same way, as one that "cannot be read".

Resuming. Every submission and result is appended to results.jsonl as soon as it happens, and the latest line per email wins. On --resume:

• completed, failed and timeout emails are skipped
• submitted emails (job created, result not collected) are polled again
• error emails (submission failed) are resubmitted

--resume reads back from config.json only whether the run is recorded on the platform, and the id of its record there (Stored runs). Pass the same --dataset, --workflow, --label, --max-emails, --shuffle and --seed as the original run; otherwise every normalized dataset is loaded and default-workflow is used for the remaining emails.

Results are stored in $SPOT_BENCHMARK_DIR/runs/<run_id>/. Run IDs have the form run_YYYY-MM-DD_HH-MM-SS (UTC). The platform records the run as well, unless --no-store is given (Stored runs).

benchmark report¶

Compute and display accuracy metrics for a run.

# Report on a specific run
spot-cli benchmark report run_2026-04-08_14-30-00

# Report on the most recent run
spot-cli benchmark report --latest

# Show per-dataset breakdown, as percentages, with column descriptions
spot-cli benchmark report --latest --per-dataset --percent --legend

# Print metrics as JSON
spot-cli benchmark report --latest -o json

An email counts as predicted phishing when its verdict is malicious or suspicious, and as predicted legitimate when it is spam or benign. A completed email whose verdict is unknown (no analyzer could decide) is left out of the confusion matrix and counted under Unknown; failed, timeout and error emails are counted under Failed. Completed counts the emails in the matrix, so Total is Completed + Unknown + Failed. The Score column is the mean score (0-100, how likely the email is malicious) of the emails in the matrix.

Every report writes the computed metrics to metrics.json in the run directory.

benchmark compare¶

Compare two runs side-by-side (e.g., different workflows, before/after a change). Both runs need a metrics.json, so run benchmark report on each first.

spot-cli benchmark report run_2026-04-08_14-30-00
spot-cli benchmark report run_2026-04-08_15-00-00
spot-cli benchmark compare run_2026-04-08_14-30-00 run_2026-04-08_15-00-00

Deltas are shown from the first run to the second, colored green for an improvement (higher precision, recall, F1, F2 and accuracy; lower FPR and FNR). The counts of completed and unknown emails are shown side by side.

benchmark list¶

List all benchmark runs. A run shows as reported once benchmark report has been run on it.

spot-cli benchmark list

Stored runs¶

benchmark run records each run on the platform, so that a detection engineer can read it later, compare it with another, and know what it ran. The local files in $SPOT_BENCHMARK_DIR/runs/ stay as they are: report, compare and --resume read them.

What is stored.

• The snapshot of what ran: the workflow's id and version, the commit of spot.yaml's history in force, SPOT's version, and each analyzer's version and image.
• The selection: each dataset file read, with the SHA-256 of its emails.jsonl and the emails taken from it, the --label filter, --shuffle, --seed and --max-emails.
• Each email's result: its key in its dataset, its truth, the workflow's verdict, score and decided_by, its time, and each analyzer's status, verdict, score and time.
• The run's metrics, counted when the run ends over the results recorded, as benchmark report counts them: overall, per dataset and per analyzer, with the median and 95^th percentile of the times. The workflow's times are those of the emails it completed. An analyzer's are those of all its answers but the ones its condition skipped: its errors and timeouts count, and so do its answers on emails without a truth.

An email's content is never stored: no subject, address, body or analyzer summary. A viewer can read a run for that reason.

Who records it. The account spot-cli signs in with, an analyst's or an admin's. A viewer's account cannot record a run: run says so and stops, and --no-store runs it without recording. Starting a run is in the audit log as benchmark.start, with the workflow as its target, cancelling one as benchmark.cancel, and deleting one as benchmark.delete (Audit log).

Bounds. A run records up to 50 000 emails. A run of more is refused before it starts: cap it with --max-emails, or keep it on this machine with --no-store.

Interrupted and resumed runs. --resume goes on with the same record, and sends every result already collected again: the platform keeps one result per email, so nothing is counted twice. The run is resumed as it was started, recorded or --no-store. A record that has ended takes no more results: run says that the emails sent again are kept on this machine only, and goes on. A record deleted from the platform is gone: run says so, and goes on with the run kept on this machine only. A refused record stops the run with the platform's reason, and the local results are kept for --resume. A refused start stops the run before any email is submitted, and says what to fix: the account, the workflow, or what the platform names.

Platform runs. The platform runs a benchmark too, a backtest, on the stored mail of a period or on a dataset of promoted copies, started from the dashboard's Playground (Dashboard tour) or with POST /api/v1/benchmarks. It stores the same run, and each result also records the production analysis its email was compared with (baseline_job_id); a run over stored mail also counts production's metrics on the same labelled emails (baseline). Backtesting a workflow gives its rules and its load. A platform run needs the message broker.

Where the runs show. The dashboard's Benchmarks page (Dashboard tour) and GET /api/v1/benchmarks (API). A run can be deleted by an analyst, from the dashboard or with DELETE /api/v1/benchmarks/{run_id}; its results go with it.

Which model an analyzer ran. The snapshot names an analyzer's image, and the version it reported when the run started, else the one spot.yaml holds. A model an analyzer takes as a setting is not recorded on its own: read it in spot.yaml at the run's configuration commit (spot-cli config diff <commit> shows that commit).

Datasets¶

Included Datasets¶

Excluded: nc3 (email bodies stored as SHA256 hashes, not usable for content analysis).

Normalization Details¶

Datasets come in different formats with different fields. The normalizer turns each email into an RFC 5322 message, the form SPOT takes at intake, so the gateway parses every one as it parses real mail:

• Body-only CSVs (alhuzali, subhajournal, miltchev): A message is built around the body, as its text/plain part, with synthetic headers (From, To, Message-ID, Date, and Subject: (no subject)). Only the body text is real.
• Rich CSVs (champa): Headers taken from CSV columns where available (subject, sender, receiver, date). Missing fields use synthetic values; a field on several lines is joined into one header line. A sender or receiver that is no address a header can hold (john smith@x.example, x@y..z) is replaced by a synthetic one, and the row is kept with a parse warning.
• JSON directories (nahmiasd): Subject and body from JSON keys; a body that looks like HTML is a text/html part, and the sender's name is the display name of a synthetic address. Labels from directory name (e.g., spear_phishing/ = phishing).
• Maildir (pashakhin): Each file is kept byte for byte, its own headers and MIME structure included. Labels from the X-Spam-Flag header or fixed assignment. An empty file is skipped with a parse warning.

Synthetic addresses and message IDs use the reserved example.com domain (e.g. miltchev-12@benchmark.example.com), and the synthetic date is Sat, 01 Jan 2000 00:00:00 +0000, so they are clearly identifiable.

Metrics¶

The following metrics are computed from the confusion matrix, which leaves out the emails whose verdict is unknown:

Where:

• TP (True Positive): Phishing email correctly detected
• TN (True Negative): Legitimate email correctly classified
• FP (False Positive): Legitimate email incorrectly flagged as phishing
• FN (False Negative): Phishing email missed (classified as legitimate)

Precision, recall and FPR each come with a 95 % Wilson score interval (precision_ci, recall_ci and fpr_ci in metrics.json, as [low, high]; --ci shows them). A rate with nothing to divide by has none (null, shown as n/a). The intervals assume independent emails, so a set full of near-identical messages makes them too narrow: the runs that report held-out numbers score distinct messages.

The counts, the rates and the intervals are defined once, in the SDK (spot_sdk.testing.metrics): spot-cli plugin test counts a plugin's corpus with the same function, so the two give the same numbers for the same answers.

Key metrics for phishing detection: - Recall is critical -- missing a phishing email has higher cost than a false alarm - F2 weights recall higher than precision, making it the best single metric for phishing detection - FPR should be minimized to avoid alert fatigue

Typical Workflow¶

The benchmark results page gives the held-out numbers of SPOT's LLM-free workflows, measured with the commands below on the distinct messages of three public sets (spot-models distinct writes <set>-distinct, which keeps one message per normalised body and label: near-identical messages would make the intervals too narrow).

Initial accuracy baseline¶

# Normalize the small validation set
spot-cli benchmark normalize --dataset miltchev

# Quick sanity check
spot-cli benchmark run --dataset miltchev --workflow default-workflow
spot-cli benchmark report --latest

Compare workflows¶

spot-cli benchmark run --dataset miltchev --workflow nlp-only-workflow
spot-cli benchmark report --latest
spot-cli benchmark run --dataset miltchev --workflow default-workflow
spot-cli benchmark report --latest
spot-cli benchmark compare run_<first> run_<second>

Large-scale test¶

# Normalize everything
spot-cli benchmark normalize --all

# Run with higher concurrency
spot-cli benchmark run --all --workflow default-workflow --concurrency 50

# Detailed report
spot-cli benchmark report --latest --per-dataset

Testing specific phishing types¶

# Test against spear phishing specifically
spot-cli benchmark run --dataset nahmiasd_spear_phishing --workflow default-workflow

# Test against traditional phishing
spot-cli benchmark run --dataset nahmiasd_traditional_phishing --workflow default-workflow

File Structure¶

$SPOT_BENCHMARK_DIR/
    normalized/                     # Pre-normalized datasets
        miltchev/
            manifest.json           # Count, label distribution, parse warnings
            emails.jsonl            # One NormalizedEmail per line (raw message, base64)
        alhuzali_balanced/
            ...
    runs/                           # Benchmark results
        run_2026-04-08_14-30-00/
            config.json             # Run parameters, and the id of the run on the platform
            results.jsonl           # Append-only BenchmarkResult log
            metrics.json            # Computed metrics (after report)

Adding New Datasets¶

1. Place the dataset in $SPOT_DATASETS_DIR
2. Add an entry to services/cli/src/spot_cli/commands/benchmark/datasets.py with the appropriate DatasetInfo (path is relative to $SPOT_DATASETS_DIR)
3. If the format is new, add a parser function in normalize.py
4. Run spot-cli benchmark normalize --dataset <name> to verify