Benchmark results¶

How SPOT's LLM-free workflows score on three public benchmark sets that no model of SPOT trained on. Every number below comes from the runs listed in What was measured and from the model cards linked there. The intervals are 95 % Wilson score intervals (see Metrics); a set shows only the rates its own counts allow, and n/a marks a rate it cannot measure.

What was measured¶

• Platform. SPOT 2.0.0-beta.7 on the test VM: 4 vCPU, 7.7 GiB, no GPU. The plugins keep their default limits of 1 CPU and 1 GiB; sampled during the baseline, analyzer-nlp was at 100 % of its CPU, which is what bounds the speed of the runs that include it. The platform's rate limit (100 requests per minute) is unchanged.
• Plugins and models.
• analyzer-nlp 2.0.0b1: the baseline. Its DistilBERT model was trained before spot-models; no repository records its training data, and it was never checked against these sets.
• analyzer-ml 2.0.0b3: serves spot-tfidf-phishing 1.0.0, a TF-IDF model with a logistic regression, trained by spot-models on legitimate mail (TREC 2005-2007 ham, through MeAJOR v2) against phishing (the Nazario and Nigerian Fraud corpora). Its model card gives the data, the training and the limitations.
• Workflows. nlp-only-workflow (core's, analyzer-nlp), ml-only-workflow (the same with analyzer-ml) and nlp-ml-workflow (both in one parallel stage, weight 1.0 each, required: false, weighted mean with the default thresholds 80 and 50).
• Commands. On the VM, with the set in $SPOT_BENCHMARK_DIR:

spot-cli benchmark run -d nahmiasd_spear_phishing-distinct -d miltchev-distinct \
  -d pashakhin-distinct --workflow <workflow> -c 4 -b 100
spot-cli benchmark report <run id> --per-dataset --ci --json

The runs used the spot-cli of SPOT 2.0.0-beta.7 on the VM; the reports were made with spot-cli at core a56d5a79, which is part of 2.0.0-beta.7.

The baseline was started with the defaults (batches of 500, concurrency 10); the gateway answered 503 once its queue filled, so after 28 minutes it was resumed with -c 4 -b 100 (--resume re-submits the failed messages and re-polls the submitted ones). Every message has exactly one result. - A model not released. A DistilBERT model was trained for spot-models 1.0.0 and not released: scored on these sets, it flagged 7,960 of the 13,061 distinct pashakhin messages (false-positive rate 0.609, interval 0.601-0.618), all legitimate, where the TF-IDF model flags 1.6 %. A retrain follows in a later release of spot-models, and its numbers will join this page.

The held-out sets¶

Counted over distinct messages: two messages with the same normalised text and the same label count once (spot-models distinct), so that near-identical messages do not make the intervals too narrow.

• miltchev (Zenodo 13474746, CC BY 4.0): 2,000 emails, bodies only, 1,000 phishing and 1,000 legitimate. 93 distinct bodies: 19 phishing and 74 legitimate. A small set: its intervals are wide.
• nahmiasd_spear_phishing (GitHub, CC BY-NC-SA 4.0, non-commercial and share-alike: the numbers are published here, the set is not): 334 spear-phishing emails, all phishing, 334 distinct.
• pashakhin (the Google Scholar Emails Dataset, Zenodo 14831438, CC BY 4.0): 26,900 messages, all legitimate, all HTML; 13,061 distinct. Not all in English: of the 13,061 distinct messages, 58 have more than half of their letters in Cyrillic, 807 more than 30 %, and 3,869 (30 %) have a subject written mostly in Cyrillic (letters counted in the visible text and the subject, Cyrillic against Latin).

Why they are held out: spot-models' leak check removed every training row it found in them (five MeAJOR ham rows, all one common closing phrase, as spot-models' 1.0.0 changelog states; the exclusion list holds their hashes), by the words of the text or by half of its 8-word shingles. It cannot find a short anonymised copy of a held-out email, nor one quoted inside a longer training row.

Results¶

A message counts as phishing when its verdict is malicious or suspicious, and as legitimate when it is benign or spam, as benchmark report defines it (the tables below); the next section also gives the counts for malicious alone. Unknown is the messages whose verdict is unknown, left out of the counts; Failed those that failed or timed out: both are 0 in every row. Mean duration is the time between a message's submission and its result, over the messages that report one, queueing included: it measures the deployment under this load, not one inference.

The baseline: nlp-only-workflow¶

The 13,488 messages took 8 h with the resume; 0 unknown, 0 failed. Its pashakhin false-positive rate is the first thing to read: 72 % of the legitimate Google Scholar alerts are called phishing.

ml-only-workflow¶

The 13,488 messages took 4 h 34 min (the platform's rate limit bounds this run); 0 unknown, 0 failed. These counts are the ones on the model's card: the workflow adds nothing to the model's answer here.

nlp-ml-workflow¶

The 13,488 messages took 8 h 4 min; 0 unknown, 0 failed.

Counting malicious alone¶

Counting a suspicious verdict as phishing is what the tables above do. A reader who acts only on malicious gets these rates from the same runs (computed from each run's results.jsonl, with the same intervals):

The convention changes the combination's pashakhin false-positive rate from 0.653 to 0.010: its verdicts there are almost all suspicious. Quote a rate with its convention.

Reading the numbers¶

What each set measures: nahmiasd_spear_phishing, recall (it holds phishing only); pashakhin, the false-positive rate (it holds legitimate mail only); miltchev, precision, recall and F1 as well, but on 93 distinct bodies, so its intervals are wide and overlap between the workflows.

• The baseline calls most of the legitimate alerts phishing. 9,369 of the 13,061 distinct pashakhin messages (72 %), and it finds half of nahmiasd's spear phishing.
• The TF-IDF model is the reverse. It finds 83 % of nahmiasd and flags 1.6 % of pashakhin, but it flags 39 % of miltchev's legitimate bodies (29 of 74).
• The two together keep the baseline's direction. With equal weights and the default thresholds, the combination calls 65 % of pashakhin's legitimate alerts phishing (0.645-0.661, against the baseline's 0.710-0.725: the intervals do not overlap) and finds 54 % of nahmiasd (the baseline's 49 % overlaps it). What differs is the verdicts: on pashakhin the baseline gives 8,944 malicious and 425 suspicious, the combination 136 malicious and 8,393 suspicious. A score of 100 from one analyzer and 0 from the other averages 50, which the default thresholds call suspicious; the weights and thresholds are a workflow's to tune.
• Model and workflow agree. The ml-only counts are the TF-IDF model's card counts, to the message: the workflow adds nothing to the model's answer.
• Speed. A message takes 0.2 s through ml-only, and from 2 to 19 s through the workflows that include analyzer-nlp, which was at its 1-CPU limit: the durations include the time a message waits in the queue.
• Small sets. miltchev has 93 distinct bodies (19 phishing, 74 legitimate): no difference between the workflows on it is established by these intervals.
• Not a verdict on the models' worth. pashakhin is one sender family of Google Scholar alerts, and nahmiasd 334 spear-phishing emails: the numbers say how each workflow does on these sets, not on mail in general.

No LLM workflow¶

The test VM has no GPU, and an 8B model on 4 vCPU cannot answer within the stage timeouts. An LLM's training data is not known either, so it could not be said to be held out.

Sets that are not held out¶

• champa_trec05, champa_trec06 and champa_trec07 are the mail MeAJOR's ham comes from.
• champa_nazario and champa_nigerian are the training positives.
• champa_ceas08 and champa_enron trained the earlier analyzer-nlp model (its merged CEAS and Enron dataset).

Their numbers are never quoted as held out.

How to reproduce¶

1. Get the three sets from their sources (see The held-out sets) and normalise them with spot-cli (core f5eb5d4b): spot-cli benchmark normalize --dataset <set>. The emails.jsonl sha256 were:

1. Write the distinct sets with spot-models 1.0.0 (spot-models distinct --benchmarks $SPOT_BENCHMARK_DIR). Their emails.jsonl sha256 were:

1. Install analyzer-ml 2.0.0b3 beside analyzer-nlp 2.0.0b1, create the two workflows below (spot-cli workflow create <file>; nlp-only-workflow is core's default one), and run the commands of What was measured for each workflow, then report with --per-dataset --ci --json.

id: "ml-only-workflow"
name: "ML-Only Detection Workflow"
version: 1
stages:
  - name: "ml-analysis"
    type: "sequential"
    depends_on: []
    analyzers:
      - id: "analyzer-ml"
        weight: 1.0
        timeout_ms: 30000
        required: true
        failure_strategy: "retry"
        retry_config: {max_attempts: 3, backoff_ms: 1000, max_backoff_ms: 5000, exponential_backoff: true}
timeout_ms: 60000
verdict_policy: {method: "most_confident", decision_stages: ["ml-analysis"]}

id: "nlp-ml-workflow"
name: "NLP and ML Detection Workflow"
version: 1
stages:
  - name: "analysis"
    type: "parallel"
    depends_on: []
    analyzers:
      - id: "analyzer-nlp"
        weight: 1.0
        timeout_ms: 30000
        required: false
        failure_strategy: "retry"
        retry_config: {max_attempts: 3, backoff_ms: 1000, max_backoff_ms: 5000, exponential_backoff: true}
      - id: "analyzer-ml"
        weight: 1.0
        timeout_ms: 30000
        required: false
        failure_strategy: "retry"
        retry_config: {max_attempts: 3, backoff_ms: 1000, max_backoff_ms: 5000, exponential_backoff: true}
timeout_ms: 60000
verdict_policy: {method: "weighted_mean", decision_stages: ["analysis"]}

A run of the 13,488 messages takes from 4.5 hours (ml-only) to 8 hours (the workflows with analyzer-nlp) on a 4 vCPU VM.