Context Providers Guide¶

Context providers ingest organisational data (employee directory, threat intelligence, wiki pages, policies, ...) into the SPOT Knowledge Store. They run as standalone HTTP services. They do not see emails and do not return verdicts: core asks each one to sync on a schedule, the provider writes its documents into the store, and analyzers retrieve them on demand by tag and semantic similarity.

When to write a context provider¶

Use a context provider when:

• You have an external source of organisational facts (LDAP, HRIS, threat feed, wiki, ticketing system) from which analyzers can benefit.
• The data refreshes on a schedule, not per email.
• Analyzers can find what they need by tag expression and free-text query.

The routes a provider serves¶

The SDK's create_plugin_app(manifest, Settings, sync=sync) serves them all:

• POST /internal/sync: core's scheduler calls it on the provider's sync_schedule, and an admin through the manual trigger. It answers 202 at once with the provider's SyncState and runs the sync handler in the background. While a sync runs, another call starts nothing and answers 202 with the running sync's state.
• GET /internal/state: the SyncState, which the dashboard's sync panel shows: in_progress, last_sync_at (when the last sync started), last_finished_at, last_sync_count (the documents the last successful sync wrote) and last_error (the exception's type when it failed).
• GET /health always answers 200 with {"status", "version"}: the status is healthy, starting or unhealthy, with a short reason when there is one. It comes from the health check the plugin gives create_plugin_app (healthy without one; unhealthy when the check fails). Core reads the status, the reason and the version, and the plugin views show them as plugin_health, beside the container's state. Core keeps the status whatever the reason and the version: it shows the reason with each character that is not printable (a control, format or separator character, a no-break space among them) replaced with U+FFFD, cut at 200 characters, and leaves out a version that is not printable text of at most 64 characters. GET /manifest answers the plugin's manifest. Both are open.

Every /internal/* route checks the provider's own token, sent by core as Authorization: Bearer, and answers 401 without it. Core reads at most 1 MiB of an answer to /internal/sync or /internal/state, and refuses an encoded one.

Implementing a provider (Python)¶

from pydantic import SecretStr
from spot_sdk import (
    Egress,
    KnowledgeClient,
    KnowledgeDocument,
    KnowledgeTag,
    Manifest,
    PluginKind,
    PluginSettings,
    SettingField,
    SettingType,
    create_plugin_app,
)

MANIFEST = Manifest(
    id="provider-employee-dir",
    version="2.0.0",
    kind=PluginKind.CONTEXT_PROVIDER,
    egress=Egress.NONE,
    settings={
        "LDAP_URL": SettingField(type=SettingType.URL, required=True),
        "LDAP_BIND_PASSWORD": SettingField(type=SettingType.SECRET, secret=True),
    },
)


class Settings(PluginSettings):
    ldap_url: str
    ldap_bind_password: SecretStr


async def sync(settings: Settings) -> int:
    employees = await fetch_from_ldap(settings)
    documents = [
        KnowledgeDocument(
            id=f"employee:{e.email}",
            content=f"{e.name}, {e.title}, {e.department}. Email: {e.email}.",
            tags=[
                KnowledgeTag.EMPLOYEE,
                *([KnowledgeTag.EXECUTIVE] if e.is_executive else []),
                e.department.lower(),
            ],
            metadata={"email": e.email, "title": e.title},
        )
        for e in employees
    ]
    await KnowledgeClient.from_settings(settings).replace_source(documents)
    return len(documents)


app = create_plugin_app(MANIFEST, Settings, sync=sync)

• KnowledgeClient.from_settings reads the store's URL (SPOT_KNOWLEDGE_URL), the token (/run/secrets/spot_plugin_token) and the provider's id (SPOT_PLUGIN_ID), which is the source of its documents.
• A document is identified by its source and its id: the ids need only be unique among the provider's own documents. Another provider may use the same id; each keeps its own document under it.
• replace_source(documents) makes them the provider's whole set, in one transaction (PUT /sources/{source}/documents): the store deletes the provider's other documents, so a record gone from the source system disappears. upsert, bulk_upsert and delete change documents one by one instead.
• For long documents (wiki pages, policy PDFs, ...), split with spot_sdk.chunk_text(text, max_chars=2000, overlap=200) and write each chunk with metadata["parent_id"] pointing back at the source: embeddings are better per chunk and retrieval is finer-grained.

provider-employee-dir is the reference implementation.

Authentication¶

A provider holds its own opaque token (spot_pt_...). It is the only credential a plugin gets.

• File. The token is the file /run/secrets/spot_plugin_token, and each secret setting the file /run/secrets/<NAME>. The container environment holds no credential: the other settings, SPOT_GATEWAY_URL, SPOT_KNOWLEDGE_URL, SPOT_PLUGIN_ID, SPOT_PLUGIN_KIND and SPOT_PLUGIN_VERSION. From SPOT 2.1 a plugin gets no proxy variable and no proxy credentials file: the egress gateway goes through the operator's proxy for it. The credentials file, /run/secrets/spot_proxy_credentials, is SPOT 2.0's only.
• Header. Every call to SPOT carries Authorization: Bearer <token>. Nothing else authenticates a plugin.
• Scopes. They come from the plugin's kind:

• Own documents only. A provider writes, replaces and deletes only documents whose source is its plugin id. An empty source takes the id; a write naming another source, or a deletion of another source's document (DELETE /sources/{source}/documents/{id}), is refused with 403. A write never touches another provider's document, whatever id it uses.
• Core's calls. The scheduler sends the provider's own token on /internal/sync and /internal/state. A provider without a usable token is not called: the trigger is recorded as failed, with the reason.

The platform stores the token's SHA-256 for lookups and a copy encrypted with ENCRYPTION_KEY (table plugin_tokens), so it can hand the token back when it recreates the container. Removing a plugin from spot.yaml revokes its token: at once through the config API, and for a hand edit at the next api-gateway start or config reload (POST /api/v1/config/reload).

Manual plugins¶

A plugin that is not installed by the platform (run by your own compose file) gets its token from the CLI:

spot-cli plugin token provider-employee-dir --kind context_provider > spot_plugin_token

--kind adds the plugin to spot.yaml as a manual entry when it is not there yet. Mount the file as the secret spot_plugin_token, and set SPOT_PLUGIN_ID, which replace_source needs:

services:
  provider-employee-dir:
    environment:
      SPOT_PLUGIN_ID: provider-employee-dir
    secrets:
      - spot_plugin_token
    networks:
      - spot-plugins
secrets:
  spot_plugin_token:
    file: ./spot_plugin_token
networks:
  spot-plugins:
    external: true

spot-plugins is the network of the plugins you run yourself: it reaches only the api-gateway, the knowledge service and the analyzer-orchestrator. A plugin left on the old spot_spot-network keeps working but is not isolated: it still reaches the mail-orchestrator, the plugin-manager and, in dev, adminer.

Running the command again rotates the token: the old one stops working at once. For an installed plugin, the platform also recreates the container with the new token.

Tags and the shared vocabulary¶

Tags are the only categorisation axis the Knowledge Store understands. Use the shared constants from spot_sdk.knowledge_tags (EMPLOYEE, EXECUTIVE, WIKI_PAGE, POLICY, FINANCE, ...) so analyzers and providers stay aligned. Custom tags like acme:jira_issue are fine where the shared vocabulary doesn't fit. Tokens must match [a-z0-9][a-z0-9_\-:]*; the Knowledge Store guide documents the AND / OR query grammar analyzers use against them.

Registering a provider¶

Context providers live under plugins.context_providers in spot.yaml:

plugins:
  context_providers:
    employee-dir:
      enabled: true
      url: http://provider-employee-dir:8000
      sync_schedule: "0 */6 * * *"   # every 6 hours
      sync_timeout_ms: 120000
      settings:
        LDAP_URL: ldap://ldap.example.com

Supported cron syntax: fixed values, *, comma lists (0,15,30,45), ranges (3-6), steps (*/10), and @hourly / @daily / @weekly / @monthly aliases. Absent or blank sync_schedule means "manual-only": the scheduler skips the provider but the manual trigger still works.

You can also manage providers via the API or the Plugins page in the web dashboard, which writes to the same spot.yaml:

The same URL shape with analyzer, mail_retriever and responder covers the other plugin kinds.

Triggering a sync manually¶

The dashboard's "Sync now" button on the provider's detail page calls:

POST /api/v1/plugins/context_provider/{id}/sync   # admin: ask the provider to sync now
GET  /api/v1/plugins/context_provider/{id}/sync   # viewer: the last trigger and the sync state

Both answer the provider's SyncState (read live from its /internal/state for the GET) with the last trigger: last_triggered_at, trigger_status (started or failed) and trigger_error. The GET adds the schedule, its next_sync_at and any schedule_error.

Failure handling¶

• Trigger failed: the provider is unreachable, answers an error, has no usable token, or does not answer within sync_timeout_ms. The trigger is recorded as failed, with the reason, and the scheduler tries again at the next scheduled time. sync_timeout_ms bounds only the wait for that answer: the sync itself runs in the provider's background.
• Sync failed: the sync handler raised. The provider records the exception's type in last_error and logs its type and where it was raised, never its message, which may hold a directory URL with credentials. Documents already in the store stay: a replace_source that did not complete changed nothing, so analyzers keep retrieving the last good snapshot.
• Knowledge Store unreachable, or its embedding backend (Ollama) down: the store answers 503 to writes, and KnowledgeClient raises SpotAPIError (httpx2 errors when it cannot connect). Let it raise: the sync fails as above.

The dashboard's Knowledge readiness banner aggregates these signals so operators see a single status line.

Packaging¶

1. Package the app as a Docker image; it listens on the manifest's port (8000 by default).
2. Label the image with the labels the manifest renders (Manifest.labels()), so the catalog reads what GET /manifest answers:
3. spot.plugin.id, spot.plugin.version;
4. spot.plugin.kind=context_provider;
5. spot.plugin.contract=2: the catalog refuses any other contract;
6. spot.plugin.egress: none, indicators or content. An image without it counts as content, and installing it needs an admin's consent;
7. spot.plugin.settings: each setting as an object with a type (string, integer, number, boolean, url, secret) and optionally required, default, description, secret and enum. Mark a secret setting (a bind password, an API key) secret so SPOT stores it encrypted and delivers it as the file /run/secrets/<NAME>. A field that does not parse is left out, with a schema warning in the catalog.
8. Publish it where a plugin source reaches it, and install it from the catalog, or run it yourself (see Manual plugins) and register it via spot.yaml or the dashboard.

The installer gives the container its token, its secrets as files, and SPOT_KNOWLEDGE_URL and SPOT_PLUGIN_ID in its environment. A provider built on SDK 1 (contract 1) is refused by the catalog: port it to SDK 2 (Migrating from 1.x).

Providers can be written in any language as long as they serve the routes of the context provider's document and call the knowledge service as its document describes, with the token as Bearer.