Knowledge Store¶

The SPOT Knowledge Store is the platform's RAG (retrieval-augmented generation) layer. Context providers deposit tagged company documents; analyzers fetch them on demand by semantic similarity + tag expression. Neither side knows the other exists ; they share only the tag vocabulary and the document vectors.

Two phases¶

Ingestion (out-of-band, scheduled) ; the api-gateway runs a cron-style scheduler that calls each enabled context provider's POST /internal/sync on a cadence declared in spot.yaml, with the provider's own token. The provider reads from its source system and writes KnowledgeDocuments to the store: replace_source for a full snapshot, bulk_upsert for changes.

Consumption (per-email, on-demand) ; an analyzer calls KnowledgeClient.for_analysis(request).fetch(...) whenever it needs context during analysis. The store embeds the query, filters by tag expression, returns top-K documents by cosine similarity.

Data model¶

Every document has the same shape:

class KnowledgeDocument(BaseModel):
    id: str                    # stable id, unique among the source's documents
    content: str               # text payload an LLM or analyzer reads
    tags: list[str] = []       # AND/OR-filterable at query time
    metadata: dict = {}        # structured fields (email, title, url, ...)
    source: str = ""           # the provider that wrote it
    updated_at: datetime | None = None
    expires_at: datetime | None = None   # optional TTL, cleaned up hourly
    score: float | None = None # populated on query results, 0.0 – 1.0

A document is identified by its source and its id. A provider chooses ids unique among its own documents: two providers may use the same id, and each has its own document under it. The source is the provider's plugin id; a provider leaves it empty and the store sets it.

A document's id appears in URL paths and in the logs (the access line of each view or deletion), so a provider never puts personal data in it, an email address or a name: it uses a stable key or a hash, and keeps such data in content or metadata.

tags is the single categorisation axis ; there is no type field. Use the shared-vocabulary constants from spot_sdk.knowledge_tags (EMPLOYEE, WIKI_PAGE, POLICY, EXECUTIVE, FINANCE, ...); custom tags like acme:jira_issue are fine.

Tag expressions¶

Queries filter on a tiny mini-language:

+ (AND) binds tighter than | (OR). No parentheses. Tokens must match [a-z0-9][a-z0-9_\-:]* ; anything else is rejected, so the expression is always safe to splice into SQL.

Authentication¶

Every route of the store except /health and /readiness needs a credential:

• the internal key (X-Internal-API-Key), for core services: whoever sends it is core, so it must not reach a plugin;
• a plugin token (Authorization: Bearer spot_pt_...) with knowledge:read for reads (/query, /stats, /sources, /documents, GET /sources/{source}/documents/{id}) or knowledge:write for writes (/upsert, /bulk-upsert, PUT /sources/{source}/documents, DELETE /sources/{source}/documents/{id}).

Analyzers hold knowledge:read, context providers knowledge:write. A provider writes and deletes only its own source's documents: an empty source takes its plugin id, and a write naming another source, or a deletion of another source's document, is refused with 403. A write never touches another source's document, whatever id it uses. Only the provider whose id is the source replaces a source's documents: core and any other plugin are refused with 403. Core names each document's source: a core write of a document without one answers 422. A plugin reads its token from /run/secrets/spot_plugin_token; see Context Providers for the details and for manual plugins.

Ingestion side¶

Ingestion is the provider's responsibility: each provider's sync handler builds KnowledgeDocuments and writes them with KnowledgeClient.replace_source(...), which makes them the provider's whole set in one transaction, or bulk_upsert(...). A re-sync leaves a document whose content, tags, metadata and expiry are unchanged as it is, without embedding it again: updated_at is the time a document last changed, not the time of the last sync (the provider's last_sync_at). The Context Providers guide is the canonical home for the routes a provider serves, the SDK code recipe, packaging, and the sync_schedule / manual-trigger lifecycle. This page focuses on what ends up in the store and how consumers read it back.

For long wikis or policy docs, providers split into chunks with spot_sdk.knowledge.chunk_text(text, max_chars=2000, overlap=200) and upsert each chunk with metadata["parent_id"] pointing at the source doc ; embeddings are better per-chunk and retrieval is finer-grained.

Querying from an analyzer¶

from spot_sdk import (
    AnalyzeRequest,
    AnalyzerVerdict,
    KnowledgeClient,
    PluginSettings,
    Signal,
    VerdictClass,
)


async def analyze(request: AnalyzeRequest, settings: PluginSettings) -> AnalyzerVerdict:
    sender = request.email.from_
    if sender is None:
        return AnalyzerVerdict(
            verdict=VerdictClass.UNKNOWN, score=0, confidence=0, summary="no sender"
        )
    kb = KnowledgeClient.for_analysis(request)

    # "Does this email claim to be from an executive?"
    hits = await kb.fetch(
        tags="employee+executive|director",
        text=sender.display_name or sender.address,
        top_k=1,
    )
    if hits and hits[0].metadata.get("email") != sender.address:
        return AnalyzerVerdict(
            verdict=VerdictClass.SUSPICIOUS,
            score=70,
            confidence=80,
            signals=[
                Signal(
                    name="exec.display_name_spoof",
                    weight=60,
                    evidence=f"{sender.display_name} is {hits[0].id}",
                    location="header:From",
                )
            ],
        )
    ...

Use KnowledgeClient.for_analysis(request): it reads the URL from SPOT_KNOWLEDGE_URL and the token from /run/secrets/spot_plugin_token, and applies the stage's retrieval_limits, which the orchestrator sends in request.retrieval_limits.

A retrieved document can go into a Signal's evidence, so the analyst UI can explain why.

Retrieval limits (operator policy)¶

Workflow stages can declare caps in spot.yaml:

stages:
  - name: context-checks
    retrieval_limits:
      max_top_k: 10          # cap top_k requested by any analyzer here
      min_score_floor: 0.3   # raise the floor even if analyzer asked lower
    analyzers:
      - id: analyzer-llm
      - id: analyzer-rules

The orchestrator sends them in each AnalyzeRequest of the stage (retrieval_limits) and the SDK enforces them transparently ; caps only, never raises what the analyzer asked for.

Provider ingestion schedule¶

Each context provider declares its own sync_schedule (cron) and sync_timeout_ms on its spot.yaml entry, and the api-gateway scheduler ticks it from there. The supported cron grammar, the manual admin-trigger endpoints, and the dashboard's "Sync now" button are all documented in the Context Providers guide.

Stats¶

GET /api/v1/knowledge/stats returns:

{
  "total": 1234,
  "per_tag": {"employee": 500, "wiki_page": 700, "policy": 34}
}

Embedding backend (Ollama)¶

Embeddings are computed by Ollama (configurable via the Embedder protocol if you need a different backend). The knowledge service expects it at OLLAMA_URL (default http://ollama:11434) with the model named in EMBEDDING_MODEL (default bge-m3).

Operators have two options:

1. External Ollama. Run Ollama anywhere reachable from the spot_spot-network Docker network and set OLLAMA_URL in /opt/spot/deploy/.env. An installed analyzer-llm calls Ollama too, through the egress gateway, by host name (not an address), through the operator's proxy unless noProxy lists it, so it must be reachable from there as well. The knowledge service calls OLLAMA_URL directly and ignores HTTP_PROXY / HTTPS_PROXY, so an external Ollama must be reachable without an egress proxy.
2. Bundled side-car. Add ollama to COMPOSE_PROFILES in /opt/spot/deploy/.env ; the deployment files ship an ollama + ollama-init service pair under that profile. ollama-init runs once on first boot, pulls EMBEDDING_MODEL (and anything in OLLAMA_EXTRA_MODELS, space-separated), and exits.

Readiness¶

The knowledge service exposes two endpoints for this:

• GET /health ; liveness only (returns {"status": "ok"}).
• GET /readiness ; probes Ollama with a lightweight GET /api/version and returns:

{
  "status": "ok",   // or "degraded"
  "embedding": {
    "url": "http://ollama:11434",
    "model": "bge-m3",
    "reachable": true
  }
}

The api-gateway proxy at GET /api/v1/knowledge/readiness (any role) combines that probe with installed-context-provider state so the dashboard can render a single banner ; see the next section.

Failure mode¶

When Ollama is unreachable, any request that touches embeddings (/upsert, /bulk-upsert, /query) returns 503 with a payload like:

{
  "detail": "Embedding backend unreachable at http://ollama:11434. Ensure Ollama is running and reachable, and that the 'bge-m3' model has been pulled.",
  "embedding": {
    "url": "http://ollama:11434",
    "model": "bge-m3",
    "reachable": false
  }
}

Raw httpx2.ConnectErrors are caught by a FastAPI exception handler and translated into that shape ; callers never see an opaque 500.

Web dashboard¶

The Knowledge Store is managed from the Knowledge top-level page, which every signed-in user reads; admins also delete documents. Three surfaces:

/knowledge ; browse & search¶

• Readiness banner: shown (amber) whenever any required component is missing ; knowledge service unreachable, embedding backend down, or zero enabled context providers. Each row explains how to fix the specific gap (point OLLAMA_URL somewhere real or enable the bundled side-car; install a context_provider plugin). Disappears once all three components report green.
• Summary cards: total docs, distinct sources, top tags (each tag chip is a filter link).
• Semantic search panel: text + optional tag expression + top_k + min_score. Mimics an analyzer's retrieval and lists ranked hits with cosine-similarity scores.
• Browse table: paginated list with filters for tag expression, source, and a substring match on id/content. Admin users can delete a document inline (confirmation required).

/knowledge/{source}/{id} ; document inspector¶

Metadata grid, clickable tag chips (each filters back to /knowledge?tags=...), raw content, and pretty-printed JSON metadata. Admin users get a Delete button.

Context-provider integration¶

• /plugins/context_provider/{id} shows sync info (last run, next run, doc count, last error) and links to the Knowledge Store pre-filtered by source=<provider_id>.
• /config/plugin/context_provider/{id} has a Scheduling section to edit the sync_schedule cron and sync_timeout_ms from the dashboard. Saving there reconciles the running scheduler immediately ; no process restart.

All dashboard operations go through the api-gateway proxies at /api/v1/knowledge/* (documents list, query, delete, stats, sources); the dashboard never talks to the knowledge service directly.

Architecture (short)¶

• services/knowledge ; in-core microservice (port 8000). FastAPI, asyncpg, pgvector. Embedding via Ollama (pluggable via an Embedder protocol). Redis caches the embed lookups and the full-query results.
• Postgres: knowledge_documents table with HNSW index on embedding vector(1024), GIN index on tags, partial index on expires_at. Alembic migration 008 creates it; migration 017 keys it by (source, id).
• api-gateway: KnowledgeScheduler (one asyncio task per provider, calling /internal/sync with the provider's token) and the proxies at /api/v1/knowledge/* and /api/v1/plugins/context_provider/{id}/sync.
• SDK: KnowledgeClient, KnowledgeDocument, KnowledgeTag, chunk_text, content_hash, plus FakeKnowledgeClient for plugin unit tests.

Not in scope¶

• Multi-tenancy / per-org isolation ; global store.
• Replacing pgvector with Qdrant / Weaviate ; KnowledgeStore keeps the backend abstract enough to swap later.
• LLM-driven adaptive retrieval ; the analyzer decides what to query.