Corpus Grounder

---
name: corpus-grounder
description: >
  Answer questions grounded in a pile of documents too big to skim. Use when the user
  drops a folder of PDFs, notes, transcripts, or docs and asks questions about them, or
  says "search these", "what do these say about X", "find where they discuss Y". Builds
  a local hybrid keyword + vector index in the sandbox and answers from retrieved,
  cited chunks. Do NOT use when the content already fits comfortably in context (just
  read it) or for a single short file.
---

# Corpus Grounder

Once a document pile is bigger than one context window, reading "as much as fits" gives
confidently wrong answers because the relevant passage was in the part that got dropped.
This skill builds a real retrieval layer first, then answers only from what it retrieves.

Distilled from two patterns: hybrid-fts5-vector-search (combine BM25 full-text via
SQLite FTS5 with cosine similarity from sqlite-vec, then rank) and openie-graph-indexing
(extract entity-relation triples into a graph plus a document-to-entity index).
Provenance: docs-mcp-rs and OpenIE-style indexers.

## Procedure

### 1. Ingest and chunk
Extract text from each file (PDF, docx, md, txt, transcripts). Chunk into passages of a
few hundred tokens with slight overlap, preserving source path and location so every
chunk can be cited back to its origin.

### 2. Build a hybrid index in the sandbox
Create a SQLite database with an FTS5 table for BM25 keyword search and a sqlite-vec
table for embeddings. Embed each chunk and store both representations. Optionally, for
corpora where relationships matter (who did what, which entity connects to which),
extract entity-relation triples into a small graph plus a document-to-entity map.

### 3. Retrieve hybrid, then answer grounded
For each question, run both the BM25 query and the vector query, merge and re-rank the
results (reciprocal-rank fusion or weighted score), and pull the top chunks. Answer
ONLY from retrieved chunks. Cite the source path and location for each claim. If the
retrieved context does not contain the answer, say so rather than filling from prior
knowledge.

### 4. Persist for the session
Keep the index on disk so follow-up questions reuse it instead of re-embedding. Report
corpus size, chunk count, and index location so the user knows what was searched.

## Acceptance check
Ask a question whose answer lives in exactly one document; the skill returns it with a
citation pointing at that document. Ask a question the corpus does not answer; the skill
declines rather than confabulating. Keyword-only and semantic-only queries both surface
their obvious targets.