Mothership

Knowledge

Apps' documents are extracted and indexed privately, searched within the documents chosen, and answered with citations. Source permissions are checked again on every read.

A document's life

  1. Upload and queue — the app uploads a file and queues ingestion (queued by default since SDK 1.1). The gateway records the job and answers at once.
  2. Extract and index — the Knowledge worker extracts text (PDF, Office, OCR'd images, text) and indexes it, with vectors when an embedding model is set.
  3. Search and cite — the app searches chosen collections and documents; results carry citable excerpts.
  4. Delete — deleting a document removes its searchable content and source file together.
ts
const file = await ctx.files.upload(selectedFile);
const queued = await ctx.knowledge.ingest({
  collectionId: "review-documents", fileId: file.fileId,
  title: file.name, scope: "user_private",
}, { idempotencyKey: `ingest:${file.fileId}` });
await ctx.jobs.wait(queued.jobId);
const results = await ctx.knowledge.search({
  collections: ["review-documents"], documentIds: [queued.documentId],
  query: "What requires approval?", topK: 6,
});
The Document Review app searches only the chosen documents, and gives no answer without evidence.The Document Review app searches only the chosen documents, and gives no answer without evidence.
The Document Review app searches only the chosen documents, and gives no answer without evidence.

Sharing

Scope Who can see it
user_private (default) The uploader
space_shared Members of the team space the host selected (never an ID the app sends)
org_shared Members of the same company, app and environment

The source file stays private even when its extracted text is shared. Every list, search, citation and download checks the collection declaration and current membership again.

How it searches, and its limits

  • PostgreSQL: a GIN full-text index shortlists candidates; with embeddings configured, a bounded set is re-ranked by meaning. No unbounded vector scan, no ANN vector database.
  • SQLite: a bounded lexical search, for development.
  • Results report the mode used and whether they were truncated. After changing the embedding provider or model, re-index before old vectors join semantic ranking.
  • Documents are processed in a separate process with CPU, memory, page, pixel, output-size and time limits: by default 25 MiB in, 25 pages, 120 seconds. Decrypt PDFs before uploading.

Workers

make dev (development) and the production Compose workers service run the Knowledge worker. On their own:

bash
.venv/bin/python backend/manage.py knowledge_worker
.venv/bin/python backend/manage.py document_worker   # PDF and OCR conversion
Esc