Crux
GuidesRetrieval & RAG

Indexing Documents

Turn documents into chunked, embedded retrieval records.

Before a retriever can answer anything, your source material has to be written somewhere searchable: split into chunks, embedded, and stored. Indexing is that write side of retrieval. Loaders produce documents, the indexer turns those documents into active chunks, and retrievers query those chunks later.

loader.load() -> indexer.indexDocuments() -> chunker -> embedding() -> RecordStore + VectorStore

In production you usually call the indexer through corpus.sync() so Crux can skip unchanged sources and delete stale ones safely.

The shortest useful version looks like this. The embeddings are defined once and shared by the indexer and retriever.

import { embedding } from "@use-crux/core/embedding";
import { corpus, indexer } from "@use-crux/core/indexing";
import {
  inMemoryRecordStore,
  inMemoryVectorStore,
} from "@use-crux/core/storage";
import { filesSource } from "@use-crux/ingest/files";

const dense = embedding({
  kind: "dense",
  name: "text-embedding-3-small",
  dimensions: 1536,
  maxInputTokens: 8191,
  embed: async (inputs) => ({
    embeddings: await embedDense(inputs.map((input) => {
      if (input.type !== "text") throw new Error("text-only provider invariant");
      return input.text;
    })),
  }),
});
const records = inMemoryRecordStore();
const vectors = inMemoryVectorStore();

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
});

const docsCorpus = corpus({
  id: "docs",
  namespace: "product-docs",
  records,
  indexer: docsIndexer,
});

const source = filesSource(
  { directory: "./docs", recursive: true },
  { namespace: "product-docs" },
);

await docsCorpus.sync(source.load(), {
  sourceSet: "complete",
  stale: "delete",
});

That is the normal product path. corpus.sync() skips unchanged sources, reindexes changed sources, records failed sources, and deletes stale sources only when you say the input is complete.

Add sparse too when your retriever will run sparse or hybrid search:

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  sparse,
});

const docs = retriever({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  sparse,
  search: { mode: "hybrid" },
});

The rule is simple: index the vector types you want to query later.

Index Media Documents

A media document uses asset shorthand or explicit media parts. The dense embedding must declare the matching modality, and storage must provide an AssetStore when the source needs a durable retrievable ref:

await productsIndexer.indexDocuments([
  {
    namespace: "products",
    sourceId: "rex",
    title: "Rex",
    asset: dogPhoto,
  },
  {
    namespace: "products",
    sourceId: "setup-pdf",
    parts: [
      {
        id: "page:1",
        kind: "media",
        modality: "document",
        asset: manualPdf,
        sourceLocation: { type: "page", pageNumber: 1 },
      },
    ],
  },
]);

Each media part becomes one chunk and one vector. Split pages, image regions, audio intervals, or video intervals before indexing when they need independent hits. Direct indexDocuments() is the supported authored-media path; corpus source hashing for arbitrary authored Blob media remains deferred.

The record and vector stores receive only allowlisted attribution. Bytes remain in AssetStore; signed URLs, provider file ids, filenames, and base64 never enter indexed records, vector metadata, stage caches, or traces. Media sources bypass pre-embedding pipeline caches.

The first vector write records the namespace's dense embedding space. A later write with a different model, dimensions, normalization, modalities, or role tasks throws before any vector mutation. Use clear() and reindex, or choose a new namespace.

Add A Pipeline

Make the pipeline explicit once indexing becomes product infrastructure. This gives every transform and chunking choice a name, version, and fingerprint.

import {
  chunker,
  corpus,
  indexer,
  indexingPipeline,
  transform,
} from "@use-crux/core/indexing";

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  sparse,
  cache: true,
  pipeline: indexingPipeline({
    documents: [
      transform.document({
        name: "normalize-docs",
        version: "1",
        run(document) {
          return {
            ...document,
            content: document.content.trim(),
            metadata: {
              ...document.metadata,
              corpus: "product-docs",
            },
          };
        },
      }),
    ],
    chunker: chunker.structured({
      maxChars: 1200,
      overlapChars: 150,
      tableRowsPerChunk: 25,
    }),
  }),
});

Document transforms run before chunking. Chunk transforms run after chunking and before embedding:

const pipeline = indexingPipeline({
  documents: [
    transform.document({
      name: "strip-draft-comments",
      version: "1",
      run: (doc) => ({
        ...doc,
        content: doc.content.replace(/<!-- draft -->/g, ""),
      }),
    }),
  ],
  chunker: chunker.structured(),
  chunks: [
    transform.chunk({
      name: "tag-chunks",
      version: "1",
      run: (chunks) =>
        chunks.map((chunk) => ({
          ...chunk,
          metadata: { ...chunk.metadata, searchable: true },
        })),
    }),
  ],
});

Choose A Chunker

The default chunker.structured() works for most ingest output. Plain prose, parent-child layouts, semantic boundaries, and custom domain rules are also supported. See Chunkers for the full menu and configuration options.

indexingPipeline({
  chunker: chunker.structured({ maxChars: 1200, overlapChars: 150 }),
});

Cache Expensive Stages

Pipeline caching covers document preparation and final dense/sparse embedding bundles. Vectors are stored per source and vector kind, so an unchanged source can be written into a fresh generation without another provider call. A change to one source invalidates only that source; all misses are still batched once per embedding kind.

Enable stage caching on the indexer:

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  pipeline,
  cache: true,
});

Then control cache behavior per sync or write:

await docsCorpus.sync(source.load(), { cache: "readwrite" });
await docsCorpus.sync(source.load(), { cache: "refresh" });
await docsCorpus.sync(source.load(), { cache: "bypass" });

await docsIndexer.indexChunks(chunks, { cache: "refresh" });

Use readwrite for normal jobs, refresh when you want to recompute and replace cached stages, and bypass when you are debugging the pipeline itself.

Dry runs may read existing safe stage entries, but mutate nothing: no cache writes, asset puts, vector writes, indexed records, namespace-space records, or corpus ledger records. Cache hits and misses appear as embedding stages with embeddingKind: "dense" or "sparse".

Embedding-stage reuse requires a vector-semantic fingerprint. Built-in embedding() instances and provider helpers supply one; a hand-written structural embedding without fingerprint is recomputed on every run. See Embeddings for the per-text versus pipeline-cache decision.

Preview A Sync

Dry runs execute the same planning path but do not write chunks or source records.

const plan = await docsCorpus.sync(source.load(), {
  sourceSet: "complete",
  stale: "delete",
  dryRun: true,
});

console.log(plan.added, plan.changed, plan.deleted, plan.failed);

Dry-run document indexing also returns prepared chunks and parent records:

const preview = await docsIndexer.indexDocuments(docs, {
  dryRun: true,
});

console.log(preview.chunks);
console.log(preview.parents);
console.log(preview.stages);

Query What You Indexed

Retrievers query active child chunks. Parent records and inactive generations are filtered out automatically.

import { retriever } from "@use-crux/core/retrieval";

const docs = retriever({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  sparse,
  search: {
    mode: "hybrid",
    fusion: "dbsf",
    limit: 8,
  },
  context: {
    query: ({ question }) => question,
    limit: 5,
  },
});

const hits = await docs.retrieve("how do billing credits work?");

If your retriever uses a different id from the indexer that wrote the records, set indexerId so parent refs, chunk hydration, and cleanup all use the same indexed read model:

const docs = retriever({
  id: "docs-search",
  indexerId: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
});

Use it as prompt context:

const answer = prompt({
  id: "docs-answer",
  use: [docs],
  input: z.object({ question: z.string() }),
  system: "Answer using the retrieved docs. Cite sourceId/chunkId.",
  prompt: ({ input }) => input.question,
});

Or expose it as an agent tool:

const searchableDocs = retriever({
  id: "docs-tools",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  sparse,
  search: {
    mode: "hybrid",
    fusion: "dbsf",
    limit: 8,
  },
  inject: "tool",
  tools: { prefix: true },
});

const assistant = prompt({
  id: "assistant",
  use: [searchableDocs],
  system: "Use search before answering product questions.",
});

Inspect The Ledger

Every synced source records its latest pipeline stages:

const sourceRecord = await docsCorpus.getSource("docs/pricing.md");

console.table(
  sourceRecord?.stages?.map((stage) => ({
    name: stage.name,
    kind: stage.kind,
    cache: stage.cache,
    chunks: stage.chunkCount,
    parents: stage.parentCount,
    ms: stage.durationMs,
  })),
);

Those same stage records flow through devtools, CLI/TUI, and OTel. When a sync surprises you, start with the source ledger.

Direct Writes

Use indexDocuments() for tests, demos, and controlled one-off writes:

await docsIndexer.indexDocuments([
  {
    namespace: "product-docs",
    sourceId: "roadmap",
    title: "Roadmap",
    content: roadmapText,
    metadata: { section: "planning" },
  },
]);

Use indexChunks() only when you already own chunking upstream:

await docsIndexer.indexChunks([
  {
    namespace: "product-docs",
    sourceId: "roadmap",
    chunkId: "roadmap:overview",
    ordinal: 0,
    content: "Q2 roadmap overview...",
    metadata: { section: "overview" },
  },
]);

Chunkers

Pick how documents are split before embedding: structured, plain text, parent-child, semantic, or custom.

Retrieval Architecture

How indexing fits into the full stack.

Corpus Sync

Keep indexed sources up to date incrementally.

Querying

Search the chunks you indexed.

Pipelines

Use parent expansion and compression over indexed chunks.

Indexer Reference

Full `indexer()` API.

Ingesting Text, Files, and URLs

Use @use-crux/ingest to produce documents for indexing.

Chunkers

Choose how documents are split before embedding: structured, plain text, parent-child, semantic, or custom.

On this page

Index Media DocumentsAdd A PipelineChoose A ChunkerCache Expensive StagesPreview A SyncQuery What You IndexedInspect The LedgerDirect WritesRelated