Indexing

import {
  chunker,
  corpus,
  indexer,
  indexingPipeline,
  transform,
} from "@use-crux/core/indexing";
import type {
  CruxDocument,
  CruxChunk,
  CruxParentChunk,
  ChunkingOptions,
  IndexingPipeline,
  DocumentTransform,
  ChunkTransform,
  Chunker,
  IndexResult,
  CorpusSyncResult,
  Indexer,
  Corpus,
} from "@use-crux/core/indexing";

Overview

indexer() owns write-time document preparation:

• document -> chunks
• versioned document transforms, chunkers, and chunk transforms
• dense and sparse embedding at write time
• generation-aware replacement
• RecordStore and VectorStore writes
• source replacement and clearing

It does not load files or URLs. That belongs to @use-crux/ingest.

The easiest way to think about an indexer is that it turns source material into a retrieval corpus. If retrieval is the read API, indexing is the write API.

Copy-Paste Patterns

Minimal document indexing

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
});

await docsIndexer.indexDocuments([
  {
    namespace: "product-docs",
    sourceId: "intro.md",
    title: "Intro",
    content: "# Intro\n\nWelcome to the docs.",
  },
]);

Incremental corpus sync

const docsCorpus = corpus({
  id: "docs",
  namespace: "product-docs",
  records,
  indexer: docsIndexer,
});

await docsCorpus.sync(source.load(), {
  sourceSet: "complete",
  stale: "delete",
});

Hybrid indexing

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  sparse,
});

Pipeline with cache

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  cache: true,
  pipeline: indexingPipeline({
    documents: [
      transform.document({
        name: "normalize",
        version: "1",
        run: (document) => ({
          ...document,
          content: document.content.trim(),
        }),
      }),
    ],
    chunker: chunker.structured({ maxChars: 1200 }),
  }),
});

Parent-child indexing

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  pipeline: indexingPipeline({
    chunker: chunker.parentChild({
      parentMaxChars: 6000,
      childMaxChars: 900,
      childOverlapChars: 120,
    }),
  }),
});

Semantic chunking

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  pipeline: indexingPipeline({
    chunker: chunker.semantic({
      strategy: "embedding",
      dense,
      minChars: 300,
      maxChars: 1200,
      similarityThreshold: 0.76,
    }),
  }),
});

Signature

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
  sparse,
  cache: true,
  pipeline: indexingPipeline({
    documents: [
      transform.document({
        name: "normalize",
        version: "1",
        run: (doc) => ({ ...doc, content: doc.content.trim() }),
      }),
    ],
    chunker: chunker.structured({ maxChars: 1200 }),
  }),
});

Canonical models

CruxDocument

type CruxDocument = {
  namespace: string;
  sourceId: string;
  content: string;
  parts?: CruxIngestPart[];
  title?: string;
  metadata?: Record<string, unknown>;
  warnings?: CruxIngestWarning[];
};

CruxChunk

type CruxChunk = {
  namespace: string;
  sourceId: string;
  chunkId: string;
  generationId?: string;
  active?: boolean;
  ordinal: number;
  content: string;
  metadata: Record<string, unknown>;
  provenance?: {
    partIds?: string[];
    pages?: number[];
    sheets?: string[];
    tables?: string[];
    jsonPaths?: string[];
    sourceSpans?: Array<{ start: number; end: number; partId?: string }>;
    confidence?: "exact" | "derived";
  };
  parent?: {
    parentId?: string;
    key?: string;
    title?: string;
    summary?: string;
  };
};

Stored chunks also carry timestamps plus optional embedding and sparseEmbedding. Parent/child pipelines may also write CruxParentChunk records. Parent records are not searchable by default; retrievers filter for active child chunks.

These models are intentionally plain. The point is not to hide chunking from users; it is to give them a stable, explicit shape that both indexing and retrieval can agree on.

parts is accepted from @use-crux/ingest. The default chunker still chunks the derived content, but it preserves coarse provenance on produced chunks so downstream UI and custom retrieval logic can recover source pages, sheets, tables, and part IDs.

Pipeline API

indexingPipeline() is the production customization point. It keeps the source processing order explicit:

const pipeline = indexingPipeline({
  documents: [
    transform.document({
      name: "strip-drafts",
      version: "1",
      run(document) {
        return {
          ...document,
          content: document.content.replace(/<!-- draft -->/g, ""),
        };
      },
    }),
  ],
  chunker: chunker.parentChild({
    parentMaxChars: 6000,
    childMaxChars: 900,
    childOverlapChars: 120,
  }),
  chunks: [
    transform.chunk({
      name: "tag-corpus",
      version: "1",
      run(chunks) {
        return chunks.map((chunk) => ({
          ...chunk,
          metadata: { ...chunk.metadata, corpus: "product-docs" },
        }));
      },
    }),
  ],
});

Document transforms run before chunking. Chunk transforms run after chunking and before embedding/writes. Every transform and chunker has a name, version, and fingerprintable options so corpus.sync() can detect when unchanged source text should still be reindexed.

Built-in chunkers:

chunker.text(options?)

Use this for plain text sources when you do not need structured provenance beyond source spans.

indexingPipeline({
  chunker: chunker.text({
    maxChars: 1200,
    overlapChars: 150,
  }),
});

Behavior:

• Splits paragraph-first.
• Falls back to sentence splitting for oversized paragraphs.
• Falls back to hard character windows for very long sentences.
• Produces stable chunk_<hash> IDs from source id, ordinal, content, and provenance.

chunker.structured(options?)

Use this as the default for loaded files, URLs, PDFs, CSV, JSON, DOCX, and XLSX because it preserves provenance from document.parts.

indexingPipeline({
  chunker: chunker.structured({
    maxChars: 1200,
    overlapChars: 150,
    tableRowsPerChunk: 25,
  }),
});

Behavior:

• If parts are present, chunks each part according to its kind.
• Text and page parts use text splitting with provenance.
• Table parts chunk rows and repeat headers.
• Sheet and JSON parts become direct chunks.
• Produced chunks preserve partIds, pages, sheets, tables, jsonPaths, and sourceSpans when available.

chunker.parentChild(options?)

Use this when search should target small child chunks but rendering or compression should have access to larger parent context.

indexingPipeline({
  chunker: chunker.parentChild({
    parentMaxChars: 6000,
    childMaxChars: 900,
    childOverlapChars: 120,
  }),
});

Behavior:

• Builds parent records from structured chunks.
• Persists non-searchable parent records through the indexed read model.
• Persists active searchable child chunks through the same read model.
• Child chunks include parent refs and optional parent title.
• Store-backed retrievers search active child chunks only through the read-model boundary.
• expandParents({ records }) resolves parent refs through the indexed read model to enrich hits with parent content while preserving child identity and score.

chunker.semantic(options)

Use this when chunk boundaries should follow meaning rather than just size. Semantic chunking currently operates on document.content, not on individual ingest parts.

indexingPipeline({
  chunker: chunker.semantic({
    strategy: "embedding",
    dense,
    minChars: 300,
    maxChars: 1200,
    similarityThreshold: 0.76,
  }),
});

Embedding strategy:

chunker.semantic({
  strategy: "embedding",
  dense,
  minChars: 300,
  maxChars: 1200,
  similarityThreshold: 0.76,
});

Model/custom strategy:

chunker.semantic({
  strategy: "custom",
  minChars: 300,
  maxChars: 1200,
  async segment({ segments }) {
    return segments
      .filter((segment) => segment.text.startsWith("# "))
      .map((segment) => ({
        start: segment.start,
        end: segment.end,
        reason: "heading",
        confidence: 0.9,
      }));
  },
});

Hybrid strategy:

chunker.semantic({
  strategy: "hybrid",
  dense,
  minChars: 300,
  maxChars: 1200,
  similarityThreshold: 0.76,
  async segment(input, ctx) {
    return detectDomainBoundaries(input.document.content, ctx);
  },
});

Behavior:

• Sentence-segments document.content.
• embedding derives boundaries from adjacent sentence embedding similarity.
• model and custom use the provided segment function.
• hybrid uses segment first and falls back to embedding boundaries when no boundaries are returned.
• Chunks include metadata.semanticReason and optional metadata.semanticConfidence.
• Chunks include source-span provenance over document.content.

Custom chunkers

Write a custom chunker when your domain already has better boundaries than generic text splitting.

const markdownSections: Chunker = {
  _tag: "Chunker",
  name: "markdown-sections",
  version: "1",
  fingerprint: () => "markdown-sections:v1",
  chunkDocument(document) {
    return {
      chunks: splitByHeading(document.content).map((section, ordinal) => ({
        namespace: document.namespace,
        sourceId: document.sourceId,
        chunkId: section.slug,
        ordinal,
        content: section.content,
        metadata: {
          ...document.metadata,
          heading: section.heading,
        },
      })),
    };
  },
};

const pipeline = indexingPipeline({ chunker: markdownSections });

Custom chunkers must preserve the document namespace and source id, produce stable chunk ids, and return parent records only when later retrieval should be able to expand from child hits to parent context.

Stage cache

Set cache: true on the indexer to cache expensive document transforms, chunkers, chunk transforms, and final dense/sparse source bundles in the same record store. Per-call options can override behavior:

await docsIndexer.indexDocuments(docs, { cache: "refresh" });
await docsIndexer.indexDocuments(docs, { cache: "bypass" });
await docsIndexer.indexChunks(chunks, { cache: "refresh" });

Preparation-stage keys include source and stage identity. Embedding-stage keys include source id, vector kind, embedding fingerprint, and ordered chunk-content hashes. Cache entries are reused only after strict count and vector-shape validation. Cached embeddings produce SourceStageRecord entries with kind: "embedding", embeddingKind, and the cache outcome.

Indexer API

chunk(documents, options?)

Run chunking and transforms without writing.

Default chunking is character-based:

• maxChars = 1200
• overlapChars = 150
• paragraph-first, sentence fallback, then hard split

indexDocuments(documents, options?)

await indexer.indexDocuments(docs, {
  replaceSources: true,
  chunking: { maxChars: 1000, overlapChars: 120 },
});

Defaults:

• replaceSources = true

This is the direct document-write path.

Successful writes return IndexResult with the exact indexing.pipeline operation pair in _meta. A dry run returns IndexDryRunResult with the same ownership rule for its own pipeline invocation.

Use it for tests, one-off writes, demos, and deliberately manual updates where you want Crux to own chunking and replacement semantics without source-ledger tracking. For repeated ingestion jobs, prefer corpus().sync().

indexChunks(chunks, options?)

await indexer.indexChunks(chunks, {
  replaceSources: false,
  cache: "readwrite",
});

Defaults:

• replaceSources = false

Use this when chunking is handled upstream.

Use it when chunk boundaries are already part of your domain logic and Crux should not infer them again.

deleteSource(sourceId)

Deletes all stored chunks for one namespace/source pair.

clear()

Deletes all stored chunks owned by the indexer namespace.

fingerprint(options?)

Returns a stable hash of the indexing pipeline. The fingerprint includes indexer identity, namespace, index version, chunking configuration, embedding metadata, and versioned chunk transforms. corpus() stores this hash per source so unchanged content can still be reindexed when the indexing pipeline changes meaningfully.

Dry runs

indexDocuments() and indexChunks() accept { dryRun: true }.

const plan = await docsIndexer.indexDocuments(docs, { dryRun: true });

Dry runs prepare chunks and embeddings but do not delete or write indexed chunks or corpus source records. With indexer caching enabled, they may read and write stage-cache entries so a later real run can reuse the completed work. They return chunk and embedding counts so callers can preview work before committing it.

Dry runs also return prepared parent records and pipeline stages when the configured pipeline produces them.

Chunk arrays, stored source records, progress items, and numeric deleteSource()/clear() results remain ordinary domain values without result metadata. This content-indexing contract does not change Project Index compiler facts, snapshots, HTTP read models, or caches. See Operation Result Correlation.

Corpus API

corpus() wraps an indexer with a source ledger for repeated sync jobs.

const docsCorpus = corpus({
  id: "docs",
  namespace: "product-docs",
  records,
  indexer: docsIndexer,
});

const result = await docsCorpus.sync(docs, {
  sourceSet: "complete",
  stale: "delete",
});

sync(documents, options?)

await docsCorpus.sync(loader.load(), {
  mode: "replaceChanged",
  sourceSet: "partial",
  stale: "keep",
  dryRun: false,
});

documents may be an array or async iterable of CruxDocument. It may also be an async iterable of ingest load results from @use-crux/ingest. Failed source results are recorded in the corpus ledger and do not stop the whole sync.

mode defaults to replaceChanged. It indexes new sources, reindexes changed sources, and skips unchanged sources. appendOnly indexes new sources but skips changed existing sources.

sourceSet defaults to partial. Use complete when the input represents the full authoritative source list. stale: 'delete' requires a complete source set and throws otherwise.

dryRun computes the same plan and chunk counts without writing chunks or mutating the source ledger.

type CorpusSyncResult = {
  corpusId: string;
  namespace: string;
  mode: "replaceChanged" | "appendOnly";
  sourceSet: "partial" | "complete";
  stalePolicy: "keep" | "delete";
  dryRun: boolean;
  added: number;
  changed: number;
  unchanged: number;
  stale: number;
  skipped: number;
  deleted: number;
  failed: number;
  chunkCount: number;
  durationMs: number;
  sources: Array<{
    sourceId: string;
    action:
      | "added"
      | "changed"
      | "unchanged"
      | "skipped"
      | "failed"
      | "stale"
      | "deleted";
    reason?: string;
    chunkCount?: number;
    stages?: SourceStageRecord[];
    error?: { message: string; stack?: string };
  }>;
  _meta: OperationResultMeta;
};

The pair identifies the outer corpus.sync operation; per-source progress and ledger records remain unobserved domain data.

Source ledger methods

await docsCorpus.getSource("guide.md");
await docsCorpus.listSources({ includeDeleted: false });
await docsCorpus.deleteSource("guide.md");
await docsCorpus.clearSources();

The ledger stores source hashes, index hashes, status, chunk counts, timestamps, metadata, recent errors, and the latest pipeline stage records. It is intentionally separate from chunk records so operational sync state does not pollute retrieval results.

type SourceStageRecord = {
  name: string;
  kind?:
    | "parser"
    | "document-transform"
    | "chunker"
    | "chunk-transform"
    | "embedding"
    | "promotion"
    | "sync";
  version?: string;
  status: "pending" | "success" | "failed" | "skipped";
  cache?: "hit" | "miss" | "write" | "refresh" | "bypass";
  inputHash?: string;
  outputHash?: string;
  durationMs?: number;
  chunkCount?: number;
  parentCount?: number;
  error?: { message: string; stack?: string };
  updatedAt: number;
};

Change detection

corpus.sync() decides whether a source needs reindexing by comparing two hashes stored in the ledger:

• the source hash, computed from source content plus stable metadata
• the index hash, the indexer pipeline fingerprint recorded at last index time

Changing source content reindexes the source. Changing stable metadata reindexes the source. Changing the indexer fingerprint also reindexes the source, so versioned pipeline changes can intentionally rebuild unchanged content.

Volatile metadata fields should not count as changes. Exclude them through hash.excludeMetadata:

const docsCorpus = corpus({
  id: "docs",
  namespace: "product-docs",
  records,
  indexer: docsIndexer,
  hash: {
    excludeMetadata: ["mtimeMs", "lastFetchedAt"],
  },
});

Replacement model

indexDocuments() uses generation-aware replacement. New chunks and parent records are written with a fresh generationId and active: true. Only after the new generation succeeds does Crux mark previous records for that source inactive.

Store-backed retrievers query through the indexed read model, which applies the active child-chunk contract and keeps old generations and parent records out of normal query results while preserving enough history for debugging and rollback-oriented tooling.

Chunking invariants

The built-in chunker optimizes for predictability over sophistication.

It is intended to be:

• easy to understand
• easy to replace
• good enough for prose and markdown

If your corpus needs domain-aware chunking, provide a custom chunker through indexingPipeline({ chunker }).

That tradeoff is deliberate. A predictable default is easier to debug and easier to replace than a "smart" default that users cannot reason about.

Embedding behavior

Dense indexing uses:

• dense.embedMany()

Sparse indexing uses:

• sparse.embedMany()

Hybrid indexing uses both.

Dense indexing accepts text and any media modality declared by the embedding. CruxDocument.asset is single-media shorthand; explicit kind: "media" parts carry an asset, optional caption, inferred or explicit modality, and page/time source location. Each media part produces one chunk and one dense vector. Sparse embedding skips media chunks.

When storage includes an AssetStore, media is materialized before provider execution and persisted records retain only its AssetRef, MIME type, source location, and allowlisted scalar attribution. Bytes and provider locators are never written to record/vector metadata or pipeline caches. Dry runs perform no asset, cache, record, vector, or namespace-space writes.

Before any vector mutation, indexing compares the configured dense space with the namespace record. EmbeddingSpaceMismatchError requires clear() plus a full reindex, or a new namespace. deleteSource() preserves the space record.

The important contract is that indexing batches by chunk content rather than making one provider call per chunk.

That keeps indexing aligned with the batching, telemetry, and cost-accounting model from embedding().

With cache: true, Crux stores one ordered vector bundle per source and kind. Dense runs complete before sparse runs. Hits scatter into the current invocation order; misses across sources flatten into one embedMany() call for that kind. Embeddings without fingerprint are always recomputed and never receive a stage cache outcome.

Hooks emitted

Indexing operations emit:

• index:start
• index:end with optional pipeline stages

Corpus sync emits:

• corpus:sync:start
• corpus:source:added | changed | unchanged | skipped | failed | stale | deleted with optional source stages
• corpus:sync:end

The payload distinguishes operations such as:

• indexDocuments
• indexChunks
• deleteSource
• clear

Intended usage

Reach for indexer() when you want Crux to standardize:

• chunk shape
• source replacement semantics
• write-time embedding
• cleanup operations

If you already own all of those concerns elsewhere, you can skip the indexer and write directly to your store. The indexer is a public primitive, not a mandatory layer.

That is an important boundary for advanced users. Crux should make the common path clean without making the explicit path awkward.

Related

• Guide: Retrieval Architecture
• Guide: Embeddings
• Guide: Ingestion
• Guide: Indexing Documents
• Guide: Corpus Sync
• Guide: Querying
• Reference: @use-crux/core/retrieval
• Reference: @use-crux/ingest