Syncing a Corpus

Your docs re-index nightly. Without sync state, every run re-embeds thousands of unchanged files, and you pay embedding cost and index churn for content that did not change. corpus() fixes that: it wraps an indexer with a source ledger that remembers what was indexed, so repeated ingestion jobs only reprocess sources that actually changed.

indexer.indexDocuments() -> direct write
corpus.sync()            -> repeated ingestion job

Basic Sync

import { embedding } from "@use-crux/core/embedding";
import { corpus, indexer } from "@use-crux/core/indexing";
import {
  inMemoryRecordStore,
  inMemoryVectorStore,
} from "@use-crux/core/storage";
import { filesSource } from "@use-crux/ingest/files";

const dense = embedding({
  kind: "dense",
  name: "text-embedding-3-small",
  dimensions: 1536,
  maxInputTokens: 8191,
  embed: async (inputs) => ({
    embeddings: await embedDense(inputs.map((input) => {
      if (input.type !== "text") throw new Error("text-only provider invariant");
      return input.text;
    })),
  }),
});
const records = inMemoryRecordStore();
const vectors = inMemoryVectorStore();

const docsIndexer = indexer({
  id: "docs",
  namespace: "product-docs",
  records,
  vectors,
  dense,
});

const docsCorpus = corpus({
  id: "docs",
  namespace: "product-docs",
  records,
  indexer: docsIndexer,
});

const source = filesSource(
  { directory: "./docs", recursive: true },
  { namespace: "product-docs" },
);

const result = await docsCorpus.sync(source.load(), {
  sourceSet: "complete",
  stale: "delete",
});

console.log(result);

Add a sparse embedding to the indexer when your retriever runs sparse or hybrid search. See Embeddings for defining both.

Typical result:

const result = {
  added: 3,
  changed: 1,
  unchanged: 42,
  stale: 2,
  deleted: 2,
  failed: 0,
  chunkCount: 84,
};

Partial Updates

Use partial syncs for webhooks, CMS callbacks, file watchers, or any job that only sees some sources.

await docsCorpus.sync(changedFiles.load(), {
  sourceSet: "partial",
});

Partial syncs can add and update sources. They do not delete missing sources because Crux cannot know whether those sources are actually gone.

Complete Syncs

Use complete syncs when the input is the full authoritative source set.

await docsCorpus.sync(allDocs.load(), {
  sourceSet: "complete",
  stale: "delete",
});

stale: 'delete' requires sourceSet: 'complete'. This fails fast:

await docsCorpus.sync(changedFiles.load(), {
  sourceSet: "partial",
  stale: "delete",
});

Crux throws instead of guessing.

Dry Run

Preview a sync without writing chunks or source records.

const plan = await docsCorpus.sync(source.load(), {
  sourceSet: "complete",
  stale: "delete",
  dryRun: true,
});

console.table(
  plan.sources.map((source) => ({
    sourceId: source.sourceId,
    action: source.action,
    reason: source.reason,
    chunks: source.chunkCount,
  })),
);

Use this for admin previews, deploy checks, and tests.

Append-Only

Use append-only mode when existing sources should not be rewritten by routine syncs.

await docsCorpus.sync(source.load(), {
  mode: "appendOnly",
  sourceSet: "partial",
});

Changed existing sources are reported as skipped:

const sourceResult = {
  sourceId: "audit-log-2026-05-10",
  action: "skipped",
  reason: "appendOnly",
};

Pipeline Cache

Pipeline caching covers document/chunk preparation and final dense/sparse source bundles. An indexVersion-only reindex can therefore write a fresh generation without another embedding provider call when ordered chunk content is unchanged. See Embeddings for the cache-layer decision.

If the indexer has cache: true, sync can control the pipeline cache per run.

await docsCorpus.sync(source.load(), {
  cache: "readwrite",
});

Force recomputation:

await docsCorpus.sync(source.load(), {
  cache: "refresh",
});

In appendOnly mode, an embedding or index fingerprint change is reported as indexChanged and skipped without updating the source record. It will be reported and skipped again on later append-only syncs until a replace-mode sync accepts the new index identity.

Bypass cache while debugging:

await docsCorpus.sync(source.load(), {
  cache: "bypass",
});

Inspect A Source

const sourceRecord = await docsCorpus.getSource("guides/retrieval.md");

console.log(sourceRecord?.status);
console.log(sourceRecord?.chunkCount);
console.log(sourceRecord?.lastError);

The stage ledger shows what happened during the latest successful index:

for (const stage of sourceRecord?.stages ?? []) {
  console.log({
    name: stage.name,
    kind: stage.kind,
    cache: stage.cache,
    chunks: stage.chunkCount,
    parents: stage.parentCount,
    durationMs: stage.durationMs,
  });
}

Example output:

[
  {
    name: "normalize-docs",
    kind: "document-transform",
    cache: "hit",
    durationMs: 1,
  },
  {
    name: "structured",
    kind: "chunker",
    cache: "miss",
    chunks: 12,
    parents: 0,
    durationMs: 9,
  },
];

List Sources

const failed = await docsCorpus.listSources({ status: "failed" });
const active = await docsCorpus.listSources({ includeDeleted: false });

Retry failed sources by loading the source IDs again:

await docsCorpus.sync(
  retryLoader(failed.map((source) => source.sourceId)).load(),
  {
    sourceSet: "partial",
    cache: "refresh",
  },
);

Delete Sources

Delete one source:

await docsCorpus.deleteSource("old-guide.md");

Clear all active sources for a corpus:

await docsCorpus.clearSources();

What Counts As Changed

Content changes, stable-metadata changes, and indexer pipeline changes all trigger a reindex. Exclude volatile metadata that should not count as a change:

const docsCorpus = corpus({
  id: "docs",
  namespace: "product-docs",
  records,
  indexer: docsIndexer,
  hash: {
    excludeMetadata: ["mtimeMs", "lastFetchedAt"],
  },
});

See change detection in the indexing reference for the exact hash and fingerprint rules.

Observability

Corpus sync, ingest parsing, and indexing each emit start/end and per-source lifecycle events, and the same stage records flow through source records, devtools, CLI/TUI, and @use-crux/otel. The full event lists live in the indexing reference and the ingest reference.