Observability

import {
  CRUX_OBSERVABILITY_CHANNEL,
  CruxGraphRecordBatchSchema,
  createHttpObservabilityTransport,
  createCruxRunId,
  createCruxSpanId,
  normalizeObservedError,
  observe,
  observabilityDiagnostics,
  subscribeObservability,
  teeObservabilityTransport,
} from "@use-crux/core/observability";

@use-crux/core/observability is the canonical graph contract for Crux execution traces. It defines the record shapes emitted by the TypeScript runtime and consumed by the local Go devtools backend.

Wire records, branded IDs, and taxonomies live in the internal contract.ts module and are covered by the observability schema version. Presentation/read-model types such as CruxRunDetail, CruxPresentationView, and live notification payloads are exported from the root observability barrel through presentation.ts; those shapes are versioned independently so devtools UI projections can evolve without changing the wire record contract.

The contract is lifecycle based:

• run:start
• span:start
• span:event
• artifact
• edge
• span:end
• run:end

The backend turns these write records into read models such as run lists, RunDetail, resource activity, artifact previews, filters, and search facets. Web devtools and the TUI should read those backend models instead of inferring graph semantics themselves.

RunDetail is the default one-run inspection model. It includes the rooted presentation tree, flattened rows, display labels, attached details, events, artifacts, relation overlays, diagnostics, facets, aggregate metric rollups, status rollups, inspection sections, source metadata, request/context composition, and a span index. Detail ownership is resolved centrally by the Go service: explains edges and presentation.ownerSpanId attach prompt/context/memory/routing metadata to the generation, agent, tool, step, or composition they explain, while chronology is only a fallback. Routing decisions fold onto their selected concrete generation even when the canonical graph wraps the generation under routing.router / routing.cascade; quiet constraint, guardrail, citation, scoring, and security warning spans fold as safety/detail evidence, while execution-changing governance remains visible. If a collector only receives completion-side records for a span, RunDetail keeps them as details instead of inventing an anonymous operation row.

Generation nodes and aggregating nodes can include request?: CruxRunDetailRequest. Exact generation requests are built from the consumed request-shaped messages artifact, linked context contributions, linked prompt budget, and request tool names. The backend also collects context contributions and prompt budgets produced under the nearest enclosing request scope before the generation starts, which covers framework agents whose Crux prompt resolves under agent.run before the provider stream begins. Nested framework agent steps that only emit output-shaped message artifacts inherit the nearest enclosing generation request and expose it with mode: "inherited" plus representative.strategy: "nearest-ancestor-request". When Convex Agent emits both call-args and thread-context request artifacts, the backend prefers thread-context and preserves prior-turn fields such as allMessages, recent, inputMessages, inputPrompt, existingResponses, and search on request.messages; inherited agent steps also include earlier sibling generation outputs as previousStepMessages. Aggregating run, stream, agent, composition, and flow nodes use the backend-selected final descendant generation as the representative effective request and include turns[] for the descendant generation breakdown. request.basePrompt.sourceId uses the concrete prompt id when known and otherwise reports messages.system / messages.prompt; base prompt segments come from systemBlocks, which preserve explicit segments and infer direct string-template interpolation from unambiguous primitive input values. request.modelSummary groups concrete generation models for exact, inherited, and aggregate requests, marks mixed-model aggregates, and turns[] carries per-generation model, provider, status, and promptId; flattened rows[] carries the row's effective model and provider as well. Convex Agent aggregate and per-step generation spans emit their configured languageModel so framework turns have model provenance, while nested tool-call or flow generations retain their own model identity. Clients should render these fields directly instead of walking descendant artifacts or deduping context contributions.

RunDetail also reconciles missing terminal lifecycle records from reliable canonical signals. Expired operation.deadline events mark the timed operation and its still-open ancestors as incomplete observability in the presentation tree, while future deadlines keep active long calls from being marked stale too early. Execution-changing governance rolls ancestors up to blocked, and intentional waits roll ancestors up to suspended. The canonical records remain unchanged. This is a telemetry diagnostic, not an application error.

Contextual retrieval, memory, and embedding spans fold into attached details when they are request-input evidence for a generation. Operational retrieval inside a tool, flow, composition, or agent boundary remains visible even when that branch lives under an agent generation stream; retrieval query/embed internals can still fold into the retrieval pipeline node.

DefinitionRef (runtime ↔ Project Index join)

Runtime records may carry definitionRefs?: DefinitionRef[] on run:start, span:start, and full span records when the runtime already holds a compiled definition handle:

interface DefinitionRef {
  id: string; // matches store.ProjectDefinition.ID exactly
  kind: ProjectDefinitionKind;
  role: DefinitionRefRole; // e.g. resolved-prompt, invoked-agent, invoked-tool
  source?: { file: string; line: number; column?: number }; // repo-relative only
}

Refs are omitted for anonymous handles, Crux never invents an id from a stack trace. source is root-redacted via sanitizeDefinitionSource (see the privacy guide). Catalog View runs and Run Detail Catalog links use this envelope; the user-facing walkthrough is in Catalog runtime evidence.

role describes how the definition participated (resolved-prompt, invoked-agent, contributed-tool-policy, invoked-flow-step, ...). kind is the full Project Index kind; role is a coarser relationship label used for display and filtering.

Plane ownership

The Project Index answers what is authored; the observability graph answers what ran. Catalog relates them without mixing the two planes: runtime evidence is a derived rollup over graph records, never part of the compiler snapshot or cache identity.

Coverage manifest and runtimeIdentity

Catalog classifies each definition's Observability section from the compiler-owned DEFINITION_KIND_COVERAGE manifest plus the server rollup GET /api/observability/definitions/{id}/activity:

The manifest also declares runtimeIdentity for definitions that are not top-level execution owners:

• definition-ref: the runtime holds the exact authored identity and records it directly.
• parent-derived: the index gives a structural child one exact containment parent; Catalog reports that parent's activity and says the child was not independently observed. Contributors without canonical runtime identity remain truthful zero-runtime states rather than selecting an arbitrary owner.
• eval: authoritative correlation through persisted Eval evidence.
• none: no truthful definition-level identity path exists. The kind may still emit runtime spans; the manifest's primary treatment distinguishes that case from genuinely static definitions.

Per-kind coverage facts:

• Executed flow steps, parallel branches, and recipe steps carry canonical child refs because their parent identity and authored child key are both available at runtime.
• Other structural children inherit evidence from an indexed parent and are labelled not independently observed.
• Knowledge bases and tool policies have canonical contributor refs on their owner spans (runtimeIdentity: definition-ref).
• Injectables and storage ports do not expose canonical runtime ids (runtimeIdentity: none); Catalog shows truthful zero-runtime / non-independent evidence for them.
• Deferred work, media operations, and ingest sources emit real runtime primitives, but those records do not carry the authored definition id; Catalog labels them runtime-observed but unjoined.
• Scorers remain Eval-primary; an authored judge({ id }) also attaches a scorer ref to its live scoring.judge span.
• File-loaded skills without a registry composite id do not invent a join key; registry-loaded skills do emit DefinitionRef.
• Compositions (parallel / pipeline / swarm / consensus) require an authored id so runtime and index share composition.<kind>:<id>.

Catalog View N runs opens Runs with definitionId set (GET /api/observability/definitions/{definitionId}/runs); the backend filters via the derived run_definition_activity projection, rebuildable from raw records with no separate revision stream. Run Detail lists every DefinitionRef on the run/span. Refs whose definition was renamed or deleted since the run keep role, kind, id, and any sanitized source, and are marked unresolved (Not found in the current Project Index: the source definition may have been renamed or deleted.) instead of becoming dead links. The projection stores runtime-emitted identity only; it is never validated against the current index at write time, and runs whose refs point at a since-deleted definition still appear in the filtered Runs list.

Error evidence

Thrown errors are represented with a shared normalized shape. observe.span() and observe.openSpan().error() attach:

• A compact error summary to the terminal span:end or run:end.
• A span:event named exception with OpenTelemetry-style attributes.
• An error.stack artifact when a stack trace exists.
• An error.raw artifact containing a bounded, redacted, JSON-safe representation of the thrown value.

The Go backend promotes span.error, error.stack, and error.raw into RunDetail.inspection.errors. Web devtools and the TUI render that section for any primitive, including tools, generation, retrieval stages, flow steps, eval cases, routing attempts, and custom spans.

Crux only uses exception evidence for actual thrown failures. Non-throwing outcomes stay in their domain-specific records: approval denial is tool.approval plus model-facing tool output, guardrail blocks are guardrail.report / guardrail.blocked, constraint retries are constraint.retry, retrieval zero hits are result counts and retrieval.hits, invalid citations are citation.report, cascade tier rejection is routing metadata, flow suspension/cancellation is flow status, and stream length or content_filter is a finish reason.

Runtime bridge and eval-runner failures can happen outside a normal span. Those responses carry the same normalized details under command.error.details, so UI and TUI surfaces can show name, message, stack, phase, kind, and raw details even when no span exists.

Exports

Built-in primitives use canonical names such as generation.call, generation.stream, prompt.resolve, prompt.budget, context.predicate, context.resolve, memory.read, memory.write, knowledge.lifecycle, retrieval.retrieve, retrieval.recipe, retrieval.step, indexing.pipeline, ingest.parse, corpus.sync, embedding.call, cache.lookup, cost.record, compaction.run, eval.run, eval.case, scoring.judge, citation.check, feedback.record, workspace.operation, plan.operation, task.operation, skill.load, security.warning, routing.router, routing.split, routing.retry, routing.cascade, routing.fallback, fallback.attempt, constraint.check, constraint.retry, guardrail.run, agent.run, flow.run, flow.step, flow.suspension, composition.parallel, composition.pipeline, composition.consensus, composition.swarm, handoff.prepare, delegate.invoke, and tool.call.

Agent compositions share one runtime lifecycle contract. parallel() creates a composition.parallel span with sibling agent.run children. pipeline() creates a composition.pipeline span with one flow.step per executable step and nested agent.run spans for agent steps. consensus() creates a composition.consensus span with voter agent.run children directly under that span. swarm() creates a composition.swarm span with agent turns, handoff.prepare spans, handoff.payload artifacts, and triggered edges between turns.

Custom edge types and artifact kinds must use the custom.* namespace. Unknown custom values are preserved and can be displayed generically, but specialized indexing and layouts are reserved for canonical values.

Common detail artifacts use stable kinds so RunDetail.inspection can route them without UI-specific parsing:

Both enforce- and report-mode media strips emit a guardrail.report artifact and a guardrail.action span event with action: 'strip'. Report mode preserves the canonical input or result; it changes mutation behavior, not whether the action is observed. guardrail.stripped is only the Turn Decision Report reason code. When an enforced strip violates a required group—an empty media-only message, final generated image, required speech/transcription audio, or retained image-edit mask without a reference—the strip report/action are followed by a guardrail.blocked edge and the audit entry records escalation. Transcript rewrites record when timed segments/words were removed. Media records expose only the boundary, mode, action, primitive/phase/field, canonical part type, stable original indexes, safe model id when known, and escalation/detail-loss flags. Sources, bytes, URLs, provider file ids, filenames, raw responses, and provider options are never included.

Turn Explanation (decisionReport)

TurnDecisionReport is a deterministic per-generation-turn read model that powers the Run Detail Explain tab. It is a projection over evidence Crux already records (request composition, context contribution and prompt-budget artifacts, routing/fallback reports, guardrail/constraint/security/cache/compaction reports, retrieval/memory artifacts, Project Index source refs, and Eval records), not a new trace format and not model chain-of-thought. It summarizes and links into the existing detail views rather than duplicating their payloads.

The local Go projection attaches one report to each generation node and folded generation detail, exposed as the optional decisionReport field on CruxRunDetailNode and CruxRunDetailDetail. Consumers read it directly; when it is absent the existing detail tabs are unaffected. Run Insight is derived by the UI from per-turn reports that share the same runId; V1 does not define a separate run-level decisionReport projection.

Cache and freshness are deliberately separate concepts and types: freshness is correctness (was the data current enough?), cache is efficiency (what did reuse save?). Every decision, disposition, and diagnostic carries an evidenceLevel (declared / observed / inferred / missing) so the report stays honest about what it could and could not prove; reason.source: "not-recorded" maps to evidenceLevel: "missing". Reason codes are stable strings (for example routing.fallback.fired, context.freshness.stale_used, cache.freshness.rejected) suitable for Eval assertions, while human-facing text can change independently. The same codes back the ctx.expect.decisionReport Eval matchers.

TurnDecisionReport V1 compatibility

schemaVersion: 1 is the compatibility boundary for this read model:

• Required top-level collections remain arrays.
• Future V1 changes are additive unless schemaVersion changes.
• Human-facing text (reason.text, turn.readout, labels, and suggestions) is display copy and not matcher-stable.
• Stable reason codes are matcher-stable.
• Coverage area ids are stable for matchers, filtering, and localization.
• Cache and freshness remain separate evidence concepts.
• Unknown, missing, unresolved, and inferred states remain explicitly representable.

String unions stay open through the contract's literal-extension types where the TypeScript API allows custom or unknown values.

Runtime emitters

Use observe.run() for user-facing execution roots and observe.span() for inspectable work inside a run. Standalone spans automatically create an implicit run, so helpers like pipeline, consensus, or swarm can be called directly and still produce a complete trace.

Crux's built-in orchestration helpers emit these records automatically. parallel() creates sibling agent.run spans, pipeline() records flow.step plus nested agent.run spans, flow() records flow.run, step children, and flow.suspension timeline markers for intentional waits, consensus() records voter agent.run spans directly under composition.consensus, and swarm / delegate handoffs create handoff.prepare spans, canonical input/output artifacts, handoff.payload artifacts, and relation edges.

Generation and streaming orchestration record operation deadlines when a structured timeout is configured. Adapters pass timeout.totalMs into core orchestration, which records totalTimeoutMs / deadlineAt and emits operation.deadline before the provider call begins.

Generation terminal span records also carry client-side performance metrics in the existing metrics field:

These metrics are additive and do not change the graph schema version. Custom metric keys must use the custom.* namespace. Span families are derived from canonical primitive names at emit time, so observe.span() and observe.openSpan() do not accept a separate family option.

Prompt and safety helpers emit the same graph contract automatically:

• prompt.resolve() opens prompt.resolve, records conditional context inclusion/exclusion as context.predicate, and records resolved context text as context.resolve spans with context.contribution artifacts. Contributions that provide tools include injectedTools, including dropped text contributions whose tools still participate in the request. Direct injectables, retrievers/grounding, memory, and blackboards emit tool-only context.contribution previews when they contribute tools without resolved text. Segmented system content preserves segments: { text, dynamic, source?, observedAt?, sourceVersion? }[], staticTokens, and dynamicTokens on context previews, inspect parts, and budget-dropped previews. Context previews and inspect parts also include servedFrom, resolvedAt, memo-hit age, and summarized segment source facts when available. Generation spans that consume the resolved prompt link back to included context artifacts and prompt-budget artifacts, and their messages artifact includes request toolNames; the Go read model projects those records into RunDetailNode.request. Token-budget decisions emit prompt.budget artifacts with dropped contribution previews and prefixOverflow: true when the stable provider-cache prefix exceeds the requested budget.
• Memory blocks and blackboards emit memory.read / memory.write spans, attach memory.snapshot artifacts when a read or write has inspectable state, attach memory.recall artifacts for non-empty read result previews, attach memory.diff artifacts for before/after write summaries, and connect those artifacts with semantic memory edges. Empty reads still emit the span but omit memory.recall, so UIs should not render a recalled-block card for zero-result reads. Raw namespaces are represented by namespaceHash; blackboard snapshots and diffs carry memoryType: "blackboard". Budgeted composed-memory renders add budgetMaxTokens, budgetIncludedBlocks, budgetTrimmedBlocks, and budgetDroppedBlocks span attributes and mirror the same decision under memory.snapshot.preview.budget. Proposal-oriented writes can include writeMode and proposalStatus so review state remains visible in traces.
• Adapter-managed tools, including live @use-crux/ai tool calls, emit tool.call spans, consume tool.args artifacts, produce raw and model-facing tool.result artifacts, and record thrown execute() failures as rich error evidence on the same tool span while preserving the model-facing error output. Model-emitted tool intents attach to the generation as tool.request artifacts and are linked to the eventual tool execution by tool call id. Convex Agent streams expose an outer generation.stream plus child generation.call spans for each AI SDK step, so tool-call turns and later text turns are visible as consecutive streamed generations. Convex Agent fallback tool-call parts follow the same contract, so stop-condition tools remain inspectable even when no handler executes. Approval request, approval, denial, and token mismatch paths emit tool.approval spans.
• Retrieval calls emit retrieval.query spans with bounded retrieval.hits artifacts and retrieval.returned edges. Retrieval recipes open a parent retrieval.recipe span and record fanout/retrieve/hit steps as child retrieval.step spans with inspectable step output artifacts.
• Indexing and corpus helpers emit indexing.pipeline, ingest.parse, and corpus.sync spans. Document transforms, chunkers, chunk transforms, dry runs, cache outcomes, loader results, and source-ledger summaries are visible through child spans plus indexing.report, ingest.report, and corpus.report artifacts.
• Embeddings emit embedding.call spans with provider name, kind, dimensions, batch shape, usage, cost, governance metrics, and bounded embedding.report artifacts. Configured embedding caches emit nested cache.lookup spans with hit/miss/write counts and per-entry hit/miss events.
• Semantic cache middleware emits cache.lookup spans for lookup, write, and skip decisions. Hit, miss, write, and skip events carry prompt id, scope hash, version, query hash, score, result kind, and timing metadata; hit/write decisions also attach cache.report artifacts.
• Cost tracking emits cost.record spans with call attribution, token/cost totals, and cost.warn / cost.limit events. Token budget checks emit prompt.budget spans with source breakdowns and pressure levels. Compaction helpers emit compaction.run spans and compaction.report artifacts with before/after token counts, compression ratios, focus, model labels, and bounded summary previews.
• Eval runs emit eval.run spans and eval.case children for each Case and Variant. LLM judges emit scoring.judge spans with bounded score.report artifacts. Citation validation emits citation.check spans with bounded citation.report artifacts; structured citations may carry output-text anchors (start, end, and outputQuote) so clients can place inline superscripts without reparsing the generated answer. Durable feedback emits feedback.record spans linked to the canonical run.
• Workspaces emit workspace.operation spans for list/read/write/edit/delete operations, with namespace hashes and bounded output artifacts. Plans and tasks emit plan.operation / task.operation spans for mutations, including task create/update/remove/discard paths. File and registry skills emit skill.load spans, and configured prompt-injection diagnostics emit security.warning spans plus security.report artifacts.
• Router, split, retry, fallback, and cascade wrappers emit routing.router, routing.split, routing.retry, routing.fallback, routing.cascade, and child fallback.attempt spans. Router spans record classification and selected route/model; split spans record deterministic bucket selection without exposing the raw seed; retry spans record repeated attempts against one child model; cascade spans record attempted tiers, rejection/acceptance, budget exits, provider errors, and full ordered configured tier metadata for skipped/not-reached tiers; fallback attempt spans record model attempts, bounded error classifications, success metadata, and fallback.attempt edges between failed and next attempts. Each top-level routed resolution attaches one routing.report artifact whose preview is the caller-visible receipt { model, cost, firstTokenAt?, trace }; the outer artifact kind is routing.report, while the receipt itself has no inner kind discriminator. At the JSON observability boundary, unavailable cost values are represented as null and treated as absent by consumers. Run Detail renders TTFT, bounded attempt errors, and cascade note/budget from that receipt.
• The Safety session's constraint phase opens a grouped constraint.check span, records each individual check as a child span, attaches constraint.report artifacts, and emits constraint.retry spans plus constraint.retry edges when feedback triggers regeneration.
• The Safety session's guardrail phases open grouped and per-guard guardrail.run spans, attach guardrail.report artifacts for pass, warn, redact, transform, hold, and block actions, and emit guardrail.blocked edges for blocks.

import { observe } from "@use-crux/core/observability";

await observe.run(
  { name: "support reply", rootPrimitive: "agent.run" },
  async () => {
    await observe.span(
      { name: "retrieve docs", primitive: "retrieval.query" },
      async () => {
        observe.event({
          name: "query.built",
          attributes: { terms: ["refund", "policy"] },
        });
      },
    );
  },
);

The runtime uses async context propagation on Node.js and preserves parent span stacks across normal async work. For async work that crosses a boundary where context is not preserved, pass an explicit captured context:

const context = observe.captureContext();

queueMicrotask(() => {
  void observe.withContext(context, async () => {
    await observe.span(
      { name: "delegate", primitive: "delegate.invoke" },
      async () => {
        // delegated work
      },
    );
  });
});

Context propagation contract

Three rules keep the run graph correct end-to-end:

1. Boundary primitives open an observed span. Every primitive that opens a boundary in the run graph (tool, delegate, flow, handoff, composition, retrieval, embedding, compaction) records a canonical span through observe.span(), which creates the span:start and span:end records, pushes the span id onto the active observability context for the duration of the callback, and records errors automatically.
2. New runs and spans capture the current parent at creation. The runtime reads the active span stack when a run or span starts and writes the deepest open span as parentSpanId on span:start. In-process this works automatically via AsyncLocalStorage. When AsyncLocalStorage is unavailable (browser-like or edge runtimes), run lifecycles and synchronous withContext() spans are still recorded; contextless observe.event(), observe.artifact(), and observe.edge() calls are skipped and counted in observabilityDiagnostics().contextlessRecords instead of throwing.
3. Cross-boundary transports pack and unpack the context explicitly. AsyncLocalStorage does not survive process boundaries (ctx.runAction() in Convex, edge-worker fetch(), Lambda invocation, an HTTP call to another service). The sending side packs the captured observability context into the call payload; the receiving side restores it with observe.withContext() before any SDK call. Anything that runs before the seed records parentSpanId: undefined and falls back to inference.

The backend builds the run graph purely from these canonical records:

1. run:start / run:end define the initiating run.
2. span:start / span:end define timed work with spanId and parentSpanId.
3. span:event records point-in-time details inside a span.
4. artifact records carry bounded payload previews or references.
5. edge records carry non-tree relationships such as handoff payloads, delegated invocations, retrieval results, citations, feedback, or replay links.

The Go backend parents each child directly under its recorded parentSpanId, with no heuristics and no time-window guessing. span:start and span:end share the same span id; the backend uses that id to pair lifecycle records and resolve children that name the span as their parent. Once every boundary primitive uses observe.span() and every async-context boundary propagates the captured context, sibling, parent-child, and many-to-many relationships render without client-side inference.

Delivery

emit() is the event spine for the TypeScript runtime. Every record is born once, then delivered to three downstream sinks:

• Synchronous in-process subscribers registered with subscribeObservability().
• The Node diagnostics channel named by CRUX_OBSERVABILITY_CHANNEL, when that channel has subscribers.
• The async observability transport, when a transport is configured.

Use an in-process subscriber when your code runs in the same runtime and wants the canonical graph records directly:

const unsubscribe = subscribeObservability((record) => {
  if (record.type === "span:end") {
    console.log(record.primitive, record.status);
  }
});

unsubscribe();

Use the narrowed overload when you only need specific record variants:

const unsubscribeSpanEnds = subscribeObservability(
  ["span:end"] as const,
  (record) => {
    console.log(record.spanId, record.status);
  },
);

unsubscribeSpanEnds();

Use the diagnostics channel when an external Node integration wants to subscribe without depending on Crux plugin wiring:

import { channel } from "node:diagnostics_channel";
import {
  CRUX_OBSERVABILITY_CHANNEL,
  type CruxObservabilityChannelMessage,
} from "@use-crux/core/observability";

channel(CRUX_OBSERVABILITY_CHANNEL).subscribe((message) => {
  const { record } = message as CruxObservabilityChannelMessage;
  // map record to your tracing backend
});

In-process subscriber errors are isolated from user code and counted in observabilityDiagnostics().subscriberErrors.

The default runtime path is non-blocking: transport failures are captured as diagnostics and never throw into user code. In serverless runtimes, await a bounded flush before returning so queued records are not killed with the request:

await observe.flush({ timeout: 5000 });

Observability delivery is queued into batches. The first delivery starts immediately for live devtools, and later records coalesce per microtask before being sent as bounded concurrent batches. This avoids head-of-line blocking when a collector, tunnel, or Convex network hop is slow. Failed batches retry on a capped backoff without waiting for another emitted record, while runtime reset and transport replacement count stale in-flight failures as dropped instead of requeueing them into a later transport. HTTP delivery normalizes unknown preview values into JSON-safe shapes, sends observability.token / CRUX_DEVTOOLS_TOKEN as bearer auth when configured, and isolates rejected records inside a failed batch, so one malformed or oversized detail artifact does not prevent terminal lifecycle records from reaching the backend. The Go observability backend reconciles out-of-order lifecycle records by stable ids and timestamps. The runtime never blocks user code on normal emits, but flush() and shutdown() wait for pending deliveries so serverless and Convex actions can preserve span starts, ends, artifacts, and events before the worker exits.

createCruxConvex(), action/internalAction from @use-crux/convex/server, and the agent/swarm wrappers all bind to a bounded final flush automatically, no per-function setup is required for first-party Crux Convex functions. The default budget is a short fixed window (DEFAULT_CONVEX_OBSERVABILITY_FLUSH_TIMEOUT_MS, 3 seconds), not the 20-second window from earlier releases: Convex exposes no per-invocation deadline API, so the bound must stay well under Convex's own action timeout instead of being derived from remaining host time. flushObservability() always returns a structured ObservabilityFlushResult (status, delivered, rejected, remaining, deadlineExceeded); pass onDrain to inspect it, or let the default reporter log a console warning whenever a drain does not fully complete so an incomplete flush is never silently discarded.

A raw, hand-written Convex function (one not created through action/internalAction/flow) opts in once with withObservabilityFlush from @use-crux/convex/observability:

import { withObservabilityFlush } from "@use-crux/convex/observability";

export const run = internalAction({
  args: {},
  handler: withObservabilityFlush(
    async (ctx, args) => {
      // Crux work here
    },
    { onDrain: (result) => console.log("crux drain", result) },
  ),
});

Pass observabilityFlushTimeoutMs: false on an action/internalAction/flow definition only when an outer serverless boundary already flushes.

Runs, execution segments, and delivery receipts

A logical run can physically execute across more than one process, isolate, or invocation, a suspended Flow, a resumed Runtime Engine wake, a Convex action that hands off to another action, a Worker that answers a follow-up request. Three identities stay distinct so none of them has to be inferred from another:

Successful public operation envelopes use { traceId, spanId } to point at their exact producing span. They do not include runId or segmentId, and a provider _meta.responseId remains a separate identifier. See Operation Result Correlation for the result, streaming, Eval evidence, and adapter-author contracts.

segmentSeq is monotonic only within its segmentId, it is not a distributed total order and the backend never treats it as one. Ordering across segments is a partial order: exact edges and parent IDs first, then lifecycle boundaries, then (segmentId, segmentSeq), then timestamp, then recordId as a stable tie-breaker. Concurrent segments are shown as concurrent, not collapsed into an invented wall-clock order.

The lifecycle union is run:start (first segment), run:suspend (the current segment ends without terminalizing the run), run:resume (a fresh segment starts on the same logical run), and run:end (terminalizes the run exactly once; a duplicate byte-identical terminal record is idempotent, a conflicting one is diagnosed and never overwrites the first). Lifecycle mutation always goes through an explicit owner, observe.openRun(), the run/span handle's own .suspend() / .end() / .error(), or observe.resumeRun(carrier, ...), never through restoring a captured context. observe.withContext() is context-only: it re-establishes the active span/attribute stack for a callback and cannot resume, suspend, or end a run by itself. See the observability context guide for the full suspend/resume pattern used by Flow and Convex swarms.

import { observe } from "@use-crux/core/observability";

const run = observe.openRun({ name: "import job", rootPrimitive: "flow.run" });
// ... do work in this segment ...
const carrier = run.suspend({ reason: "awaiting-approval" }); // CruxPropagationCarrier, JSON-safe

// Persist `carrier` with your workflow state. In the invocation that resumes:
const resumed = observe.resumeRun(carrier, { reason: "approval-received" });
// ... a fresh segmentId; segmentSeq starts at 1 again ...
resumed.end();

A host boundary alone is not a suspension. A request can end successfully and start a separately linked downstream run for queued or background work; only an operation that is truly the same logical continuation emits run:suspend/run:resume.

Delivery success is per record, not per HTTP status. The transport treats a 2xx response as complete only when every sent record has an accounted disposition (accepted or rejected, with a retryable flag); a malformed or partial receipt is retried in full rather than assumed successful. recordId identifies immutable content: the first occurrence persists, an exact duplicate is accepted idempotently without re-applying rollups, and a different payload under the same recordId is rejected permanently as a diagnosed conflict rather than overwriting the original. Bounded diagnostics (observabilityDiagnostics()) report accepted/retried/permanently-rejected/overflow-dropped/deadline-dropped counts and last-error state so a process that never reconnects can still report what it knows locally.

Host lifecycle adapters

Core exposes a small provider-neutral host lifecycle port (context, defer, deadline) instead of importing any runtime. First-party wrappers bind it per host:

Every wrapper reports a structured ObservabilityFlushResult (status, delivered, rejected, remaining, deadlineExceeded) instead of a boolean. Pass onDrain to inspect it; the default reporter logs a warning whenever a drain does not fully complete, so a truncated flush is never silently discarded. No configuration can keep an already-returned invocation alive if the adapter was never given the host's lifetime capability, a raw Worker handler or custom serverless entry point that never passes its ExecutionContext/deadline to a wrapper can still lose unflushed records exactly as it would without Crux.

OpenTelemetry bridge

@use-crux/otel's withTelemetry() plugin activates a real OTel span around the instrumented callback (trace.getActiveSpan() resolves correctly inside provider/tool/agent/flow work, and nested spans parent correctly), not just after the fact. run:suspend ends the segment's root OTel span; run:resume starts a fresh root span sharing the original traceId rather than reopening the old one. No SDK span object crosses a suspension boundary. Inject/extract helpers (injectCruxPropagationCarrier() / extractCruxPropagationCarrier()) round-trip the same CruxPropagationCarrier used for Crux-native resume through standard W3C traceparent/tracestate headers, and an explicit baggageAttributeAllowlist controls which baggage keys (if any) become span attributes, untrusted baggage is never copied by default. observe.flush()/observe.shutdown() and the host lifecycle wrappers above also force-flush the installed telemetry manager's exporter/processor work, bounded by the same deadline, so a serverless/Workers freeze does not orphan pending OTel export calls either.

Deployment identity follows OTel Resource ownership. Construct the application's immutable Resource with createCruxResourceAttributes(identity) before starting its provider, then pass the same object to config({ observability: { identity } }). Standard OTel spans inherit those values from the Resource only; lightweight exporters carry them per span. DefinitionRefs map to bounded primary attributes and up to 16 ordered crux.definition.ref events without source or prompt content.

Local development vs. production telemetry

The HTTP transport and local SQLite read model described in this page are a development topology: crux dev (optionally tunneled) for one user's own machine. Production telemetry should go through an OTLP-compatible collector or platform-native tracing (withTelemetry(), or your platform's own OTel integration) rather than pointing production traffic at a local devtools server. Both can run at once: observability.serverUrl for local inspection, withTelemetry() for your production APM, since they are complementary sinks over the same graph-record spine, not competing sources of truth.

Graph v3 deployment identity migration

Writers emit graph v3 (CRUX_OBSERVABILITY_SCHEMA_VERSION === 3) with optional validated deployment identity. TypeScript and Go readers retain persisted v2 as explicitly deployment-unspecified and reject a v2 record that attempts to carry the new field. On first startup against a pre-v2 local database, the Go backend transactionally discards pre-v2 observability rows (they carry no truthful execution-segment identity to migrate) and rebuilds the schema automatically. Every other SQLite table (Eval, workspace, etc.) is untouched, and you never need to delete .crux by hand.

Serverless and edge hosts

A physical invocation can end before a background timer ever fires, so wrap handlers instead of relying on observe.flush() placement alone.

withObservableInvocation() (from @use-crux/core/observability) wraps a generic async handler, binds one scoped host lifecycle per call, and awaits a bounded final drain before returning or rethrowing:

import { withObservableInvocation } from "@use-crux/core/observability";

export const handler = withObservableInvocation(
  async (event) => {
    // Crux work here
  },
  (event) => ({ remainingTimeMs: 5000 }),
);

withNodeObservableInvocation() (from @use-crux/core/observability/node) adapts this to Node Lambda-style (event, context) handlers, reading context.getRemainingTimeInMillis() automatically.

Cloudflare Workers

Workers isolates can outlive the returned Response via ExecutionContext.waitUntil, so withCrux() (from @use-crux/cloudflare) registers the final drain with ctx.waitUntil() instead of awaiting it inline, telemetry drains in the background with no added response latency:

import { defer } from "@use-crux/core";
import { withCrux } from "@use-crux/cloudflare";

const handleFetch = withCrux(
  async (request, env, ctx) => {
    defer(() => env.ANALYTICS.writeDataPoint({ blobs: [request.url] }));
    return new Response("ok");
  },
  {
    context: (request, env, ctx) => ctx,
    invocation: () => ({ flushTimeoutMs: 5000 }),
  },
);

export default { fetch: handleFetch };

A raw Worker fetch handler must pass its ExecutionContext to the wrapper once, as shown above, @use-crux/core/observability has no way to discover ctx on its own. @use-crux/cloudflare needs no nodejs_compat compatibility flag; WorkersExecutionContext is a structural type satisfied by anything exposing waitUntil(promise). The provider-neutral low-level alternative is withWaitUntilDefer() from @use-crux/core/defer/serverless.

Explicit propagation is the correctness path across Workers requests and isolates: use observe.openRun(...).suspend({ reason }) to obtain a serializable CruxPropagationCarrier, hand it off (queue message, KV, a follow-up request body), and call observe.resumeRun(carrier, { reason }) in the next request to continue the same logical run in a fresh segment. Neither call depends on AsyncLocalStorage or any state shared between isolates.

Use createHttpObservabilityTransport() to send canonical batches to the local Crux backend endpoint:

import { config } from "@use-crux/core";

config({
  observability: {
    serverUrl: "http://localhost:4400",
    delivery: { maxPendingDeliveries: 1000 },
  },
});

Use teeObservabilityTransport() when a library needs its own capture sink while preserving an existing transport:

import {
  createHttpObservabilityTransport,
  createInMemoryObservabilityTransport,
  setObservabilityTransport,
  teeObservabilityTransport,
} from "@use-crux/core/observability";

const capture = createInMemoryObservabilityTransport();
const auditSink = createHttpObservabilityTransport({
  serverUrl: "http://localhost:4400",
});
const restore = setObservabilityTransport(
  teeObservabilityTransport(capture, auditSink),
);

restore();

Trace batches fan out to every leg. Feedback is a durable mutation, so the tee delegates it to exactly one feedback-capable leg. Construction fails clearly if multiple legs expose submitFeedback() instead of guessing which destination owns the Review record.

Capture policy

Input/output artifact previews are enabled by default for local inspection. Disable them centrally when traces leave a trusted boundary:

config({
  observability: {
    recordInputs: false,
    recordOutputs: false,
    capture: {
      default: "evidence",
      overrides: {
        "guardrail.report": "safe",
        "constraint.report": "safe",
        "tool.args": "off",
      },
      redactRecord: (record) => record,
    },
  },
});

capture.default accepts "full", "safe", "evidence", or "off". Per-artifact overrides use canonical artifact kinds. The Safety family includes guardrail.report, constraint.report, score.report, approval.request, validation.feedback, memory.write, tool.args, and tool.result.

When disabled, input-family artifacts (messages, input, system, prompt, tool.args, tool.request) and output-family artifacts are still emitted, but as encoding: 'reference' with sizeBytes and hash instead of preview. Output-family artifacts include model outputs, stream timelines, tool results, retrieval hits, memory snapshots/recalls/diffs, guardrail and constraint reports, handoff payloads, delegate reports, composition outputs, compaction summaries, judge reasoning, citation quotes, rendered context, and raw error payloads. The runtime's canonical artifact-kind map is exhaustive, so adding a new built-in artifact kind requires an explicit input, output, safety, or exempt privacy decision before it can compile. Custom artifact kinds default to captured payloads only when the active capture policy allows their declared direction.

The emit path also strips payload-shaped attributes, converts JSON-hostile values, and drops invalid optional metrics before fan-out, so subscribers, the diagnostics channel, plugins, and transports all consume the same sanitized JSON-safe record. A failure inside a configured redactRecord hook drops the record (fail-closed) instead of emitting it unredacted. The shared helper toSafeJsonValue() truncates long strings (default 8000 characters) and redacts sensitive object keys.

Eval coverage map

Eval uses the same graph contract as production traces. The table below lists every Eval-owned observability kind, the core or local path that emits it, and where local tooling surfaces it.

Eval provenance can point to task evidence observed before the current Eval run. Consumers must preserve that historical link and must not require same-run membership. Baseline and reuse truth comes from the Eval run read model, not from reconstructing it from graph edges.

The observability domain is explicit export configuration. config() with no observability block does not install telemetry, cloud upload, or raw-content capture. Use devtools only for local UI/control/tunnel/bridge behavior.

The HTTP transport posts { records } to /api/observability/records. POST /api/observability/records is intentionally partial-accepting: parseable batches return 202 with accepted and rejected record counts, malformed JSON returns 400, and transient local storage failures return 503 with Retry-After: 1. The local read models keep broad dashboard/resource queries bounded, including latest-500 caps for full-history devtools pulls and resource activity. When a devtools transport is installed, the SDK also sends runtime prompt/context/tool snapshots through POST /api/index/snapshot to enrich the Project Index. Queued records are capped with oldest-record drops, and generation streams close their spans from the raw stream's completion or error signal, so stopped or failed streams do not stay visually running while later usage metadata is collected. The Go server accepts the configured bearer token only on the observability ingest route; tokenized tunnel URLs such as https://example.ngrok.app?t=... are still supported as serverUrl values for compatibility, with the token preserved on the final ingest endpoint. The server stores the raw payload for every accepted graph record, including forward-compatible unknown record types, so newer SDKs can preserve data even when an older local read model skips normalized projection. The Go backend owns validation, persistence, read-model building, filtering, search, and subscriptions. It joins Eval annotations onto the same run identity server-side and publishes one monotonic revision per affected run after each ingest transaction commits; devtools never merges a second, independently-filtered list on the client. Web devtools polls an open run's detail once per second while it is live, then once every 5 seconds for the following 60 seconds after a terminal signal (catching a late Convex/serverless flush that lands after the terminal update), before stopping; the Runs list polls once per second while any row is live and once every 5 seconds otherwise, and is revision-gated so a WebSocket push only triggers a bounded catch-up or refetch when its revision is actually new. A reconnect after a dropped WebSocket sends the last known revision and performs a bounded catch-up, or a full invalidation when that revision has aged out of the server's retained change log. Convex Agent container streams fold into details in the Go read model, while step-level streamed generation turns, tools, handoffs, and delegated flows render as chronological agent children. Promoted tool executions preserve source.canonicalParentSpanId and sort after their request generation by relation, even when Convex action timestamps arrive slightly out of order.

Runs and run detail distinguish suspended (a non-terminal boundary was observed, not an error) and incomplete (a stale segment with no suspend/end, most often a host crash or freeze) from ordinary success/error/cancelled terminal states, and conflicted when an immutable-identity or incompatible terminal-evidence conflict was diagnosed. A compact segment indicator on each row shows segment count and the active segment; run detail adds a reliability badge for sequence gaps, missing parents, partial ordering, and trace-alias conflicts when any of those are present, a normal single-segment run renders with no extra chrome. Delivery/export health is unknown, healthy, or degraded; unknown (no source has reported health yet, e.g. a process that never connected) is never presented as healthy.

Run status and delivery-health copy

DevTools uses shared plain-language explanations on Runs and Run Detail (one helper, so list and detail copy do not drift):

suspended

• Default: This run is durably suspended, waiting on a signal, event, or timer.
• With multiple segments: This run is durably suspended, waiting on a signal, event, or timer. N execution segments have been observed so far.

incomplete

• Default: Telemetry ended without a run:end record, the run may still be executing out of view, or its process exited before reporting a terminal status.
• With sequence gaps: Telemetry ended without a run:end record, and N sequence gap(s) or missing parent reference(s) were observed, the run may still be executing out of view, or its process exited before reporting a terminal status.

conflicted

• Trace alias conflict: A trace alias identifies more than one logical run, so the server could not establish one immutable identity for this trace.
• Otherwise: The stored terminal evidence for this run conflicts with what was previously recorded, so its identity could not be resolved.

Delivery-health badge tooltips (exact copy):

• healthy: The server observed a clean terminal run with causal ordering and no gaps, conflicts, or rejected delivery.
• degraded: Some telemetry records were rejected or delivery could not complete cleanly.
• unknown: The server has not observed enough delivery evidence to call this run healthy or degraded.

Degraded explanation variants:

• Default: Delivery/export health is degraded, some records were rejected during export.
• With partial ordering: Delivery/export health is degraded, and the server could not establish one causal display order across segments, some records may be missing or rejected.

When is healthy reachable?

The local Go read model promotes delivery health to healthy only when all of the following hold:

• The run has ended (endedAt set) and is not running / incomplete / conflicted
• orderingConfidence === 'causal'
• gapCount === 0
• No traceAliasConflict
• No persisted ingest-health rejections for that run

Still-running runs stay unknown. Incomplete or partially ordered runs stay unknown unless a rejection marks them degraded. The server never invents healthy.

Segment and ordering signals

Alongside status and delivery health, run rows may surface:

Reliability chips may also say "N segments", "N sequence gaps", "conflicting trace alias", "partial ordering", or "delivery degraded".

Local retention and reconciliation

Crux Local keeps observability history bounded. The Go server reconciles abandoned running traces from cheap rollup columns, uses recent last_activity_at updates (such as streaming token.chunk records) to avoid false stale states, and persists reconciled lifecycle state so crashed runs are not rebuilt on every tick. Local retention defaults to 14 days, 2000 runs, and 256KB artifact previews; override with CRUX_OBSERVABILITY_RETENTION_DAYS, CRUX_OBSERVABILITY_RETENTION_RUNS, and CRUX_OBSERVABILITY_PREVIEW_MAX_BYTES.

Live token visualization uses the same observability channel without turning every provider delta into a whole-run refetch. The SDK coalesces stream deltas into span:event records named token.chunk; the local server keeps a capped per-span ring, excludes those events from RunDetail, and broadcasts append-only token.chunk notifications keyed by runId, spanId, and eventId.

Plugin composition

Plugins passed to config({ plugins }) are processed in order. Each plugin's install(hooks) receives a frozen snapshot of the cumulative CruxHooks state from all prior plugins, and the framework merges returned hook fields, so a plugin never manually fans out to prior handlers.

• Middleware: later plugins wrap earlier ones; the outer middleware controls when next() is called.

Plugin A middleware (inner)
  ↕ wraps
Plugin B middleware (outer)
  ↕ wraps
Adapter generate() call

• Transports / capture / activation hooks: merged according to mergeHooks rules (last writer or layered activation depending on the field). Prefer one observability transport plus OTel as a graph subscriber.
• Graph-record fan-out: every installed subscriber, diagnostics-channel listener, HTTP transport, and @use-crux/otel projection observes the same sanitized record stream. Subscriber errors are isolated from user code and counted in observabilityDiagnostics().subscriberErrors.
• Dispose order: plugin dispose() functions are called in reverse order during registry.dispose(): last installed, first disposed.

Prompt lifecycle hook argument types

Per-prompt hooks declared with prompt({ hooks }) receive typed argument objects.

PrepareHookArgs

Passed to onPrepare after the prompt's system message and contexts are composed.

interface PrepareHookArgs {
  promptId: string; // ID of the prompt being prepared
  systemTokens: number; // total tokens in composed system message
  droppedContexts: DroppedContext[]; // contexts dropped due to token budget
}

droppedContexts includes the source name and tokens count for each context that was trimmed. Use this to detect when important context is being lost.

GenerateHookArgs

Passed to onGenerate after a successful generation completes.

interface GenerateHookArgs {
  promptId: string; // ID of the prompt that generated
  durationMs: number; // generation wall-clock time in milliseconds
  result: unknown; // the generation result (adapter-specific)
}

The result type depends on your adapter: for Vercel AI SDK it is the generateText or generateObject result, for OpenAI the chat completion response.

ErrorHookArgs

Passed to onError when an error occurs during preparation or generation.

interface ErrorHookArgs {
  promptId: string; // ID of the prompt that errored
  error: unknown; // the error that occurred
  phase: "prepare" | "generate"; // which phase the error happened in
}

The phase field distinguishes context composition failures (prepare) from LLM call failures (generate).