Observability context propagation

When you run an agent that delegates work, to a subagent, a flow step, a parallel branch, the resulting run graph should reflect what actually happened: each child span nests under the specific span that triggered it. Getting that hierarchy right under concurrency, async fan-out, and serverless function boundaries (Convex, edge workers, Lambda) is the job of observability-context propagation.

Span hierarchy works automatically

You don't normally write any of this yourself. Every built-in boundary primitive (tool, delegate, flow, handoff, composition, retrieval, embedding, compaction) already records a canonical span, and every new span captures the currently active span as its parentSpanId at creation:

import { observe } from "@use-crux/core/observability";

await observe.span(
  { name: "handoff", primitive: "handoff.prepare" },
  async () => {
    await observe.span(
      { name: "delegate", primitive: "delegate.invoke" },
      async () => {
        // The delegate span's parentSpanId points at the handoff span.
      },
    );
  },
);

In-process this works via AsyncLocalStorage, modeled on W3C traceparent: the backend parents the child directly under the recorded span, no heuristics, no time-window guessing. When AsyncLocalStorage is not available, such as in browser-like or edge runtimes, Crux still records run lifecycles and synchronous withContext() spans. Contextless observe.event(), observe.artifact(), and observe.edge() calls are skipped and counted in observabilityDiagnostics().contextlessRecords instead of throwing.

The full three-rule propagation contract and what the backend does with the records are documented in the Observability reference.

That leaves two things you actually do by hand: share a traceId across intentionally separate run roots, and pack/unpack context across serverless boundaries.

Share a traceId across multiple run roots

When one workflow intentionally owns several run roots, pass traceId to observe.openRun() for the child roots. Eval uses this for evaluation traces: the eval.run umbrella and every eval.case run share one trace while staying separate runs.

Correlators ride on the same context machinery but do not define execution parentage. Use propagateAttributes({ sessionId, userId, metadata }, fn) to group records by session or user without adding graph nodes:

import { observe, propagateAttributes } from "@use-crux/core/observability";

await propagateAttributes(
  { sessionId: "checkout-123", userId: "user-456", metadata: { plan: "team" } },
  () =>
    observe.run(
      { name: "checkout assistant", rootPrimitive: "agent.run" },
      runCheckout,
    ),
);

Every record emitted inside the callback carries sessionId and userId. Metadata appears as attributes named meta.plan, meta.<key>, and so on, with values capped for transport safety.

Pack and unpack context across serverless boundaries

AsyncLocalStorage does not survive process boundaries, ctx.runAction() in Convex, edge-worker fetch(), AWS Lambda invocation, an HTTP call to another service. Cross-boundary transports must pack the captured observability context into the call payload and the receiving side must restore it before any SDK call.

@use-crux/convex/server does this automatically:

// Calling side, inside a Crux-aware action or tool:
const result = await ctx.crux.runAction(
  "research",
  iref.agent.subagent.runResearch,
  {
    taskId,
    projectId,
    query,
    userId,
  },
);

// Receiving side, via @use-crux/convex/server:
export const runResearch = internalAction({
  args: {
    taskId: v.id("subAgentTasks"),
    projectId: v.string(),
    query: v.string(),
  },
  handler: async (ctx, args) => {
    // Anything started here records the calling action's parentSpanId.
    return researchFlow.handler(ctx, args);
  },
});

ctx.crux.runAction() captures the current observability context and packs it into the hidden __crux action args. @use-crux/convex/server actions restore that context before running the handler and await a bounded flush so Convex/serverless workers do not drop queued records after the handler returns.

For other runtimes the contract is the same; only the packing differs:

The receiving side must seed the context before any SDK call. Anything that runs before the seed will record parentSpanId: undefined and fall back to inference.

For scheduled serverless workflows that resume in a later invocation, do not persist run.captureContext() and restore it with observe.withContext(), withContext() is context-only and never resumes, suspends, or ends a run. A logical run that suspends across a physical worker boundary needs an explicit lifecycle owner: call run.suspend({ reason }) (or openedRun.suspend(...)) to close the current execution segment and get back a serializable CruxPropagationCarrier, persist that carrier with your workflow state, and call observe.resumeRun(carrier, { reason }) in the resumed worker to open a fresh segment on the same logical run before any child span/event records. Only the segment that actually completes the operation calls .end() / .error(). @use-crux/convex uses this shape for multi-action swarm runs (packages/convex/src/swarm.ts) so a resumed turn continues the same run in a new segment instead of either starting an unrelated run or reviving a stale in-memory context.

Advanced: authoring a custom boundary primitive

Only when you build a primitive that wraps user work in a span the timeline should show do you need to write the span yourself. Follow this template:

import { observe } from "@use-crux/core/observability";

export async function myBoundary<T>(
  input: Input,
  fn: () => Promise<T>,
): Promise<T> {
  return observe.span(
    {
      name: "my-boundary",
      primitive: "custom.operation",
      attributes: { inputKind: input.kind },
    },
    fn,
  );
}

observe.span() creates the span:start and span:end records, pushes the span id onto the active observability context for the duration of fn, and records errors automatically.

For non-lexical lifetimes, use observe.openSpan():

import { observe } from "@use-crux/core/observability";

const span = observe.openSpan({
  name: "my-stream",
  primitive: "custom.operation",
});

try {
  await span.withContext(async () => {
    await streamWork();
  });
  span.setAttributes({ chunkCount: 12 });
  span.end({ attributes: { streamCompleted: true } });
} catch (error) {
  span.error(error);
  throw error;
}

span.end() accepts terminal options only. Use span.setAttributes() for metadata discovered during the span, or pass terminal metadata as span.end({ attributes }); raw attribute bags are rejected so { error } consistently records an error end.

Custom primitive records must use canonical primitive names where possible, or custom.operation for app-specific work. The runtime derives span family from the primitive. The span:start and span:end records share the same span id; the Go backend uses that id to pair lifecycle records and resolve children that name this span as their parent.

Detail-only spans can pass implicitRun: false when they should enrich an existing trace but must not create a standalone run. Router and cascade resolution use this so direct generations are not mislabeled as router.resolve; inside an active run those spans are still fully recorded and inspectable.

Troubleshooting

My delegated child span shows up as a sibling, not a child. Either (a) the spawning primitive doesn't emit a canonical span yet, (b) the async-context boundary loses the captured observability context, or (c) the receiving handler runs SDK code before restoring context. Search for parentSpanId in the span:start record, if absent, the SDK never saw the parent.

Two adjacent spans both have parentSpanId but only one nests correctly. Confirm the parent span exists in the same run graph and that the receiving side restored context before opening child spans.

I want to inspect raw graph records. Use the devtools graph endpoint or the TUI run detail view. Both read backend-owned observability records instead of rebuilding relationships in the client.