Production Telemetry

Crux devtools is great for development, but production environments need traces in your existing observability stack. @use-crux/otel emits OpenTelemetry spans for every instrumented event: one integration covers Datadog, Honeycomb, Grafana, New Relic, and any OTel-compatible platform.

Setup

import { config } from "@use-crux/core";
import { withTelemetry } from "@use-crux/otel";

export default config({
  plugins: [withTelemetry({ serviceName: "my-app" })],
});

That's it. Spans flow to whatever observability platform you've configured.

Export paths

@use-crux/otel supports two strategies depending on your runtime environment.

Standard OTel (Node.js servers)

For long-lived servers (Next.js, Express, Fastify), set up the OTel SDK once in your application entrypoint. @use-crux/otel creates spans via @opentelemetry/api which routes them through your globally registered TracerProvider.

import { NodeSDK } from "@opentelemetry/sdk-node";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { createCruxResourceAttributes } from "@use-crux/otel";
import type { CruxDeploymentIdentity } from "@use-crux/core/project-index";

export const identity = {
  projectId: "checkout",
  manifestId:
    "pim_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  deploymentId: "production-42",
} satisfies CruxDeploymentIdentity;

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    ...createCruxResourceAttributes(identity),
  }),
  traceExporter: new OTLPTraceExporter({
    url: "https://otel-collector.example.com/v1/traces",
    headers: { "DD-API-KEY": process.env.DD_API_KEY },
  }),
});
sdk.start();

import { withTelemetry } from "@use-crux/otel";
import { identity } from "./instrumentation";

// No exporter needed: uses the global TracerProvider
config({
  observability: { identity },
  plugins: [withTelemetry({ serviceName: "my-app" })],
});

The Resource is immutable application-owned setup. Use the same validated identity for the Resource and observability.identity; changing a process-wide deployment identity requires reconstructing and restarting the provider. Crux does not mutate a registered Resource, access SDK-private state, or install a second provider. Lightweight exporters have no Resource object, so they carry the same three semantic identity keys on each span.

Lightweight (Lambda, Convex, Cloudflare Workers)

For ephemeral runtimes where the full OTel SDK isn't available, pass an exporter option:

export const telemetry = withTelemetry({
  serviceName: "my-app",
  exporter: {
    url: "https://collector.example.com/v1/traces",
    headers: { "X-Api-Key": collectorApiKey }, // supplied by this host
  },
});

export const telemetry = withTelemetry({
  serviceName: "my-app",
  // `sendTelemetry` is an application-injected host adapter.
  exporter: (spans) => sendTelemetry(spans),
});

The URL exporter uses fire-and-forget HTTP POST (5-second timeout): failures are silently ignored so telemetry never impacts your application.

withTelemetry() is safe to compose defensively: duplicate active installs warn once and become no-ops, and open span registries are bounded. Spans evicted because of registry pressure are force-ended with crux.expired: true and UNSET status.

What gets traced

Every instrumented Crux event produces a span:

Tool spans include privacy-safe toModelOutput() metadata only. OTel records the output shape and byte-size difference; it does not record raw tool results, model-facing tool content, queries, files, or metadata.

Generate/stream spans follow Crux's pinned genai-dev-2026-06 GenAI semantic convention table. Attributes like gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.client.operation.duration, and gen_ai.server.time_to_first_token work with dashboards and alerting rules while keeping semconv churn isolated to one versioned table.

Running alongside devtools

Both local devtools and OTel can run simultaneously. Use observability.serverUrl for a local server or tunnel, and keep production export explicit through @use-crux/otel or observability config. The plugin system's fan-out semantics ensure every event reaches both configured destinations:

config({
  observability: {
    serverUrl: process.env.DEVTOOLS_URL,
    token: process.env.CRUX_DEVTOOLS_TOKEN,
  },
  plugins: [withTelemetry({ serviceName: "my-app" })],
});

Devtools uses the visual UI for development. OTel sends spans to your production dashboards. Same events, deliberately configured destinations.

Options

Payload capture is configured centrally with config({ observability: { recordInputs, recordOutputs, redactRecord } }). recordInputs and recordOutputs accept inline, reference, or off capture modes, so devtools, subscribers, diagnostics-channel listeners, and OTel all observe the same redacted or full graph records. The same central emit path strips payload-shaped attributes, converts JSON-hostile values, and strips invalid optional metrics before records reach those consumers. redactRecord can replace a graph record or return null; hook failures drop the record fail-closed.

OTel message content is disabled by default, even when local graph capture is inline. Set captureMessageContent: true or OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true to export gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions from generation artifacts. Each content attribute is capped at 32KB.

Crux emits W3C-format trace and span IDs. Lightweight OTel exports reuse those IDs directly; standard OTel exports keep provider-issued IDs and include Crux IDs as correlation attributes.

Active span bridge and W3C propagation

withTelemetry() activates a real OTel span around instrumented callbacks: trace.getActiveSpan() resolves inside provider, tool, agent, and flow work, and nested spans parent correctly. This is not post-hoc span creation after the work already finished.

Suspend/resume never reuses one SDK span object across a host boundary:

• run:suspend ends the current segment's root OTel span
• run:resume starts a fresh root span that shares the original Crux traceId

For custom queues or RPC hops, round-trip the same CruxPropagationCarrier Flow and Convex use for in-process resume through standard W3C headers:

import {
  injectCruxPropagationCarrier,
  extractCruxPropagationCarrier,
} from "@use-crux/otel";
import { observe } from "@use-crux/core/observability";

// Header-like carrier: any object with get/set/keys.
const headers = new Map<string, string>();
const headerCarrier = {
  get: (key: string) => headers.get(key),
  set: (key: string, value: string) => {
    headers.set(key, value);
  },
  keys: () => [...headers.keys()],
};

// Producer: serialize W3C headers from a suspend carrier.
const carrier = run.suspend({ reason: "queued-work" });
injectCruxPropagationCarrier(carrier, headerCarrier);
await queue.send({ body, headers: Object.fromEntries(headers) });

// Consumer: restore correlation before resuming work.
const extracted = extractCruxPropagationCarrier(headerCarrier);
if (extracted.carrier) {
  const resumed = observe.resumeRun(extracted.carrier, {
    reason: "worker-start",
  });
  // ...
  resumed.end();
}

Incoming carriers are untrusted: format/length limits and baggage caps apply. Baggage keys become crux.baggage.* span attributes only when listed in withTelemetry({ baggageAttributeAllowlist: ['tenant', …] }); nothing is copied by default.

observe.flush() / observe.shutdown() and the host lifecycle wrappers also force-flush the installed telemetry manager (lightweight in-flight exports or the standard TracerProvider.forceFlush()), bounded by the same deadline.

Next steps