Observability

LLMs fail in subtle ways. The output looks fine but the model never saw the brand context. The retry logic kicked in twice. The token budget dropped your most important fragment. None of this is visible from outputs alone.

Without observability:

• You don't know what system message the model actually saw
• You don't know which contexts were dropped under token pressure
• You don't know how long each step of a flow took, or where it stalled
• You don't know if the constraint retry kicked in, or how many times
• You don't know which prompt, flow, or model is driving spend
• You can't reproduce a bad output because you don't have the inputs

With observability, every interesting moment in the pipeline emits one graph record, and every consumer (devtools, subscribers, plugins, OTel) reads the same sanitized stream. Eval runs use the same graph, so Cases, managed scorers, Baselines, and production traces share one provenance model. How records are sanitized, fanned out, linked, and retained: Observability reference.

The layers

All layers are built on the same canonical graph-record spine:

• Devtools: a local UI that shows every run live while you develop.
• Catalog runtime evidence: open a definition and see which runs actually used it.
• Runs and delivery health: what incomplete, suspended, and degraded mean, and what to do.
• Adapter outcomes: handle rate limits, timeouts, and refusals the same way on every provider.
• Runtime setup: keep telemetry from being lost when serverless hosts freeze.
• Privacy: control what payloads are captured and how they are redacted.
• Troubleshooting: fix missing runs, empty Catalog activity, and unclean statuses.
• Cost tracking: see which prompt, model, or session is driving spend.
• Telemetry: send production traces to Datadog, Honeycomb, or Grafana via OpenTelemetry.
• Subscribers: consume raw graph records in-process with subscribeObservability() or the crux:observability diagnostics channel.
• Plugins: package your own instrumentation as a composable CruxPlugin.
• Middleware: log, time, or wrap every generate() / stream() call in one place.

When should I use which?

• Always: install devtools in dev. It's zero-cost when disabled.
• Production: install @use-crux/otel if you have an existing OTel-compatible APM (Datadog, Honeycomb, Grafana, New Relic). Otherwise the lightweight HTTP/callback exporter works.
• Custom logic: use subscribeObservability() for raw graph records, or a CruxPlugin when you need broader runtime composition.
• Per-call wrapping: use middleware for things that wrap every generation: request logging, timing, multi-tenant scoping.
• Ephemeral hosts: bind the host lifecycle wrappers in Runtime setup so runs do not end as incomplete solely because the worker froze mid-flush.

When should I NOT use middleware?

• To mutate inputs or outputs. Middleware should observe, not transform. Use guardrails for transforms or hooks for per-prompt behavior.
• For logic that depends on the prompt, that's per-prompt hooks, not global middleware.

Pick a topic