Multimodal Content

import {
  contentText,
  hasMediaParts,
  messageText,
  textPart,
} from "@use-crux/core";
import type { ContentPart, MediaSource, MessageContent } from "@use-crux/core";

Types

MessageContent is the content type used by canonical generation messages:

type MessageContent = string | readonly ContentPart[];

MediaSource is an Asset, HTTPS/data URL string, URL, Uint8Array, ArrayBuffer, or Blob. AssetRef is not model input; hydrate it with assetStore.get(ref) first.

ContentPart is a closed model-visible content union:

Every model-visible part may carry providerOptions. Assistant content may additionally contain ordered ReasoningPart and ToolCallPart values. System, user, and tool messages cannot. GenerateResult.content is authoritative; text is its text-part projection.

Parts

textPart(text)

Create a text part.

const part = textPart("Summarize this image.");

const image: ContentPart = {
  type: "image",
  source: pngBytes,
  mediaType: "image/png",
};

const file: ContentPart = {
  type: "file",
  source: "https://example.com/report.pdf",
  mediaType: "application/pdf",
  filename: "report.pdf",
};

const audio: ContentPart = {
  type: "audio",
  source: wavBytes,
  mediaType: "audio/wav",
};

const video: ContentPart = {
  type: "video",
  source: videoBlob,
  mediaType: "video/mp4",
};

Model calls normalize and validate media before provider I/O. They do not persist media.

Text Projection

contentText(content)

Project a MessageContent value into bounded text.

const projected = contentText([
  textPart("Inspect this."),
  { type: "image", source: pngBytes, mediaType: "image/png" },
]);

Text parts are emitted verbatim. Other parts become bounded semantic descriptors. This projection may include a locator or filename and is intended for model-facing text, not logs.

messageText(message)

Project a canonical message by reading message.content.

const projected = messageText(message);

Use this where a widened Message.content must become semantic text: guardrails, compaction, memory capture, cache query text, or custom provider fallbacks. It is not a logging redactor: URL and filename descriptors can retain sensitive locators. Crux observability applies its stricter media sanitizer before records are emitted.

hasMediaParts(content)

Return true when a content array contains any non-text part.

Media Errors

Malformed media throws InvalidMediaSourceError. Valid media that the selected adapter or model cannot send throws UnsupportedCapabilityError before provider I/O.

Specialized operations

generateImage(), transcribe(), and generateSpeech() are separate bounded operations for callers that require a guaranteed image, transcript, or audio result. They share cancellation, timeout, warning, execution, provider-metadata, and raw-result conventions, but they are not a universal executor and do not stream or persist.

Each successful result carries _meta.traceId and _meta.spanId for its exact outer media.generate_image, media.transcribe, or media.generate_speech operation. A composed child generation shares the trace but owns a different span. Provider validators construct ID-free payloads, and raw is never rewritten. See Operation Result Correlation.

Adapters omit unsupported operations structurally. Portable options are validated before I/O; provider-native options belong in the operation's typed extra record. Returned media is immediately usable and may be persisted explicitly with assetStore.put(asset).

Completed operations accept boundary-targeted Safety directly. Image and speech calls expose guardrails and safety; transcription additionally exposes one-shot output-text constraints. Image references/masks and transcription audio are guarded before provider normalization. Generated images and speech audio are guarded before canonical results return. result.safety contains the canonical audit when an entry was recorded, while raw, provider metadata, and warnings stay provider-owned and unguarded. See Safety for content primitives for the exact matrix and strip/escalation rules.

Privacy

Observability uses a stricter descriptor projection than contentText(). At every capture level it excludes bytes, base64/data URLs, bearer refs, provider file IDs, filenames, signed URL details, and native media payloads. Local text/transcript capture defaults on and production export defaults off; neither mode enables raw media.

Convex Mirror

@use-crux/convex re-exports ContentPart, MessageContent, textPart, contentText, messageText, and hasMediaParts from its root.