Semantic Response Cache

Skip repeated model calls when a new request is semantically equivalent to a previous prompt result.

You classify, route, or normalize thousands of near-identical requests, and most of those model calls repeat work: "cancel my subscription" and "I want to cancel my plan" deserve the same answer without paying for a second generation. Semantic response caching returns the previous result when a new request means the same thing. Install the plugin once and opt in per prompt:

import { config, prompt } from "@use-crux/core";
import { createSemanticCache } from "@use-crux/core/cache";
import { embedding } from "@use-crux/openai";
import { inMemoryStorage } from "@use-crux/core/storage";
import { z } from "zod";

const classifyIntent = prompt({
  id: "classify-intent",
  input: z.object({
    userId: z.string(),
    message: z.string(),
  }),
  output: z.object({
    intent: z.enum(["billing", "support", "sales"]),
  }),
  cache: {
    semantic: {
      version: "v1",
      query: ({ input }) => input.message,
    },
  },
  prompt: ({ input }) => `Classify this message: ${input.message}`,
});

config({
  plugins: [
    createSemanticCache({
      storage: inMemoryStorage(),
      embedding: embedding({
        name: "cache",
        model: "text-embedding-3-small",
      }),
      ttl: 60_000,
      scope: ({ input }) => `user:${input.userId}`,
    }),
  ],
});

On the first request, Crux runs the model and writes the result. On a later semantically similar request, Crux embeds the query, searches the cache namespace, and returns the cached text or object when the score is above the threshold.

This is the third of Crux's three caching layers. Provider caching reduces token cost for stable prefixes, context caching skips expensive context resolvers, and semantic response caching skips the model call itself.

When To Use It

Use semantic caching for classification, extraction, routing, policy checks, normalization, and other prompts where similar inputs should produce the same result. It is a poor fit for highly personalized answers, time-sensitive answers, tool-heavy prompts, or prompts where the exact wording should always affect the output.

The important design rule is that semantic cache scope is a safety boundary. If two users should not see each other's cached result, do not use scope: 'global'. Scope by tenant, user, project, or another domain boundary.

Prompt Options

The shortest form is cache: { semantic: true }, which means read and write with the plugin defaults.

Use the object form when a prompt needs its own version, stricter threshold, shorter TTL, or custom query text:

cache: {
  semantic: {
    mode: 'readwrite',
    version: 'intent-v2',
    threshold: 0.98,
    ttl: 30_000,
    query: ({ input }) => input.message,
  },
}

Prompt settings can only make the plugin safer. A prompt-level TTL can shorten the plugin TTL, not extend it. A prompt-level threshold can make matching stricter, not looser. Change version whenever the prompt, output schema, safety policy, or cache assumptions change.

mode supports:

Mode	Behavior
`readwrite`	Look up first, write on miss.
`readonly`	Look up only. Useful during rollout.
`writeonly`	Populate cache without serving hits. Useful for warmup.
`off`	Disable semantic cache for this prompt.

If a prompt opts in but no plugin is installed, Crux warns in development and continues normally. That makes shared prompt definitions safe to import in tests and worker processes that do not install the cache.

Policies

The default write policy caches successful responses with a normal stop finish reason. For prompts with tools or other side effects, pass explicit policies:

import {
  createSemanticCache,
  semanticCachePolicies,
} from "@use-crux/core/cache";

createSemanticCache({
  store,
  embedding,
  ttl: 60_000,
  scope: ({ input }) => `tenant:${input.tenantId}`,
  shouldLookup: semanticCachePolicies.skipWhenToolsPresent(),
  shouldCache: semanticCachePolicies.all([
    semanticCachePolicies.finishReason("stop"),
    semanticCachePolicies.skipWhenToolCallsPresent(),
  ]),
});

If the built-in helpers are not enough, provide your own callback. See the semantic cache reference for the arguments policy callbacks receive.

Lookup uses dense embeddings only; the reference explains why sparse and hybrid embeddings are not accepted here.

Durable Stores

The cache needs records with TTL support and a dense vector store with pre-filter support:

import { upstashRedisRecordStore, upstashVectorStore } from "@use-crux/upstash";

createSemanticCache({
  records: upstashRedisRecordStore({
    redis,
    prefix: "semantic-cache:",
  }),
  vectors: upstashVectorStore({
    index,
    namespace: "semantic-cache",
  }),
  embedding,
  ttl: 60_000,
  scope: ({ input }) => `tenant:${input.tenantId}`,
});

Use a dedicated namespace/table/index for cache entries. Do not point the cache at the same vector namespace used for RAG chunks or long-term memory.

For Convex, use component-backed records with an explicit vector backend and a dedicated semantic-cache namespace:

import { convexRecordStore } from "@use-crux/convex";
import { upstashVectorStore } from "@use-crux/upstash";

const cache = createSemanticCache({
  records: convexRecordStore({ component: components.crux, ctx }),
  vectors: upstashVectorStore({
    index: vectorIndex,
    namespace: "semantic-cache",
  }),
  embedding,
  ttl: 60_000,
});

See the store capability rule for why shared indexes are refused.

Redis records alone are not enough for semantic cache lookup. Pair upstashRedisRecordStore() with upstashVectorStore() or another pre-filter-capable VectorStore.

Observability

Semantic cache emits semantic-cache:* events for lookup start/end, hit, miss, write, skip, and stream replay. They flow through devtools, OTel, the dev server, CLI stats, and the TUI dashboard.

Cached stream responses are replayed as synthetic streams, so stream consumers do not need a separate code path.