Middleware

Middleware lets you hook into every stage of the chat() lifecycle — from configuration to streaming, tool execution, usage tracking, and completion. You can observe, transform, or short-circuit behavior at each stage without modifying your adapter or tool implementations.

Common use cases include:

• Logging and observability — track token usage, tool execution timing, errors
• Configuration transforms — inject system prompts, adjust temperature per iteration, filter tools
• Stream processing — redact sensitive content, transform chunks, drop unwanted events
• Tool call interception — validate arguments, cache results, abort on dangerous calls
• Side effects — send analytics, update databases, trigger notifications

Quick Start#

Pass an array of middleware to the chat() function:

import { chat, type ChatMiddleware } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";

const logger: ChatMiddleware = {
  name: "logger",
  onStart: (ctx) => {
    console.log(`[${ctx.requestId}] Chat started`);
  },
  onFinish: (ctx, info) => {
    console.log(`[${ctx.requestId}] Finished in ${info.duration}ms`);
  },
};

const stream = chat({
  adapter: openaiText("gpt-5.5"),
  messages: [{ role: "user", content: "Hello" }],
  middleware: [logger],
});

Just want to see chunks flowing through your middleware during development? Use debug: { middleware: true } on your chat() call — no custom middleware required. See Debug Logging.

Lifecycle Overview#

Every chat() invocation follows a predictable lifecycle. Middleware hooks fire at specific phases:

graph TD
    A["chat() called"] --> B["onConfig (phase: init)"]
    B --> C[onStart]
    C --> D["onConfig (phase: beforeModel)"]
    D --> E["Adapter streams response"]
    E --> F["onChunk (for each chunk)"]
    F --> G{Tool calls?}
    G -->|No| H[onUsage]
    G -->|Yes| I[onBeforeToolCall]
    I --> J[Tool executes]
    J --> K[onAfterToolCall]
    K --> L{Continue loop?}
    L -->|Yes| D
    L -->|No| H
    H --> SO{"Structured output path?"}
    SO -->|None| M{Outcome}
    SO -->|"Native combined"| SOH["Post-loop structured-output harvest (onChunk)"]
    SOH --> M
    SO -->|"Separate finalization"| SOC[onStructuredOutputConfig]
    SOC --> SOM["onConfig (phase: structuredOutput)"]
    SOM --> SOS["Structured-output finalization (onChunk, onUsage)"]
    SOS --> M
    M -->|Success| N[onFinish]
    M -->|Abort| O[onAbort]
    M -->|Error| P[onError]

    style I fill:#e1f5ff
    style J fill:#ffe1e1
    style SOC fill:#e1f5ff
    style SOM fill:#e1f5ff
    style SOS fill:#e1f5ff
    style SOH fill:#e1f5ff
    style N fill:#e1ffe1
    style O fill:#fff4e1
    style P fill:#ffe1e1

Phase Transitions#

The context's phase field tracks where you are in the lifecycle:

Hooks Reference#

onConfig#

Called once during init (startup) and once per iteration during beforeModel (before each model call). On the separate-finalization path, onConfig additionally re-fires at the structured-output boundary with ctx.phase === 'structuredOutput', receiving the post-onStructuredOutputConfig view of the config. A single-iteration separate-finalization run therefore fires onConfig three times (init + beforeModel + structuredOutput). Native-combined output does not add this third call. Use onConfig to transform the configuration that the model receives.

Return a partial config object with only the fields you want to change — they are shallow-merged with the current config automatically. No need to spread the existing config.

import { type ChatMiddleware } from "@tanstack/ai";

const dynamicTemperature: ChatMiddleware = {
  name: "dynamic-temperature",
  onConfig: (ctx, config) => {
    if (ctx.phase === "init") {
      // Add a system prompt at startup — only systemPrompts is overwritten
      return {
        systemPrompts: [
          ...config.systemPrompts,
          "You are a helpful assistant.",
        ],
      };
    }

    if (ctx.phase === "beforeModel" && ctx.iteration > 0) {
      // Increase temperature on retries. Sampling params live in the
      // provider-native modelOptions object — `temperature` is universal,
      // so it's the same key across providers. Spread the existing
      // modelOptions so other model options stay unchanged.
      const current =
        typeof config.modelOptions?.temperature === "number"
          ? config.modelOptions.temperature
          : 0.7;
      return {
        modelOptions: {
          ...config.modelOptions,
          temperature: Math.min(current + 0.1, 1.0),
        },
      };
    }
  },
};

Sampling parameters (temperature, top_p / topP, the various max*Tokens keys) live inside modelOptions under each provider's native name — they are no longer root config fields. temperature happens to be spelled the same across every provider, so the example above is provider-agnostic; if you mutate a token limit instead, use the provider-native key (e.g. max_output_tokens for OpenAI, num_predict nested under modelOptions.options for Ollama). See Moving Sampling Options into modelOptions.

Config fields you can transform:

When multiple middleware define onConfig, the config is piped through them in order — each receives the merged config from the previous middleware.

Return providerMessages when a transform must affect only the model call. For compatibility, returning messages also updates provider input unless the same result sets providerMessages explicitly.

Change the tool choice of a call#

Your agent hits its last model call, and the model calls a tool again. The run ends after the tool result, and the user gets no answer. Return toolChoice: 'none' on that last call, and the model answers in text:

import { chat, maxIterations, toolDefinition, type ChatMiddleware } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { z } from "zod";

const MAX_CALLS = 10;

const search = toolDefinition({
  name: "search",
  description: "Search the docs",
  inputSchema: z.object({ query: z.string() }),
}).server(async ({ query }) => ({ hits: [`A page about ${query}`] }));

const answerOnLastCall: ChatMiddleware = {
  name: "answer-on-last-call",
  onConfig: (ctx) => {
    if (ctx.phase === "beforeModel" && ctx.iteration === MAX_CALLS - 1) {
      return { toolChoice: "none" };
    }
  },
};

const stream = chat({
  adapter: openaiText("gpt-6.1-sol"),
  messages: [{ role: "user", content: "How do I add a tool?" }],
  tools: [search],
  agentLoopStrategy: maxIterations(MAX_CALLS),
  middleware: [answerOnLastCall],
});

• config.toolChoice is the chat() option.
• The value you return applies to the next model call only. The call after it starts again from the chat() option.
• Return it in the beforeModel phase. A value from the init phase does not reach a model call.
• For the values, and what each provider does with them, see Choose when the model calls a tool.

Change the HTTP requests of a call#

Your proxy wants a header on each request. Or you want to log each model call. You do not need your own client for that. Return wrapFetch from onConfig:

import { type ChatMiddleware } from "@tanstack/ai";

const traceHeader: ChatMiddleware = {
  name: "trace-header",
  onConfig: (ctx) => {
    if (ctx.phase !== "beforeModel") return;
    return {
      wrapFetch: (next) => (input, init) => {
        const headers = new Headers(init?.headers);
        headers.set("x-trace-id", `${ctx.requestId}-${ctx.iteration}`);
        return next(input, { ...init, headers });
      },
    };
  },
};

You want the same wrapper on every call of the run? Pass it to chat():

import { chat } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";

const stream = chat({
  adapter: openaiText("gpt-5.6"),
  messages: [{ role: "user", content: "Hi" }],
  wrapFetch: (next) => async (input, init) => {
    const started = Date.now();
    const response = await next(input, init);
    console.log(`model call: ${response.status} in ${Date.now() - started} ms`);
    return response;
  },
});

• A wrapper gets the next fetch and gives back a new one. It can change the URL, the headers, the request, or the response.
• A wrapper from onConfig applies to the next model call only.
• Wrappers chain. The chat() wrapper runs first, then the middleware wrappers in middleware order, then the fetch of the adapter.
• Each adapter page says if that adapter supports wrapFetch.

Change the prompt cache of a call#

A slow tool, like a build or a test run, can pause your agent for more than 5 minutes. The short Claude cache is gone by then. Return promptCache from onConfig to keep the cache longer from the next call on:

import { type ChatMiddleware } from "@tanstack/ai";

const longCacheAfterTools: ChatMiddleware = {
  name: "long-cache-after-tools",
  onConfig: (ctx, config) => {
    if (ctx.phase !== "beforeModel" || ctx.iteration === 0) return;
    return { promptCache: { ...config.promptCache, retention: "long" } };
  },
};

• config.promptCache is the current value, with its retention and its key.
• The value you return goes to the adapter on the next model call. It stays until a middleware returns another one.
• For the retention values and what they cost, see Prompt Caching.

onStructuredOutputConfig#

Called once at the start of the final structured-output adapter call — only when chat() was invoked with outputSchema and supportsCombinedToolsAndSchema() does not return true for the current model/options. Pipes through middleware in order, like onConfig, but with access to the JSON Schema being sent to the provider. Use this hook when you need to transform the schema (e.g., inject $defs, strip vendor-incompatible keywords) or apply structured-output-specific behavior (e.g., suppress system prompts on the final call).

Native-combined adapters (modern OpenAI, Claude 4.5+, Gemini 3.x, Grok 4.x — see issue #605) skip the separate finalization call and never invoke this hook. The engine passes the converted schema directly to chatStream after onConfig runs, so middleware cannot transform the native-combined schema.

Return a partial StructuredOutputMiddlewareConfig with only the fields you want to change — they are shallow-merged with the current config. Return void to pass through.

import { type ChatMiddleware } from "@tanstack/ai";
import { sharedDefs } from "./defs";

const injectDefs: ChatMiddleware = {
  name: "inject-defs",
  onStructuredOutputConfig: (_ctx, config) => {
    // `config.outputSchema` is the JSON Schema being sent to the provider
    return {
      outputSchema: {
        ...config.outputSchema,
        $defs: { ...sharedDefs },
      },
    };
  },
};

Config fields you can transform:

Ordering at the structured-output boundary:

1. onStructuredOutputConfig fires first, piping through every middleware in array order.
2. onConfig then re-fires at the same boundary with ctx.phase === 'structuredOutput', receiving the post-onStructuredOutputConfig view of the config (minus outputSchema). Use onConfig for general-purpose transforms that apply to every adapter call; use onStructuredOutputConfig when you need access to the schema.

When multiple middleware define onStructuredOutputConfig, the config is piped through them in order — each receives the merged config from the previous middleware.

onStart#

Called once after the initial onConfig completes. Use it for setup tasks like initializing timers or logging.

import { type ChatMiddleware } from "@tanstack/ai";

const timer: ChatMiddleware = {
  name: "timer",
  onStart: (ctx) => {
    console.log(`Request ${ctx.requestId} started at iteration ${ctx.iteration}`);
  },
};

onChunk#

Called for every chunk streamed from the adapter. You can observe, transform, expand, or drop chunks.

import { type ChatMiddleware } from "@tanstack/ai";

const redactor: ChatMiddleware = {
  name: "redactor",
  onChunk: (ctx, chunk) => {
    if (chunk.type === "TEXT_MESSAGE_CONTENT") {
      // Transform: redact sensitive content
      return {
        ...chunk,
        delta: chunk.delta.replace(/\b\d{3}-\d{2}-\d{4}\b/g, "[REDACTED]"),
      };
    }
    // Return void to pass through unchanged
  },
};

Return values:

When multiple middleware define onChunk, chunks flow through them in order. If one middleware drops a chunk (returns null), subsequent middleware never see it.

Chunk types you'll see#

onChunk receives every AG-UI event the run produces — not just text. Narrow on chunk.type (a discriminated union) before reading type-specific fields. The common ones:

See the AG-UI protocol docs for the full event catalogue and exact field shapes.

Transforming structured-output chunks#

There is no separate onStructuredOutputChunk hook — and you don't need one. When chat() is invoked with outputSchema, the structured-output chunks (the JSON TEXT_MESSAGE_CONTENT deltas, plus the structured-output.start / structured-output.complete CUSTOM events and any finalization RUN_ERROR) flow through the same onChunk hook as everything else. You transform, expand, or drop them exactly like any other chunk.

How you distinguish them depends on which finalization path the adapter takes:

• Separate-finalization adapters (supportsCombinedToolsAndSchema() does not return true for the current model/options): ctx.phase === 'structuredOutput' during the finalization call. Discriminate on the phase.
• Native-combined adapters (modern OpenAI Chat Completions / Responses, Claude 4.5+, Gemini 3.x, Grok 4.x — see issue #605): the schema-constrained JSON is produced on the model's natural final turn, so ctx.phase stays 'modelStream' — the 'structuredOutput' phase never fires. Discriminate on the CUSTOM event name (structured-output.start / structured-output.complete) instead.

import { type ChatMiddleware } from "@tanstack/ai";

const redactStructuredOutput: ChatMiddleware = {
  name: "redact-structured-output",
  onChunk: (ctx, chunk) => {
    // Separate-finalization path: the JSON streams as TEXT_MESSAGE_CONTENT
    // during the 'structuredOutput' phase. Transform the delta like any
    // other text chunk — here, redact anything that looks like an SSN before
    // it reaches the client.
    if (
      ctx.phase === "structuredOutput" &&
      chunk.type === "TEXT_MESSAGE_CONTENT"
    ) {
      return {
        ...chunk,
        delta: chunk.delta.replace(/\b\d{3}-\d{2}-\d{4}\b/g, "[REDACTED]"),
      };
    }

    // Both paths: the completed typed payload arrives as a CUSTOM
    // `structured-output.complete` event. On the native-combined path this is
    // your only signal (ctx.phase never flips to 'structuredOutput'), so key
    // off the event name, not the phase. `chunk.value` carries { object, raw }.
    if (chunk.type === "CUSTOM" && chunk.name === "structured-output.complete") {
      console.log("final structured output:", chunk.value);
    }

    // Return void to pass everything else through unchanged.
  },
};

Why is there onStructuredOutputConfig but no onStructuredOutputChunk? Because the config shape genuinely differs at the structured-output boundary — it carries an outputSchema field that plain ChatMiddlewareConfig doesn't (see onStructuredOutputConfig). Chunks are all just StreamChunk regardless of phase, so one onChunk plus ctx.phase (or the CUSTOM event name) covers every case — a parallel chunk hook would be redundant.

onShouldContinue#

Called when the engine is deciding whether to start another agent-loop iteration (after a tool phase or between model turns). Combined with AND semantics across middleware and with agentLoopStrategy — any explicit false stops the loop. Return true, void, or undefined to allow continuation.

Does not abort the run: the stream finishes normally with the current messages. Use ctx.abort() only for a hard abort.

import { type ChatMiddleware } from "@tanstack/ai";

const budget: ChatMiddleware = {
  name: "tool-budget",
  onShouldContinue: (_ctx, state) => {
    // Stop further turns once 20 tool calls have been emitted
    if (state.toolCallCount >= 20) return false;
  },
};

For a full per-turn + cumulative tool budget recipe, see Tool-call budgets.

onModelError#

Some model calls fail for a reason your middleware can fix. For example, the prompt is too long for the model. onModelError gets the RUN_ERROR before anyone else sees it. Fix the request and call ctx.retry(). While the call has retries left, the run goes on as if the call never failed.

Say your app sends retrieved documents as a second system prompt. When the call is too long, this middleware drops them and tries again:

import { CONTEXT_OVERFLOW_CODE } from "@tanstack/ai";
import type { ChatMiddleware, ChatMiddlewareContext } from "@tanstack/ai";

// The runs that go on without the documents.
const withoutDocs = new WeakSet<ChatMiddlewareContext>();

const dropDocsWhenTooLong: ChatMiddleware = {
  name: "drop-docs-when-too-long",
  onModelError: (ctx, error) => {
    if (error.code !== CONTEXT_OVERFLOW_CODE) return;
    withoutDocs.add(ctx);
    ctx.retry();
  },
  onConfig: (ctx, config) => {
    if (ctx.phase !== "beforeModel" || !withoutDocs.has(ctx)) return;
    // Keep the main system prompt. Drop the documents after it.
    return { systemPrompts: config.systemPrompts.slice(0, 1) };
  },
};

What ctx.retry() does while the call has retries left:

• It drops the failed call. onChunk and the caller never see its RUN_STARTED or RUN_ERROR.
• It runs onConfig again (phase beforeModel), then calls the model again.
• It works only inside onModelError. In any other hook it throws.

When the hook runs:

• Only when the adapter sends a RUN_ERROR before any output. A call that already sent text is not retried, so no output repeats.
• Not for an exception that the adapter throws. That error goes to onError.
• Not for the separate structured-output call that runs after the agent loop. The model calls of the agent loop get the hook as usual.
• Every time, also after the last retry. A model call retries at most 2 times. After that, the error goes to the caller.

withCompaction uses this hook to compact and retry after a context overflow.

onInterruptBoundary and onInterruptResolution#

Use these hooks when middleware needs data from the client. Define the request with defineInterrupt() and register it with chat({ interrupts }) and useChat({ interrupts }). Do not emit raw AG-UI events from middleware.

onInterruptBoundary runs at four points in an agent iteration:

• beforeModel, before the adapter starts.
• afterModel, after the model response is complete.
• beforeTools, before tool execution starts.
• afterTools, after the tool phase is complete.

Each middleware can return requests from one boundary. The engine combines all requests from that boundary into one AG-UI interrupt batch. The batch ends the run with one interrupt outcome.

This hook cannot change config. Its only legal return is { interrupts } or nothing. The continuation is a new chat() call, so the hook runs again. Skip the emit when ctx.parentRunId is set if this pause belongs to the original request only.

What is in ctx at each phase, and when to use each phase, is in Lifecycle Boundaries.

Create one shared definition. Both the server and the client import this value, so the definition ID and response shape stay the same on both sides.

import { defineInterrupt, type ChatMiddleware } from '@tanstack/ai'
import { z } from 'zod'

export const reviewPlan = defineInterrupt({
  id: 'review-plan',
  payloadSchema: z.object({ title: z.string() }),
  responseSchema: z.object({ approved: z.boolean() }),
})

export const reviewMiddleware: ChatMiddleware<unknown, typeof reviewPlan> = {
  name: 'review-plan',
  onInterruptBoundary(ctx) {
    if (ctx.phase !== 'beforeTools') return
    if (ctx.parentRunId) return
    return {
      interrupts: [
        reviewPlan.interrupt({
          key: 'release-plan',
          reason: 'review-required',
          message: 'Approve this plan?',
          payload: { title: 'Release plan' },
        }),
      ],
    }
  },
  onInterruptResolution(_ctx, resumedInterrupts) {
    for (const result of resumedInterrupts.for(reviewPlan)) {
      if (result.status === 'resolved' && !result.response.approved) {
        return { toolResume: 'stop' }
      }
    }
  },
}

Register the definition on the server. Forward parentRunId and resume so a client resolution starts the continuation with its full context.

import {
  chat,
  chatParamsFromRequestBody,
  toServerSentEventsResponse,
} from '@tanstack/ai'
import { openaiText } from '@tanstack/ai-openai'
import { reviewMiddleware, reviewPlan } from './review-plan'

export async function POST(request: Request) {
  const params = await chatParamsFromRequestBody(await request.json())
  const stream = chat({
    adapter: openaiText('gpt-5.5'),
    messages: params.messages,
    threadId: params.threadId,
    runId: params.runId,
    ...(params.parentRunId ? { parentRunId: params.parentRunId } : {}),
    ...(params.resume ? { resume: params.resume } : {}),
    interrupts: [reviewPlan],
    middleware: [reviewMiddleware],
  })

  return toServerSentEventsResponse(stream)
}

Register the same definition on the client. Check kind and definitionId. TypeScript then treats the item as GenericInterrupt<typeof reviewPlan>. resolveInterrupt uses the response shape from reviewPlan.responseSchema.

import { fetchServerSentEvents, useChat } from '@tanstack/ai-react'
import type { GenericInterrupt } from '@tanstack/ai-react'
import { reviewPlan } from './review-plan'

function ReviewCard({
  interrupt,
}: {
  interrupt: GenericInterrupt<typeof reviewPlan>
}) {
  return (
    <button
      onClick={() => interrupt.resolveInterrupt({ approved: true })}
    >
      Approve plan
    </button>
  )
}

export function ReviewPlanPanel() {
  const { interrupts, sendMessage } = useChat({
    connection: fetchServerSentEvents('/api/chat'),
    interrupts: [reviewPlan],
  })

  return (
    <>
      <button onClick={() => sendMessage('Review the release plan')}>
        Start review
      </button>
      {interrupts.map((interrupt) => {
        if (interrupt.kind !== 'generic') return null
        if (!('definitionId' in interrupt)) return null
        if (interrupt.definitionId !== reviewPlan.id) return null
        return <ReviewCard key={interrupt.id} interrupt={interrupt} />
      })}
    </>
  )
}

onInterruptResolution does not run in the chat() call that paused. It runs once at the start of the next chat() call, after the client answers.

setup
onConfig                 (phase is init)
onInterruptResolution    (phase is still init)
onStart
then stop, or continue the agent loop

useChat sends parentRunId and resume on that second request. Each generic resume item includes the original request in metadata. If resume is present and parentRunId is missing, the server throws.

Use resumedInterrupts.for(definition) for one typed definition. Use resumedInterrupts.all() for every registered definition. Use resumedInterrupts.all(definitionA, definitionB) to read a typed subset.

The hook can return toolResume: 'continue', 'cancel', or 'stop'. Results from all middleware combine by the most restrictive rule: stop wins over cancel, and cancel wins over continue.

This hook cannot change prompts, tools, or messages. Store the answer on a capability, then return those fields from onConfig when ctx.phase === 'beforeModel'.

The full resume order, plus an example that writes a user note into the system prompt, is in Apply Answers.

onBeforeToolCall#

Called before a server tool runs, and before a client tool goes to the client. The first middleware that returns a non-void decision skips the remaining middleware for that call.

Each tool call goes through these steps:

1. The engine parses the JSON arguments from the model.
2. onBeforeToolCall gets the parsed input.
3. The tool's inputSchema checks the final input. An async validator is awaited. A schema transform applies once, here.
4. The server tool runs, or the client gets the call, with the checked input.

The schema checks the input after your hook:

• hookCtx.args has no schema defaults or transforms yet, and it can be invalid. Check its shape before you read it.
• A transformArgs value or an in-place edit must fit the schema. If it does not fit, the call gets a tool error, onAfterToolCall runs with ok: false, and the tool does not run.

When the hook runs:

• A client tool runs the hook before the server sends the call to the client. The hook does not run when the client result comes back.
• A client tool gets onAfterToolCall only when the hook skips the call or the schema check fails. The result from the client does not run it. If your middleware pairs the two hooks, return early when hookCtx.tool?.execute is missing.
• A call that waits for approval does not run the hook. A denied call does not run it either.
• After approval, the hook gets the approved input, including any editedArgs. See Tool Approval.

import { type ChatMiddleware } from "@tanstack/ai";

function isRecord(v: unknown): v is Record<string, unknown> {
  return typeof v === "object" && v !== null;
}

const guard: ChatMiddleware = {
  name: "guard",
  onBeforeToolCall: (ctx, hookCtx) => {
    // Block dangerous tools
    if (hookCtx.toolName === "deleteDatabase") {
      return { type: "abort", reason: "Dangerous operation blocked" };
    }

    // Validate and transform arguments
    if (hookCtx.toolName === "search" && isRecord(hookCtx.args) && !hookCtx.args.limit) {
      return {
        type: "transformArgs",
        args: { ...hookCtx.args, limit: 10 },
      };
    }
  },
};

Decision types:

The hookCtx provides:

onAfterToolCall#

Called after each tool execution (or skip). Every middleware runs, in array order.

import { type ChatMiddleware } from "@tanstack/ai";

const toolLogger: ChatMiddleware = {
  name: "tool-logger",
  onAfterToolCall: (ctx, info) => {
    if (info.ok) {
      console.log(`${info.toolName} completed in ${info.duration}ms`);
    } else {
      console.error(`${info.toolName} failed:`, info.error);
    }
  },
};

The info object provides:

Replace a tool result#

Some tools return way more than the model needs: 500 rows, a huge log, a secret you want to hide. The model reads every token of it. Return { type: 'replaceResult', result } and the model gets your version instead:

import { type ChatMiddleware } from "@tanstack/ai";

const firstTwentyRows: ChatMiddleware = {
  name: "first-twenty-rows",
  onAfterToolCall: (ctx, info) => {
    if (!info.ok || !Array.isArray(info.result)) return;
    if (info.result.length <= 20) return;
    return {
      type: "replaceResult",
      result: {
        rows: info.result.slice(0, 20),
        omitted: info.result.length - 20,
      },
    };
  },
};

Good to know:

• Each middleware gets the result of the one before it as info.result. The last replacement wins.
• A failed call stays failed. Your replacement becomes the error that the model reads.
• A skip result from onBeforeToolCall goes through this hook too. info.result is the parsed skip result, the same value the model gets.
• A call that the output limit cut goes through this hook too. It did not run, so ok is false, and error has the code max_tokens. See When the output limit cuts a tool call.

Tool hook order in one turn#

When the model calls several server tools in one turn, the tools run at the same time:

• onBeforeToolCall runs for each call in call order, before any tool starts.
• onAfterToolCall runs for each tool when it finishes, so in finish order.
• A call that onBeforeToolCall skips gets its onAfterToolCall right away, before the tools start.
• If the run aborts before the tools start, no tool runs. Each call gets onAfterToolCall with ok: false and the error "Operation aborted".
• If onBeforeToolCall aborts or throws for a call, the calls before it still run. The error is thrown after they finish.

With toolExecution: 'sequential', each call finishes its onBeforeToolCall, its run, and its onAfterToolCall before the next call starts. See Run tools one at a time.

onUsage#

Called once per model iteration when the RUN_FINISHED chunk includes usage data. Receives the usage object directly.

import { type ChatMiddleware } from "@tanstack/ai";

const usageTracker: ChatMiddleware = {
  name: "usage-tracker",
  onUsage: (ctx, usage) => {
    console.log(
      `Iteration ${ctx.iteration}: ${usage.totalTokens} tokens`
    );
  },
};

The usage object:

Terminal Hooks: onFinish, onAbort, onError#

Exactly one terminal hook fires per chat() invocation. They are mutually exclusive:

onFinish also fires when the invocation pauses at a client-tool or approval wait. The run is not complete yet. To tell a pause from a completion, look in onChunk for a RUN_FINISHED chunk with outcome.type === 'interrupt'.

Separate-finalization path: Adapters without native-combined support make a separate structured-output provider call after the agent loop.

• onStructuredOutputConfig fires before the separate provider call, and ctx.phase is 'structuredOutput' for its chunks.
• onIteration does not fire for finalization; it only fires for agent-loop iterations.
• onFinish fires after finalization completes. Its info object reflects the agent loop's terminal state.
• info.content — the agent loop's accumulated text. Separate-finalization JSON deltas are not included. Middleware can observe the completed result through the structured-output.complete CUSTOM event in onChunk.
• info.usage — the agent loop's last RUN_FINISHED.usage. For a tools-less structured-output run (no agent-loop iteration produces RUN_FINISHED), this is undefined. To capture finalization tokens, use onUsage — that hook fires for every RUN_FINISHED carrying usage, including the finalization call.
• info.finishReason — the agent loop's last finishReason. null when no agent-loop iteration produced RUN_FINISHED (e.g. a tools-less structured-output run).
• info.duration — wall-clock duration of the entire chat() invocation, including finalization.

Native-combined output: Adapters with native-combined support produce the schema-constrained JSON in the regular agent-loop stream. onStructuredOutputConfig does not fire, ctx.phase remains 'modelStream', and onIteration fires for the iteration that produces the JSON. The JSON is agent-loop text, so info.content includes it. Middleware observes the structured-output.complete event in onChunk during the same phase.

On successful completion in either path, onFinish receives the complete canonical transcript in ctx.messages. Native-combined output keeps the structured result on its terminal assistant message. The separate-finalization path can preserve the agent loop's plain-text assistant message followed by a distinct structured-output assistant message. This transcript is separate from the path-specific fields on info.

To aggregate usage across the whole run, accumulate from onUsage callbacks rather than relying on info.usage.

import { type ChatMiddleware } from "@tanstack/ai";

const terminal: ChatMiddleware = {
  name: "terminal",
  onFinish: (ctx, info) => {
    console.log(`Finished: ${info.finishReason}, ${info.duration}ms`);
    console.log(`Content: ${info.content}`);
    if (info.usage) {
      console.log(`Tokens: ${info.usage.totalTokens}`);
    }
  },
  onAbort: (ctx, info) => {
    console.log(`Aborted: ${info.reason}, ${info.duration}ms`);
  },
  onError: (ctx, info) => {
    console.error(`Error after ${info.duration}ms:`, info.error);
  },
};

The info object for onFinish (FinishInfo):

Context Object#

Every hook receives a ChatMiddlewareContext as its first argument. It provides request-scoped information and control functions:

Typed Runtime Context#

ChatMiddleware accepts a context generic. This lets reusable middleware declared outside chat() access the same typed runtime context as your tools.

import { chat, type ChatMiddleware } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { session, audit } from "./server";

type AppContext = {
  userId: string;
  audit: {
    write(event: { userId: string; requestId: string }): Promise<void>;
  };
};

export const auditMiddleware: ChatMiddleware<AppContext> = {
  name: "audit",
  onStart(ctx) {
    ctx.defer(
      ctx.context.audit.write({
        userId: ctx.context.userId,
        requestId: ctx.requestId,
      })
    );
  },
};

chat({
  adapter: openaiText("gpt-5.5"),
  messages: [{ role: "user", content: "Hello" }],
  middleware: [auditMiddleware],
  context: {
    userId: session.user.id,
    audit,
  },
});

When typed middleware or typed tools are present, chat() checks that the provided context matches the required shape. Existing middleware typed as plain ChatMiddleware still works; its ctx.context remains unknown and does not force a context option.

Runtime context is process-local application state. It is separate from AG-UI RunAgentInput.context, which is protocol metadata parsed by chatParamsFromRequest. See Runtime Context for server, client, and client-to-server handoff patterns.

Aborting from Middleware#

Call ctx.abort() to gracefully stop the run. This triggers the onAbort terminal hook:

import { type ChatMiddleware } from "@tanstack/ai";

const timeout: ChatMiddleware = {
  name: "timeout",
  onChunk: (ctx) => {
    if (ctx.chunkIndex > 1000) {
      ctx.abort("Too many chunks");
    }
  },
};

Deferred Side Effects#

Use ctx.defer() to register promises that run after the terminal hook without blocking the stream:

import { type ChatMiddleware } from "@tanstack/ai";

const analytics: ChatMiddleware = {
  name: "analytics",
  onFinish: (ctx, info) => {
    ctx.defer(
      fetch("/api/analytics", {
        method: "POST",
        body: JSON.stringify({
          requestId: ctx.requestId,
          duration: info.duration,
          tokens: info.usage?.totalTokens,
        }),
      })
    );
  },
};

Composing Multiple Middleware#

Middleware execute in array order. The ordering matters for hooks that pipe or short-circuit:

import { chat, type ChatMiddleware } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { authMiddleware, loggingMiddleware, cachingMiddleware } from "./middleware";

const stream = chat({
  adapter: openaiText("gpt-5.5"),
  messages: [{ role: "user", content: "Hello" }],
  middleware: [authMiddleware, loggingMiddleware, cachingMiddleware],
});

Composition Rules#

Capabilities#

Middleware often need to share state. A provider middleware sets something up (a database handle, a per-request counter, a sandbox), and a consumer middleware reads it back later in the same run. Capabilities make that hand-off type-safe and order-checked: the consumer declares what it needs, the provider declares what it offers, and chat() refuses to run (at compile time and at runtime) if a required capability was never provided.

Creating a capability#

A capability is created with createCapability<TValue>()('name') — a curried call:

import { createCapability } from "@tanstack/ai";

const counterCapability = createCapability<{ value: number }>()("counter");
const [getCounter, provideCounter] = counterCapability;

The currying is deliberate: you supply the value type explicitly (<{ value: number }>) while the name literal is inferred from the argument ("counter"). A single createCapability<T>('name') call can't do both — supplying T explicitly stops TypeScript inferring the name, collapsing it to string and defeating the compile-time coverage check that keys on the literal name.

The returned counterCapability is a hybrid value:

• It destructures to [get, provide] — the two accessors you use inside hooks.
• It is itself the identity you list in requires / provides. There is no separate token to import.

The accessors:

Equivalently, the context exposes ctx.get(capability), ctx.getOptional(capability), and ctx.provide(capability, value) — pass the capability handle directly. These are typed by the handle you pass (ctx.get(counterCapability) returns the value type), so getCounter(ctx) and ctx.get(counterCapability) are interchangeable — use whichever reads better in your hook.

Capability names must be unique across your app. The compile-time coverage check keys on the name literal (runtime keys on the handle reference), so two capabilities sharing a name will conflate in the type-level check.

The setup hook#

Provisioning happens in a dedicated setup(ctx) hook. It runs first — before any onConfig (init), across all middleware in array order — so that by the time the rest of the lifecycle begins, every capability is in place. setup receives the stable ChatMiddlewareContext (not the mutable config), and may be async.

requires / provides / optionalRequires#

Three array fields on a middleware declare its capability contract. Each is a ReadonlyArray<CapabilityHandle> — you list the capability handles themselves:

Array example#

Author middleware with defineChatMiddleware — it sharpens the requires / provides tuple types so the coverage check and builder can read them precisely. Here a provider sets up a counter in setup, and a consumer reads it in a hook:

import {
  chat,
  createCapability,
  createChatMiddleware,
  defineChatMiddleware,
} from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";

const counterCapability = createCapability<{ value: number }>()("counter");
const [getCounter, provideCounter] = counterCapability;

// Provider: declares `provides` and provisions the value in `setup`.
const withCounter = defineChatMiddleware({
  name: "with-counter",
  provides: [counterCapability],
  setup(ctx) {
    provideCounter(ctx, { value: 0 });
  },
});

// Consumer: declares `requires` and reads the value via `get` in a hook.
const countsChunks = defineChatMiddleware({
  name: "counts-chunks",
  requires: [counterCapability],
  onChunk(ctx) {
    getCounter(ctx).value++;
  },
  onFinish(ctx) {
    console.log(`Saw ${getCounter(ctx).value} chunks`);
  },
});

const stream = chat({
  adapter: openaiText("gpt-5.5"),
  messages: [{ role: "user", content: "Hello" }],
  // Provider must come before the consumer.
  middleware: [withCounter, countsChunks],
});

If you drop withCounter from the array, chat() reports a compile-time error at the middleware option naming the missing "counter" capability — and throws at runtime before the adapter is ever called.

Builder example#

createChatMiddleware() builds the array through chained .use() calls and enforces provider-before-consumer ordering at compile time: each .use() requires that the middleware's requires are already covered by capabilities provided by earlier .use() calls.

import {
  chat,
  createCapability,
  createChatMiddleware,
  defineChatMiddleware,
} from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";

const counterCapability = createCapability<{ value: number }>()("counter");
const [getCounter, provideCounter] = counterCapability;

const withCounter = defineChatMiddleware({
  name: "with-counter",
  provides: [counterCapability],
  setup(ctx) {
    provideCounter(ctx, { value: 0 });
  },
});

const countsChunks = defineChatMiddleware({
  name: "counts-chunks",
  requires: [counterCapability],
  onChunk(ctx) {
    getCounter(ctx).value++;
  },
});

const middleware = createChatMiddleware()
  .use(withCounter) // provides "counter"
  .use(countsChunks) // requires "counter" — OK, already provided above
  .build();

const stream = chat({
  adapter: openaiText("gpt-5.5"),
  messages: [{ role: "user", content: "Hello" }],
  middleware,
});

Swap the two .use() calls (.use(countsChunks).use(withCounter)) and the builder rejects it at the .use(countsChunks) line — the consumer is ordered before its provider, so "counter" isn't in the provided set yet.

Validation guarantees#

The capability system fails loudly and early:

• Compile-time coverage. A required capability that nothing provides surfaces as a type error at the middleware option. This is enforced two ways: an array coverage check on middleware: [...], and the order-aware createChatMiddleware() builder (which additionally enforces ordering).
• Runtime coverage. Even if types are bypassed, chat() validates coverage and throws before the adapter runs if a required capability is missing.
• Post-setup assertion. If a middleware declares a capability in provides but never calls its provide accessor during setup, chat() throws after the setup phase — you can't silently forget to provision.
• Duplicate provide → last-wins + warning. If two middleware provide the same capability, the last write wins and a development warning is emitted.
• Unique names. Capability names must be unique across your app; the compile-time coverage check keys on the name literal (runtime keys on the handle reference).

Append records to the session log#

A host is a middleware that keeps a session log and provides LogRecordsCapability. chat() does not provide it, so without a host there is no log. With a host, your middleware can write its own records to that log, for example a note that a run started. Read the writer with getLogRecords:

import {
  LogRecordsCapability,
  defineChatMiddleware,
  getLogRecords,
} from "@tanstack/ai";

const noteStart = defineChatMiddleware({
  name: "note-start",
  optionalRequires: [LogRecordsCapability],
  async onStart(ctx) {
    const log = getLogRecords(ctx, { optional: true });
    await log?.append([{ type: "app.note", text: "run started" }]);
  },
});

• Each record needs a type. The other fields are yours.
• The host decides which record types it accepts. If the host refuses a record, append rejects.
• Without a host log, getLogRecords(ctx, { optional: true }) returns undefined. Use the optional read when your middleware also runs without a log.
• If your middleware cannot work without the log, list the capability in requires and call getLogRecords(ctx). Then chat() throws when no host provides the log.

If you write a host, provide the log in setup and list it in provides:

import {
  LogRecordsCapability,
  defineChatMiddleware,
  provideLogRecords,
} from "@tanstack/ai";

const withSessionLog = defineChatMiddleware({
  name: "with-session-log",
  provides: [LogRecordsCapability],
  setup(ctx) {
    // One log per run. A real host writes to its durable store here.
    const records: Array<{ type: string }> = [];
    provideLogRecords(ctx, {
      append: async (batch) => {
        records.push(...batch);
      },
    });
  },
});

Built-in Middleware#

TanStack AI ships ready-made middleware for common cases — caching tool results, redacting streamed text, and OpenTelemetry tracing:

See Built-in Middleware for full options and examples for each. The recipes below show how to build your own.

Recipes#

Live custom events#

A long onConfig hook can send progress to the client while it waits. Call ctx.emitCustomEvent when the work starts, then again when it finishes:

import { type ChatMiddleware } from "@tanstack/ai";

async function prepare() {
  await new Promise<void>((resolve) => {
    setTimeout(resolve, 1);
  });
}

const progress: ChatMiddleware = {
  name: "progress",
  async onConfig(ctx) {
    if (ctx.phase !== "beforeModel") return;
    ctx.emitCustomEvent("job:started", { step: "prepare" });
    await prepare();
    ctx.emitCustomEvent("job:ended", { step: "prepare" });
  },
};

The engine yields each CUSTOM chunk as soon as you call emitCustomEvent. If RUN_STARTED is not on the wire yet, the engine sends it first. Durability then flushes the event so the client can render a live indicator. Read these events on the client the same way as tool emitCustomEvent calls. See Custom Events.

Rate Limiting#

Limit the number of tool calls per request:

import { type ChatMiddleware } from "@tanstack/ai";

function rateLimitMiddleware(maxCalls: number): ChatMiddleware {
  let toolCallCount = 0;
  return {
    name: "rate-limit",
    onBeforeToolCall: (ctx, hookCtx) => {
      toolCallCount++;
      if (toolCallCount > maxCalls) {
        return {
          type: "abort",
          reason: `Rate limit: exceeded ${maxCalls} tool calls`,
        };
      }
    },
  };
}

Audit Trail#

Log every action for compliance:

import { type ChatMiddleware } from "@tanstack/ai";
import { db } from "./db";

const auditTrail: ChatMiddleware = {
  name: "audit-trail",
  onStart: (ctx) => {
    ctx.defer(
      db.auditLog.create({
        requestId: ctx.requestId,
        event: "chat_started",
        timestamp: Date.now(),
      })
    );
  },
  onAfterToolCall: (ctx, info) => {
    ctx.defer(
      db.auditLog.create({
        requestId: ctx.requestId,
        event: "tool_executed",
        toolName: info.toolName,
        success: info.ok,
        duration: info.duration,
        timestamp: Date.now(),
      })
    );
  },
  onFinish: (ctx, info) => {
    ctx.defer(
      db.auditLog.create({
        requestId: ctx.requestId,
        event: "chat_finished",
        duration: info.duration,
        tokens: info.usage?.totalTokens,
        timestamp: Date.now(),
      })
    );
  },
};

Per-Iteration Tool Swapping#

Expose different tools at different stages of the agent loop:

import { type ChatMiddleware } from "@tanstack/ai";

const toolSwapper: ChatMiddleware = {
  name: "tool-swapper",
  onConfig: (ctx, config) => {
    if (ctx.phase !== "beforeModel") return;

    if (ctx.iteration === 0) {
      // First iteration: only allow search
      return {
        tools: config.tools.filter((t) => t.name === "search"),
      };
    }
    // Later iterations: allow all tools
  },
};

Content Filtering#

Drop or transform chunks before they reach the consumer:

import { type ChatMiddleware } from "@tanstack/ai";
import { containsProfanity } from "./filters";

const contentFilter: ChatMiddleware = {
  name: "content-filter",
  onChunk: (ctx, chunk) => {
    if (chunk.type === "TEXT_MESSAGE_CONTENT") {
      if (containsProfanity(chunk.delta)) {
        // Drop the chunk entirely
        return null;
      }
    }
  },
};

Error Recovery with Retry Logging#

import { type ChatMiddleware } from "@tanstack/ai";
import { alertService } from "./services";

const errorRecovery: ChatMiddleware = {
  name: "error-recovery",
  onError: (ctx, info) => {
    ctx.defer(
      alertService.send({
        level: "error",
        message: `Chat ${ctx.requestId} failed after ${info.duration}ms`,
        error: String(info.error),
      })
    );
  },
};

TypeScript Types#

The core middleware types are exported from @tanstack/ai:

import type {
  ChatMiddleware,
  ChatMiddlewareContext,
  ChatMiddlewarePhase,
  ChatMiddlewareConfig,
  StructuredOutputMiddlewareConfig,
  ToolCallHookContext,
  BeforeToolCallDecision,
  AfterToolCallInfo,
  AfterToolCallDecision,
  IterationInfo,
  ToolPhaseCompleteInfo,
  UsageInfo,
  FinishInfo,
  AbortInfo,
  ErrorInfo,
} from "@tanstack/ai";

The option/type surfaces for the built-in middleware are exported from the @tanstack/ai/middlewares subpath (not the main barrel):

import type {
  ToolCacheMiddlewareOptions,
  ToolCacheStorage,
  ToolCacheEntry,
  ContentGuardMiddlewareOptions,
  ContentGuardRule,
  ContentFilteredInfo,
} from "@tanstack/ai/middlewares";

Next Steps#

• Built-in Middleware — toolCacheMiddleware, contentGuardMiddleware, otelMiddleware
• Compaction: keep long chats under the context limit with withCompaction
• OpenTelemetry — emit traces and metrics via otelMiddleware- Tools — Learn about the isomorphic tool system
• Agentic Cycle — Understand the multi-step agent loop
• Streaming — How streaming works in TanStack AI