Isn't model routing just picking the right default?

No — a default is per-agent, routing is per-turn. Same user, same conversation: the plan turn goes to a reasoning model, the next five execution turns go to a cheap model, the summary turn goes to Flash-Lite. Routing decisions need signals (task type, input length, retry count) that a static default can't capture.

Where does routing logic live?

Either in a thin custom classifier LLM that runs before each turn, or in the framework (LangGraph conditional edges, LlamaIndex routers). The heuristic form — a small function that inspects the input — is cheaper and easier to debug; the LLM-classifier form catches cases the heuristic misses. Most production stacks use both: heuristics for the 80% obvious cases, LLM classifier for the tail.

Model Routing

Written by Max Zeshut

Founder at Agentmelt · Last updated Jul 22, 2026

The practice of choosing which LLM (and which reasoning effort) to invoke per agent turn based on task class, input signals, or previous-turn outcome — rather than running everything on one model. Common patterns include two-tier (cheap default, Reasoning Model escalation on hard turns), reasoning-at-plan / cheap-at-execution (one expensive planning call, cheap execution turns), and judge-and-repair (cheap draft, reasoning judge, reasoning repair on failure). Model routing is the single highest-leverage cost lever in 2026 agent deployments — well-routed workloads typically cost 3-10× less than same-quality single-model equivalents. Requires observability (see Agent Observability) to measure whether the routing is actually working.

Example

A coding agent's turn types: plan-time (10% of turns) → o3 with medium reasoning; write-diff (30%) → Sonnet 4 with extended thinking budget 8K; execute-test / read-file (55%) → Haiku or Gemini 2.5 Flash; format-summary (5%) → Gemini Flash-Lite. Total cost per task drops from $2.40 (all-o3) to $0.31 (routed) with equivalent quality on the acceptance benchmark. Wins hold only if the router is measured — bad routing degrades quality without warning, so tie routing changes to your eval suite.

Frequently asked questions

Isn't model routing just picking the right default?: No — a default is per-agent, routing is per-turn. Same user, same conversation: the plan turn goes to a reasoning model, the next five execution turns go to a cheap model, the summary turn goes to Flash-Lite. Routing decisions need signals (task type, input length, retry count) that a static default can't capture.
Where does routing logic live?: Either in a thin custom classifier LLM that runs before each turn, or in the framework (LangGraph conditional edges, LlamaIndex routers). The heuristic form — a small function that inspects the input — is cheaper and easier to debug; the LLM-classifier form catches cases the heuristic misses. Most production stacks use both: heuristics for the 80% obvious cases, LLM classifier for the tail.

Related glossary terms

Related niches

Back to glossary

Loading…