Model Routing
The practice of choosing which LLM (and which reasoning effort) to invoke per agent turn based on task class, input signals, or previous-turn outcome — rather than running everything on one model. Common patterns include two-tier (cheap default, Reasoning Model escalation on hard turns), reasoning-at-plan / cheap-at-execution (one expensive planning call, cheap execution turns), and judge-and-repair (cheap draft, reasoning judge, reasoning repair on failure). Model routing is the single highest-leverage cost lever in 2026 agent deployments — well-routed workloads typically cost 3-10× less than same-quality single-model equivalents. Requires observability (see Agent Observability) to measure whether the routing is actually working.
Written by Max Zeshut
Founder at Agentmelt · Last updated Sep 9, 2026
Example
A coding agent's turn types: plan-time (10% of turns) → o3 with medium reasoning; write-diff (30%) → Sonnet 4 with extended thinking budget 8K; execute-test / read-file (55%) → Haiku or Gemini 2.5 Flash; format-summary (5%) → Gemini Flash-Lite. Total cost per task drops from $2.40 (all-o3) to $0.31 (routed) with equivalent quality on the acceptance benchmark. Wins hold only if the router is measured — bad routing degrades quality without warning, so tie routing changes to your eval suite.
Frequently asked questions
- Isn't model routing just picking the right default?
- No — a default is per-agent, routing is per-turn. Same user, same conversation: the plan turn goes to a reasoning model, the next five execution turns go to a cheap model, the summary turn goes to Flash-Lite. Routing decisions need signals (task type, input length, retry count) that a static default can't capture.
- Where does routing logic live?
- Either in a thin custom classifier LLM that runs before each turn, or in the framework (LangGraph conditional edges, LlamaIndex routers). The heuristic form — a small function that inspects the input — is cheaper and easier to debug; the LLM-classifier form catches cases the heuristic misses. Most production stacks use both: heuristics for the 80% obvious cases, LLM classifier for the tail.