The right model for every prompt.
Automatically.
Mesh places a single OpenAI-compatible API in front of 1,000+ models from 30+ providers. Routing is split into two independent planes: a reliability plane that keeps a request alive across provider failures, and a performance plane that selects which model should serve each prompt. Every decision is observable per request, governed by organization-level policy, and identical across SaaS, VPC, and on-premises deployments.
Two routing planes, fully independent.
The performance plane picks the target model; the reliability plane guarantees the attempt chain around it. Configured independently, identical across SaaS, VPC, and on-prem.
Smart routing is deterministic and local. No model is invoked to pick a model.
Classified deterministically from surface signals. Same prompt, same route, every time.
Only Pareto-non-dominated models are eligible, so no weight setting can select a worse-and-pricier model.
An unknown task shape, a missing table, or an empty candidate set returns your pinned model.
Per response in band, per request in the logs, per decision on demand: ranked candidates with capability, cost, and speed components.
Request and response bodies are not stored; the entire routing decision trail is retained in full. Auditability without retaining prompts.
Two routing planes, fully independent.
Every request passes through both planes, and they are configured independently. The performance plane picks the target; the reliability plane guarantees the attempt chain around it.
Reliability plane
What happens when this model or provider fails?
Organization default or per API key
Requests survive provider incidents
Performance plane
Which model should serve this prompt?
Per API key or per request
Best available intelligence per dollar, per prompt
Failover as policy, not retry code.
Failover in Mesh is a declarative routing policy, not client-side retry code. A policy defines the ladder a request may climb: a configurable number of retries against the original target, provider fallback where the same model is served by an alternate provider, and model fallback down an ordered list supplied in the request body (models: [...]) or set in the policy. The order of provider-first versus model-first fallback is definable.
Budgets the entire chain across all stages, so a request can never spiral through unbounded retries.
Caps time spent switching (for example, at most one second), so failover can never violate the caller's service-level objective. When the budget is exhausted, the chain aborts to a clean error rather than silently exceeding the response-time commitment.
Policies live at the organization level and can be overridden per API key: the platform team sets a safe default, and individual workloads tighten or relax it. The provider and model actually used are reported on every response, so a fallback is never invisible.
A dial from full control to full dynamism.
Model selection is a dial from full control to full dynamism. The three modes coexist in one organization and are chosen per API key or per request.
A deterministic local engine classifies the prompt and picks from pre-computed per-task model frontiers. No model is invoked to route.
The request names a concrete model. No routing; the governance floor.
The request sends model: "auto" and a gateway-side classifier model selects a concrete model per request. The resolved model is returned on every response (X-Resolved-Model-Id); POST /v1/router/select previews the pick without running inference.
Six deterministic stages. One pick.
The smart router is a pipeline of deterministic stages, engineered so the routing layer can be neither a failure mode nor a cost center. Because every stage is deterministic, the same prompt under the same settings routes identically, which makes routing behavior testable and auditable.
Assigns the prompt to one of 16 task cohorts (coding, agentic tool use, extraction, mathematical reasoning, writing, vision, long context, web research, bulk classification, and more) using deterministic signals. Conversational follow-ups inherit the conversation's cohort.
Estimates per-prompt difficulty from surface signals such as constraint density and task markers. Easy prompts shift weight toward cost; hard prompts shift weight toward capability. Callers can force a fixed level.
Removes models whose context window cannot fit the request, and requires probe-verified support for tool calling and vision rather than trusting catalog metadata.
Ranks the cohort's candidate frontier under the caller's cost, capability, and speed weights.
Grants the incumbent model a bonus while it remains competitive, preventing mid-conversation model changes that would discard provider prompt caches.
An unknown task shape, a missing table, or an empty candidate set at any stage returns the caller's pinned model. The routing layer cannot break a request.
See it route, in the Mesh CLI.
The screenshots below show the routing surface in the Mesh CLI, captured from a live session against the production gateway. They are examples rather than benchmarks: concrete picks, scores, and prices vary with the configured weights, live catalog pricing, and the routing-table version.
Dominated models never get picked.
The economics of model choice follow an efficiency frontier: at any spend level there is a best-available capability, and most models sit strictly below that curve because some other model is both better and cheaper. Mesh makes the frontier the routing data structure itself.
Within each task cohort, only Pareto-non-dominated models are eligible. A model that is worse and more expensive than an alternative is excluded outright, so no weight setting, however misconfigured, can select a dominated model. Callers express preference as weights over cost, capability, and speed: equal weights select the balanced point on the frontier, and skewing the weights slides the pick along the curve. A quality floor is enforced underneath: candidates below the cohort's competence floor are dropped whenever anything better is available. Finally, the difficulty tilt turns this into marginal-gain routing: the router pays frontier prices only when the frontier model's advantage is real for the prompt at hand.
The difficulty tilt in practice
Continuing the session above: the weights are skewed toward capability, a structurally hard prompt classifies as mathematical reasoning at high difficulty, and the router selects a frontier model. The same session that answered a trivial question for fractions of a cent escalates only when the prompt warrants it, and the per-request cost is reported either way.
Benchmarked in the open, caveats included.
A routing claim is easy to make and hard to check, so Mesh evaluates its auto router the same way it builds it: with evidence, published in the open. The evaluation harness and its results are public at github.com/aifiesta/mesh-benchmarks, and the evaluation runs in two passes: a cheap, fully reproducible replay on public data to check the decision logic, then a live pass on real traffic with an independent judge. Figures below are from the September 2026 run: seven routing strategies plus four baselines, 36,497 offline tasks, and 91 live prompts graded by Claude Opus 4.8.
routing vs a random model pick, offline replay (n = 36,497)
judged quality of what production actually served: the bar every strategy had to beat
lower cost than the served reference on live traffic, at comparable or better judged quality
harness, results, and caveats at github.com/aifiesta/mesh-benchmarks
Offline replay on a public benchmark
The router's picks are replayed over 36,497 tasks from RouterBench, a public routing benchmark that ships precomputed correctness scores and costs for 11 models. For each task the strategy names a model and the harness looks up that model's known score and cost: no live calls, no spend, fully reproducible. The router's own overhead is charged in every number; the internal classifier call adds a 4.8 percent cost surcharge to table-routed requests, and the harness counts it.
The ranking table beats random assignment by 42.7 percent relative, which confirms the routing signal is real. Two honest qualifiers sit next to that result. On RouterBench's fixed model set, which dates from early 2024, the table does not beat "always use the most expensive model" on quality. And a perfect-hindsight oracle reaches 0.912 at roughly 14 times lower cost than the premium baseline, because on most tasks a cheap model is also correct: the headroom for cost-aware routing is large, and it is exactly what the difficulty tilt targets. The offline pass tests decision logic on a small, dated model pool, not the current 1,000+ model catalog, which is why the live pass exists.
| Policy | Mean score | Cost / request |
|---|---|---|
| Hindsight oracle (ceiling) | 0.912 | $0.000242 |
| Always the most expensive model | 0.781 | $0.003293 |
| Mesh ranking table | 0.741 | $0.003267 |
| Weighted (quality + cost blend) | 0.700 | $0.003217 |
| Random assignment | 0.519 | $0.000824 |
| Always the cheapest model | 0.306 | $0.000046 |
Costs include the classifier surcharge.
Live traffic, independently judged
Each strategy routes 91 real, multilingual prompts over the live catalog; the chosen model produces a genuine answer, and a strong LLM judge (Claude Opus 4.8) scores every answer from 0 to 1 on correctness, helpfulness, and instruction-following. Roughly 1,270 live calls, deduped and cached. The reference point is the model that was actually served for each prompt in production.
| Strategy | Judged quality | Cost / request |
|---|---|---|
| Heuristic router | 0.675 | $0.00694 |
| Mesh ranking table | 0.673 | $0.00687 |
| Registry classifier | 0.660 | $0.00386 |
| Actually served (reference) | 0.657 | $0.01436 |
| Weighted (default) | 0.636 | $0.00322 |
The quality leaders beat the served reference at about half its cost. The default weighted profile gives up about 3 percent quality for a 4.5 times cost cut, steering visibly toward cheaper models: that trade is the reason the profile exists, and it is the dial the weight settings expose. The registry classifier's score is an artifact: it picked one model for all 91 prompts because a reasoning model's "thinking budget" consumed its small token budget, truncating the model-ID response; the workaround (disable reasoning) is in place. Baselines built on public capability scores are omitted: the vendor-published numbers behind them proved stale enough that some models could not answer basic factual prompts. Full board, seven strategies and four baselines, in the repository.
Tuning the weighted dial
The weighted router scores candidates on three components. The evaluation measured each preference profile on the live pass and re-tuned the component weights against the observed quality-cost frontier; the current vectors:
| Profile | Quality | Cost | Speed |
|---|---|---|---|
| Quality-first | 0.75 | 0.15 | 0.10 |
| Balanced (default) | 0.40 | 0.35 | 0.25 |
| Cost-first | 0.15 | 0.70 | 0.15 |
| Latency-first | 0.15 | 0.25 | 0.60 |
The Balanced profile moves slightly toward cost relative to its pre-evaluation setting (0.40 / 0.30 / 0.30). None of these profiles has yet been run as a full live-pass variant; that head-to-head is next in the evaluation queue.
The public benchmark and the real-traffic pass rank the approaches the same way: ranking table at 0.74 offline and 0.67 live, weighted at 0.70 and 0.64, random at 0.52 and 0.48. Scores dip a little on real traffic, which is expected; real prompts are harder and free of benchmark contamination. That consistency is what lets the cheap offline pass predict live behavior between full evaluations.
Benchmarks that do not change the product are theater. This one shipped changes: the fast lane migrated from 5% to 100% of auto traffic; the registry classifier's truncation bug was fixed; the weighted profiles behind the Balanced / Quality / Cost / Fastest preference picker were re-tuned against the measured frontier (table above); and admins can now edit strategy weights at runtime, with no redeploy. The evaluation also surfaced a catalog-health gap (models listed that cannot serve basic calls); probe-verified capability checks in the feasibility filter are the remediation, scoped as a data-quality pass.
Stated plainly, because a benchmark that hides its own caveats is not one. The live pass uses a single judge model, one pass, rubric-based; judge biases are not corrected and multi-judge agreement is not yet checked. Ninety-one prompts is enough to see the cost separation clearly but too few for statistical significance, which is scoped for a larger run. Offline scores come from a dated, partly contaminated public set and are valid only for comparing strategies, never as absolute quality claims for any model. Cost comparisons are directionally right; exact multiples depend on token caps that differ between the evaluation and production.
The offline pass runs from the public repository with one command (python -m router_eval.replay --source routerbench --shots 0) against the public dataset. The per-strategy aggregates are committed alongside the harness.
Routing you can audit, down to the decision.
Routing that cannot be audited cannot be trusted in production. Mesh exposes its decisions at three levels:
Every response carries the resolved model, routing attempts made and any fallback taken, provider latency, cache status, and a unique request ID that keys the dashboard log entry.
Model requested versus model served, token counts, cost and platform fee, latency, status, and the API key and team member behind the call, filterable by organization, team, project, and key.
The router's ranked view lists the candidates considered with their capability, cost, and speed components: why this model won, and what came second and third.
Mesh runs with zero data retention (ZDR) by default: request and response bodies are not stored, while routing metadata, that is, the entire decision trail above, is retained in full. Auditability does not depend on retaining prompts, a combination regulated customers typically require. Organizations that want payload-level logs can disable ZDR explicitly.
The router optimizes inside your limits.
The routing engine optimizes strictly inside limits the organization sets. Controls compose from the organization down to the individual key.
Default routing policy (retries, fallbacks, attempts, latency budget), roles, teams, organization-wide rate limits.
Per-member rate limits (e.g. tokens per minute) and spend caps (e.g. $1,000 per month).
Allowed-model lists; requests per minute and per day; tokens per minute; total spend cap; per-model spend within the key; routing-policy override; expiry.
API behavior itself is versioned with a dated header (X-Mesh-Version). Traffic that does not pin a version stays on the baseline indefinitely; behavior moves only when the customer opts in.
One engine, three deployment models.
The same routing engine runs in three deployment models, all in production use today.
Gateway, dashboard, and routing run in Mesh's cloud. Fastest to adopt.
The gateway and dashboard deploy inside the customer's cloud account. With custom per-model endpoints, models point at the customer's own Azure OpenAI, AWS Bedrock, or Google Vertex deployments, and that traffic never leaves the customer's cloud perimeter.
The full stack (gateway, dashboard, routing) deploys on customer infrastructure, with the same custom-endpoint model.
Customer-held provider keys (OpenAI, Azure, AWS, GCP, and others) plug in directly, preserving negotiated pricing and existing credits, with Mesh's routing, failover, observability, and governance layered on top. Coverage spans 30+ providers, including regional and sovereign options, and expands on customer demand.
Data protections include zero data retention by default, encryption in transit, no-training clauses in enterprise provider agreements for routed traffic, prompt-injection prevention, DDoS protection, and PII protection at the gateway, with private connectivity (e.g. SSH tunneling) available.
A base-URL change. That's it.
Mesh is OpenAI-compatible: adopting it is a base-URL and API-key change in any OpenAI SDK, with no request-shape migration. An Anthropic-compatible surface (/v1/messages) is also available and is not limited to Anthropic models. Routing features are additive request fields (model: "auto"; models: [...] for fallback), so teams can adopt them incrementally, per workload.
Documentation is published for humans and for machines: full developer documentation at developers.meshapi.ai, machine-readable indexes (llms.txt, llms-full.txt), and a complete OpenAPI 3.1 specification.
# OpenAI SDK → Meshbase_url = "https://api.meshapi.ai/v1"api_key = "rsk_..." # Opt into routing, per workloadmodel = "auto"models = ["gpt-4o", "claude-sonnet"]Frequently asked questions
For a team already using an OpenAI-compatible SDK, adoption is a base-URL and API-key change. Routing features are opt-in request fields adopted per workload. Enterprise onboarding (SSO, roles, policies, security review, deployment model) proceeds alongside and does not block the first request.
- Mesh API, "Router evaluation: offline RouterBench replay and live judged pass," public harness and results, September 2026. github.com/aifiesta/mesh-benchmarks
- Q. J. Hu et al., "RouterBench: A Benchmark for Multi-LLM Routing Systems," 2024. Dataset: huggingface.co/datasets/withmartian/routerbench, doi:10.57967/hf/1996.
Asia's largestsovereign AI gateway.
1,000+ AI models. One endpoint. Your data stays inside your border.