The right model. Every request.
Arbitex Gateway routes across 9+ LLM providers with cost-based selection, latency-aware fallback, and automatic circuit breakers — transparent to the application layer.
Three modes for every use case
From cost-optimized single routing to multi-model synthesis — one gateway, one API, any pattern.
Single
Route each request to the best-matched provider based on cost, latency, and routing rules. A single response is returned. Ideal for production workloads where determinism and cost efficiency matter most.
Compare
Send the same prompt to multiple providers simultaneously and return all responses side by side. Useful for evaluating model quality, running A/B experiments, and auditing output consistency across providers — without changing the application layer.
Summarize
Broadcast the prompt to multiple providers, collect all responses, then synthesize a single consolidated answer. Reduces hallucination risk by cross-referencing multiple models — effective for research tasks, document analysis, and high-stakes queries.
Every major provider. One gateway.
- Anthropic
- OpenAI
- Google Gemini
- Ollama
- Mistral
- Cohere
- AWS Bedrock
- Azure OpenAI
- Groq
- + BYOE
Cost-Based Routing
Set cost thresholds per org, per team, and per request type. The router selects the cheapest provider capable of handling the request — falling back to higher-cost providers only when cheaper options are unavailable or degraded. Budget caps and usage quotas integrate directly with routing decisions.
Latency-Aware Fallback
Real-time latency tracking scores each provider based on recent response times. When a provider's latency exceeds threshold or a request times out, the router automatically falls back to the next provider in the configured chain — without the application needing to handle the failure.
Provider Health Monitoring
Circuit breakers track error rates per provider. When a provider exceeds the error threshold, it's automatically removed from the routing pool for 5 minutes and traffic shifts to healthy alternatives. Providers re-enter the pool automatically when the reset window expires — no manual intervention needed.
Bring Your Own Endpoint
BYOE lets teams route to internal or fine-tuned models hosted outside the standard provider list — private LLMs, on-premises deployments, or experimental models. Any OpenAI-compatible endpoint can be registered and used in routing chains alongside managed providers.
Fallback Chains
Define ordered fallback sequences per org or per request type. If the primary provider is unavailable, the router steps through the chain until a healthy provider responds. Chain configuration is centrally managed — no code changes required to update routing priorities.
Model Selection Rules
Route based on request content, identity, and context — not just availability. Send compliance-sensitive requests to on-premises models, route high-volume low-stakes tasks to cost-efficient providers, and keep regulated data within approved infrastructure automatically.
How it works
Request arrives, routing rules evaluated
Each request entering Arbitex Gateway is evaluated against the configured routing rules — cost thresholds, content type, identity context, and compliance requirements. The routing engine selects the appropriate mode (single, compare, or summarize) and identifies the provider pool for this request.
Provider selection with health and cost scoring
Candidate providers are scored in real time against latency history, error rates, and cost. Circuit breakers automatically exclude degraded providers. The router picks the optimal provider — or multiple providers if the mode requires — from the healthy pool.
Fallback executes transparently if needed
If the selected provider fails or times out, the router steps through the fallback chain immediately — no error surfaces to the application. All routing decisions, provider selections, and fallback events are logged to the tamper-evident audit trail for full observability.