# Choosing an AI Gateway in 2026: A performance-based analysis of self-hosted options

*Note: this is the first in a new series on AI tool bakeoffs — looking at the numbers and features across hot, emerging AI tool categories.*

For just about anyone running AI inference in production, a dream quickly emerges: can you treat inference as a commodity market? Can you take any unit of inference you need, give it a simple classification, and route it to whichever model is cheapest, fastest, or not currently experiencing downtime **at that moment**?

Enter the AI gateway.

Every team running multiple LLMs in production eventually hits a similar wall: one provider becomes five, one SDK becomes five different client libraries, and someone has to own routing, retries, fallbacks, cost tracking, and budgets across all of them. An AI gateway is a tool built to handle all of that — it sits in front of every model call and makes the rest of the stack simpler.

It's also, structurally, hot-path software. Every request, retry, stream, and timeout passes through it. That makes gateway overhead worth taking seriously, especially if your engineering organization has SLAs to meet — which is most enterprises.

## The numbers

At Coolhand, we've been using various AI gateways and wanted to take a closer look at this critical hot path across a few emerging, popular solutions. The numbers below come from one of the providers, [GoModel](https://github.com/ENTERPILOT/GoModel), so weigh that into your evaluation — but they more or less match our own more limited internal tests with each provider.

| Metric | [GoModel](https://github.com/ENTERPILOT/GoModel) | [Bifrost](https://github.com/maximhq/bifrost) | [Portkey](https://github.com/portkey-ai/gateway) | [LiteLLM](https://github.com/BerriAI/litellm) |
| --- | --- | --- | --- | --- |
| Runtime | Go | Go | Node.js | Python |
| p50 latency overhead | 1.8 ms | 2.5 ms | 9.7 ms | 30.6 ms |
| p99 latency | 6.9 ms | 18.3 ms | 30.5 ms | 39.3 ms |
| Sustained throughput | 4,900 req/s | 3,100 req/s | 950 req/s | 324 req/s |
| Peak RAM under load | 37 MB | 143 MB | 112 MB | 2.3 GB |
| Cold start | 0.56 s | 7.1 s | 1.1 s | 25.5 s |
| Docker image | 16 MB | 77 MB | 59 MB | 372 MB |

All requests were run against an identical mock backend on the same hardware (a single AWS `c7i.large`, 2 vCPU / 4 GiB RAM), so the comparison isolates gateway overhead rather than provider or network latency.

## Why this matters more for some teams than others

Cloud LLM calls hide gateway overhead — when a model call takes 2-3 seconds, 10-30ms of proxy time barely registers. That math changes as calls get faster: local models, vLLM, Ollama, and smaller specialized models all shrink model-side latency, which makes gateway overhead a larger share of total request time. As inference technologies continue to evolve and speed up, this overhead only becomes more critical.

## Why we pick GoModel

[GoModel](https://github.com/ENTERPILOT/GoModel) performs well on the hot path because it's small and focused — a feature most enterprises appreciate. With [security becoming a bigger concern](https://everythingengineer.substack.com/publish/posts/detail/196139422), you want a smaller surface area of dependencies, and you don't want to carry the weight of features other users asked for that you'll never use. GoModel's low memory footprint also matters to us: it's part of delivering an AI experience that's as cost-efficient as possible.

## Why LiteLLM hurt

[LiteLLM](https://github.com/BerriAI/litellm) is popular right now. It's easy to start with and carried the AI gateway conversation early — props to them as pioneers in this space.

But 2.3 GB of RAM under load, a 25.5-second cold start, and a 372 MB compressed Docker image feels pretty bloated for the simple routing solution most enterprises want.

## Portkey and Bifrost are worth looking at

[Portkey](https://github.com/portkey-ai/gateway) is not just a proxy. It's closer to a production LLM platform, with routing, observability, guardrails, and managed features around the gateway. If you want a commercial control plane and you're comfortable with that shape, Portkey is a valid option.

[Bifrost](https://github.com/maximhq/bifrost) is different. It's written in Go, focuses on performance, and did well in the benchmark — close to GoModel on median latency, with strong throughput. The gap showed up more clearly in tail latency, memory, cold start, and image size. If either offers features you need that GoModel doesn't, they're worth exploring.

## The question worth asking

Before adopting any gateway, benchmark it against your own workload shape, not just a mock backend. The question that actually matters: how much does this gateway cost you on every single request, retry, and fallback — and is that cost still acceptable once your model calls get faster than they are today?

For now, GoModel looks like the winner on the fundamentals, but we'll update this post (or add follow-ups) as more data emerges in this fast-moving space.