LiteLLM vs Portkey vs OpenRouter: Which LLM Gateway Actually Fits Your Stack

A client asked me a version of this question last week: "What happens to our product if OpenAI has an outage, or raises prices again, or we just want to try Gemini for one feature?" The honest answer, if you've hardcoded the OpenAI SDK into your app, is "you go down with them, or you spend a weekend rewriting integration code." The fix isn't picking a better model. It's putting a gateway between your product and every model provider, so switching, falling back, and load-balancing become config changes instead of code changes. Here's how that actually works, and which of the real options — LiteLLM, Portkey, and OpenRouter — fits a given team.

The mechanism: what a gateway actually does between your app and the model

An LLM gateway sits between your application code and every model provider you use, and does three jobs that have nothing to do with intelligence and everything to do with reliability:

• **Protocol translation.** Anthropic, OpenAI, Google, and every open-weight host behind a different API have slightly different request and response shapes. A gateway normalizes all of them to one format (almost always OpenAI's), so your app code calls one interface no matter which provider actually answers.
• **Retry and fallback logic.** When a request to your primary model fails, times out, or gets rate-limited, the gateway can automatically retry it, and if retries exhaust, reroute the same request to a different model or provider without your app ever seeing the failure. Portkey's own documentation describes this as retrying failed requests "up to 5 times, with exponential backoff" before falling back.
• **Load balancing and key management.** Instead of one API key eating one provider's rate limit, a gateway spreads requests across multiple keys or providers using weights, and tracks spend per key so you can see which feature or customer is actually burning budget.

None of this touches model quality. It's the same reason a load balancer in front of web servers doesn't make any one server faster — it makes the system as a whole stop depending on any single server. This is also a different mechanism from the caching we covered in our guide to semantic caching for LLM APIs: caching avoids paying for a repeated question, a gateway avoids depending on a single provider for every question. Most serious production stacks eventually run both.

LiteLLM: fully open source, fully your ops problem

LiteLLM is an MIT-licensed proxy you run yourself (a separate `enterprise/` directory ships under its own commercial license, but the core gateway is MIT). Its README states it plainly: "Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready." You start it with a single command (`litellm --model gpt-4o`), point your existing OpenAI client at `http://localhost:4000`, and it proxies every call through whichever provider you've configured — Azure, Anthropic, Bedrock, a local Ollama model, or anything else on its list of 100+ providers.

The tradeoff is exactly what "self-hosted" implies: there's no markup on tokens, but there's also no one else operating the proxy. You own uptime, you own securing the endpoint, and you own building the rules for which model a request should fall back to. If your team already runs infrastructure and wants zero vendor dependency in the routing layer itself, this is the honest default.

Portkey: a gateway with the guardrails built in

Portkey's AI Gateway is also MIT-licensed and self-hostable, but it ships more of the production layer out of the box: guardrails, observability, and semantic caching sit next to the routing logic instead of being something you bolt on separately. Worth noting as a trust check, not a knock: Portkey's own README is inconsistent about scale, claiming "250+ LLMs" in its header and "1,600+ models" in its body text, which is a reminder to verify a vendor's own numbers before quoting them in a pitch deck. What's consistently documented is the mechanism — conditional routing rules, retries with backoff, and weighted load balancing across keys, configured in JSON rather than code.

Portkey fits teams that want fallback and routing but don't want to also build the guardrail and observability layer from scratch. You still run it (or use Portkey's hosted version), so it sits a step above LiteLLM in built-in features and a step below full outsourcing.

OpenRouter: zero infrastructure, a real fee for it

OpenRouter is the hosted alternative: one API, no proxy to run, access to "400+ models and providers" per its own docs. There's no self-hosting option and no gateway repo to audit, which is the real distinction from the two tools above — you're trusting OpenRouter's infrastructure, not your own.

The fee structure is worth reading before you commit a product to it. OpenRouter's BYOK documentation states a flat 5% charge "of what the same model/provider would cost normally on OpenRouter" when you bring your own provider key, deducted from your credits, with a $25,000 monthly free allowance before it kicks in on pay-as-you-go accounts. That's a real, disclosed cost for the convenience of not running anything yourself, and at meaningful volume it's worth modeling against what a self-hosted LiteLLM instance would cost you in server time instead.

Which one actually fits

A rough way to decide, based on what each tool is actually optimized for:

• Team already running infrastructure, wants zero per-token markup, comfortable owning the proxy: LiteLLM.
• Team wants routing plus guardrails and caching without assembling them separately, still willing to self-host: Portkey.
• Small team, no ops capacity, fine paying a disclosed fee to never touch a server: OpenRouter.
• Any team choosing between a bigger frontier model and a cheaper open-weight one for a given feature — our Llama 4 vs. Mistral vs. DeepSeek vs. Qwen3 guide covers the model side of that decision; a gateway is what makes switching between them at runtime actually safe to try.

Where this fits in a real build

A gateway is infrastructure, not a feature — users never see it, and that's the point. But the model, architecture, and build-versus-buy decisions it sits on top of are exactly what separates a product that survives its first provider outage from one that goes down with it. That's the kind of decision AI Product Development Bootcamp is built around: not just which model to call, but how to architect the calling so a price change, an outage, or a better model next quarter doesn't mean rewriting your product.