gateway

1. Project Overview

Portkey AI Gateway is an open-source, blazing-fast AI gateway that lets developers route requests to 1,600+ LLMs across 50+ providers through a single unified API, adding automatic reliability, security guardrails, and cost controls without changing application code.

2. Background & Positioning

Portkey was built to solve a problem every team building on LLMs eventually hits: provider APIs are inconsistent, individual providers go down or rate-limit unpredictably, and there is no single place to enforce security, budget, and compliance policies across dozens of models. The project's core mission is to make switching, combining, or falling back between LLM providers as simple as changing one header — with production-grade reliability and safety built in rather than bolted on afterward. In production, the gateway already processes over 10 billion tokens a day.

What sets it apart from a plain API proxy or a single-provider SDK wrapper:

• Universal API surface — one OpenAI-compatible interface in front of 1,600+ models from 50+ providers, so switching providers is a config change, not a rewrite.
• Built-in reliability engineering — automatic fallbacks, retries with exponential backoff, load balancing, and timeouts are native gateway behavior, not something each application team has to reimplement.
• Guardrails as a first-class citizen — 50+ pre-built input/output guardrails run inline in the request path, rather than being an external, bolt-on moderation service.
• Deployment flexibility — the same gateway runs as a managed cloud service, a self-hosted Docker/Node.js instance, or an edge deployment on Cloudflare Workers.

3. Feature Categories

🔀 Reliable Routing

Keeps requests succeeding even when individual providers fail.

• Automatic fallback across providers and models
• Exponential backoff retries (up to 5 attempts)
• Load balancing across multiple providers and API keys
• Request timeout management
• Multi-modal and Realtime API (websocket) support
• Purpose: absorb provider outages, rate limits, and latency spikes without the calling application noticing.

🛡️ Security & Guardrails

Inline checks and access control around every request.

• 40+ pre-built guardrails for input and output validation
• Role-based access control with granular permissions
• Virtual key management for secure credential handling
• SOC2, HIPAA, GDPR, and CCPA-aligned compliance controls
• Purpose: enforce content, safety, and access policy centrally instead of per-application.

💰 Cost Management

Visibility and savings on LLM spend.

• Simple and semantic response caching
• Usage analytics and per-request monitoring
• Cross-provider cost optimization and comparison
• Purpose: cut redundant API spend and give teams a real-time view of what LLM usage actually costs.

🤝 Collaboration & Integrations

Connects the gateway into the tools teams already build agents with.

• Agent framework integrations: Autogen, CrewAI, LangChain, LlamaIndex, Phidata, Control Flow
• Prompt template management
• MCP Gateway for managing Model Context Protocol servers
• Purpose: let teams standardize on Portkey as the routing layer no matter which agent framework or protocol they build on top of.

4. Key Highlights

• 1,600+ models behind one API — switch or combine providers via configuration, not code changes.
• 10B+ tokens/day in production — the routing and reliability logic is proven at real production scale.
• 50+ inline guardrails — safety and compliance checks run in the request path instead of as an afterthought.
• Sub-2-minute integration — designed so a new provider or model can be wired in almost immediately.
• Multiple deployment modes — managed cloud, self-hosted Docker/Node.js, or Cloudflare Workers at the edge.
• MCP Gateway support — first-class management of Model Context Protocol servers alongside traditional LLM routing.

5. Use Cases by Role

General developers — Point existing OpenAI-SDK-based code at the gateway's OpenAI-compatible endpoint to instantly gain fallback, retries, and caching with minimal code changes.

DevOps/SRE — Deploy the gateway via Docker or Cloudflare Workers to add a resilient routing layer in front of LLM traffic, with load balancing and automatic failover reducing on-call incidents caused by provider outages.

Security engineers — Configure the 40+ guardrails, role-based access control, and virtual key management to enforce input/output policy and SOC2/HIPAA/GDPR/CCPA-aligned controls across all LLM traffic centrally.

Data/research scientists — Use semantic caching and usage analytics to control experiment costs while comparing outputs across many providers/models through the same interface.

Project managers — Use built-in usage analytics and cost monitoring dashboards to track LLM spend and reliability across teams without needing to touch each integration individually.

6. Getting Started

Find what you need — browse the full documentation for provider guides, guardrails, and framework integrations:

https://portkey.ai/docs

Install and run — the fastest way to try the gateway locally is via npx (requires Node.js):

npx @portkey-ai/gateway

This starts the gateway at http://localhost:8787/v1 with a console at http://localhost:8787/public/. Docker, Cloudflare Workers, Replit, and managed Portkey Cloud deployment are also available.

Contribute — clone the repo, check issues tagged "good first issue", and open a pull request:

git clone https://github.com/Portkey-AI/gateway
npm install
npm run build

The project also runs weekly AI Engineering Hours (Fridays, 8 AM PT) for contributors to get direct engagement.

7. Project Structure

gateway/
├── src/          # Core gateway source code (TypeScript)
├── plugins/      # Guardrail and integration plugins
├── docs/         # Documentation source
├── cookbook/     # Usage examples and tutorials
├── tests/        # Test suite
├── patches/      # Dependency patches
├── Dockerfile           # Container build definition
├── docker-compose.yaml  # Multi-container orchestration
└── wrangler.toml         # Cloudflare Workers deployment config

Key entry points: src/ contains the request routing, fallback, and retry logic that forms the gateway core; plugins/ is where guardrails and third-party integrations live; cookbook/ is the best starting point for copy-pasteable examples against specific providers or frameworks.

8. Related Ecosystem

• Upstream dependencies: 50+ LLM providers including OpenAI, Azure OpenAI, Google Gemini, Anthropic, Cohere, Mistral, and Ollama; the Model Context Protocol (MCP) for tool/server integration.
• Deployment platforms: Portkey Cloud (managed), Docker, Node.js, Cloudflare Workers, Replit.
• Complementary tools: agent frameworks Autogen, CrewAI, LangChain, LlamaIndex, Phidata, and Control Flow; client SDKs for Python, JavaScript, and REST, plus drop-in compatibility with existing OpenAI SDKs.

9. License

Portkey AI Gateway is released under the MIT License.

• ✅ Use, modify, and distribute the software for personal or commercial purposes
• ✅ Create and distribute derivative works, including proprietary products
• ✅ Use it in production without paying license fees
• ❌ Hold the authors or contributors liable for damages arising from use of the software
• ❌ Use the Portkey name or trademarks to imply official endorsement without permission
• ℹ️ Retain the original copyright notice and license text in redistributed copies

10. FAQ

Q: Can I use my existing OpenAI SDK code with Portkey?
A: Yes. The gateway exposes an OpenAI-compatible API, so existing OpenAI SDK calls can point at the gateway endpoint with minimal changes.

Q: How many providers and models does it support?
A: 1,600+ models across 50+ providers, including OpenAI, Azure OpenAI, Google Gemini, Anthropic, Cohere, Mistral, and Ollama.

Q: What's the fastest way to try it locally?
A: npx @portkey-ai/gateway, which starts the gateway at http://localhost:8787/v1 with a local console.

Q: Does it handle provider outages automatically?
A: Yes. Built-in automatic fallback, retries with exponential backoff, and load balancing keep requests succeeding when a provider fails or rate-limits.

Q: Can it manage MCP (Model Context Protocol) servers too?
A: Yes, via the MCP Gateway feature, which manages Model Context Protocol servers alongside standard LLM routing.

11. Quick Links

• Repository: https://github.com/Portkey-AI/gateway
• Official docs: https://portkey.ai/docs
• Contributing guide: https://github.com/Portkey-AI/gateway (see CONTRIBUTING references and "good first issue" labels)
• Community: Discord and Twitter (@PortkeyAI) linked from the repository README

12. Summary

Portkey AI Gateway gives teams a single, production-proven API in front of 1,600+ LLMs, handling reliability, security, and cost concerns that would otherwise be rebuilt by every application. It is best suited for developers and platform/DevOps teams who need dependable multi-provider LLM routing, and for security-conscious organizations that want centralized guardrails and compliance controls — all under a permissive MIT license.