Integrating with the Open-Source AI Gateway
The Noveum AI Gateway is a lightweight proxy that sits between your application and AI model providers like OpenAI, Anthropic, DeepSeek, or any future model. It offers a drop-in replacement for direct API calls and automatically forwards traffic while capturing essential metrics.
1. Why Use the AI Gateway?
- Unified API: Your code doesn’t need to change every time you switch providers.
- Real-Time Metrics: Track cost, latency, TTFB, token usage, and error rates in one place.
- Open Source & Flexible: Deploy to Cloudflare Workers, Docker, or Kubernetes in minutes.
2. Deployment Options
Cloudflare Workers
- Clone the Gateway repository:
- Configure your
wrangler.toml
with your Cloudflare account info. - Deploy:
Docker
- Pull the Docker image:
- Run the container:
Kubernetes
- Add the official Helm chart:
- Provide environment variables (like API keys, logging config, etc.) as Kubernetes secrets or config maps.
3. Minimal Code Change
Simply update your AI calls to point to http://[GATEWAY_URL]/v1/chat/completions
instead of the provider’s endpoint. For example, if using OpenAI’s Python library:
Your requests are now routed through the Gateway, and Noveum automatically collects:
- Cost and token usage (if available from the provider)
- Latency and TTFB (Time to First Byte)
- Request and response logs
That’s it! You now have a unified pipeline for multiple providers without refactoring your application code.
4. Configuration
Configure the Gateway with a simple YAML or JSON file:
Tip: All environment variables can be stored securely, then referenced in the config file to keep secrets out of source control.
Next Steps
- Learn how to create Datasets from real-world logs
- Set up Eval Jobs to automatically compare models
- Integrate external providers beyond OpenAI or Anthropic
Get Early Access to Noveum.ai Platform
Be the first one to get notified when we open Noveum Platform to more users. All users get access to Observability suite for free, early users get free eval jobs and premium support for the first year.