Integrating with the Open-Source AI Gateway

The Noveum AI Gateway is a lightweight proxy that sits between your application and AI model providers like OpenAI, Anthropic, DeepSeek, or any future model. It offers a drop-in replacement for direct API calls and automatically forwards traffic while capturing essential metrics.

1. Why Use the AI Gateway?

• Unified API: Your code doesn’t need to change every time you switch providers.
• Real-Time Metrics: Track cost, latency, TTFB, token usage, and error rates in one place.
• Open Source & Flexible: Deploy to Cloudflare Workers, Docker, or Kubernetes in minutes.

2. Deployment Options

Cloudflare Workers

1. Clone the Gateway repository:

   git clone https://github.com/Noveum/ai-gateway.git
   cd ai-gateway

2. Configure your wrangler.toml with your Cloudflare account info.
3. Deploy:

   wrangler deploy

Docker

1. Pull the Docker image:

   docker pull noveum/ai-gateway:latest

2. Run the container:

   docker run -p 3000:3000 \
     -e GATEWAY_CONFIG=/path/to/your/config.yml \
     --name noveum-gateway \
     noveum/ai-gateway:latest

Kubernetes

1. Add the official Helm chart:

   helm repo add noveum https://noveum.ai/charts
   helm install my-gateway noveum/ai-gateway

2. Provide environment variables (like API keys, logging config, etc.) as Kubernetes secrets or config maps.

3. Minimal Code Change

Simply update your AI calls to point to http://[GATEWAY_URL]/v1/chat/completions instead of the provider’s endpoint. For example, if using OpenAI’s Python library:

import openai
 
openai.api_base = "http://localhost:3000/v1"
openai.api_key = "YOUR_OPENAI_KEY"  # still your real provider key

Your requests are now routed through the Gateway, and Noveum automatically collects:

• Cost and token usage (if available from the provider)
• Latency and TTFB (Time to First Byte)
• Request and response logs

That’s it! You now have a unified pipeline for multiple providers without refactoring your application code.

4. Configuration

Configure the Gateway with a simple YAML or JSON file:

providers:
  - name: openai
    api_key: ${OPENAI_API_KEY}
  - name: anthropic
    api_key: ${ANTHROPIC_API_KEY}
 
logging:
  output: elasticsearch
  es_url: http://elasticsearch:9200
  index_prefix: noveum-gateway

Tip: All environment variables can be stored securely, then referenced in the config file to keep secrets out of source control.

Next Steps

• Learn how to create Datasets from real-world logs
• Set up Eval Jobs to automatically compare models
• Integrate external providers beyond OpenAI or Anthropic