Documentation

Integrating with the Open-Source AI Gateway

The Noveum AI Gateway is a lightweight proxy that sits between your application and AI model providers like OpenAI, Anthropic, DeepSeek, or any future model. It offers a drop-in replacement for direct API calls and automatically forwards traffic while capturing essential metrics.

1. Why Use the AI Gateway?

  • Unified API: Your code doesn’t need to change every time you switch providers.
  • Real-Time Metrics: Track cost, latency, TTFB, token usage, and error rates in one place.
  • Open Source & Flexible: Deploy to Cloudflare Workers, Docker, or Kubernetes in minutes.

2. Deployment Options

Cloudflare Workers

  1. Clone the Gateway repository:
    git clone https://github.com/Noveum/ai-gateway.git
    cd ai-gateway
  2. Configure your wrangler.toml with your Cloudflare account info.
  3. Deploy:
    wrangler deploy

Docker

  1. Pull the Docker image:
    docker pull noveum/ai-gateway:latest
  2. Run the container:
    docker run -p 3000:3000 \
      -e GATEWAY_CONFIG=/path/to/your/config.yml \
      --name noveum-gateway \
      noveum/ai-gateway:latest

Kubernetes

  1. Add the official Helm chart:
    helm repo add noveum https://noveum.ai/charts
    helm install my-gateway noveum/ai-gateway
  2. Provide environment variables (like API keys, logging config, etc.) as Kubernetes secrets or config maps.

3. Minimal Code Change

Simply update your AI calls to point to http://[GATEWAY_URL]/v1/chat/completions instead of the provider’s endpoint. For example, if using OpenAI’s Python library:

import openai
 
openai.api_base = "http://localhost:3000/v1"
openai.api_key = "YOUR_OPENAI_KEY"  # still your real provider key

Your requests are now routed through the Gateway, and Noveum automatically collects:

  • Cost and token usage (if available from the provider)
  • Latency and TTFB (Time to First Byte)
  • Request and response logs

That’s it! You now have a unified pipeline for multiple providers without refactoring your application code.

4. Configuration

Configure the Gateway with a simple YAML or JSON file:

providers:
  - name: openai
    api_key: ${OPENAI_API_KEY}
  - name: anthropic
    api_key: ${ANTHROPIC_API_KEY}
 
logging:
  output: elasticsearch
  es_url: http://elasticsearch:9200
  index_prefix: noveum-gateway

Tip: All environment variables can be stored securely, then referenced in the config file to keep secrets out of source control.


Next Steps

Exclusive Early Access

Get Early Access to Noveum.ai Platform

Be the first one to get notified when we open Noveum Platform to more users. All users get access to Observability suite for free, early users get free eval jobs and premium support for the first year.

Sign up now. We send access to new batch every week.

Early access members receive premium onboarding support and influence our product roadmap. Limited spots available.

On this page