Integrating External Providers

Noveum is designed to be provider-agnostic, so you can connect any AI model API—commercial or open source—within minutes. Whether it’s AWS Bedrock, Azure OpenAI, Google PaLM, or new specialized platforms, our integration flow remains consistent.

1. Provider Onboarding Steps

1. Credentials: Obtain the required API key or token from the provider.
2. Gateway Config: Add an entry for the provider in your Gateway config:

   providers:
     - name: openai
       api_key: ${AWS_PALM_KEY}
       endpoint: https://api.openai.com/v1/
     - name: deepseek
       api_key: ${DEEPSEEK_KEY}
       endpoint: https://api.deepseek.ai/v1

3. Dashboard Setup: In the Noveum dashboard, navigate to Providers → Add New → select your provider from the list (or create a custom one).
4. Test Connection: Noveum will ping the provider with a simple request to confirm connectivity.

2. Custom Providers

If your team hosts an internal model or uses an open-source model with a custom endpoint:

1. Provide the Base URL (e.g., https://my-llm.internal.com/v1).
2. Define the request/response schema so Noveum can map usage and costs properly.
3. Optionally, define custom metrics. If your model logs extra info (e.g., GPU cycles, memory usage), you can attach it to the Gateway’s extended metrics.

3. FAQ

Q: Do I need separate Noveum accounts for each provider?
A: No. One Noveum project can store multiple provider credentials. Just keep your secrets in a secure location (e.g., environment variables) and map them in your Gateway config.

Q: Can I override default cost calculations for lesser-known providers?
A: Yes. Under Providers → Advanced, set a costPerToken or costPerRequest to accurately reflect the provider’s pricing. This ensures correct cost metrics for your Eval Jobs.

Q: What if the provider uses different fields (e.g., prompt vs. text) in their API?
A: The Gateway supports request transformers. You can define small rules in your config to adapt fields from the standard OpenAI-like schema to the provider’s schema.

4. Troubleshooting

• Connection Errors: Make sure the provider’s endpoint is public or accessible from your environment.
• Authentication Failures: Double-check API keys and tokens, especially for region-specific endpoints.
• Model Not Showing in Dashboard: Confirm you selected the correct provider name. The Gateway uses the name field to display it in the UI.

Once set up, you can immediately include the new provider in your Eval Jobs and compare its performance to existing options.

---

Next Steps

• Review how to run Eval Jobs
• Dive into the results and see which provider is best