Documentation

Noveum SDK - Python Client

Professional Python SDK for programmatic access to the Noveum AI platform with 180+ API endpoints

The Noveum SDK is a comprehensive Python client library that provides both high-level convenience methods and low-level access to 180+ API endpoints for AI/ML evaluation, testing, and observability.

Key Features

✨ Complete API Coverage

180+ endpoints fully implemented across all API categories

🚀 Full IDE Support

Complete type hints, autocomplete, and docstrings

⚡ Async & Sync

Both async/await and synchronous support

🔐 Secure

API key authentication, HTTPS only, proper error handling

🧪 Production-Ready

Extensive test suite with integration and unit tests

🎯 Easy to Use

High-level wrapper for common operations

Installation

pip install noveum-sdk-python

From Source

git clone https://github.com/Noveum/noveum-sdk-python.git
cd noveum-sdk-python
pip install --upgrade pip setuptools wheel
pip install -e .

Quick Start

Basic Usage

import os
from noveum_api_client import NoveumClient
 
# Get API key from environment
api_key = os.getenv("NOVEUM_API_KEY")
 
# Initialize client
client = NoveumClient(api_key=api_key)
 
# List datasets
datasets = client.list_datasets(limit=10)
print(f"Found {len(datasets['data'])} datasets")
 
# Get dataset items
items = client.get_dataset_items("my-dataset", limit=50)
for item in items["data"]:
    print(f"Item: {item}")
 
# Get evaluation results
results = client.get_results(dataset_slug="my-dataset")
print(f"Results: {results['data']}")

Setting Your API Key

Option 1: Environment Variable (Recommended)

export NOVEUM_API_KEY="nv_your_api_key_here"

Then use it in code:

import os
from noveum_api_client import NoveumClient
 
api_key = os.getenv("NOVEUM_API_KEY")
client = NoveumClient(api_key=api_key)

Option 2: Direct Initialization

from noveum_api_client import NoveumClient
 
client = NoveumClient(api_key="nv_your_api_key_here")

Option 3: .env File

echo "NOVEUM_API_KEY=nv_your_api_key_here" > .env

Then load it:

import os
from dotenv import load_dotenv
from noveum_api_client import NoveumClient
 
load_dotenv()
api_key = os.getenv("NOVEUM_API_KEY")
client = NoveumClient(api_key=api_key)

High-Level Client API

The NoveumClient class provides convenient methods for common operations.

List Datasets

response = client.list_datasets(limit=10)
print(f"Status: {response['status_code']}")
print(f"Datasets: {response['data']}")

Parameters:

  • limit (int): Number of datasets to return (default: 20)
  • offset (int): Pagination offset (default: 0)

Get Dataset Items

items = client.get_dataset_items("my-dataset", limit=100)
for item in items["data"]:
    print(f"Item ID: {item['id']}, Input: {item['input']}")

Parameters:

  • dataset_slug (str): The dataset slug (required)
  • limit (int): Number of items to return (default: 20)
  • offset (int): Pagination offset (default: 0)

Get Evaluation Results

# Get all results
results = client.get_results()
 
# Filter by dataset
results = client.get_results(dataset_slug="my-dataset")
 
# Filter by item
results = client.get_results(item_id="item-123")
 
# Filter by scorer
results = client.get_results(scorer_id="factuality_scorer")

Parameters:

  • dataset_slug (str): Filter by dataset slug (optional)
  • item_id (str): Filter by item ID (optional)
  • scorer_id (str): Filter by scorer ID (optional)
  • limit (int): Number of results to return (default: 100)
  • offset (int): Pagination offset (default: 0)

Common Use Cases

Use Case 1: CI/CD Regression Testing

Test your model/agent quality in CI/CD pipelines:

from noveum_api_client import NoveumClient
 
def test_agent_quality():
    client = NoveumClient(api_key="nv_...")
    
    # Get test dataset
    items = client.get_dataset_items("regression-tests")
    
    # Evaluate each item
    failed = 0
    for item in items["data"]:
        # Run your agent/model
        output = my_agent.run(item["input"])
        
        # Get evaluation results
        results = client.get_results(item_id=item["id"])
        
        # Check quality
        for result in results["data"]:
            if result.get("score", 0) < 0.8:
                print(f"❌ Item {item['id']} failed: {result['score']}")
                failed += 1
    
    # Assert
    assert failed == 0, f"{failed} items failed quality check"
    print("✅ All items passed quality check")
 
test_agent_quality()

Use Case 2: Batch Processing

Process all items in a dataset with pagination:

from noveum_api_client import NoveumClient
 
client = NoveumClient(api_key="nv_...")
 
# Get all items (with pagination)
offset = 0
while True:
    items = client.get_dataset_items("my-dataset", limit=100, offset=offset)
    
    if not items["data"]:
        break
    
    # Process each item
    for item in items["data"]:
        print(f"Processing item {item['id']}")
        # Your processing logic here
    
    offset += 100

Use Case 3: Result Analysis

Analyze evaluation results:

from noveum_api_client import NoveumClient
 
client = NoveumClient(api_key="nv_...")
 
# Get all results
results = client.get_results(limit=1000)
 
# Analyze
total = len(results["data"])
passed = sum(1 for r in results["data"] if r.get("passed"))
avg_score = sum(r.get("score", 0) for r in results["data"]) / total if total > 0 else 0
 
print(f"Total: {total}")
print(f"Passed: {passed} ({passed/total*100:.1f}%)")
print(f"Average Score: {avg_score:.2f}")
 
# Find failures
failures = [r for r in results["data"] if not r.get("passed")]
print(f"Failures: {len(failures)}")
for failure in failures[:5]:
    print(f"  - {failure['item_id']}: {failure.get('reason', 'Unknown')}")

Use Case 4: Async Operations

Use async for concurrent operations:

import asyncio
from noveum_api_client import Client
from noveum_api_client.api.datasets import get_api_v1_datasets
 
async def main():
    api_key = "nv_..."
    client = Client(
        base_url="https://api.noveum.ai",
        headers={"Authorization": f"Bearer {api_key}"}
    )
    
    # Async call
    response = await get_api_v1_datasets.asyncio_detailed(client=client)
    print(f"Status: {response.status_code}")
    print(f"Datasets: {response.parsed}")
 
asyncio.run(main())

Advanced Configuration

Custom Base URL

client = NoveumClient(
    api_key="nv_...",
    base_url="https://custom.api.noveum.ai"
)

Custom Timeout

import httpx
from noveum_api_client import Client
 
client = Client(
    base_url="https://api.noveum.ai",
    timeout=httpx.Timeout(30.0)  # 30 second timeout
)

Context Manager

from noveum_api_client import NoveumClient
 
# Automatically closes connection
with NoveumClient(api_key="nv_...") as client:
    datasets = client.list_datasets()
    # Connection automatically closed

Response Format

All high-level client methods return a dictionary with:

{
    "status_code": 200,           # HTTP status code
    "data": {...},                # Response data (parsed JSON)
    "headers": {...}              # Response headers
}

Check status_code to verify success:

response = client.list_datasets()
 
if response["status_code"] == 200:
    print(f"Success: {response['data']}")
else:
    print(f"Error: {response['status_code']}")

Error Handling

Handle API Errors

from noveum_api_client import NoveumClient
 
client = NoveumClient(api_key="nv_...")
 
try:
    response = client.list_datasets()
    
    if response["status_code"] != 200:
        print(f"API Error: {response['status_code']}")
        print(f"Response: {response['data']}")
    else:
        print(f"Success: {response['data']}")
        
except Exception as e:
    print(f"Error: {e}")

Handle Network Errors

import httpx
from noveum_api_client import NoveumClient
 
client = NoveumClient(api_key="nv_...")
 
try:
    response = client.list_datasets()
except httpx.ConnectError:
    print("Connection error - check your internet connection")
except httpx.TimeoutException:
    print("Request timeout - API is slow or unreachable")
except Exception as e:
    print(f"Unexpected error: {e}")

Best Practices

1. Use Environment Variables

import os
from noveum_api_client import NoveumClient
 
# Never hardcode API keys
api_key = os.getenv("NOVEUM_API_KEY")
if not api_key:
    raise ValueError("NOVEUM_API_KEY environment variable not set")
 
client = NoveumClient(api_key=api_key)

2. Handle Pagination

client = NoveumClient(api_key="nv_...")
 
# Paginate through all datasets
offset = 0
all_datasets = []
 
while True:
    response = client.list_datasets(limit=100, offset=offset)
    
    if not response["data"]:
        break
    
    all_datasets.extend(response["data"])
    offset += 100
 
print(f"Total datasets: {len(all_datasets)}")

3. Use Context Managers

from noveum_api_client import NoveumClient
 
# Ensures proper cleanup
with NoveumClient(api_key="nv_...") as client:
    datasets = client.list_datasets()
    # Connection automatically closed

4. Check Status Codes

response = client.list_datasets()
 
if response["status_code"] == 200:
    # Success
    print(response["data"])
elif response["status_code"] == 401:
    # Unauthorized - check API key
    print("Invalid API key")
elif response["status_code"] == 404:
    # Not found
    print("Resource not found")
else:
    # Other error
    print(f"Error: {response['status_code']}")

5. Add Logging

import logging
from noveum_api_client import NoveumClient
 
# Configure logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
 
client = NoveumClient(api_key="nv_...")
 
response = client.list_datasets()
logger.info(f"Listed datasets: {len(response['data'])} found")

Architecture

Two-Layer Architecture

Layer 1: Generated API Client

  • Auto-generated from OpenAPI schema
  • Low-level access to all endpoints
  • Full control over parameters
  • Both sync and async support

Layer 2: High-Level Wrapper (NoveumClient)

  • Convenient methods for common operations
  • Simplified API for typical use cases
  • Automatic error handling
  • Better developer experience

The Noveum ecosystem includes multiple specialized packages:

  • noveum-trace - Lightweight tracing SDK for LLM applications with decorator-based API
  • noveum-sdk-python - This package - comprehensive API client for evaluation and platform management

Support

Next Steps

Ready to build with Noveum? Install the SDK and start integrating with your AI applications!

Exclusive Early Access

Get Early Access to Noveum.ai Platform

Be the first one to get notified when we open Noveum Platform to more users. All users get access to Observability suite for free, early users get free eval jobs and premium support for the first year.

Sign up now. We send access to new batch every week.

Early access members receive premium onboarding support and influence our product roadmap. Limited spots available.

On this page