Noveum SDK - Python Client

Professional Python SDK for programmatic access to the Noveum AI platform with 180+ API endpoints

The Noveum SDK is a comprehensive Python client library that provides both high-level convenience methods and low-level access to 180+ API endpoints for AI/ML evaluation, testing, and observability.

Key Features

✨ Complete API Coverage

180+ endpoints fully implemented across all API categories

🚀 Full IDE Support

Complete type hints, autocomplete, and docstrings

⚡ Async & Sync

Both async/await and synchronous support

🔐 Secure

API key authentication, HTTPS only, proper error handling

🧪 Production-Ready

Extensive test suite with integration and unit tests

🎯 Easy to Use

High-level wrapper for common operations

Installation

From PyPI (Recommended)

pip install noveum-sdk-python

From Source

git clone https://github.com/Noveum/noveum-sdk-python.git
cd noveum-sdk-python
pip install --upgrade pip setuptools wheel
pip install -e .

Quick Start

Basic Usage

import os
from noveum_api_client import NoveumClient

# Get API key from environment
api_key = os.getenv("NOVEUM_API_KEY")

# Initialize client
client = NoveumClient(api_key=api_key)

# List datasets
datasets = client.list_datasets(limit=10)
print(f"Found {len(datasets['data'])} datasets")

# Get dataset items
items = client.get_dataset_items("my-dataset", limit=50)
for item in items["data"]:
    print(f"Item: {item}")

# Get evaluation results
results = client.get_results(dataset_slug="my-dataset")
print(f"Results: {results['data']}")

Setting Your API Key

Option 1: Environment Variable (Recommended)

export NOVEUM_API_KEY="nv_your_api_key_here"

Then use it in code:

import os
from noveum_api_client import NoveumClient

api_key = os.getenv("NOVEUM_API_KEY")
client = NoveumClient(api_key=api_key)

Option 2: Direct Initialization

from noveum_api_client import NoveumClient

client = NoveumClient(api_key="nv_your_api_key_here")

Option 3: .env File

echo "NOVEUM_API_KEY=nv_your_api_key_here" > .env

Then load it:

import os
from dotenv import load_dotenv
from noveum_api_client import NoveumClient

load_dotenv()
api_key = os.getenv("NOVEUM_API_KEY")
client = NoveumClient(api_key=api_key)

High-Level Client API

The NoveumClient class provides convenient methods for common operations.

List Datasets

response = client.list_datasets(limit=10)
print(f"Status: {response['status_code']}")
print(f"Datasets: {response['data']}")

Parameters:

limit (int): Number of datasets to return (default: 20)
offset (int): Pagination offset (default: 0)

Get Dataset Items

items = client.get_dataset_items("my-dataset", limit=100)
for item in items["data"]:
    print(f"Item ID: {item['id']}, Input: {item['input']}")

Parameters:

dataset_slug (str): The dataset slug (required)
limit (int): Number of items to return (default: 20)
offset (int): Pagination offset (default: 0)

Get Evaluation Results

# Get all results
results = client.get_results()

# Filter by dataset
results = client.get_results(dataset_slug="my-dataset")

# Filter by item
results = client.get_results(item_id="item-123")

# Filter by scorer
results = client.get_results(scorer_id="factuality_scorer")

Parameters:

dataset_slug (str): Filter by dataset slug (optional)
item_id (str): Filter by item ID (optional)
scorer_id (str): Filter by scorer ID (optional)
limit (int): Number of results to return (default: 100)
offset (int): Pagination offset (default: 0)

Common Use Cases

Use Case 1: CI/CD Regression Testing

Test your model/agent quality in CI/CD pipelines:

from noveum_api_client import NoveumClient

def test_agent_quality():
    client = NoveumClient(api_key="nv_...")
    
    # Get test dataset
    items = client.get_dataset_items("regression-tests")
    
    # Evaluate each item
    failed = 0
    for item in items["data"]:
        # Run your agent/model
        output = my_agent.run(item["input"])
        
        # Get evaluation results
        results = client.get_results(item_id=item["id"])
        
        # Check quality
        for result in results["data"]:
            if result.get("score", 0) < 0.8:
                print(f"❌ Item {item['id']} failed: {result['score']}")
                failed += 1
    
    # Assert
    assert failed == 0, f"{failed} items failed quality check"
    print("✅ All items passed quality check")

test_agent_quality()

Use Case 2: Batch Processing

Process all items in a dataset with pagination:

from noveum_api_client import NoveumClient

client = NoveumClient(api_key="nv_...")

# Get all items (with pagination)
offset = 0
while True:
    items = client.get_dataset_items("my-dataset", limit=100, offset=offset)
    
    if not items["data"]:
        break
    
    # Process each item
    for item in items["data"]:
        print(f"Processing item {item['id']}")
        # Your processing logic here
    
    offset += 100

Use Case 3: Result Analysis

Analyze evaluation results:

from noveum_api_client import NoveumClient

client = NoveumClient(api_key="nv_...")

# Get all results
results = client.get_results(limit=1000)

# Analyze
total = len(results["data"])
passed = sum(1 for r in results["data"] if r.get("passed"))
avg_score = sum(r.get("score", 0) for r in results["data"]) / total if total > 0 else 0

print(f"Total: {total}")
print(f"Passed: {passed} ({passed/total*100:.1f}%)")
print(f"Average Score: {avg_score:.2f}")

# Find failures
failures = [r for r in results["data"] if not r.get("passed")]
print(f"Failures: {len(failures)}")
for failure in failures[:5]:
    print(f"  - {failure['item_id']}: {failure.get('reason', 'Unknown')}")

Use Case 4: Async Operations

Use async for concurrent operations:

import asyncio
from noveum_api_client import Client
from noveum_api_client.api.datasets import get_api_v1_datasets

async def main():
    api_key = "nv_..."
    client = Client(
        base_url="https://api.noveum.ai",
        headers={"Authorization": f"Bearer {api_key}"}
    )
    
    # Async call
    response = await get_api_v1_datasets.asyncio_detailed(client=client)
    print(f"Status: {response.status_code}")
    print(f"Datasets: {response.parsed}")

asyncio.run(main())

Advanced Configuration

Custom Base URL

client = NoveumClient(
    api_key="nv_...",
    base_url="https://custom.api.noveum.ai"
)

Custom Timeout

import httpx
from noveum_api_client import Client

client = Client(
    base_url="https://api.noveum.ai",
    timeout=httpx.Timeout(30.0)  # 30 second timeout
)

Context Manager

from noveum_api_client import NoveumClient

# Automatically closes connection
with NoveumClient(api_key="nv_...") as client:
    datasets = client.list_datasets()
    # Connection automatically closed

Response Format

All high-level client methods return a dictionary with:

{
    "status_code": 200,           # HTTP status code
    "data": {...},                # Response data (parsed JSON)
    "headers": {...}              # Response headers
}

Check status_code to verify success:

response = client.list_datasets()

if response["status_code"] == 200:
    print(f"Success: {response['data']}")
else:
    print(f"Error: {response['status_code']}")

Error Handling

Handle API Errors

from noveum_api_client import NoveumClient

client = NoveumClient(api_key="nv_...")

try:
    response = client.list_datasets()
    
    if response["status_code"] != 200:
        print(f"API Error: {response['status_code']}")
        print(f"Response: {response['data']}")
    else:
        print(f"Success: {response['data']}")
        
except Exception as e:
    print(f"Error: {e}")

Handle Network Errors

import httpx
from noveum_api_client import NoveumClient

client = NoveumClient(api_key="nv_...")

try:
    response = client.list_datasets()
except httpx.ConnectError:
    print("Connection error - check your internet connection")
except httpx.TimeoutException:
    print("Request timeout - API is slow or unreachable")
except Exception as e:
    print(f"Unexpected error: {e}")

Best Practices

1. Use Environment Variables

import os
from noveum_api_client import NoveumClient

# Never hardcode API keys
api_key = os.getenv("NOVEUM_API_KEY")
if not api_key:
    raise ValueError("NOVEUM_API_KEY environment variable not set")

client = NoveumClient(api_key=api_key)

2. Handle Pagination

client = NoveumClient(api_key="nv_...")

# Paginate through all datasets
offset = 0
all_datasets = []

while True:
    response = client.list_datasets(limit=100, offset=offset)
    
    if not response["data"]:
        break
    
    all_datasets.extend(response["data"])
    offset += 100

print(f"Total datasets: {len(all_datasets)}")

3. Use Context Managers

from noveum_api_client import NoveumClient

# Ensures proper cleanup
with NoveumClient(api_key="nv_...") as client:
    datasets = client.list_datasets()
    # Connection automatically closed

4. Check Status Codes

response = client.list_datasets()

if response["status_code"] == 200:
    # Success
    print(response["data"])
elif response["status_code"] == 401:
    # Unauthorized - check API key
    print("Invalid API key")
elif response["status_code"] == 404:
    # Not found
    print("Resource not found")
else:
    # Other error
    print(f"Error: {response['status_code']}")

5. Add Logging

import logging
from noveum_api_client import NoveumClient

# Configure logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

client = NoveumClient(api_key="nv_...")

response = client.list_datasets()
logger.info(f"Listed datasets: {len(response['data'])} found")

Architecture

Two-Layer Architecture

Layer 1: Generated API Client

Auto-generated from OpenAPI schema
Low-level access to all endpoints
Full control over parameters
Both sync and async support

Layer 2: High-Level Wrapper (NoveumClient)

Convenient methods for common operations
Simplified API for typical use cases
Automatic error handling
Better developer experience

The Noveum ecosystem includes multiple specialized packages:

noveum-trace - Lightweight tracing SDK for LLM applications with decorator-based API
noveum-sdk-python - This package - comprehensive API client for evaluation and platform management

Support

Platform: //
PyPI Package: https://pypi.org/project/noveum-sdk-python/
API Documentation: //docs
GitHub Repository: https://github.com/Noveum/noveum-sdk-python
GitHub Issues: https://github.com/Noveum/noveum-sdk-python/issues
Email: support@noveum.ai

Next Steps

Getting Started - Set up your first project
NovaEval - Learn about AI model evaluation
Integration Examples - See SDK in action
Platform Guide - Explore the dashboard

Ready to build with Noveum? Install the SDK and start integrating with your AI applications!

Noveum SDK - Python Client

✨ Complete API Coverage

🚀 Full IDE Support

⚡ Async & Sync

🔐 Secure

🧪 Production-Ready

🎯 Easy to Use

On this page