A Correct HTTP Response Can Still Produce a Bad Agent Outcome

An AI agent calls an article-extraction API and gets a successful response. It then writes a summary.

Has the agent completed its task?

Only if the response contained the material needed for the task, the summary stayed grounded in that material, and the workflow met the user's requirements. The network status alone cannot settle those questions.

I work on these boundaries as the founder and CEO of Noveum AI and creator of API.market.

Separate the checks

For an illustrative article-summary workflow, I would ask four different questions.

Did the request complete? Did the extracted content include the relevant section? Did the summary preserve the source's meaning? Did the agent act within the agreed scope?

Each question can fail while the others pass. A request can return valid JSON containing a navigation menu. A summary can be well written while describing a detail that was not in the extracted text. An agent can produce a useful result after making an unapproved paid call.

A single “success” field hides those distinctions.

Here is a small test matrix a team could start with. The examples are invented to make the checks concrete.

The matrix separates retrieval coverage, grounding and authorized behavior. A team can add format or latency requirements without pretending those measure factual quality.

Start with the user's outcome

Write the task in a form someone can review: “Summarize the article's proposed method, limitations and evidence, using only the supplied article.”

Then create examples that test the important boundaries. Include an article without a limitations section. Include a page that cannot be fetched. Include a source containing a number that resembles a result but is actually a hypothetical example.

Specify what a good response should do when information is missing. An honest limitation can be the correct outcome.

Keep the evidence with the result

When a check fails, preserve the input, relevant output, evaluation condition and configuration version. A teammate should be able to understand the disagreement without recreating the entire conversation from memory.

Also allow for several valid approaches. If two tool paths produce an acceptable, authorized outcome, the evaluation should not reject one merely because its sequence differs.

Decide how to review disagreements

Some checks can be deterministic. A parser can check required fields, and an allowlist can check whether the workflow used an approved tool. Other checks need judgment: whether a summary changes the meaning of a source, for example.

For judgment-based checks, save the evidence that supports the score and review a sample of both passing and failing cases. If a scorer repeatedly disagrees with the team's definition of a good outcome, revise the rubric or the scorer before using its aggregate score to approve a release.

Keep a separate regression set for previously observed failures. After changing the prompt, model or extraction path, rerun those examples and inspect what changed. An improved average does not resolve a specific failure if that case still fails.

Noveum connects tracing, evaluation and investigation. The underlying discipline is useful with any stack: define what the task requires, evaluate the different failure modes, and keep enough evidence to review the result.

For a broader development workflow, see my guide to eval-driven development. The narrow lesson here is to define success at the level of the user's task, then retain enough evidence to test that definition.

More of my work on production AI is on LinkedIn.

This is an illustrative engineering framework, not a benchmark or a customer case study. I founded Noveum AI and created API.market. Drafted with AI assistance.