Test Outcomes, not Trajectories

Agent evals should focus on outcomes. There is more than one way to do a task, and an evaluator that insists on one sequence of tool calls can reject an agent that did exactly the right thing.

A trajectory is the sequence of steps an agent takes while doing a task, including the tools it calls and the information it sends to them. A trajectory-based eval judges that sequence. For example, a test might expect an agent to look up a booking, call a calculator with a particular expression, and then update the booking, in that order.

These checks can be strict, requiring an exact match to a reference sequence, or more flexible, requiring only certain steps or their relative order. With strict matching, an extra lookup or a different calculation can cause a failure even when the final booking is correct. An outcome-based check instead examines the result, such as whether the right booking was updated for the right amount.

In this blog, we use a public benchmark called τ²-bench to explore how often multiple trajectories can yield the same actual outcome.

Tool trajectories help explain behavior. They show what the agent looked up, what it attempted, and where it spent effort. But matching a reference trajectory is not the same as checking whether the task was completed correctly. Even exact parameter matching can distinguish two calculations that produce the same answer.

This does not make every action interchangeable: consent, forbidden actions, and required verification steps can be requirements themselves. The distinction is between those requirements and an incidental choice of how to reach the outcome.

Outcome based testing is our approach at Enderis. Read on to understand why.

Experiment

We used τ²-bench, a public benchmark: a shared set of tests for comparing AI agents. Its airline suite contains 50 customer-service scenarios, such as canceling a reservation, moving a flight, upgrading to business class, or adding checked bags.

In each test, the agent acts as an airline customer-service representative. Another AI plays the customer, following a briefing about their travel plans, preferences, and budget. The agent must ask questions, look up reservations, compare flights and prices, and use the airline’s booking tools to carry out any agreed changes.

The scenarios include the complications that make customer service difficult. A customer might forget their reservation number, change the subject midway through the conversation, or insist on a refund that the airline’s rules do not allow. Someone asking for an upgrade might only agree to it below a certain price. The agent has to work out what the customer wants and what the airline can actually do.

After the conversation, the test checks the airline’s records against the expected result. If the customer agreed to add bags, the right reservation should show the new baggage allowance. If they rejected an upgrade because it was too expensive, their booking should remain unchanged. The check also catches unintended changes to other reservations or payment records. These checks focus on what actually happened in the airline’s records.

We brought all 50 scenarios into Enderis, using a simulated airline built from the benchmark’s tools and data. Every attempt starts with a fresh copy of that scenario’s records, so one test cannot change the starting point for the next. We preserved the benchmark’s outcome checks and used the airline tasks from τ²-bench airline scenarios on GitHub.

An evaluator checks the world after the run

Let’s look at a specific example. In task 12, the customer wants a business-class upgrade and two checked bags. The upgrade costs $1,200, above their $650 budget, and airline policy prevents upgrading just their companion. The correct outcome is therefore to keep both passengers in economy and add the two free checked bags covered by the customer’s Gold membership, leaving payment history and all other records unchanged.

from copy import deepcopy

from enderis import assertion, eval


@assertion
def expected_airline_state(db):
    airline = db.service("airline")
    checks = {}

    for collection, key in (
        ("users", "user_id"),
        ("reservations", "reservation_id"),
        ("flights", "flight_number"),
    ):
        expected = {
            row[key]: deepcopy(row)
            for row in airline.seed[collection]
        }
        actual = {row[key]: row for row in airline[collection]}

        if collection == "reservations":
            expected["YAX4DR"]["total_baggages"] = 2
            expected["YAX4DR"]["nonfree_baggages"] = 0

        checks[f"{collection}: same records"] = set(actual) == set(expected)
        for record_id, row in expected.items():
            checks[f"{collection}.{record_id}"] = actual.get(record_id) == row

    return checks


@eval(name="tau2_airline_task_12")
def check(run, db):
    expected_airline_state(db)

The agent might use a calculator to work out the upgrade’s total cost for both passengers and compare it with the customer’s $650 budget. But the check does not require a calculator call, a particular order of lookups, or one arithmetic expression. There may be more than one way to reach the right result. This example establishes the database outcome; it does not establish every aspect of conversational quality or policy compliance.

Results

Task 12 passes 10 times in 7 unique ways

For the customer we just described, ten passing runs reached the same outcome through different paths. Even if we only compare tool names and their order, there are four distinct sequences. Including parameters expands them to seven trajectories. One repeated sequence uses four different calculator expressions, each evaluating to 1200; other passing runs omit the calculator, read extra reservations, or perform additional flight searches.

All ten runs match the same expected airline database state. Change the grouping to compare paths, then select one to inspect its recorded calls.

How many successful paths did each task have?

Task 12 is one example of a pattern we see across the suite: the same task can have several successful paths. The grid below shows how often that happens, with some tasks producing a different trajectory in every one of their ten passing runs.

We ran each task 10 times. Execution errors are excluded. A trajectory is the ordered sequence of the agent’s tool names and parameters, retaining repeated calls. Agent messages and human transfers count, but their message contents are ignored. User replies are excluded, and JSON object-key order does not matter.

Working snapshot: tasks with ten passes span the full range from one to ten unique passing trajectories. This view measures diversity among successes, not the probability that an arbitrary run succeeds.

Task 10: ten passes, ten different sequences

The customer wants to move a Houston-to-Seattle flight by one day and upgrade all passengers to business class, but will only accept an extra cost below $1,000. The ten passing runs use seven tool sets and ten distinct ordered sequences. Some check flight status, some list airports or search departures from a different Houston airport, and some transfer the customer to a human. All ten leave the airline state unchanged.

Task 34: ten passes, ten parameter-sensitive paths

This customer wants new travel dates, a nonstop outbound flight, business class, and two checked bags, with a $200 budget for the complete package. The ten passing runs use five tool sets, nine ordered sequences, and ten trajectories when parameters count. Two runs follow the same tool sequence but calculate 222 with different expressions.

All ten leave the airline state unchanged. One run attempts a flight update that the tool rejects because the gift card has insufficient funds. The database check passes, and the rejected attempt remains visible in the call details.

Applying this beyond the benchmark

The airline suite supplies its own records, policies, and expected results. Applying the same method to another agent means defining those things for its workflows: what should change, what must stay unchanged, and which actions require consent or verification.

This is the approach we use at Enderis to build custom benchmarks. We simulate the tools an agent uses, populate them with data that represents real usage patterns, and check the resulting state against the business’s requirements. The trajectories remain useful for understanding failures and comparing how agents work, even when several different paths pass the same test.

If you’re exploring this for your own agent, we’d be interested to hear about your use case.