AI Agent Evaluation Needs Full Trajectories, Real-World Evidence and Independent Review
- Cargo
- Machine Learning
- Shipped
- Shipper
- Sayash Kapoor
- Readers
- 604
A new research agenda maps 33 trends in agent evaluation, from costly trace analysis to the need for independent, real-world testing.

Evaluating an AI agent means assessing more than whether it reached the right answer. Evaluators need evidence about the steps it took, how it used tools, what happened when it interacted with people, and whether its performance holds up outside a controlled test. A paper titled Emerging Trends in AI Agent Evaluation maps 33 trends across evaluation methods, research priorities and the wider ecosystem. Its central message: agent evaluation needs stronger methods and a broader set of participants as the systems become more costly and harder to observe.
The paper brings together researchers from Princeton, Stanford, the UK AI Security Institute and Brookings. The Princeton group says it has worked on improving agent evaluation for about three years. The authors aim to help new evaluators build the skills this work requires, while giving evaluators, funders and the public questions to ask and steps they can take.

Why agent tests are harder to interpret
Benchmark scores have long been used to support claims about how AI systems perform in practical settings. For agents, a final result alone can conceal important details. An agent may reach the target by taking a shortcut, and access to tools and simulated environments can make that behavior harder to detect. Evaluators need to inspect the full sequence of actions, or trajectory, to understand what the system did and whether the result reflects the ability being tested.
That fuller view also makes it harder to identify why performance changes. An agent combines multiple components, so a change in results may not point clearly to the component responsible. Building evaluations that capture those differences takes substantial engineering work. The cost can also be high: evaluating several agents on ExploitBench has cost tens of thousands of dollars in credits, according to the benchmark’s evaluation information. That figure leaves out the engineering needed to build and maintain the infrastructure.

Volume creates another problem. Agents produce many traces, and evaluators need ways to inspect and monitor them at scale. Automated review can help, but those review tools also need scrutiny. In a post hoc investigation of an OpenAI and Hugging Face episode, evaluators used other agents to search for harmful actions. The investigators acknowledged that this method brought its own limits and biases.
Evaluation conditions also need to reflect the settings where agents operate. Dynamic environments, including settings where several agents interact, make it difficult to define metrics that capture the intended qualities and to create valid tests for them. The authors point to better evaluation tools, more rigorous benchmarks, open-world testing and more observational evidence from real-world use as parts of the response.
Capability does not predict impact on its own

A strong result in a lab test does not establish that an agent will work well in a particular job or improve outcomes for the people using it. The paper cites a randomized controlled trial by METR that found software developers took longer to complete tasks when using agent tools, contrary to expectations. The result illustrates why task capability and effects on human work need separate evaluation.
The research agenda, the authors argue, should examine agents at several levels: what they can do, how people interact with them, and how they affect organizations and incentives. That requires technical evaluation of the system as well as sociotechnical study of the agent in context. Evaluators can also frame questions around automation or augmentation. Testing whether an agent can take over a task helps identify work it may replace; it does not show how an agent could help a person do that work better.
Access and independence shape the evidence
A wider research agenda calls for evaluators from different disciplines, with room to choose their questions and methods. Agreement across teams using different approaches can strengthen confidence in a result. Disagreement can reveal unclear definitions or limits that a single evaluation might miss.
Independence depends in part on resources and access. As frontier evaluations grow more expensive, independent teams need enough funding to participate. They also need access to the systems they are assessing. For agents, access can include chain-of-thought traces: some labs now provide summarized versions rather than raw traces, limiting what outside evaluators can inspect. Evaluators also need to be able to publish findings transparently and do their work without fear of retaliation.
Building a stronger evaluation practice
The paper assigns practical work across the evaluation ecosystem. Prospective evaluators can identify questions that fit their expertise and consider what independence their work requires; the authors specifically encourage people from disciplines not yet active in the field to contribute. Current evaluators can revisit their assumptions and draw on methods from other areas, where related ideas may need adaptation.
Funders, coalitions and conference organizers can help establish shared standards for validity and rigor, and create opportunities for exchange across disciplines. Leaderboard aggregators can improve reporting so results are more useful to people making adoption decisions, while explaining evaluation costs. Companies and evaluators working with deployed systems can contribute evidence from real use and connect it with laboratory findings. Labs can build relationships with independent evaluators, with credibility resting on independence, transparency and sound methods.
The authors also call for evaluation to be continuous from the start of public sector deployments, and identify AI security institutes as potential conveners for new forms of collaboration and public input. Across these recommendations, the practical priority is clear: agent evaluation needs methods that track behavior, evidence from actual use, and enough access and independence for different teams to challenge one another’s conclusions.