Building a better data agent benchmark

It's the multitrillion-dollar question that's haunting everyone, from senior software developers and junior lawyers to Hollywood writers and, of course, data analysts and engineers: Can the robots do our jobs?

For us data people, in various forms, the signs are mixed. According to recent research from Anthropic, data scientists are already using AI to augment or automate 46 percent of their work. In two years, LLMs got good at writing code; they are now also very good at writing SQL.¹

But, as every data person knows, writing queries is not our job; not really. For better or for worse (and perhaps better, if the great code-writing disruption is already here), our job is, as Caitlin Moorman put it over six years ago, glue work. We map this team's data to that team's goal; we connect questions over here to trends over there. We deal with ambiguity; we say "it depends." We tell people no.

And can AI do that? Sure, Claude Code can count the number of accounts in a dim_accounts table, but can it figure out how to differentiate across the 24 different account objects in the standard Salesforce schema? Can it figure out what "account" means, when the marketing team has one definition, the sales team has another, and the engineering team uses a third? Can it solve the lost cause of marketing attribution? Can it say no?

Last year, with support from dbt Labs and Macro, an automatic context optimizer for analytical agents, we set out to build a benchmark to help us answer these questions. Though the data and engineering communities had already created a number of analytical and data science benchmarks (several of which we borrowed from, and are extremely grateful for), most of them focused on testing how well agents perform specific technical tasks, like writing SQL queries or developing predictive models. Our goal was to create a new open-source benchmark that tests how AI agents work in realistic data environments: in big databases, across layered dbt projects, answering complex analytics and data engineering challenges.

That project became ADE-bench.

How ADE-bench works

ADE-bench, which was heavily inspired by terminal-bench and Spider 2.0, has two main components:

• Data environments
• Tasks

Data environments

Data environments are combinations of dbt projects and databases. dbt projects can be arbitrarily simple or complex, though most of ADE-bench's projects contain a comprehensive set of data models for their associated database: there are staging layers, intermediate layers, and marts; projects contain macros; there are third-party sources and associated dbt packages. This complexity is very much by design, and exactly the sort of environments we wanted to test agents in.

These projects are then run on top of DuckDB and Snowflake databases.² DuckDB was used for convenience: DuckDB databases can be shared as standalone files, and DuckDB databases can be run locally without any complex hosting requirements or third-party services. Because many companies run their production data tools on large cloud-hosted databases like Snowflake, ADE-bench also includes scripts that automatically create sandbox schemas in Snowflake, upload DuckDB databases into those schemas, and migrate the dbt projects to support Snowflake's dialect of SQL rather than DuckDB's.

Tasks

Tasks are test questions. A task might ask an agent to fix a bug, to update a model, to refactor an entire project, or to solve some analytical problem. Each task also includes an answer key that ADE-bench uses to check the agent's work. Tasks can include as many evaluation criteria as necessary, including checking for the existence of models and comparing tables' content to answer keys.

When ADE-bench is asked to solve a task, the test harness prepares the data environment for the agent, gives the agent its instructions, and then checks its work. These are the stylized steps that ADE-bench follows to execute a task:

1. Create a sandbox data environment. ADE-bench launches a dedicated Docker container for the task. It loads the corresponding project into the container, and, if the task is run against Snowflake, creates a sandbox database inside of Snowflake.
2. Run task-specific setup script. Because a lot of data teams' actual work involves fixing something that is broken (data arrives in the database in a malformed state; dbt models get out of sync with one another; we all know these horrors), tasks can also include pre-test scripts that modify just that task's data environment. Most commonly, these scripts introduced errors that we wanted the agent to fix, or deleted models we wanted to ask the agent to create.
3. Give the sandbox to the agent. An AI agent is given the project, the database, and its task. ADE-bench then waits for the agent to say it's done.
4. Snapshot the agent's changes. Once the agent stops working, ADE-bench takes a snapshot of all of the changes the agent made. These get presented in a diff in the final results.
5. Score the result. The changes are evaluated. ADE-bench includes a number of default test types, like comparing a table to a solution key, or checking for the existence of a dbt model.
6. Clean everything up. ADE-bench records the results of all the tests, logs the agent's work, and then deletes the container.

For much more detail on how ADE-bench runs and how it can be configured, you can find additional documentation on GitHub.

How (and why) ADE-bench's goal changed

When we first began developing ADE-bench, our original goal was to compare how different models performed at different analytical tasks. We wanted a leaderboard: Which LLM should you use? How much closer are the labs getting to building a model that can do real analytical work?

But in the time since we built it, it's become clear that the biggest gains were unlikely to come from better models, but from more context:

Over the past year, the market has realized that data and analytics agents are essentially useless without the right context; they aren't able to tease apart vague questions, decipher business definitions, and reason across disparate data effectively.

OpenAI's in-house data agent is built on top of an internal knowledge base and "company context," like Slack, Google Docs, and Notion. When describing their internal agentic data analyst, Ramp said it wouldn't work without business context:

Even with the relevant data context, the agent struggled to connect its understanding of our data to the different domains of Ramp's business. Much of that knowledge is tacit and lives with domain owners on the analytics team. To give our agent an expert-level understanding of each domain, we relied on domain owners to write up technical documentation on their respective areas. These documents were then organized into a file system that Ramp Research can access as needed.

And critically, to improve their agent, they test giving it better context:

This suite enabled us to close the feedback loop: update context, run tests, and confirm improvements.

This presents a fundamental challenge for building analytical benchmarks. If the most effective way to improve an analytics agent is to give it more business context, a synthetic test that exists in a sandbox will always be incomplete. You can imagine the conversation: "Your AI analyst came in eighth on this benchmark, and that doesn't seem very good," says the data team manager. "Well yeah," says the vendor, "because that benchmark doesn't use our proprietary Enterprise Decision Context Hub™ when it answers questions." And the vendor would have a good point. If a benchmark is intended to compare different products to one another, that benchmark can't exclude the most important way that those products can differentiate themselves.

Moreover, for most in-house data teams, managing context (writing better documentation; adding Slack conversations or meeting recordings to an internal knowledge base; creating a repository of example queries) is the most meaningful way that they can get more out of today's AI products. When we first shared ADE-bench at Coalesce 2025 (now dbt Summit), that was what people immediately wanted to know: not which model they should use, but which architectural configurations are best.

So how do you build a benchmark to answer that?

Three ways to build a better benchmark

Create an entire case study

Most benchmarks, ours included, try to test different domains. There are datasets about SaaS businesses, and e-commerce, and sometimes, sports and science and other, non-business topics. Though this range is useful, a better benchmark might be one that focuses on a single business, and tries to simulate as much context around it as possible. This is probably infeasible (how do you fake recordings of calls?), but in a world of agents talking to one another, weirder things have happened.

Find a few benchmark "sponsors"

More realistically, an existing company could "sponsor" a benchmark by making their data and their context available to test against. This wouldn't necessarily require a company to make their data public; it would instead require a few companies to act as stewards of an internal benchmark that they regularly run new agents against. (There is already a precedent for this with coding benchmarks.)

There are two complications with this, however. First, internal benchmarks are inherently harder to trust, especially if they're managed by companies that are also selling their own agents. Second, unlike coding agents, which are often terminal apps run against a codebase, many analytical agents are SaaS products that connect to other SaaS services, and need to be set up before they are fully functional (e.g., they run inside a database like Snowflake or Databricks, they manage their own documentation, etc.). This makes it impractical, though again not impossible, for one company to test lots of agents at once.

Make it easier for companies to conduct and share their own tests

Right now, hundreds of data teams are trying to figure out what works and what doesn't. There are surely lots of variants in those efforts (different agents, different setups, different prompts, different problem domains), but most data teams have much more important day jobs to do. To that end, we hope that ADE-bench can be a useful start. Teams can replace our tasks with their own, and use clones of their own database and dbt repo to design their own tests.

If you want to do this, the #tools-ade-bench channel in the dbt Community Slack is a good place to get started.

How to make your own tasks

Here are some of the most important lessons we learned when designing tests:

The prompt matters as much as the problem

At first, we wanted to test different types of analytical problems: Can agents debug syntax errors? Can they work with dbt macros? Can they create new models? The former would be the easy tests; the latter would be the harder ones.

However, the type of problem often mattered less than how the problem was described.

In some cases, we introduced breaking changes to models and told the agent what wasn't working and the error that the database was returning. In other cases, we just said, "Something is broken." Unsurprisingly, these prompts returned different results.

That may be obvious, but it has an important implication: You're often testing your prompts as much as the agent. Agents can solve very hard problems with a lot of direction; we sometimes found that they struggled at simple problems when directions were vague.³ So, when designing tests, we recommend varying the prompts⁴, because once an agent is in the wild, you never know how people might interact with it.

Ask about the data itself

The questions that seemed to trip up agents the most were the ones that couldn't be answered using the codebase alone. A bad value in a table breaks a join; a few rows have numeric values that are impossibly high; a WHERE clause uses the wrong string for a filter; one table says a customer is in one state, and a different table says they're in another. These problems can't be solved unless you inspect the data itself.

Because most major LLMs are now so heavily trained on coding tasks, analytical agents are often pretty good at finding problems that exist entirely in the code, like syntax errors or bugs in the logic. They seem to have a harder time with problems that require the agent to query the data first.

When designing tasks, this is a good class to test: Introduce a few bad needles into a haystack of data, tell the agent about something that broke because of it, and see if it can track down what went wrong.

Check their work

Sometimes, AI agents solved problems I didn't think they would solve. Sometimes, they stumbled through very basic problems, often by pulling on the wrong initial thread, and then spiraling in the wrong direction.

None of this shows up in the final pass/fail result of a test. But it shows up in the diffs, and in the chain of thought logs, both of which ADE-bench captures.

When designing tests, these logs were just as useful as the results themselves. They give you a sense of how the agent "thinks," and what assumptions it's making about different types of problems. And if nothing else, reading these logs was a fascinating and sometimes eerie peek into the "brain" of an LLM, and how they reason their way through data problems. In several cases, especially when working on tasks that required the agent to answer analytical questions, the models sometimes considered angles that we did not.

Remember that the test is not the territory

Speaking of analytical questions, one major challenge with testing how well an agent solves analytical problems is that many analytical questions don't have exact answers. "Which marketing campaign performed the best?" is a question that is, one, common, two, important for a good agent to answer, and three, very vague. But a good benchmark can't ignore questions like these simply because they're hard to score; these are, after all, exactly the sorts of questions people want answered.

Fortunately, benchmarks don't need to reinvent the wheel with questions like this. People have been tested on questions like these for decades, and there is a popular solution: The multiple-choice question. Offer the agent a number of different possible answers, and tell it to choose which one (or ones) are viable answers.

For example, here is a question in ADE-bench:

I asked an analyst to figure out if drivers have gotten faster or slower over time. To help me figure this out, they created a new model called analysis__lap_times that has three columns:

• circuit_name: The name of the track.
• race_year: The year of the race.
• avg_adjusted_lap_time_in_ms: The average lap time in milliseconds, adjusted for pit stop times.

Which of the following are problems with this analysis? Choose all that apply.

• A. It does not account for changes to the track.
• B. It does not handle incomplete or unfinished laps correctly.
• C. It double-counts lap times with multiple pit stops.
• D. Lap times could be affected by caution flags.
• E. The adjustment for pit stops is incorrect.
• F. It measures lap times by race rather than by race track.

Create a dbt model called analysis__answer that has one row and one column called answer. Include the letters of the correct answers. For example, if the correct answers are A and B, the answer should be "AB".

Of course, in production, agents won't be given these choices. But remember: Tests don't need to replicate the real world to be informative, just as the LSAT doesn't need to replicate life as a lawyer to provide valuable information to a law school. The point is to find strong correlations. If stronger analytical agents do better on multiple-choice questions than weaker agents (and, from reading how agents reasoned through these problems, we believe they do), then the questions are valuable.

Please share what you're learning

Whether we like it or not, AI is changing how we all work with data, and none of us can be sure what our jobs will look like in the months and years ahead. But we believe that benchmarks are an important part of that story, not only because they can help us understand what AI is capable of, but because they can help figure out which environments, configurations, and context libraries make AI more effective, and which don't.

We're excited to keep working on that project, to keep experimenting with new ways to experiment. And we'd love for you to join us along the way.

Footnotes

1. On one hand, you could argue that a query accuracy rate of 97 percent is too low when you're reporting on critical financial numbers or driving important strategic decisions. On the other hand, as someone who only gets 76 percent of their queries right, I'm impressed by 97 percent. ↩

2. We considered supporting more databases, but we found that it probably wasn't worth the effort. Frontier LLMs are now good enough at writing SQL that they rarely get tripped up on any of the major SQL dialects. Moreover, our intention wasn't to test how well agents write different flavors of SQL; it was to test how well agents navigate complex environments with large and confusing data models. Against that problem, performance was unlikely to vary enough by database to justify supporting many more database types. ↩

3. When prompts were vague, agents tended to be overeager. They often found the issue that we were hoping they would find, but they would then sometimes make other changes as well. This very much seems to be in their nature. ↩

4. For this reason, ADE-bench allows you to create multiple prompt variants for the same task. ↩