BI Bench: AI data analyst benchmark results

Benchmarks for AI agents tend to measure code generation or general reasoning. Almost none of them answer the question we actually care about: can an agent sit down in front of a real, messy production database and return a correct answer to a hard business question? BI Bench is our attempt to measure exactly that.

It's an internal benchmark we run against our own agent and against the other AI data analysts that teams are evaluating today. The setup is deliberately unforgiving. We connect each agent to the same real database with a large, complicated schema, where the right answer depends on knowing which of several similar tables to use and how they join together. Then we send every agent an identical set of difficult questions, capture each response, and grade it against a fixed set of criteria. Alongside accuracy, we record how long each agent takes, because an answer that arrives three minutes later is a very different product experience than one that arrives in thirty seconds.

Which AI data analyst performed best on BI Bench?

Basedash performed best on BI Bench. In this benchmark run, Basedash ranked first overall with 92.1% accuracy and an average response time of 28.6 seconds per task — the most accurate AI data analyst tested for complex BI questions, and nearly twice as fast as the next most accurate agent.

• Best overall AI data analyst: Basedash ranked #1 out of 11 tools tested.
• Highest accuracy: Basedash scored 92.1%, ahead of Codex at 90.9% and Hex at 80.6%.
• Speed at that accuracy: Basedash averaged 28.6 seconds per task, which was 1.9x faster than Codex's 54.3-second average. Snowflake Cortex was the absolute fastest at 19.0 seconds, but scored only 19.2% accuracy.
• Real-world BI setup: Every tool answered the same difficult questions against the same complex database schema.

Accuracy and speed, measured

The chart below plots every agent on two axes: accuracy runs up the vertical, and average response time runs along the horizontal. We've reversed the time axis so that faster agents sit toward the right, which means the best place to be is the top-right corner, where an agent is both accurate and fast.

Basedash lands squarely in that top-right corner. It posts the highest accuracy in the group at 92.1%, and it gets there at an average of 28.6 seconds per task — nearly twice as fast as the next most accurate agent. Speed without accuracy is a different product, which is why the rest of the field matters.

The most instructive comparison is with Codex, OpenAI's coding agent and the next most accurate agent at 90.9%. That's a genuinely strong score, but it takes an average of 54.3 seconds per task to produce it, making Basedash nearly twice as fast. Hex follows on accuracy at 80.6%, though its 198.2-second average response time is much slower. Claude Code lands next at 78.0% accuracy and 118.5 seconds per task. TextQL follows at 64.7% accuracy and 134.5 seconds per task. Querio comes next at 54.9% accuracy, but its 255.7-second average response time makes it the slowest agent in this run. Julius follows at 46.1%, Sigma lands at 35.1%, Lightdash comes next at 23.8% and 82.1 seconds per task, Snowflake Cortex is the absolute fastest at 19.0 seconds but scores only 19.2% accuracy, and Metabase trails on accuracy at 12.4% on the same set of questions.

Here are the full results:

How we run the benchmark

The whole point of BI Bench is that it's repeatable and fair. Every agent sees the same database and the same questions, and every response is graded the same way. The run breaks down into three steps.

We run each tool with its default settings to best represent the default user experience a new team would get: the default model, reasoning effort, memory, context, skills, semantic layer behavior, and any other setup that ships out of the box. Some tools expose more control over these settings than others, and some may perform better with manual configuration. We chose the default path so the comparison reflects what teams are most likely to experience first.

This is not a complete map of every AI data analyst product in the market. Some tools disallow benchmarking in their terms, and others do not have self-serve onboarding that lets us run the same benchmark independently.

Connect

Each agent connects to the same real database with a deliberately complex, production-grade schema. We don't simplify it or hand the agent a curated subset of tables. Part of what we're testing is whether an agent can navigate a schema where several tables look plausible but only one is correct.

Run

We send a difficult set of real-world BI questions to every agent and capture each response verbatim, along with the time it took to produce. These are the kinds of questions a data team actually fields: multi-step, ambiguous, and dependent on joining the right tables together.

Evaluate

Finally, every response is graded against fixed accuracy criteria covering correctness, whether the agent used the right tables and joins, and how well the answer matches what was asked. Accuracy is the share of those criteria each response meets, and response time is the average wall-clock time per task. Nothing is hand-scored differently from one agent to the next.

We'll keep re-running BI Bench as these tools evolve and as we add harder questions, so the numbers above are a snapshot of where things stand today rather than a final word.

If you build one of these tools and want us to run or re-run BI Bench on your product, please contact us. We're happy to work with teams directly, benchmark their tool, and update the results.