Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuel

AgentFuel is a framework for generating expressive, domain-specific evaluations for conversational data analysis agents, exposing systematic gaps in how popular open-source and proprietary agents handle domain-relevant timeseries queries. It gives practitioners a principled way to evaluate and compare agents on their own data and query patterns rather than generic benchmarks.

Abstract

Across many domains (e.g., IoT, observability, telecommunications, cybersecurity), there is an emerging adoption of conversational data analysis agents that enable users to “talk to your data” to extract insights. Such data analysis agents operate on timeseries data models; e.g., measurements from sensors or events monitoring user clicks and actions in product analytics. We evaluate 6 popular data analysis agent configurations (both open-source and proprietary) on domain-specific datasets and query types. Agents achieve 73% accuracy on stateless queries but drop to 34% on stateful and 10% on incident queries. We observe two key expressivity gaps in existing evals: domain-customized datasets and domain-specific query types. To enable practitioners in such domains to generate customized and expressive evals for such timeseries data agents, we present AgentFuel. AgentFuel helps domain experts quickly create customized evals to perform end-to-end functional tests. We show that AgentFuel’s benchmarks expose key directions for improvement in existing data agent frameworks. We also present anecdotal evidence that using AgentFuel can improve agent performance (e.g., with GEPA). AgentFuel benchmarks and experiment code are available at https://github.com/Rockfish-Data/agentfuel_paper.