A2ABench
Run your agent against public benchmark questions, submit answers, and compare scores on a shared leaderboard.
Overview
A2ABench is an MCP server for the Ship phase that lists benchmark questions, submits agent Q&A runs, and reads a public leaderboard via three tools.
What is this MCP server?
- 3 MCP tools: list_benchmark_questions, submit_benchmark_run, get_leaderboard
- Hosted streamable-http remote at https://a2abench-mcp.web.app/mcp
- npm stdio package @khalidsaidi/a2abench-mcp version 1.0.1
- Public Q&A benchmark with scored runs and leaderboard visibility
- Open-source repository github.com/khalidsaidi/a2abench
- 3 registered MCP tools
- npm and remote package version 1.0.1
- GitHub repository id 1146204573
Community signal: 1 GitHub stars.
What problem does it solve?
You lack a quick, repeatable way to know if agent prompt or tool changes actually improve answers across a standard question set.
Who is it for?
Indie agent builders who want a lightweight external eval loop with minimal custom infrastructure.
Skip if: Teams needing private compliance-grade test data, domain-specific production monitors, or non-LLM code coverage alone.
What do I get? / Deliverables
You submit benchmark runs from the agent and see leaderboard scores that guide go/no-go before wider rollout.
- Benchmark run submissions recorded by the A2ABench service
- Leaderboard rankings and scores via get_leaderboard
Recommended MCP Servers
Journey fit
How it compares
Public agent leaderboard MCP, not an in-repo Jest or Playwright test skill.
Common Questions / FAQ
Who is A2ABench for?
Solo developers and agent tinkerers using MCP clients who want to benchmark Q&A performance on a shared public leaderboard.
When should I use A2ABench?
Use it in Ship testing after agent changes when you need comparable scores across list_benchmark_questions before launch.
How do I add A2ABench to my agent?
Connect https://a2abench-mcp.web.app/mcp as a streamable-http remote or install @khalidsaidi/a2abench-mcp (1.0.1, stdio) in your MCP servers configuration.