Ag2 Eval Comparison

Evaluation — comparing builds (variants & pairwise)

When to use

• Rank N models / prompts / configs on a leaderboard → run_variants
• Decide which of two is better, head-to-head → run_pairwise with pairwise_judge (LLM) or human_pairwise (people)

For running and grading a single agent (scorers, CI, persistence), use ag2-evaluation.

Install

pip install "ag2[openai,tracing]"

Required. Run this install before delivering the code. If you cannot run commands, state the exact pip install command.

Leaderboard — run_variants

Variants is a frozen dataclass holding a mapping of named `Agent` instances plus an axis label naming what you varied. Build each agent with the one thing that differs (config, prompt, tools, middleware, …), hold the rest fixed, score each, rank:

from autogen.beta import Agent
from autogen.beta.config import OpenAIConfig, GeminiConfig
from autogen.beta.eval import Variants, run_variants
from autogen.beta.eval.scorers import agent_judge

board = await run_variants(
    suite,
    variants=Variants(
        {
            "gpt-4o": Agent("a", prompt="Answer helpfully.", config=OpenAIConfig("gpt-4o")),
            "flash":  Agent("a", prompt="Answer helpfully.", config=GeminiConfig("gemini-3-flash-preview")),
        },
        axis="config",                  # label for what was varied (used in summary)
    ),
    scorers=[agent_judge(OpenAIConfig("gpt-4o"), criterion="Helpful and accurate.", key="quality")],
    store_dir="runs",
    repeats=5,                          # optional: N runs per variant for stability
)
print(board.summary("quality"))         # ranked leaderboard
board.best("quality")                   # winning variant name (None if tied)
board.leaderboard("quality")            # list[LeaderboardRow] — variant, score, n, rank
board.results["gpt-4o"]                 # each variant's full RunResult

Vary whatever you like across the agents — set axis to label it (e.g. "config", "prompt", "tools"). Tied scores share a rank; a 3-way tie usually means the eval isn't discriminating — make it harder, or score quality with a judge.

Head-to-head (LLM) — run_pairwise + pairwise_judge

A comparator picks a winner PER task. pairwise_judge shows the pair in BOTH orders and counts a win only if it's consistent — else a tie (cancels position bias):

from autogen.beta.eval import run_pairwise
from autogen.beta.eval.scorers import pairwise_judge

result = await run_pairwise(
    suite, variant_a=agent_v1, variant_b=agent_v2,
    comparators=[pairwise_judge(OpenAIConfig("gpt-4o"), criterion="more helpful answer", key="quality")],
    store_dir="runs",
)
wr = result.win_rate("quality")         # B's win-rate
print(wr.rate, wr.ci, wr.wins, wr.losses, wr.ties)   # ties count 0.5; ci is a Wilson 95% interval
print(result.flips("quality"))          # int — count of cases where the two orders disagreed

variant_a / variant_b are `Agent` instances; comparators= is a plural iterable. result.agreement("quality", "human") returns an Agreement (.rate, .cohen_kappa, …) between two comparator keys. Use a judge model different from the variants.

Head-to-head (human) — human_pairwise

Same unit, decided by a person. The pair is blinded and order-randomized; the default prints it and reads 1 / 2 / tie. Pass your own async ask(task, response_1, response_2) to collect a vote from a UI (returns "1", "2", or "tie"):

from autogen.beta.eval.scorers import human_pairwise

async def ask(task, response_1, response_2) -> str:
    return await my_ui.compare(task.inputs["input"], response_1, response_2)   # "1" / "2" / "tie"

result = await run_pairwise(suite, variant_a=agent_v1, variant_b=agent_v2,
                            comparators=[human_pairwise(key="quality", ask=ask)], store_dir="runs")

At scale, export a blinded manifest, label it in any tool, import it. evaluate_pairwise is the grade-only twin of run_pairwise (pairs two existing trace sources by task_id):

from autogen.beta.eval import evaluate_pairwise, DirectoryTraceSource
from autogen.beta.eval.scorers import export_pairwise_cases, human_labels

a, b = DirectoryTraceSource("runs/champion"), DirectoryTraceSource("runs/challenger")
await export_pairwise_cases(a, b, criteria=["more helpful"], out="labels.jsonl", suite=suite)   # blinded JSONL
# a person adds  "preferred": "1" | "2" | "tie"  per line, then:
result = await evaluate_pairwise(a, b, suite=suite, store_dir="runs",
                                 comparators=[human_labels("labels.jsonl", criterion="more helpful", key="helpful")])

The manifest hides which model is which; its first_variant field de-blinds it for human_labels.

Common pitfalls

• Judge == a variant's model — self-preference bias; use a different judge model.
• Bare win-rate on few pairs — report wr.ci (Wilson); a small n straddles 50%.
• Passing factories, not agents — run_variants (variants=Variants({name: Agent(...)})) and run_pairwise (variant_a=/variant_b=) take `Agent` instances, not build callables. Vary the model per task with model_config= (a dict[task_id, ModelConfig]) rather than rebuilding the agent. Keep pairwise_judge's default swap (don't set swap=False) for unbiased verdicts.

Going deeper

• website/docs/beta/evaluation/ — variants (the Variants mapping + axis), pairwise (comparators, win-rate, blinded labeling)
• ag2-evaluation — single-agent run/grade, scorers, CI, persistence