An AI agent that passes your eval once hasn’t proven much. Run it again and it may pick a different tool, phrase a tool argument differently, or fail outright. Google’s Agent Development Kit (ADK) for Python 2.15.0, released on 19 November 2026, adds proper tooling for this: run every eval case several times, then judge it with pass@k (can the agent do it at all?) and pass^k (does it do it every time?). It also adds partial-credit scoring for tool arguments.

This guide shows how to use both features from the adk eval CLI, and lists the breaking changes to check before you upgrade. Source: the ADK Python v2.15.0 release notes and the ADK source for the config fields.

Upgrade

pip install --upgrade "google-adk==2.15.0"
adk --version

pass@k vs pass^k in plain English

ADK 2.15 implements two estimators, and the difference matters:

  • pass@k (pass_at_k) is the chance that at least one of k runs passes. It measures capability. The ADK source describes it as the unbiased estimator from Chen et al. (2021), the HumanEval paper.
  • pass^k (pass_hat_k) is the chance that all k runs pass. It measures reliability. The source cites Yao et al. (2024), the tau-bench paper.

At k = 1 both equal the plain pass rate. As k grows, pass@k goes up and pass^k goes down. If your agent passes 4 out of 5 runs, pass@3 is 1.0 (every group of three runs contains a pass), but pass^3 is only 0.4 (only 4 of the 10 possible groups of three are all passes). For customer-facing agents, pass^k is usually the number that tells you whether you can ship.

ADK computes these over the runs it actually executed. Runs that crashed without producing any metric results are left out of the counts.

Step 1: Run each case more than once

The quickest way is the new --num_runs flag:

adk eval ./my_agent ./my_agent/support.evalset.json --num_runs 5

Each run uses a fresh session. In 2.15, adk eval prints a per-case multi-run summary table by default and saves MultiRunCaseSummary results to the .evalset_result.json file.

Step 2: Put the rules in test_config.json

For repeatable CI runs, configure everything in the eval config file. If you don’t pass --config_file_path, adk eval looks for test_config.json next to a single eval set file.

{
  "criteria": {
    "tool_trajectory_avg_score": 1.0,
    "response_match_score": 0.8
  },
  "multi_run_config": {
    "num_runs": 5,
    "cross_run_scoring_by_metric": {
      "response_match_score": "min_over_runs",
      "tool_trajectory_avg_score": {"method": "pass_rate", "threshold": 0.9}
    },
    "cross_run_criteria": [
      {"method": "pass_rate", "threshold": 0.8},
      {"method": "pass_hat_k", "k": 3, "threshold": 0.5},
      {"method": "pass_at_k", "k": 3}
    ]
  }
}

Here’s what each part does:

  • num_runs – run every case five times. The --num_runs CLI flag overrides this value.
  • cross_run_scoring (not shown) – how each metric’s scores across runs become one score. The default is "mean_over_invocations". Other built-in options are "mean_of_runs" and "min_over_runs" (the worst run). An object such as {"method": "pass_rate", "threshold": 0.8} counts runs where the metric passed instead.
  • cross_run_scoring_by_metric – the same, but per metric. Above, the response must score at least 0.8 in every run, while the tool trajectory must pass in at least 90% of runs.
  • cross_run_criteria – rules that judge the case on whole runs. The case fails if any rule with a threshold fails. A pass_at_k or pass_hat_k rule needs k; without a threshold it is informational only and just reports its value (like the pass_at_k entry above).

Then run:

adk eval ./my_agent ./my_agent/support.evalset.json \
  --config_file_path ./my_agent/test_config.json

Crashes

max_crashed_runs (default 0) controls how many runs of a case may crash before the case fails. With the default, any crash fails the case. A case also needs at least one run that didn’t crash to pass. Raise it only if you have a known flaky dependency and you track it separately:

"multi_run_config": { "num_runs": 5, "max_crashed_runs": 1 }

The config model rejects unknown keys, so a typo like "num_run" raises an error instead of silently running each case once.

Step 3: Partial credit for tool arguments

The existing tool_trajectory_avg_score metric is strict: one wrong argument and the whole call fails. 2.15 adds tool_parameter_match, which scores each expected argument and averages them per call. A call to book_flight with the right origin and date but a wrong cabin class gets partial credit instead of zero.

{
  "criteria": {
    "tool_parameter_match": {
      "threshold": 0.75,
      "match_type": "ANY_ORDER"
    }
  }
}

The match_type decides how expected calls are paired with actual ones:

  • EXACT – pair calls by position.
  • IN_ORDER (default) – pair each expected call with the next actual call of the same name. Note that it pairs by name alone, so a retried call is scored on its first attempt.
  • ANY_ORDER – pair calls of the same name in any order.

You can also set an args_matcher to control how individual arguments are compared; when it isn’t set, arguments must be equal. Custom matchers must be registered with register_arg_matcher before you can name them in config.

Use this metric alongside, not instead of, the strict trajectory metric. Partial credit is great for tracking progress while you tune prompts; the strict metric tells you whether the call would actually have worked.

Other notable additions in 2.15

  • GoogleMapsToolset with validate_address, reverse_geocode and get_street_view_image.
  • BigQuery ask_data_insights accepts search_scope to discover relevant tables across projects or datasets.
  • Experimental BufferedSessionService, which buffers session events in memory and flushes them in background batches, keeping slow storage off the critical path.
  • FsspecArtifactService can now write, delete and version artifacts across gs://, s3://, file:// and memory://.
  • Live streaming: run_live now runs every tool call in a background task, so voice audio and barge-ins keep flowing while slow tools run.
  • RAG memory search latency cut by up to 6x in VertexAiRagMemoryService and VertexAiRagRetrieval, according to the release notes.

Breaking and behaviour changes to check

  1. Visual Agent Builder removed from ADK Web. ADK Web now focuses on running, testing, debugging and evaluating agents. Agents you built with the builder are standard Agent Config YAML and still run with adk web, adk run and adk deploy. The /dev/apps/{app_name}/builder* routes and google.adk.cli.built_in_agents are gone.
  2. McpInstructionProvider no longer auto-fills prompt arguments from session.state. This is security hardening, so unrelated state keys aren’t sent to external MCP servers. Pass them explicitly:
from google.adk.agents.mcp_instruction_provider import McpInstructionProvider

provider = McpInstructionProvider(
    connection_params,
    "support_prompt",
    arguments=lambda ctx: {"customer_tier": ctx.state.get("tier", "free")},
)
  1. Live tool calls run in background tasks. Parallel calls now emit separate FunctionResponse events as each one finishes, and a tool exception returns an error response instead of ending the live run.
  2. Dynamic instruction routing is on by default. Instructions from load_artifacts and load_mcp_resource now go into a user-role message before the current turn, rather than being appended to system_instruction, so the static prefix stays cacheable. Set ADK_DISABLE_DYNAMIC_INSTRUCTION_ROUTING=1 to opt out.

Note that multi_run_config is read by adk eval. The release source notes that AgentEvaluator and adk web don’t read it yet; AgentEvaluator takes its own num_runs argument.

Key takeaways

  • Run each eval case several times with --num_runs or multi_run_config.num_runs.
  • Use pass^k to measure reliability and pass@k to measure capability. Add a threshold to make either one a gate.
  • Add tool_parameter_match to see how close wrong tool calls are.
  • Before upgrading, check MCP prompt arguments, any builder usage and live-mode tool error handling.