An AI agent that passes your eval once hasn’t proven much. Run it again and it may pick a different tool, phrase a tool argument differently, or fail outright. Google’s Agent Development Kit (ADK) for Python 2.15.0, released on 19 November 2026, adds proper tooling for this: run every eval case several times, then judge it with pass@k (can the agent do it at all?) and pass^k (does it do it every time?). It also adds partial-credit scoring for tool arguments.
This guide shows how to use both features from the adk eval CLI, and lists the breaking changes to check before you upgrade. Source: the ADK Python v2.15.0 release notes and the ADK source for the config fields.
Upgrade
pip install --upgrade "google-adk==2.15.0"
adk --version
pass@k vs pass^k in plain English
ADK 2.15 implements two estimators, and the difference matters:
- pass@k (
pass_at_k) is the chance that at least one of k runs passes. It measures capability. The ADK source describes it as the unbiased estimator from Chen et al. (2021), the HumanEval paper. - pass^k (
pass_hat_k) is the chance that all k runs pass. It measures reliability. The source cites Yao et al. (2024), the tau-bench paper.
At k = 1 both equal the plain pass rate. As k grows, pass@k goes up and pass^k goes down. If your agent passes 4 out of 5 runs, pass@3 is 1.0 (every group of three runs contains a pass), but pass^3 is only 0.4 (only 4 of the 10 possible groups of three are all passes). For customer-facing agents, pass^k is usually the number that tells you whether you can ship.
ADK computes these over the runs it actually executed. Runs that crashed without producing any metric results are left out of the counts.
Step 1: Run each case more than once
The quickest way is the new --num_runs flag:
adk eval ./my_agent ./my_agent/support.evalset.json --num_runs 5
Each run uses a fresh session. In 2.15, adk eval prints a per-case multi-run summary table by default and saves MultiRunCaseSummary results to the .evalset_result.json file.
Step 2: Put the rules in test_config.json
For repeatable CI runs, configure everything in the eval config file. If you don’t pass --config_file_path, adk eval looks for test_config.json next to a single eval set file.
{
"criteria": {
"tool_trajectory_avg_score": 1.0,
"response_match_score": 0.8
},
"multi_run_config": {
"num_runs": 5,
"cross_run_scoring_by_metric": {
"response_match_score": "min_over_runs",
"tool_trajectory_avg_score": {"method": "pass_rate", "threshold": 0.9}
},
"cross_run_criteria": [
{"method": "pass_rate", "threshold": 0.8},
{"method": "pass_hat_k", "k": 3, "threshold": 0.5},
{"method": "pass_at_k", "k": 3}
]
}
}
Here’s what each part does:
num_runs– run every case five times. The--num_runsCLI flag overrides this value.cross_run_scoring(not shown) – how each metric’s scores across runs become one score. The default is"mean_over_invocations". Other built-in options are"mean_of_runs"and"min_over_runs"(the worst run). An object such as{"method": "pass_rate", "threshold": 0.8}counts runs where the metric passed instead.cross_run_scoring_by_metric– the same, but per metric. Above, the response must score at least 0.8 in every run, while the tool trajectory must pass in at least 90% of runs.cross_run_criteria– rules that judge the case on whole runs. The case fails if any rule with a threshold fails. Apass_at_korpass_hat_krule needsk; without a threshold it is informational only and just reports its value (like thepass_at_kentry above).
Then run:
adk eval ./my_agent ./my_agent/support.evalset.json \
--config_file_path ./my_agent/test_config.json
Crashes
max_crashed_runs (default 0) controls how many runs of a case may crash before the case fails. With the default, any crash fails the case. A case also needs at least one run that didn’t crash to pass. Raise it only if you have a known flaky dependency and you track it separately:
"multi_run_config": { "num_runs": 5, "max_crashed_runs": 1 }
The config model rejects unknown keys, so a typo like "num_run" raises an error instead of silently running each case once.
Step 3: Partial credit for tool arguments
The existing tool_trajectory_avg_score metric is strict: one wrong argument and the whole call fails. 2.15 adds tool_parameter_match, which scores each expected argument and averages them per call. A call to book_flight with the right origin and date but a wrong cabin class gets partial credit instead of zero.
{
"criteria": {
"tool_parameter_match": {
"threshold": 0.75,
"match_type": "ANY_ORDER"
}
}
}
The match_type decides how expected calls are paired with actual ones:
EXACT– pair calls by position.IN_ORDER(default) – pair each expected call with the next actual call of the same name. Note that it pairs by name alone, so a retried call is scored on its first attempt.ANY_ORDER– pair calls of the same name in any order.
You can also set an args_matcher to control how individual arguments are compared; when it isn’t set, arguments must be equal. Custom matchers must be registered with register_arg_matcher before you can name them in config.
Use this metric alongside, not instead of, the strict trajectory metric. Partial credit is great for tracking progress while you tune prompts; the strict metric tells you whether the call would actually have worked.
Other notable additions in 2.15
GoogleMapsToolsetwithvalidate_address,reverse_geocodeandget_street_view_image.- BigQuery
ask_data_insightsacceptssearch_scopeto discover relevant tables across projects or datasets. - Experimental
BufferedSessionService, which buffers session events in memory and flushes them in background batches, keeping slow storage off the critical path. FsspecArtifactServicecan now write, delete and version artifacts acrossgs://,s3://,file://andmemory://.- Live streaming:
run_livenow runs every tool call in a background task, so voice audio and barge-ins keep flowing while slow tools run. - RAG memory search latency cut by up to 6x in
VertexAiRagMemoryServiceandVertexAiRagRetrieval, according to the release notes.
Breaking and behaviour changes to check
- Visual Agent Builder removed from ADK Web. ADK Web now focuses on running, testing, debugging and evaluating agents. Agents you built with the builder are standard Agent Config YAML and still run with
adk web,adk runandadk deploy. The/dev/apps/{app_name}/builder*routes andgoogle.adk.cli.built_in_agentsare gone. McpInstructionProviderno longer auto-fills prompt arguments fromsession.state. This is security hardening, so unrelated state keys aren’t sent to external MCP servers. Pass them explicitly:
from google.adk.agents.mcp_instruction_provider import McpInstructionProvider
provider = McpInstructionProvider(
connection_params,
"support_prompt",
arguments=lambda ctx: {"customer_tier": ctx.state.get("tier", "free")},
)
- Live tool calls run in background tasks. Parallel calls now emit separate
FunctionResponseevents as each one finishes, and a tool exception returns an error response instead of ending the live run. - Dynamic instruction routing is on by default. Instructions from
load_artifactsandload_mcp_resourcenow go into a user-role message before the current turn, rather than being appended tosystem_instruction, so the static prefix stays cacheable. SetADK_DISABLE_DYNAMIC_INSTRUCTION_ROUTING=1to opt out.
Note that multi_run_config is read by adk eval. The release source notes that AgentEvaluator and adk web don’t read it yet; AgentEvaluator takes its own num_runs argument.
Key takeaways
- Run each eval case several times with
--num_runsormulti_run_config.num_runs. - Use pass^k to measure reliability and pass@k to measure capability. Add a threshold to make either one a gate.
- Add
tool_parameter_matchto see how close wrong tool calls are. - Before upgrading, check MCP prompt arguments, any builder usage and live-mode tool error handling.
