# Reliability Differs From Capability in Repeated Agent Trials ## Claim An agent's capability — can it solve the task at all — and its reliability — does it solve the task every time — are different numbers that can diverge by a meaningful margin. <!-- atomicity: 1 idea ✓ --> ## Scope & conditions - Holds for any task run multiple times under comparable conditions. - The reported gap (10–20%) is specific to Grafana's internal tasks and shouldn't be assumed to generalize without measuring your own pass@k vs. pass^k on your own tasks. ## Evidence pass@k measures whether an agent succeeds at least once in k tries (capability); pass^k measures whether it succeeds every time out of k (reliability). Yas reports a 10–20% gap between the two on Grafana's internal task set. ## Links - [[3. Resources/Permanent Notes/Fact-Based Rubrics Catch Plausible But Wrong Agent Answers That Judges Miss]] — because applies-in; both are eval-methodology claims from the same source. - [[3. Resources/MOCs/AI Agent Safety MOC]] — because applies-in ## Tests / falsifiers - Run any agent task 3+ times and compute both metrics directly; a near-zero gap on your own tasks would mean this specific reliability concern doesn't apply there. - Track whether the gap shrinks as harness/prompt engineering improves, or is a stable property of the model itself. ## Maintenance - Last tended: 2026-08-16 - Next review: 2026-11-16 ## Source - [[3. Resources/Literature Notes/Overview of AI Evaluation (The Context Window 05)]] — the pass@k vs. pass^k distinction and reported gap.