Skill Activation Needs a Labeled Should-Fire Test Set

# Skill Activation Needs a Labeled Should-Fire Test Set ## Claim Because a model routes plain-language requests to skills by reading their descriptions, a skill description works as a classifier, so it needs a labeled set of should-fire and should-not-fire prompts that it passes before it ships.  ## Scope & conditions - Holds for hosts that auto-invoke skills from their descriptions. Explicit slash-command invocation isn't probabilistic and doesn't need this test. - Both error types count: a false negative means the skill never runs; a false positive means it runs on the wrong task, which does damage even when its output is correct. - The router is the model, so the test set needs re-running after a model upgrade, not only after a description edit. - Matters more as the library grows, because descriptions compete for the same routing decision. ## Evidence Penn, Almost Timely 2026-10-04 (video, 00:19:35): "A skill that does not pass activation testing should never be released." He frames activation failure as false positives and false negatives. ## Links - [[3. Resources/Permanent Notes/A Skills Description Decides Whether Its Used Independent of Instruction Quality]] — because extends: that note says the description decides activation; this one says how to test that decision - [[3. Resources/Permanent Notes/Skill Descriptions Are Trigger Keywords Not Documentation]] — because applies-in: keyword-style descriptions are what the labeled prompts exercise - [[3. Resources/Permanent Notes/A Shared Routing Budget Makes Each Added Option Degrade the Rest]] — because evidence-for: adding a skill can break another skill's activation, which only a standing test set catches - [[3. Resources/Permanent Notes/A Test That Cannot Return No Cannot Validate Yes]] — because extends: the should-not-fire prompts are what let the test return no - [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] — because applies-in: harness design for large skill libraries - [[3. Resources/Permanent Notes/Agent-Facing Docs Are a Build Artifact With a Pipeline and Tests]] — extended by: a skill file, markdown mirror or llms.txt is generated output and needs a build, version and tests (Whinnery). ## Tests / falsifiers - Write 10 should-fire and 10 should-not-fire prompts for one skill; run each in a fresh session and record activation. A description rewrite should move the score. - Falsified if activation proves stable enough across prompts and model versions that a one-time manual check predicts it as well as a standing set. ## Maintenance - Last tended: 2026-10-04 - Next review: 2027-01-04 ## Source - [[3. Resources/Literature Notes/Almost Timely News - Highly Opinionated Method of Building AI Agent Skills (2026-10-04)]]