# Generated Test Fixtures Are Not Evidence Until Independently Validated ## Claim An AI-generated test fixture becomes usable test evidence only after an independent check confirms its counts, relationships, build identity, and expected result — generating the fixture is not the same step as validating it. <!-- atomicity: 1 idea ✓ --> ## Scope & conditions - Holds wherever an AI assistant automates the mechanical setup of a test scenario (data volume, state, relationships) rather than the judgment of whether the scenario is correct. - Applies most sharply when the fixture is shared with other testers (a Blueprint, a seeded environment) — an unvalidated fixture then propagates its errors to everyone who reuses it. - Doesn't claim AI-generated fixtures are unreliable — only that reliability isn't established by generation alone and requires a separate confirming step. ## Evidence Anne McCarthy used WordPress Studio's AI to generate eight WordPress 7.1 test fixtures (revision-heavy posts, nested comment trees, high-volume media, etc.) and published the result as a shareable Playground Blueprint. The deep-read notes she does not publish a fixture manifest, the prompts used, or verification output confirming each fixture's actual state matches its intended state — the workflow automates setup but the article stops short of demonstrating the validation step. ## Links - [[DevRel MOC]] — the verification step between generated artifacts and usable evidence - [[3. Resources/Permanent Notes/The Maker Should Not Grade Its Own Homework]] — because extends; both name that whatever produces the artifact (AI-generated fixture, AI-generated code) shouldn't be trusted as its own verification. - [[3. Resources/Permanent Notes/A Control Is Enforceable Only When the Actor Cannot Reclassify It]] — because applies-in; a fixture manifest with independent counts is the enforceable check that a generated-and-self-reported fixture lacks. - [[3. Resources/MOCs/Open Source Governance MOC]] — because applies-in; fixture validation discipline is part of what keeps shared community test infrastructure trustworthy. ## Tests / falsifiers - Compare confirmed-defect rates between testers who validated a generated fixture's state before testing and testers who used it as-is — a measurable quality gap would support the claim directly. - Find a case where a generated fixture used without independent validation still produced a correctly-attributed, reproducible bug report — a counter-case would narrow when validation is actually load-bearing. ## Maintenance - Last tended: 2026-08-16 - Next review: 2026-11-16 ## Source - [[3. Resources/Literature Notes/Using WordPress Studio to speed up testing 7.1]] — describes the fixture-generation workflow and the absence of a published validation step.