Detecting a Property Is Not Evaluating Whether It Serves Anyone

# Detecting a Property Is Not Evaluating Whether It Serves Anyone ## Claim A rule-based check can confirm that a required property is present and still say nothing about whether it does its job at the point a person encounters it, so a clean scan bounds the space of failures rather than closing it.  ## Scope & conditions - The gap opens wherever the requirement is stated as a property but the purpose is an experience: an accessible name exists but means nothing out of context; a heading level is present but the DOM order contradicts the visual one; an error message renders but isn't associated with its field. - Applies past accessibility to any automated gate — a passing type check, a green test suite, a satisfied lint rule. - Automation is still the right first pass. It clears rule-based failures cheaply, which is what makes human attention affordable for the questions only humans can answer. - The claim is about what a clean result licenses you to say, not about tool quality. A better scanner detects more properties; it does not start evaluating them. ## Evidence > [!quote] ID:rw_01m1fjwmzh9wejvs4khjqg2jqe > "The distinction I want you to remember is that detecting something and evaluating it are not the same thing. They are the different things." > [!quote] ID:rw_01m1fjwmzh9wejvs4khjqg2jqe > "Automation asks, "Are there detectable technical issues?" Manual testing asks, "Does this actually work when someone uses it?"" Khan demonstrated six failures on purpose-built pages that ARC Toolkit and WAVE missed or under-reported: no visible focus indicator, link text ("Check Them Out," "Grab Yours") that passes as an accessible name and is useless in a screen reader's links list, DOM order 1→4→2→3 against visual order, an unassociated validation error, a dialog that doesn't trap focus, and a status message with no live region. ## Links - [[3. Resources/MOCs/WordPress Platform Design MOC]] — the platform question this raises: which of these six are fixable at the block and theme layer rather than left to per-site auditing - [[3. Resources/Permanent Notes/Accessibility Enforced at Authoring Time Beats Post-Hoc Auditing]] — extends: authoring-time enforcement is the strongest available answer, and this note names its ceiling — authoring tools enforce detectable properties too, so the evaluative gap survives the shift left - [[3. Resources/Permanent Notes/Generated Accessibility Targets Do Not Transfer Verification Responsibility]] — applies-in: a generated target is a detected property with a plausible label on it, which is the failure mode this claim predicts - [[3. Resources/Permanent Notes/Absence Of Detection Is Not Absence Of Occurrence]] — the inverse framing: that note is about what a scan missed, this one is about what a scan caught and still couldn't judge - [[3. Resources/Permanent Notes/The Maker Should Not Grade Its Own Homework]] — evidence-for: the reason a person has to close the gap is that the evaluative judgment is exactly the part the producing system cannot supply about itself - [[3. Resources/Permanent Notes/Encoding What Good Looks Like Is What Makes Agent Verification Trustworthy]] — qualifies: encoding the criterion converts some evaluations into detections, which is real progress and also the boundary of what encoding can do - [[3. Resources/Readwise/Full Document Contents/Articles/Introducing Divi AI Agents]] — evidence-for, shipped instance: Divi 5's AI agent markets "add missing alt text and improve accessibility settings" as a bulk operation, which generates the detectable property (alt text exists) without any capacity to judge whether it serves the reader who meets it ## Tests / falsifiers - Take a page that passes axe, WAVE, and ARC Toolkit cleanly and run a keyboard-and-screen-reader pass. If the evaluative failures are consistently absent, the gap is narrower in practice than the demo suggests. - Watch whether AI-led scanning closes any of the six. Khan's own trial at Arbisoft produced false positives, many false negatives, and mis-severities — one anecdote, unnamed tool, worth re-testing rather than citing as settled. ## Source - [[3. Resources/Readwise/Articles/179 What Automated Accessibility Tools Will Never Catch With Tanveer Khan - Accessibility Craft]] — WordPress Accessibility Meetup talk, recorded as Accessibility Craft ep. 179