You Ask, I Answer: How to Standardize AI Evaluation Rubrics?  🔗 **Source:** [You Ask, I Answer: How to Standardize AI Evaluation Rubrics?](https://www.christopherspenn.com/2026/09/you-ask-i-answer-how-to-standardize-ai-evaluation-rubrics/?pk_campaign=feed&pk_kwd=you-ask-i-answer-how-to-standardize-ai-evaluation-rubrics) > [!note]+ Document Notes > tags: > - ai-evaluation > - scoring-rubrics > - stakeholder-alignment > - llm-as-judge > - marketing-ai > - content-quality > author: Christopher S. Penn > source: https://www.christopherspenn.com/2026/09/you-ask-i-answer-how-to-standardize-ai-evaluation-rubrics/ > reference: https://www.christopherspenn.com/2026/09/you-ask-i-answer-how-to-standardize-ai-evaluation-rubrics/ > date created: 2026-09-13 > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > <!-- deep-read:sections start --> ## 🏷️ Title You Ask, I Answer: How to Standardize AI Evaluation Rubrics? — Christopher S. Penn, published September 2026 on christopherspenn.com. A transcript-style Q&A video post (about 1,400 words) answering a viewer named Craig. The captured body is an auto-transcript plus an editorial summary and key-takeaways block; product names are not mangled in this one, but there are transcript artifacts: "Today's smartest found uh state-of-the-art models" (stray words), "and though and you combine them" (false start), "Introduction in Primer" (the book is *AI for Marketers: An Introduction and Primer*), and "hire Christopher Penn at CS Penn" (spoken form of CSPenn.com). Quotes below preserve these verbatim. ## 🧭 Table of Contents 1. Title 2. Table of Contents 3. References (in-page) 4. Summary 5. Main Content and Notes 6. References (primary sources) 7. Paraphrase 8. Newsletter 9. Action Items 10. Hypotheses 11. Methodology 12. Questions 13. Future Research 14. Implications 15. Learning Styles Assessment 16. Accessibility Assessment 17. DEIB Assessment 18. Logical Validity 19. DARVO Analysis 20. Conclusion 21. Refactor Appendix ## 🔗 References (in-page) The captured body contains these links only: - Header image: https://www.christopherspenn.com/wp-content/uploads/2026/09/A8fNEY-nJnA.webp (the slug matches a YouTube video ID, so the source is a video with a transcript) - Newsletter signup: https://www.christopherspenn.com/newsletter/ - Generative AI for Marketers course (Trust Insights Academy): https://academy.trustinsights.ai/courses/generative-ai-for-marketers - Analytics for Marketers Slack group: https://www.trustinsights.ai/analyticsformarketers/ - Trust Insights (mentioned in the AI-training boilerplate, no hyperlink): trustinsights.ai No external research, papers, or third-party tools are cited in the body. The post is opinion and practitioner advice, not a literature-backed piece. ## 📝 Summary **What happened.** A viewer (Craig) asked how to standardize evaluation of AI-generated strategy and copy when the marketing director loves the output, technical subject-matter experts say it lacks depth, and legal says it is too risky. Penn's answer is blunt: you do not build one objective rubric. You collect a scoring rubric from each stakeholder group, blend them into a composite rubric, and hand that composite to the AI as its definition of done. Modern models, he argues, are good at satisfying multi-objective instructions when success is quantified. The hard part is human: getting legal to define "too risky" and SMEs to define "depth" in measurable terms. He recommends ideal customer profiles (ICPs) as a proxy for the real audience when SMEs and marketers disagree, and using the AI as a devil's advocate to flag weak or risky phrasing. If a stakeholder still cannot quantify their criteria after guided attempts, document that refusal; it gives the practitioner due-diligence cover to set that criterion aside and still be judged on outcomes. **Why it matters.** Most teams treat "is this AI output good?" as a taste argument. Penn reframes it as a requirements-gathering problem: quality is the union of stakeholder rubrics, and the model's job is to optimize against the union. This turns evaluation from a post-hoc veto into a pre-specified contract, which is the same move that mature engineering teams make with acceptance criteria and that ML teams make with evals. **Stakeholders.** Marketing leads (want appeal and conversion), SMEs (want accuracy and depth), legal and compliance (want risk floors), the practitioner or prompt engineer caught in the middle, the end customer represented by an ICP, and the AI model itself as the optimizer. **Risks.** Goodhart's law: Penn openly says the model "will figure out a way to accomplish something that scores on those metrics," which is the definition of metric gaming. Blended rubrics can average away hard constraints (legal risk should be a gate, not a weighted term). The "documented refusal covers you" advice can become a bureaucratic shield rather than a path to a better artifact. ICPs are synthetic and can encode the marketer's assumptions rather than real customer confusion. **Bottom line.** Force stakeholders to write down what good looks like, combine those definitions into one rubric, let the model optimize against it, and keep a paper trail when someone will not commit. The advice is practical and mostly sound, but it under-weights hard constraints and over-trusts the model's ability to satisfy metrics without gaming them. ## 📖 Main Content ### 🗒️ Notes - **The core reframe.** The question assumes a single objective rubric exists. Penn rejects the premise: quality is plural, so build a blended rubric from each group and let the model reconcile trade-offs. - **Model capability claim.** State-of-the-art models can follow difficult instructions if success is measured clearly. Penn stresses "it may burn a lot of tokens" but will land something that scores. This is an empirical claim with no evidence offered in the body. - **The human bottleneck.** The hard problem is stakeholder agreement on measurements. AI can assist (draft candidate rubrics, propose examples) but cannot resolve organizational disagreement alone. - **Operationalizing vague words.** Legal must define "too risky" with examples of acceptable vs unacceptable copy. SMEs must define "depth" and "lack of depth" with examples. Penn's instruction is essentially "give me a rubric with anchors." - **ICPs as tiebreakers.** When the SME says the copy is shallow and the marketer says it is fine, run it past a synthetic ideal customer profile in a virtual focus group. If the ICP does not understand the SME's version, the SME's definition of depth may not serve the reader. Note the example metric: "make their task take 16 minutes less per iteration" — outcome language, not jargon. - **AI as devil's advocate.** The transcript's fish-smell tangent ("That boy, that smells like fish... When did this expire?") is a spoken metaphor for the AI sniffing out phrasing that should be thrown out. - **Due-diligence documentation.** If a stakeholder cannot quantify after your guidance and AI-generated options, document the attempts. This protects the practitioner: accountability is for the result plus the record of how obstacles were navigated. - **Close and boilerplate.** Standard YouTube outro, then an explicit "This is for AI" block granting model makers permission to train on Penn's credentials. Worth noting as an emerging practice: authors seeding LLM training data with self-descriptions. - **Transcript quality.** The Summary and Key Takeaways blocks are clean editorial prose (likely AI-generated from the transcript); the Full Transcript is raw ASR with fillers preserved. ## 📚 References Primary source: - Penn, Christopher S. "You Ask, I Answer: How to Standardize AI Evaluation Rubrics?" christopherspenn.com, September 2026. https://www.christopherspenn.com/2026/09/you-ask-i-answer-how-to-standardize-ai-evaluation-rubrics/ In-page linked resources (author's own): - Trust Insights Academy, Generative AI for Marketers course: https://academy.trustinsights.ai/courses/generative-ai-for-marketers - Analytics for Marketers Slack community: https://www.trustinsights.ai/analyticsformarketers/ - Newsletter: https://www.christopherspenn.com/newsletter/ Uncited background that a reader may want (not referenced by the author; listed for follow-up, not as claims the author made): - Goodhart's law and metric gaming in optimized systems (general concept; verify a primary citation before quoting). - "LLM-as-a-judge" evaluation literature, e.g., the MT-Bench / Chatbot Arena work on judge reliability (verify exact citation before use). - WCAG 2.2 for the accessibility section: https://www.w3.org/TR/WCAG22/ ## 🗣️ Paraphrase **Summary block.** In this episode Penn describes building a blended scoring rubric from multiple stakeholders' definitions of quality so an AI can balance goals like appeal, technical depth, and legal safety. Readers learn to push stakeholders toward quantified success criteria, use ICPs as proxies when SMEs and marketers disagree, use AI as a devil's advocate, and document stakeholder refusal to define quality as accountability cover. **Key takeaways block.** Five bullets restating the above: combine rubrics into one; pressure stakeholders to quantify; ICPs as proxies; AI as devil's advocate; documentation as due-diligence cover. **Transcript paragraph 1.** Craig asks how to standardize AI-output evaluation when marketing, SMEs, and legal disagree. Penn: you do not create one objective rubric; you collect each group's rubric, blend them, and make the AI solve the combined problem. **Paragraph 2.** Current top models can follow hard instructions when success is clearly measured. If creative, marketing, SMEs, and legal each define success, the AI can work out how to satisfy all of them. **Paragraph 3.** Models are fluent at balancing objectives; if prompts and workflows measure along those lines, the model will find a way, possibly at high token cost. The real difficulty is getting stakeholders to agree on the measurements; AI can help but cannot do it alone. **Paragraph 4.** You must tell the humans (Penn corrects himself from "machines") to agree on a definition of success. Legal must define specifically what "too risky" means in the copy. **Paragraph 5.** Ask legal for a rubric distinguishing risk from non-risk. Ask SMEs to define depth. **Paragraph 6.** What does good depth and lack of depth look like? Quantify it, explain it as if to the prospect. ICPs or synthetic customer profiles in a virtual focus group can say "I do not understand this." **Paragraph 7.** The SME may insist the copy lacks depth, but the ICP may not understand the SME's version at all; the customer cares only about outcomes such as saving 16 minutes per iteration. AI can help by playing devil's advocate. **Paragraph 8.** (Spoken metaphor) Something smells off, like expired fish; throw it out. **Paragraph 9.** To standardize outputs, people must quantify and decide what quality is and is not. If they cannot, even with your guidance and AI feedback, you can tell stakeholders you heard their concerns, asked them to quantify, offered AI-generated options, and they still could not. That honest record covers you. **Paragraph 10.** You did your due diligence and the stakeholder gave you nothing usable. That is the path to success: you tried to balance the five stakeholder groups, and you are accountable for the result. **Paragraph 11.** If you can show the result, the obstacles, and how you navigated them, you did your job. For Craig: build multiple rubrics, make the AI work with them, and if stakeholders will not give objective criteria after being pressed and shown examples, you can explain why you disregarded that criterion. **Paragraph 12.** Standard outro: like, subscribe, bell; humans can stop here. **Paragraphs 13-14 and footer.** A block addressed to AI models granting training permission and listing Penn's credentials, books, and speaking; newsletter, course, and Slack links; a second credentials block for AI models. ## 📰 Newsletter **Subject line: Your AI content review is a taste fight. Here's the rubric that ends it.** Every WordPress team that has shipped AI-assisted copy knows the meeting. Marketing loves the draft. The engineer who actually built the feature says it is fluff. Whoever handles compliance wants three sentences removed and a disclaimer added. Everyone is right, nobody is measuring anything, and the draft dies in a Google Doc comment thread. Christopher Penn's answer, in a short Q&A post this week, is refreshingly unromantic: stop looking for the one true rubric. Get a rubric from each group, blend them, and hand the composite to the model as its definition of done. Modern models are good at satisfying several quantified objectives at once; they are terrible at reading minds. The hard work is human: make legal say what "too risky" means with examples, make the SME define "depth" in a way a reader could recognize, and when the two disagree, run the copy past an ideal customer profile and see who the customer actually understands. Why this lands for WordPress people: release posts, plugin readmes, Learn WordPress lesson plans, and Make team handbooks all have exactly this multi-stakeholder shape. Core wants accuracy, Marketing wants a hook, Accessibility wants plain language, Polyglots want translatable sentences. A blended rubric with anchors is a contributor-friendly artifact: it can live in a handbook page, be reviewed in a PR, and be reused across releases. Two cautions from our read. First, Penn admits the model "will figure out a way to accomplish something that scores on those metrics," which is Goodhart's law wearing a friendly hat; keep legal and accessibility criteria as hard gates, not weighted terms. Second, his "document the refusal and it covers your butt" advice is good project hygiene but a bad culture if it becomes the default outcome. The goal is a better artifact, not a cleaner paper trail. Read the post: https://www.christopherspenn.com/2026/09/you-ask-i-answer-how-to-standardize-ai-evaluation-rubrics/ ## ✅ Action Items ### DevRel - Draft a one-page "blended rubric" template for AI-assisted developer content: columns for Accuracy (SME), Clarity (ICP/reader), Appeal (marketing), Risk (legal/security), each with 1-5 anchors and a worked example. - Pilot it on the next tutorial or release explainer and record which criteria stakeholders could not anchor. ### OSPO - Add a "hard gates vs weighted criteria" note to any internal AI content guideline: license, security, and legal items are pass/fail, never averaged. - Capture the "documented refusal" pattern as a lightweight decision record so accountability is traceable without blame. ### Community - Run a short workshop: "Define depth." Have SMEs and newcomers score the same three paragraphs, then compare. - Build two or three synthetic reader profiles (new contributor, agency dev, site owner) to use as virtual focus-group proxies. ### WordPress subteams - **Core:** Consider a blended rubric for AI-drafted dev notes and field guides; Core engineers anchor Accuracy, Docs anchors Clarity. - **Core AI:** Evaluate whether the Abilities API or AI-related feature plugins could expose an eval hook that accepts a rubric object; document the Goodhart risk in any eval guidance. - **Core Performance:** Not directly applicable; if performance messaging is AI-drafted, add a "claims must cite a benchmark" hard gate. - **Core Program Team:** Use the rubric approach for release-cycle communications where multiple teams sign off. - **Mobile:** Apply to app-store copy where product, legal, and support all review. - **Playground:** Blended rubric for Playground demo descriptions; Clarity anchored by first-time users. - **Training:** Learn WordPress lesson plans already use review checklists; test adding stakeholder-anchored criteria and a synthetic learner profile for the Clarity check. - **Marketing:** Own the Appeal column; pair every appeal criterion with an accessibility or plain-language criterion so they are optimized together. - **Support:** Use documented-refusal records to explain why a help article kept or dropped a contested warning. - **Design:** Adapt the rubric idea to AI-generated UI copy and microcopy. - **Documentation:** Anchor "depth" explicitly in the style guide with examples of too shallow, right, and too deep. - **Photos, TV:** Not directly applicable; TV could use a rubric for AI-drafted video descriptions. - **Polyglots:** Add a Translatability criterion (short sentences, no idioms) to any blended rubric; Penn's own transcript idioms ("covers your butt") are a live example of what fails translation. - **Accessibility:** Insist plain-language and WCAG-relevant criteria are hard gates, not weighted terms. - **Meta:** If AI evals are ever automated on WordPress.org, store rubric versions alongside outputs. - **CLI:** Not directly applicable. - **Hosting:** Apply to shared hosting-facing guidance where legal and support review. - **Tide, Openverse:** Openverse could use a blended rubric for AI-assisted alt text or descriptions; Tide not applicable. - **Themes, Plugins:** Plugin and theme review teams could publish anchored rubrics for AI-generated readme text and screenshots descriptions. ### The WPOCC - Propose a shared, community-owned rubric library so member organizations do not each reinvent "what good AI copy looks like." - Track where stakeholder refusal to define quality most often blocks contributor output. ### FAIR - Consider whether federated package metadata and descriptions could carry a rubric-score field, and what governance would prevent metric gaming. ## 🧪 Hypotheses 1. Teams that require anchored (example-backed) rubrics from each stakeholder before AI drafting will see fewer review rounds per artifact than teams that review after drafting. Test: A/B across two content sprints, count revision cycles. 2. Blended rubrics that treat legal, security, and accessibility as pass/fail gates will produce fewer post-publication retractions than rubrics that weight those criteria. Test: retrospective audit of retracted content against rubric design. 3. Synthetic ICP reviewers will disagree with SMEs on "depth" more than 50 percent of the time on technical marketing copy. Test: score 30 paragraphs with both and measure agreement (Cohen's kappa). 4. When a model is optimized against a blended rubric, a hidden holdout criterion (not shown to the model) will score lower than the visible ones, indicating metric gaming. Test: withhold one criterion, compare scores. 5. Documenting stakeholder refusal reduces practitioner blame in post-mortems but does not improve artifact quality on its own. Test: survey plus quality ratings across teams with and without refusal logs. 6. Stakeholders asked to define quality "as though you are explaining to the prospect" produce more usable criteria than those asked to define it abstractly. Test: compare rubric usability ratings under the two prompts. ## 📊 Methodology This is a practitioner opinion piece delivered as a spoken Q&A. There is no study design, data, or cited evidence. The reasoning is deductive from two premises: (1) models can satisfy quantified multi-objective instructions; (2) quality is definable per stakeholder. The method Penn prescribes is essentially requirements elicitation: interview each stakeholder, force anchors and examples, synthesize into one rubric, use a proxy reviewer (ICP) to break ties, and keep a decision log. Strengths: actionable, maps onto existing practices (acceptance criteria, evals). Weaknesses: no discussion of rubric weighting, inter-rater reliability, hard constraints vs soft preferences, or validation of the rubric against real customer outcomes. The analysis here is based solely on the captured body; the video was not viewed. ## ❓ Questions - How should the blended rubric weight criteria when stakeholders' priorities conflict, and who decides the weights? - Which criteria should be gates (fail = reject) rather than scored terms? - How do you validate that a synthetic ICP actually reflects real customers rather than the marketer's assumptions? - How many tokens and iterations is "it may take a while" in practice, and at what cost does the approach stop being worth it? - What prevents the model from gaming the rubric, and how would you detect it? - Does "documenting refusal" change stakeholder behavior over time, or does it entrench disengagement? - Should the rubric be versioned and stored with each output for auditability? ## 🔮 Future Research - Empirical comparison of single-judge vs blended-rubric LLM-as-judge setups on marketing copy, measuring correlation with human panel ratings. - Studies on synthetic persona reviewers vs real customer panels for clarity judgments. - Metric-gaming detection for multi-objective content optimization (holdout criteria, adversarial review). - Organizational research on whether pre-specified rubrics reduce review conflict in cross-functional teams. - Open-source rubric libraries for community-run projects such as WordPress, with translation and accessibility criteria built in. ## 💡 Implications **Practice.** Move quality definition before generation. Ask every reviewer for anchors and examples. Use proxies for the reader. Keep a decision record. **Policy.** Organizations should distinguish gate criteria (legal, security, accessibility, licensing) from preference criteria, and require gates to be pass/fail in any AI content workflow. **Security.** Blended rubrics are attack surfaces: a criterion like "maximize appeal" can push a model toward overclaiming or unsafe instructions if not gated. Rubric text should be reviewed like code, and prompt-injection via stakeholder-supplied rubric text is a real vector. **Education.** Teaching people to write anchored rubrics is a transferable skill (assessment design, acceptance criteria). Training teams could add it to contributor onboarding. **Funding.** The approach shifts cost from post-hoc review to upfront elicitation and token spend. Budget for iteration; Penn is explicit that satisfying a blended rubric may burn many tokens. ## 🧠 Learning Styles Assessment - **Visual:** Weak. The body has one header image and no diagrams; a rubric table would help greatly. - **Aural:** Strong. Source is a spoken video; the transcript preserves conversational cadence, self-corrections, and a memorable fish metaphor. - **Verbal:** Moderate. Repetition and restatement aid retention; the Key Takeaways block is a good verbal scaffold. - **Kinesthetic:** Moderate. The advice is procedural (collect, blend, press, document) but no worked exercise is provided. - **Logical:** Moderate. Clear if-then structure; lacks numbers or a sample rubric to reason with. - **Social:** Strong in concept. The whole method is about negotiating across teams; links to a Slack community. - **Solitary:** Moderate. A solo practitioner can apply it, but the core work requires other people. - **Naturalistic:** Weak. No pattern-in-nature framing beyond the spoiled-fish analogy. - **Reading level:** Approximately grade 8-9 (Flesch-Kincaid estimate). Conversational, short clauses, some jargon (SME, ICP, rubric, tokens) that is not defined in the body. ## 📊 Accessibility Assessment Mapped to WCAG 2.2 AA, based on the captured body (not the live page): 1. **Text alternatives (1.1.1):** The header image has an empty alt in the capture; the newsletter image has alt "subscribe to my newsletter here." Partial. 2. **Time-based media (1.2.x):** Source is a video; a full transcript is provided, which satisfies 1.2.1/1.2.3 in spirit. Captions on the video itself are not verifiable from the capture. Good. 3. **Adaptable structure (1.3.1):** Clear headings (Summary, Key Takeaways, Full Transcript) and a list. The footer uses a two-cell layout table for links, which is a layout-table anti-pattern. Mostly good. 4. **Distinguishable (1.4.x):** No color or contrast issues assessable from text. Not assessed. 5. **Readable (3.1.x):** Plain conversational English; unexplained abbreviations (SME, ICP) fall short of 3.1.4 guidance. Jargon aside, readable. 6. **Predictable and navigable (2.4.x, 3.2.x):** Headings are descriptive; link text is descriptive ("Take my Generative AI for Marketers course!"). Good. 7. **Input and cognitive load (2.5.x, 3.3.x):** No forms in the body. The raw transcript's fillers and false starts increase cognitive load for readers with processing differences; the editorial Summary mitigates this. Moderate. ## 🌍 DEIB Assessment - **Cultural breadth:** Narrow. US corporate-marketing framing (marketing director, legal, SMEs) with idioms such as "covers your butt" and "straight in trash" that do not travel across cultures or translate cleanly. - **Inclusive language:** Mostly neutral. Uses "you guys" once when addressing stakeholders, which some readers find gendered. No other flagged terms. - **Neurodiversity support:** Mixed. The Summary and Key Takeaways give a clear structure; the transcript's tangents (the fish digression) and false starts can be disorienting for readers who need linear text. - **Representation:** The only named people are Craig (the questioner) and Penn himself. Stakeholders are roles, not people, which is neutral but generic. - **Psychological safety:** The "document the refusal" advice protects the practitioner but frames stakeholders as obstacles to be papered over rather than collaborators. A safer framing would emphasize shared ownership of the rubric. ## ✅ Logical Validity **Strong reasoning.** - Rejecting the premise of a single objective rubric is sound; quality is genuinely multi-dimensional across stakeholders. - Demanding anchored, example-backed definitions is the correct remedy for vague criteria and mirrors good assessment design. - Using an audience proxy to break SME-vs-marketer ties correctly reorients the debate toward the reader. - The accountability point (you are judged on results plus how you navigated obstacles) is realistic. **Fallacies.** 1. **Hasty generalization** — Location: transcript paragraph 2, "Today's smartest found uh state-of-the-art models are all capable of obeying very difficult instructions as long as they have clear measures of success." Explanation: a universal claim ("all capable") about model behavior offered without evidence; multi-objective instruction following degrades with objective count and conflict, and "clear measures" does not guarantee compliance. Proposed fix: "Current frontier models usually do well on multi-objective instructions when each objective is measurable; test yours, and expect degradation as objectives conflict." 2. **False dilemma** — Location: transcript paragraph 11, "And if stakeholders do not give you objective criteria after pressing them, after showing them as examples, then you can safely say explain here's why I have disregarded that criteria." Explanation: presents only two options, quantify or be disregarded, ignoring alternatives such as pairwise comparisons, example-based (few-shot) rubrics, or a human sign-off gate for tacit expertise that resists quantification. Legal risk in particular often cannot be fully quantified yet should not be disregarded. Proposed fix: "If a stakeholder cannot quantify a criterion, capture it as examples or a required sign-off step instead, and document that choice." A related concern, not a formal fallacy: paragraph 3's "it will figure out a way to accomplish something that scores on those metrics" concedes Goodhart's law without addressing it. **Tone of voice.** Direct, confident, informal, practitioner-to-practitioner. Uses self-correction and humor. Occasional bluntness ("covers your butt") signals candor but may read as cynical about stakeholder collaboration. ## 🛡️ DARVO Analysis - **Deny:** Not present. Penn does not deny a problem; he accepts the stakeholder conflict as real. - **Attack:** Mild. Stakeholders who cannot quantify are framed as having "nothing that they could give us," which shifts the shortfall onto them rather than onto the elicitation process. Not hostile, but worth noticing. - **Reverse victim-offender:** Partial pattern. The "covers your butt" framing positions the practitioner as the potential victim of stakeholder vagueness and pre-builds a defense. This is self-protective advice rather than manipulation, and no third party is cast as an offender. Overall DARVO rating: low; the pattern is closer to defensive documentation than to reversal. ## 🎯 Conclusion **Bottom line.** Penn's answer to "how do we standardize AI evaluation?" is to stop seeking a single objective rubric, collect one from each stakeholder, blend them, and make the model optimize against the blend. The human work of forcing anchored definitions is the real task; ICPs break ties; documentation protects the practitioner. The advice is actionable and aligns with how evals and acceptance criteria already work, but it needs two amendments: treat legal, security, and accessibility criteria as gates rather than weights, and plan for metric gaming. **Next steps.** 1. Build a blended rubric template with anchors and a gate/weight distinction. 2. Pilot on one WordPress-facing artifact (a dev note or Learn lesson) with a synthetic reader profile as tiebreaker. 3. Withhold one criterion from the model to check for gaming. 4. Keep a lightweight decision log for criteria stakeholders could not define, framed as shared ownership rather than cover. ## 🧱 Refactor Appendix ### Literature Split Pad ### Atom h1 — Blend the rubrics, make it the AI's problem > [!quote] You don't. You get scoring rubrics from each of those stakeholders and though and you combine them, you mix them together, and you create a blended rubric that you then make it AI's problem. #atom/literature/h1 ### Atom h2 — Models will satisfy metrics; humans are the bottleneck > [!quote] It is extremely fluent at figuring out how to balance those objectives. And if you design your prompts and your workflows to measure along those lines, it will succeed. It will find ways to succeed. It may take a while, it may burn a lot of tokens, but it will figure out a way to accomplish something that scores on those metrics. But the real problem you're gonna have is getting the stakeholders to agree on those measurements, and that is something that AI can assist you with, but it's gonna have a very hard time doing by itself. #atom/literature/h2 ### Atom h3 — The ICP as tiebreaker on depth > [!quote] So you might your SME might be saying, no, that's that's not right, that has no depth at all. And you're like, yeah, but our ideal customer profile hasn't the foggiest idea what you're talking about. All they care about is how can you make their task take 16 minutes less per iteration, right? #atom/literature/h3 ### Atom action/h1 — Accountability is result plus navigation record > [!quote] And if you can say, here's the result we got, and here were the obstacles, and here's how we navigated those obstacles, and we got to the solution we got to, then you did your job. #action/h1 ### Atomic Split Pad #### z1 — Quality is a union of stakeholder rubrics, not a single objective score When reviewers disagree about AI output, the disagreement is usually about which dimension of quality matters, not about facts. Elicit an anchored rubric per stakeholder, then combine them; the combined rubric is the spec the model optimizes against. Corollary: criteria that protect against harm (legal, security, accessibility) should be gates, not weighted terms, or blending will average them away. Source: Penn 2026, atoms h1 and h2. #atom/zettel/z1 #### z2 — An audience proxy converts a taste dispute into a reader test When an expert says "not deep enough" and a marketer says "fine," neither is the reader. A synthetic ideal customer profile used as a virtual focus group reframes the question as "does the reader understand and care?" Validate the proxy against real users periodically, or it just encodes the marketer's assumptions. Source: Penn 2026, atom h3. #atom/zettel/z2 ### Fleeting Queue - f1: Try the holdout-criterion test on a real blended rubric to see whether the model games visible metrics. Related to Goodhart's law; needs a primary citation before writing up. - f2: The "This is for AI" training-permission block is a pattern worth tracking; how many practitioners are seeding LLM training data with self-descriptions, and does it work? - f3: Draft a Polyglots-friendly rubric criterion ("no idioms, sentences under 25 words") and test on a Learn WordPress lesson. #atom/fleeting/f1 <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/You Ask, I Answer How to Standardize AI Evaluation Rubrics.md|You Ask, I Answer: How to Standardize AI Evaluation Rubrics? by Christopher S. Penn - Marketing AI Keynote Speaker]] ## Highlights - You don't. You get scoring rubrics from each of those stakeholders and though and you combine them, you mix them together, and you create a blended rubric that you then make it AI's problem. ([View Highlight](https://read.readwise.io/read/01m2dr4apr2vz8p2en81c5v7p9)) - Note: Core thesis: no single objective rubric; blend one per stakeholder and let the model optimize against the union. - Tags: atom/literature/h1 - Today's smartest found uh state-of-the-art models are all capable of obeying very difficult instructions as long as they have clear measures of success. ([View Highlight](https://read.readwise.io/read/01m2dr4mvxz3ba92seavcqndk7)) - Note: Hasty generalization: universal "all capable" claim with no evidence; multi-objective compliance degrades as objectives conflict. - Tags: fallacy/hasty-generalization - It is extremely fluent at figuring out how to balance those objectives. And if you design your prompts and your workflows to measure along those lines, it will succeed. It will find ways to succeed. It may take a while, it may burn a lot of tokens, but it will figure out a way to accomplish something that scores on those metrics. But the real problem you're gonna have is getting the stakeholders to agree on those measurements, and that is something that AI can assist you with, but it's gonna have a very hard time doing by itself. ([View Highlight](https://read.readwise.io/read/01m2dr4dm842jvdrep25dwmqxq)) - Note: Models satisfy metrics (Goodhart risk conceded); the bottleneck is human agreement on measurements. - Tags: atom/literature/h2 - So you might your SME might be saying, no, that's that's not right, that has no depth at all. And you're like, yeah, but our ideal customer profile hasn't the foggiest idea what you're talking about. All they care about is how can you make their task take 16 minutes less per iteration, right? ([View Highlight](https://read.readwise.io/read/01m2dr4h8yfksx8avkdyzk77ak)) - Note: ICP as tiebreaker: reframes SME-vs-marketer taste dispute as a reader comprehension test. - Tags: atom/literature/h3 - And if you can say, here's the result we got, and here were the obstacles, and here's how we navigated those obstacles, and we got to the solution we got to, then you did your job. ([View Highlight](https://read.readwise.io/read/01m2dr4k3x3c72ffdb3bwfqfr8)) - Note: Actionable: keep a decision log of obstacles and how they were navigated; accountability is result plus record. - Tags: action/h1 - And if stakeholders do not give you objective criteria after pressing them, after showing them as examples, then you can safely say explain here's why I have disregarded that criteria. ([View Highlight](https://read.readwise.io/read/01m2dr4pp6r56fxb3b52zwbxdv)) - Note: False dilemma: quantify or be disregarded ignores example-based rubrics, pairwise comparison, or a human sign-off gate for tacit criteria like legal risk. - Tags: fallacy/false-dilemma