# You Ask, I Answer: Measuring AI ROI Beyond Time Saved? > [!question] Lingering Questions > - What's the minimum viable baseline (how much history, which KPI) before an uplift estimate becomes trustworthy rather than noise? > - Does Penn's "resistance = underperformance" heuristic hold at the team/department level, or does group-level resistance start indicating something else (bad metric design, psychological safety, past retaliatory use of data)? > - How does this reasoning change for roles where "done" is inherently qualitative (creative or strategic work) and resists a single KPI? > [!tip] Article Ideas > - "Your AI ROI problem is a KPI problem you already had" — a DevRel/OSPO-facing piece translating the baseline-first argument for teams being asked "what's the AI ROI here?" without first being asked "what does done look like?" > - A short field-note applying uplift modeling (control period vs. treatment period) to a real AI-adoption metric already tracked in a WordPress or DevRel program. > [!link] Cross Reference > - [[3. Resources/Permanent Notes/A Metric Is Vanity Only Against Your Own Outcome Data]] — same author (Penn), same underlying discipline: a metric only means something once it's tested against your own outcome data, whether that's "vanity vs. real" or "AI ROI vs. no ROI" > - [[3. Resources/Permanent Notes/An Agent Evaluation Without an Incumbent Baseline Cannot Establish Value]] — parallel baseline argument one level down (agent-vs-incumbent-tool rather than org-vs-its-own-past-performance) > [!quote] Vocabulary > - *uplift modeling* — a biostatistics-derived technique comparing a control period against a treatment period to isolate one variable's (here, AI's) incremental effect > - *Five P framework* (Trust Insights) — Purpose, People, Process, Platform, Performance; Performance = the quantitative "definition of done" ## Bib capsule Christopher S. Penn, "You Ask, I Answer: Measuring AI ROI Beyond Time Saved?" — Trust Insights "You Ask, I Answer" YouTube series, answering a viewer question (Yuri) about measuring AI's organizational ROI when the visible benefit is individual time saved. ## Author's thesis "AI ROI" is usually a misdiagnosis: it isn't a distinct measurement category at all, but ordinary performance measurement applied to a new input. An organization that can't say what "done" looks like for a role had no baseline before AI arrived, so it has nothing to compare an "after" against — no baseline, no valid ROI claim, regardless of how good the AI tool is. ## Key arguments (≤3) 1. **No baseline, no delta.** ROI is inherently a before/after comparison. Missing task/time/output tracking isn't an ambiguity AI creates — it's a pre-existing blind spot AI makes invisible rather than fixes. 2. **Role-specific KPIs already exist, just disconnected from AI adoption** — customer service (satisfaction/resolution), sales (close rate, deal value), marketing (lead volume/quality), advertising (cost-to-traffic efficiency), PR (placement count/quality). Once a baseline metric is chosen, uplift modeling (control period vs. treatment period) isolates AI's actual incremental effect. 3. **Resistance to measurement is diagnostic, not just friction.** Penn treats pushback against introducing performance measurement as a "yellow flag" that disproportionately comes from underperformers — itself evidence of a deeper management gap, not an argument against measuring. (Flagged in the source's own logical-validity pass as an unqualified universal claim — see processing checklist.) ## Claim atoms ### No baseline performance measurement means no AI ROI measurement > [!quote] ID: rw_01kzrkr4ce8kvfph2vencmk26p > If you have no idea what your performance metrics are, you do not have an AI problem Opening reframe of the entire answer — shifts responsibility from "find a better AI metric" to "fix the underlying measurement system," a harder but more accurate fix. #xref ### Underperformers resist performance measurement most > [!quote] ID: rw_01kzrkr4ce8kvfph2vencmk26p > the strongest push back you will get on measuring performance management is from underperformers at every level Offered as a prioritization heuristic for diagnosing rollout resistance. The source's own Logical Validity pass flags this as an unqualified claim of universality — doesn't acknowledge other legitimate causes of resistance (poorly designed metrics, past retaliatory data use, privacy concerns). Treat as "often," not "always." #xref ### Uplift modeling isolates AI's effect via control-vs-treatment comparison > [!quote] ID: rw_01kzrkr4ce8kvfph2vencmk26p > there's an entire branch of statistics called uplift modeling that we borrow from biostatistics The concrete statistical method once a baseline exists — bridges "measure your KPI" to an actual technique for attributing a delta specifically to AI rather than other confounders. Named but not mechanically explained (no discussion of propensity scores or confounders) — verify before treating as a fully specified method. #xref #addendum ## Permanent candidates - **AI ROI requires a pre-existing performance baseline** — the strongest, cleanest candidate; promoted below. - **Uplift modeling isolates a treatment's effect via control-vs-treatment comparison** — a general measurement-method claim, useful independent of the AI framing; promoted below. - Underperformer-resistance-as-diagnostic — NOT promoted; the source's own logical-validity pass names it as an unqualified overgeneralization (see Lingering Questions). ## Processing checklist - [ ] Verify whether Penn's "uplift modeling" usage matches standard causal-inference uplift modeling (propensity/treatment-control) or is a looser analogy - [ ] Don't reuse the PR-measurement anecdote ("when you did the actual measurements... that would more often than not show the work didn't matter") as a citable claim — it's n=1 professional experience, not cited data - [ ] Verify Trust Insights' Five P framework write-up (trustinsights.ai) before citing it elsewhere - [x] Connect into the relevant MOC — [[3. Resources/MOCs/AI Cost & Tokenomics MOC]] ## Metadata - Source: [[3. Resources/Readwise/Podcasts/You Ask, I Answer Measuring AI ROI Beyond Time Saved]] - Duplicate Readwise import: doc `01kzrkswc8h32m9dqecswnq2pj` (same YouTube video, separate Reader capture) — this note is the single canonical Literature Note for the source; do not create a second one from the duplicate doc. ## Links - [[3. Resources/MOCs/AI Cost & Tokenomics MOC]] — this is a measurement-methodology argument for what "AI ROI" even means before any cost/value math is possible - [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] — AI-adoption evaluation in knowledge-work roles (customer service, sales, marketing, PR)