Beyond Pull Requests: Quantifying the ROI of InnerSource  🔗 **Source:** [Beyond Pull Requests: Quantifying the ROI of InnerSource](https://www.youtube.com/watch?v=oQVN1iYlf2A) > [!note]+ Document Notes > tags: [innersource, metrics, goodharts-law, developer-productivity, knowledge-sharing, ai-agents, dora] > author: founder/CEO of the presenting company (name uncertain in transcript — see note) > source: InnerSource Commons community call (YouTube) > reference: https://www.youtube.com/watch?v=oQVN1iYlf2A > date created: 2026-07-31 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > > [!warning] Names unreliable in this capture > > The auto-transcript renders the speaker as "Ununice Hayes" and the company as "Aquamo," and the > > client as "Toshima." All three are likely mis-transcriptions and none could be verified from the > > capture. **Do not cite any name from this document without checking the source video.** The > > speaker is a non-native English speaker and the auto-captioner struggles throughout. > > # 🏷️ Title > > Beyond Pull Requests: Quantifying the ROI of InnerSource — InnerSource Commons community call > > # 🧭 Table of Contents > > 1. Summary > 2. Main Content / Notes > 3. References > 4. Paraphrase > 5. Newsletter > 6. Action Items > 7. Hypotheses > 8. Methodology > 9. Questions > 10. Future Research > 11. Implications > 12. Learning Styles Assessment > 13. Accessibility Assessment > 14. DEIB Assessment > 15. Logical Validity > 16. DARVO Analysis > 17. Conclusion > 18. Refactor Appendix > 19. Template Library > > # 🔗 References > > - **DORA** (DevOps Research and Assessment) four metrics — deployment frequency, lead time, change failure rate, time to restore > - **Amazon's internal "hero rank" leaderboard** ranking developers by AI tokens burned; reportedly killed in May and replaced with "normalized deployment" > - A ~100-year-old Japanese textile company case study (name unreliable in transcript) > - **InnerSource Commons** — the hosting community > - The speaker's own InnerSource maturity assessment tool > > # 📝 Summary > > **What happened.** A ~14-minute talk on why InnerSource ROI is hard to demonstrate, which metrics actually connect to business outcomes, a textile-industry case study, and an argument that InnerSource practice is the substrate for agent adoption. > > **Why it matters.** Two things stand out. The **Amazon token leaderboard** is a fresh, concrete instance of Goodhart's Law in AI-era metrics — engineers ran pointless tasks to climb a ranking that measured token consumption, compute costs spiked, the leaderboard was killed. And the case study is **not a software company**, which is unusual and useful: InnerSource practices applied to textile design patterns produced a claimed 70% cycle-time reduction. > > **Stakeholders.** Engineering leaders justifying InnerSource investment; anyone designing developer productivity metrics; organizations preparing for agent adoption. > > **Risks.** The speaker is a vendor selling InnerSource consulting and an AI governance platform, and closes by pitching a maturity assessment tool. The case study numbers are unaudited client claims. > > **Bottom line.** One genuinely valuable warning (measure outcomes, not activity — with a live example), one useful non-software case study, and a vendor pitch wrapped around them. > > # 📖 Main Content > <!-- deep-read:sections start --> ## 🗒️ Notes **[00:00:08–00:01:00] Positioning.** The speaker founded a professional-services company building InnerSource infrastructure — platforms, workflows, engineering practices — now building an AI platform for agents to "execute in a governed way" on enterprise infrastructure. The commercial interest is disclosed up front, which is to their credit. **[00:01:30–00:03:03] Why InnerSource ROI is hard.** Four reasons, and the first is the sharpest: > [!quote] > our measurement DNA is integrated is industrial. We are great at counting deployments and uh uptime but terrible at quantifying collaboration or knowledge transfer. The other three: returns are **delayed and nonlinear** (compounding over quarters, not visible next sprint); there is **no shared language**, so cross-organization comparison fails; and **DORA is insufficient**: > [!quote] > it tells you how fast um code moves not how knowledge flows. That last line is the talk's cleanest formulation. DORA measures throughput of a pipeline; InnerSource claims to change who knows what, which the pipeline metrics cannot see. **[00:03:03–00:04:31] The strategic frame.** Three questions before measuring anything: what creates competitive advantage (CI/CD probably not; a domain-specific AI model, yes); what to own versus rent; and where speed and innovation are actually needed. The stated failure modes are outsourcing everything and losing your edge, or building everything in silos and moving too slowly. InnerSource is positioned as most valuable where **multiple teams need the same thing, can't wait for a vendor, and shouldn't rebuild it separately** — a reasonably precise scope condition. **[00:04:31–00:05:31] Activity versus outcome.** The wrong metrics: repo counts, PRs, lines of code, and — added recently — **tokens burned**. The right ones: developer productivity through reuse, knowledge sharing across teams, operational efficiency, platform adoption. > [!quote] > These tell you activities happened. They don&#39;t tell you if it created value. He is explicit that the wrong metrics dominate because they are easy to collect, not because anyone is lazy — which is the correct diagnosis and unusually charitable. **[00:05:31–00:06:32] The Amazon leaderboard — the talk's best material.** Reported as recent and observable: > [!quote] > Amazon built a an internal a leaderboard called u hero rank ranking developers by how many AI tokens they burned. Within weeks engineers were token maxing running pointless tasks just to climb the rankings. So compute cost spike uh and then Amazon uh killed the uh leaderboard in May and pivoted to what they call normal normalized deployment measuring useful shipped code not raw activity. This is textbook Goodhart: the measure became the target, engineers optimized it directly, and the optimization consumed real money while producing nothing. The speaker's framing is that the unit changed but the error did not: > [!quote] > So same mistakes we&#39;re talking u about here just with a different unit. So counting to token instead of count input request. **[00:06:32–00:07:32] Metrics mapped to executive concerns.** Shared components → time to market; redundant builds → wasted budget; **bus factor** → knowledge retention risk; adoption rate → whether teams actually trust what's shared. The argument is that each is a CFO or CIO conversation rather than an engineering one — which is the actual point of the talk, since InnerSource funding decisions happen at that level. **[00:07:32–00:09:30] The case study.** A ~100-year-old Japanese textile company. Before: designers worked in silos; sales needed physical samples for client proposals; samples took weeks to produce; **16% were never used**; no way to search existing designs. After applying InnerSource — described as open contribution and visible assets: - Pattern-to-client cycle **down ~70%** - Sample testing **from 3 weeks to 2 hours** - **60% fewer physical samples** - Sales able to test ~10× more variations The speaker's own emphasis is worth noting: "this isn't about software, it's really about collaboration." A non-software InnerSource case study is genuinely uncommon and makes the practice-versus-technology distinction concrete. **[00:09:30–00:11:02] Agents as contributors — the most transferable section.** InnerSource framed as "a living engineering system, not a policy doc, not a wiki," with specific mechanics: - The repo teaches contributors how to work - Specs define what done looks like - Structured issues let any team pick up work **without a handover meeting** - Tests committed before code, "so the agent has a contract to implement against" - Validation in an isolated sandbox, with a human as final gate > [!quote] > the key insight here is that agents are contributors. So same inner source patterns as humans um uh low on boarding costs, clear ownership, executable uh acceptance criteria. The underlying claim is that the practices that let an unfamiliar *human* team contribute — legible conventions, explicit done-criteria, self-service onboarding — are the same ones that let an agent contribute. Which reframes InnerSource maturity as agent-readiness. His reason for urgency: > [!quote] > the barrier to to to building is um is is getting lower. So everyone can just wipe code *(that is "vibe code," mis-transcribed)* **[00:11:30–00:12:31] The AI-native argument.** AI "doesn't create intelligence, it amplifies what's already there." Practising InnerSource means already measuring collaboration, making contributions visible, and producing structured data about how teams actually work — which becomes context for agents. He links this to "harness" and context engineering. **[00:12:31–00:14:01] Recommended actions.** Start small with one shared capability (a library, tool, skill, or CI template); measure usage, contribution, and reuse with no special tooling; then quantify by asking: > [!quote] > if we didn&#39;t share this, how much time would we lose rebuilding it? That counterfactual is the most practical thing in the talk — it converts an unmeasurable benefit into an estimable one. Closes with a pitch for the company's maturity assessment tool. # 📚 References Primary sources preferred: reporting on Amazon's internal leaderboard and its withdrawal (the claim is specific and checkable, and the talk cites nothing); the DORA research programme's own publications on what the four metrics do and do not measure; InnerSource Commons patterns library; and the client case study, which would need the company's own account to verify. The speaker's maturity tool is a commercial product, not a reference. # 🗣️ Paraphrase Demonstrating that InnerSource pays off is hard for four reasons: our measurement habits come from manufacturing and count throughput rather than knowledge; the returns compound slowly rather than appearing next sprint; no two organizations measure it the same way; and DORA, the standard delivery framework, tracks how fast code moves rather than how knowledge travels. Before measuring, decide what is worth measuring. What actually differentiates you, what you should own rather than rent, and where several teams need the same capability and cannot each build it. Then measure outcomes rather than activity. Repository counts, pull requests, lines of code, and now tokens consumed all record that something happened, not that it was worth doing. Amazon learned this recently and publicly: a leaderboard ranking engineers by token usage produced engineers running useless jobs to climb it, until the leaderboard was withdrawn. A textile company applied the same practices to design assets rather than code and reports substantially faster cycles and far fewer wasted physical samples. Finally, the practices that make it easy for an unfamiliar team to contribute — legible conventions, explicit acceptance criteria, work that can be picked up without a meeting — are the same ones that make it possible for an agent to contribute. Organizations already doing InnerSource are, without having planned it, prepared for that. # 📰 Newsletter **Amazon ranked engineers by tokens burned. You can guess what happened.** The best thirty seconds of this InnerSource Commons talk is a story that should go straight into every conversation about AI productivity metrics. Amazon built an internal leaderboard — reportedly called "hero rank" — ranking developers by **how many AI tokens they burned**. Within weeks engineers were *token maxing*: running pointless tasks purely to climb the board. Compute costs spiked. The leaderboard was killed in May and replaced with something measuring useful shipped code. That is Goodhart's Law with a fresh coat of paint, and the speaker's framing is exactly right: **the unit changed, the mistake didn't.** We counted lines of code, then pull requests, now tokens. Each one records that activity occurred. None records that it was worth doing. The rest of the talk is about why InnerSource ROI is so hard to prove, and one line is worth keeping: **DORA "tells you how fast code moves, not how knowledge flows."** DORA is excellent at pipeline throughput. InnerSource claims to change *who knows what* — and no deployment-frequency metric can see that. The most practical takeaway is a counterfactual you can run today: *if we didn't share this, how much time would we lose rebuilding it?* That turns an unmeasurable benefit into an estimable one, and it's the number a CFO will actually engage with. There's also a genuinely unusual case study — a **century-old Japanese textile company**, not a software firm. Designers working in silos, sales waiting weeks for physical samples, 16% of samples never used. After opening contribution and making assets visible: cycle time down ~70%, sample testing from three weeks to two hours, 60% fewer physical samples. Unaudited vendor-reported numbers, but the point that InnerSource is a collaboration practice rather than a software one lands better with a non-software example. And the argument worth stealing for anyone thinking about agents: **the practices that let an unfamiliar human team contribute are the same ones that let an agent contribute.** Legible conventions, explicit done-criteria, structured issues that don't need a handover meeting, tests committed first so there's a contract to implement against. If that's already true of your repo, you're agent-ready without having planned for it. If it isn't, no amount of tooling will substitute. Usual caveat: the speaker sells InnerSource consulting and closes with a pitch for their maturity assessment tool. # ✅ Action Items **DevRel** - The Amazon token-leaderboard story is an excellent, current teaching example for metric design — use it. - "How much time would we lose rebuilding it?" is a usable framing for justifying shared-component work. **OSPO** - The activity-versus-outcome distinction applies directly to how contribution is reported; PR counts are the classic activity metric. - Bus factor as an explicitly tracked metric is worth adopting. **Community** - The InnerSource-as-agent-readiness argument reframes documentation and convention work as forward-looking rather than housekeeping. **WordPress subteams** - **Core Program Team / Meta:** contribution metrics currently skew toward activity counts; the outcome-metric framing is directly applicable. - **Documentation:** "the repo teaches contributors how to work" is the standard this sets — self-service onboarding without handover meetings. - **Core AI:** agents-as-contributors implies the same onboarding surface serves both; worth designing for once rather than twice. - **Training:** the strategic frame (own versus rent, where multiple teams need the same thing) is teachable. - **Community:** bus-factor tracking for critical components is a concrete practice. - **Plugins / Themes:** shared-component reuse is the exact InnerSource case. - **Core / Core Performance / Mobile / Playground / Marketing / Support / Design / Photos / TV / Polyglots / Accessibility / CLI / Hosting / Tide / Openverse:** no direct action from this source. - **The WPOCC:** InnerSource patterns fit member agencies sharing tooling across client work. - **FAIR:** no direct action from this source. # 🧪 Hypotheses 1. Metrics measuring resource consumption (tokens, compute) produce gaming behaviour faster than metrics measuring output, because consumption is directly actionable. 2. DORA metrics show no correlation with knowledge distribution measures such as bus factor. 3. Organizations with mature InnerSource practice achieve productive agent contribution faster than those without, controlling for tooling. 4. The counterfactual rebuild-cost estimate produces ROI figures that survive finance scrutiny more often than activity-based justifications. 5. InnerSource benefits appear on a quarters-long lag, making sprint-level evaluation systematically misleading. 6. Non-software applications of InnerSource show similar cycle-time effects to software ones. # 📊 Methodology **None in a research sense** — a vendor presentation to a practitioner community. The case study is the only quantitative content, and every figure is client-reported and unaudited: 70% cycle reduction, 3 weeks to 2 hours, 60% fewer samples, 16% unused. No baseline methodology, no control, no time period, no confounders addressed. The company is a client of the presenting consultancy, so these are outcomes reported by the party who sold the intervention. The Amazon anecdote is presented as recent fact with a specific month ("killed in May") and a specific internal name. It is checkable and is not cited — the strongest content in the talk rests on an unsourced claim about another company's internals. The four reasons ROI is hard are analytical assertions rather than findings, and are reasonable ones. Commercial position is disclosed clearly at the start and again at the close, which is the right handling even though the content is shaped by it. # ❓ Questions - Can the Amazon leaderboard story be confirmed independently, including the "hero rank" name and the May withdrawal? - What was the measurement baseline for the textile case, and over what period? - Does the 16%-unused-samples figure predate the intervention, and how was it established? - What specifically was shared in the textile case — patterns, designs, tooling, process? - Does agent contribution actually work in their environment, or is that a projection? - What does the maturity assessment measure, and against whose leader benchmark? # 🔮 Future Research - Empirical study of gaming behaviour across metric types (consumption versus output versus outcome). - Correlation analysis between DORA metrics and knowledge-distribution measures. - Controlled comparison of agent onboarding cost in high- versus low-InnerSource-maturity repositories. - Replication of InnerSource practices in non-software domains with independent measurement. # 💡 Implications **Practice.** The Amazon token-consumption leaderboard is a clean, recent instance of Goodhart's Law: engineers ran pointless tasks to climb a ranking that measured tokens burned, compute costs spiked, and the leaderboard was killed. Any AI-adoption metric defined over consumption rather than outcome will reproduce it. **Policy.** Measuring InnerSource against DORA metrics rather than contribution counts changes what leadership can be asked to fund. A practice justified by pull-request volume is defensible only while that number rises; one justified by lead time and change-failure rate survives its own success. **Security.** Not applicable to this source. No security dimension is raised. **Education.** The non-software case study — a century-old textile company — is the more transferable teaching example precisely because it removes the confound that InnerSource only works where everyone already writes code. **Funding.** The talk's real argument is that InnerSource practice is the substrate for agent adoption, which reframes it from a cultural nicety to a prerequisite investment. That is a funding case, and it is stronger than the ROI case it opens with. # 🧠 Learning Styles Assessment - **Visual:** Significant — the talk is slide-driven, and the speaker refers to on-screen content ("what we see in the slide," left/right comparisons) that the transcript does not capture. - **Aural:** Difficult. Heavy disfluency, frequent restarts, and non-native English with auto-captioning that compounds the problem. - **Verbal:** High density of acronyms and terms — DORA, InnerSource, bus factor, harness — mostly explained, with DORA notably well glossed. - **Kinesthetic:** Moderate; the three closing actions are directly actionable. - **Logical:** Strong structure, explicitly signposted at the outset (challenge → what to measure → case study → AI future) and followed throughout. - **Social:** Community-call format; a question appears to be taken near the end. - **Solitary:** Moderate — InnerSource Commons is a discoverable follow-up, though the talk's own references are thin. - **Naturalistic:** Not applicable to this source. **Reading level estimate:** roughly grade 12–14 equivalent for the concepts. The *delivered* difficulty is higher than the conceptual difficulty because of disfluency and transcription noise — the ideas are more accessible than this transcript makes them appear. # 📊 Accessibility Assessment Mapped to WCAG 2.2 AA: 1. **Captions (1.2.2):** **Poor.** Auto-captions struggle badly with the speaker's accent — names are mangled beyond recovery ("Ununice Hayes," "Aquamo," "Toshima"), "vibe code" becomes "wipe code," and sentences fragment. Auto-captioning underperforms systematically for non-native speakers, and this is a clear instance. Fails for practical purposes. 2. **Audio description (1.2.5):** **Fails.** Slide-driven talk with explicit references to on-screen comparisons that are never read aloud in full. 3. **Keyboard/navigation (2.1):** Platform-determined. Pass. 4. **Timing / seizure safety (2.3):** No flashing content. Pass. 5. **Reading level (3.1.5):** Fails plain-language targets, largely due to delivery rather than concept. 6. **Predictable structure (3.2):** **Strong** — the agenda is stated up front and followed, which materially aids comprehension despite the delivery difficulties. 7. **Assistive-tech compatibility (4.1):** Standard platform; transcript machine-readable but unreliable. Pass with caveats. **Notable:** the combination of a non-native speaker and slide-dependent content makes this one of the least accessible sources in this batch, despite clear organization. Human-corrected captions would fix most of it. # 🌍 DEIB Assessment - **Cultural breadth:** Genuinely good, and the strongest dimension. The presenter appears to be a non-native English speaker addressing an international community, and the case study is a Japanese textile company — a non-Western, non-software, century-old organization. That is a materially broader frame than the US-software-startup default that dominates this genre. - **Inclusive language:** Clean. No group is characterised negatively. The framing of why teams measure the wrong things — "not because they're lazy, but because the obvious metrics aren't easy to collect" — is explicitly charitable, which is a small but real choice. - **Neurodiversity support:** Mixed. The explicit agenda and consistent structure genuinely help. Against that, heavy disfluency, frequent self-interruption, and slide-dependence raise processing cost considerably. - **Representation:** Single presenter, founder/CEO. Worth noting that the presenter is not from the dominant demographic of this genre, and that the accessibility failure above falls disproportionately on exactly such speakers — auto-captioning quality is a representation issue, not merely a technical one. - **Psychological safety:** Good. The talk's stance toward practitioners is non-judgemental throughout, and the "I did my share as well" admission about getting build-versus-buy wrong is a small act of self-implication that lowers the bar for the audience. # ✅ Logical Validity ## Strong reasoning - **"How fast code moves, not how knowledge flows"** is a precise and fair characterisation of DORA's scope. It identifies what the standard framework structurally cannot see rather than dismissing it. - **The Amazon example is well-chosen and well-used.** It is recent, concrete, involves a sophisticated organization, and demonstrates the exact failure the talk warns about. Crucially, the speaker draws the *general* lesson — same error, different unit — rather than treating it as an Amazon-specific blunder. - **Charitable diagnosis of bad metrics.** Attributing the prevalence of activity metrics to collection ease rather than to laziness is both more accurate and more actionable. - **The counterfactual ROI question** is genuinely useful: "if we didn't share this, how much time would we lose rebuilding it?" converts an unmeasurable into an estimable. - **A precise scope condition for InnerSource** — multiple teams need the same thing, can't wait for a vendor, shouldn't rebuild alone — rather than recommending it universally. - **The agents-as-contributors argument is structurally sound.** Legible conventions and executable acceptance criteria genuinely do serve unfamiliar humans and agents alike; the conclusion follows from the premise. ## Fallacies 1. **Unsourced load-bearing anecdote.** The Amazon story carries the talk's central warning and is presented with specific details (internal name, month of withdrawal) and no citation. **Fix:** cite it; it is checkable. 2. **Vendor-reported case data.** All case-study figures come from a client of the presenting consultancy, with no baseline methodology, control, or time period. The intervention's seller is reporting the intervention's results. 3. **Selection effect in the case study.** One client is presented. Engagements that produced weak results would not appear in this talk, and no denominator is offered. 4. **Correlation presented as causation.** The textile improvements are attributed to InnerSource adoption, but digitising a previously physical, siloed design process would plausibly produce large gains regardless of governance model. The intervention bundles several changes. 5. **Motivated conclusion.** "InnerSource is the foundation for AI-native organizations" is the conclusion most favourable to a company selling both InnerSource consulting *and* an AI agent governance platform. The argument is reasonable; the alignment should be weighted. 6. **Assertion without evidence on agent readiness.** That InnerSource-mature organizations onboard agents faster is plausible and entirely untested here. ## Tone of voice Earnest, organized, and noticeably effortful — the speaker is presenting complex material in a second language and the strain shows in the disfluency without obscuring the structure. The register is practitioner-to-practitioner rather than salesy; the commercial interest is stated plainly at both ends and does not colour the middle. The most appealing feature is the absence of evangelism. InnerSource is given a *scope condition* rather than being recommended universally, teams using bad metrics are excused rather than scolded, and the speaker admits getting build-versus-buy wrong himself. For a founder presenting to prospective customers, that restraint is worth noting. # 🛡️ DARVO Analysis **Not applicable to this source.** A practitioner talk with no accusation, defence, or contested claim. - **Deny.** Absent. The speaker acknowledges the measurement difficulty rather than minimising it, and concedes his own past errors. - **Attack.** Absent. Amazon is described as having made and then *corrected* a mistake — the framing is explicitly non-derisive, and their pivot is presented approvingly. DORA is characterised as insufficient for a purpose rather than as bad. - **Reverse Victim & Offender.** Absent. **Verdict:** No DARVO. Recorded per template. # 🎯 Conclusion **Bottom line.** Two things survive the vendor framing. **The Amazon token leaderboard** is a current, concrete Goodhart instance — engineers ranked by tokens burned started burning tokens for the ranking, costs rose, the metric was withdrawn — and the correct generalisation is that the unit keeps changing while the error does not: lines of code, then PRs, now tokens. And **"DORA tells you how fast code moves, not how knowledge flows"** precisely names what the standard delivery framework cannot see, which is exactly the thing InnerSource claims to change. The most immediately usable item is the counterfactual: *if we didn't share this, how much time would we lose rebuilding it?* The agents-as-contributors argument is the most forward-looking: the conventions that let an unfamiliar human team contribute — legible practice, explicit done-criteria, work pickable without a handover meeting — are the same ones an agent needs. InnerSource maturity is agent-readiness by another name. **What to hold loosely.** The Amazon story is uncited despite carrying the talk. Every case-study number is vendor-reported, unaudited, and bundles digitisation with governance change. The AI-native conclusion is the one most favourable to a company selling both halves of it. And names throughout this capture are unreliable. **Next steps.** Verify the Amazon leaderboard claim at source. Try the rebuild-cost counterfactual on one shared component. If agents are on your roadmap, audit whether a new *human* team could contribute to your repo without a meeting — that is the same test. # 🧱 Refactor Appendix ## Literature Split Pad ### DORA measures pipeline speed, not knowledge distribution > [!quote] > it tells you how fast um code moves not how knowledge flows. *Where:* ~00:02:30. *Why:* precisely identifies what the industry-standard framework structurally cannot observe — and it is the exact dimension InnerSource claims to change, which is why ROI proofs fail. ### The metric unit keeps changing; the error does not > [!quote] > Amazon built a an internal a leaderboard called u hero rank ranking developers by how many AI tokens they burned. Within weeks engineers were token maxing running pointless tasks just to climb the rankings. *Where:* ~00:05:31. *Why:* a fresh Goodhart instance from a sophisticated organization, with the correct generalisation attached — lines of code, PRs, tokens, all activity proxies. ### Activity metrics dominate because they are collectable, not because teams are careless > [!quote] > Most teams measure the wrong things, not because they&#39;re lazy, but because the obvious metrics aren&#39;t easy to collect. *Where:* ~00:04:31. *Why:* correct and actionable diagnosis — the fix is investment in measurement, not exhortation. ### Agents are contributors, so contributor-readiness is agent-readiness > [!quote] > the key insight here is that agents are contributors. So same inner source patterns as humans um uh low on boarding costs, clear ownership, executable uh acceptance criteria. *Where:* ~00:10:32. *Why:* the most forward-looking claim — legible conventions and explicit acceptance criteria serve unfamiliar humans and agents identically. ### The rebuild counterfactual makes an unmeasurable benefit estimable > [!quote] > if we didn&#39;t share this, how much time would we lose rebuilding it? *Where:* ~00:13:01. *Why:* the single most practical item in the talk; converts avoided duplication into a number finance will engage with. ## Atomic Split Pad #### A metric measuring resource consumption invites consuming resources When the tracked quantity is an input rather than an outcome, the cheapest way to score well is to spend more. <!-- atomicity: 1 idea ✓ --> #### Delivery-throughput metrics cannot observe knowledge distribution A framework measuring how fast work moves through a pipeline is structurally blind to who else could now do that work. <!-- atomicity: 1 idea ✓ --> #### Ease of collection determines which metrics dominate, independent of their usefulness Organizations measure what instrumentation makes cheap, so improving measurement requires investment rather than exhortation. <!-- atomicity: 1 idea ✓ --> #### The conditions that let an unfamiliar contributor start are the same ones an agent requires Legible conventions, explicit acceptance criteria, and work that needs no handover meeting are contributor-agnostic. <!-- atomicity: 1 idea ✓ --> ## Fleeting Queue - #xref **The token-leaderboard atom is a direct sibling to the eval-design material from [[Give Your Agents a Performance Review]]** — one warns that benchmarks must compare against a baseline, this warns that the wrong benchmark actively produces waste. Together they make a strong Permanent on measurement design in agent contexts. - #xref "Ease of collection determines which metrics dominate" connects to the proxy/credentialing atom from the Grandin/Partlow source — both are about a measurable stand-in displacing the thing it stood for. - #xref Agents-as-contributors pairs with the CNCF contribution-pathways material — both concern lowering the cost of a first contribution, for different kinds of contributor. - #articleseed "Your AI metric is a Goodhart trap" — built on the Amazon leaderboard. - #aphorisms "It tells you how fast code moves, not how knowledge flows." - Vocabulary: **InnerSource**, **DORA**, **bus factor**, **token maxing**, **normalized deployment**, **activity vs outcome metrics**. - Lingering question: can the Amazon leaderboard story be confirmed? It carries the talk's central lesson and is entirely uncited. - ⚠️ Names in this capture are unreliable — speaker, company, and client are all likely mis-transcribed. Verify before citing anyone. ## Things 3 - Verify the Amazon "hero rank" leaderboard story and its withdrawal - Try the rebuild-cost counterfactual on one shared component - Look up InnerSource Commons patterns library <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/Beyond Pull Requests Quantifying the ROI of InnerSource.md|Beyond Pull Requests: Quantifying the ROI of InnerSource by YouTube]] ## Highlights - DORA isn't um enough either. DORA stands for the DevOps research and assessment. So it's uh the uh four metrics uh most engineering organization use to measure delivery performance, deployment frequency, lead time, change fate and time to restore. Uh so it tells you how fast um code moves not how knowledge flows. Then so we need um different framework ([View Highlight](https://read.readwise.io/read/01kyx6yscfajryyf5erh1ck94s)) - Note: The talk's cleanest formulation: DORA measures pipeline throughput and is structurally blind to who else could now do the work — which is exactly what InnerSource claims to change. Explains why its ROI proofs fail against the standard framework rather than dismissing that framework. - Tags: metrics, dora, atom/literature/h1 - burned. Within weeks engineers were token maxing running pointless tasks just to climb the rankings. So compute cost spike uh and then Amazon uh killed the uh leaderboard in May and pivoted to what they call normal normalized deployment measuring useful shipped code not raw activity. So same mistakes we're talking u about here just with a different unit. So counting to token ([View Highlight](https://read.readwise.io/read/01kyx6ybzb1bk1fax4mtpcj5e4)) - Note: A fresh Goodhart instance in AI-era metrics: rank engineers by tokens burned and they burn tokens. The generalisation is the valuable part — the unit keeps changing (lines of code, PRs, now tokens) while the error stays identical. A metric measuring an INPUT invites spending. CAUTION: uncited, and it carries the talk. - Tags: goodharts-law, atom/literature/h1, metrics - contributors how to work. Specs define what done looks like. Uh structured issues mean any team can pick up work without a handover meeting. Uh tests get committed before code. So the agent has a contract to implement against. um the the the you know partners validates in an isolated sandbox. So human all the final gate. Um so the key insight here is that agents are contributors. So same ([View Highlight](https://read.readwise.io/read/01kyx6z60h5rc114ctgpf3nhk9)) - Note: The most forward-looking claim: the conditions that let an unfamiliar human team contribute — legible conventions, explicit done-criteria, work pickable without a handover meeting, tests as a contract — are identical to what an agent needs. Reframes InnerSource maturity as agent-readiness. Useful test: could a new human team contribute to your repo without a meeting? - Tags: contributor-onboarding, ai-agents, atom/literature/h1