AI Productivity at Enterprise Scale  🔗 **Source:** [AI Productivity at Enterprise Scale](https://www.youtube.com/watch?v=ZgxsSEUPzgI) > [!note]+ Document Notes > --- > tags: [ai-skills-assessment, mental-models, developer-productivity, pr-throughput, ai-adoption-vs-value, engineering-leadership, learning-intervention, dx-metrics, leaddev, skiller-whale, vendor-talk, uk-enterprise, correlation-not-causation, ai-training-roi] > author: LeadDev > source: youtube.com > reference: https://www.youtube.com/watch?v=ZgxsSEUPzgI > date created: 2026-09-03 07:20 > --- > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the speaker. > > Do not quote or attribute any other line to the named author. > > [!warning] Four things to know before trusting this note > > **1. Vendor talk in a sponsored slot.** The speaker co-founded the training company whose results he presents, says his co-founder "owns the budget for conference sponsorship," and closes with "come talk to us at the stand." Every number is the vendor's own assessment data or a customer's self-reported estimate. He states the correlation/causation caveat himself, twice. > > **2. Speaker and company names are ASR guesses.** Auto-captions render the speaker as "Howell" and the company as "Skillow" / "skill well." The likeliest reading is Hywel Carver of Skiller Whale (UK tech-team training), a co-founder named "Haley" in the captions. Not verified against the LeadDev programme; treat the identification as inferred. > > **3. Thin source.** 10 min 40 s, 1,779 words, 22 caption paragraphs, zero outbound links. Slides are referenced ("yellow lines here are before, blue lines here are after") but not captured; the chart claims below are as spoken. > > **4. The talk withholds one answer on purpose.** The XML-plus-max-tokens thought experiment ends with "Come see me on the stand if you want to know." The note does not supply the answer as if the speaker had. > > Published to the LeadDev channel 2026-09-02; recorded at LDX3 (LeadDev's London event). Channel name used as author per the video convention. > # 🏷️ Title: AI productivity at enterprise scale > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ### 🔗 References > - Source: https://www.youtube.com/watch?v=ZgxsSEUPzgI (LeadDev channel, LDX3 talk) > - Named in the talk, not linked: DX (developer-experience analytics vendor; supplied the PR-throughput comparison and the time-savings survey), a DX report on AI adoption vs. throughput, "two of the big four high street banks in the UK," "one of the top three telecoms in the UK," "two of the top 10 retailers in the UK." > - Zero `<a href>` links in the captured transcript. Nothing in this talk is cited in a way a reader can follow. > ## 📝 Summary > **What this is.** A ten-minute vendor talk from a UK tech-training company presenting, for the first time in eight years, trends from its own AI skills assessments of software engineers across UK and European companies. The pitch is a single sentence the speaker returns to three times: mental models, not tips and tricks, are what turn AI adoption into productivity. > **The evidence chain, in the order given.** > - *How skills are measured.* Confidence questions ("how confident are you using agentic LLM systems to write code") plus demonstration questions (explain how tokenization works). The speaker argues self-assessment alone is "kind of flawed" because "people don't normally know what they don't know." > - *Baseline distribution.* Perceived LLM understanding is bimodal: skeptics who understand little and expect little, and a second peak who understand deeply and expect large gains. The two correlate. Measured ability shows "a pretty broad range." > - *Where untrained developers are already strong.* AI tool usage and security. His explanation: tool usage "is really like AI doing function calls," and security was always close to developers' work. > - *Where they're weak.* Customizing AI systems: building RAG, knowing when to fine-tune, agentic workflows. "This is what we really want people to be doing most." > - *Perceived gaps differ from measured gaps.* People feel weak on understanding LLMs and coding effectiveness; measured weakness is in building systems. > - *Intervention.* Four to six learning sessions, then reassessment. Peak score moves from about 40% to about 70%. Confidence and expected productivity impact rise too. > - *Productivity.* With DX, two groups in one organization measured on PR throughput, both previously given a vendor's tool training. The studied group "roughly doubled" throughput while the rest of the org "slowly trailed downwards." > - *ROI.* At one of the UK's largest retailers, a DX survey after training: 44% estimate saving 1–3 hours a week, 31% estimate 3–6, 17% estimate 10 or more. DX extrapolates to £51.6 million; the vendor says its fee was under 0.1% of that. > - *The counter-case.* A DX report found a 65% increase in AI adoption produced an 8% increase in PR throughput. "Adoption is not the same thing as gaining value. Like buying and using a saw does not turn you into a carpenter." > **Bottom line.** The 65%-to-8% figure and the saw line are the transferable parts; they're the vendor's argument against its competitors' "tool-based training" and they hold regardless of who's selling. The £51.6M and the doubled throughput are self-reported or single-org comparisons the speaker himself declines to call causal. For Courtney the talk is a compact statement of the training-design position she already holds from the instructional-design side: teach the model of the system, not the incantation. > ## 📖 Main Content > **The structure of the argument.** Three moves. First, establish that measured skill and perceived skill diverge, which justifies assessment over self-report and, conveniently, justifies the vendor's product. Second, show that a short intervention moves measured skill (40% → 70%) and then moves an outcome metric (PR throughput, self-estimated hours). Third, contrast with the industry pattern (65% adoption → 8% throughput) to argue that adoption without mental models wastes money. > **The mental-model example is the best teaching moment in the talk.** The markdown tip ("ask Claude for markdown because it's token-efficient") is good advice that stops being advice the moment you run a local model, pay by seat rather than by token, or need a format that isn't markdown. The XML-plus-max-tokens question is the test: someone with a mental model of decoding and stop conditions can predict whether they'll get terse XML or truncated XML; someone with a tip list can't. He then withholds the answer to drive stand traffic, which is fair for a sponsored slot and a small loss for the recording. > **What the data can and can't carry.** > - The bimodal perception chart and the correlation between understanding and expected impact are descriptive and plausible; no sample size is given. > - The strengths/gaps pattern (tool usage and security strong, system-building weak) is the most interesting finding and the least argued. His post-hoc explanation for tool usage is reasonable; the security explanation ("something that should always have been close to the work of developers") is a hope, not a mechanism, given how often security is the gap in other surveys. > - The 40% → 70% jump is pre/post on the vendor's own assessment after the vendor's own course. Teaching to the test is the obvious confound and isn't addressed. > - The PR-throughput comparison is two groups in one company, both previously trained by an "AI vendor" on tools. Doubling versus slight decline is a striking split, but "roughly doubled" from what base, o Thanks, Mary. Hi folks. Uh, as you've heard, my name's Howell. It is really great to be back at LDX3. This is one of the highlights of my year is um coming back to this conference. Uh, as you've heard, I work at Skillow. We specialize in skills for tech teams and over the last couple of years that's meant a huge focus on mapping the existing AI skills of people who are in place as software engineers and measuring the impact of learning on productivity. And so for the first time in eight years, we're now publicly sharing the trends that we've seen in the skill assessments that we've done. And that's partly because this is a time of rapid change and because we hope that the insights we can share might help engineering leaders in deciding where to invest and what to do next. Um some caveats first. Everything I'm going to present is kind of trends data. Uh science is quite hard. So separating correlations from causations is tricky perhaps impossible given given the nature of what we do. And also, your mileage may vary. Your team might be different to these teams. I can't guarantee that everything I'm going to say will apply for you. We're all just doing our best out here. Um, the data I'm going to talk about has come from a broad range of companies. They're largely organizations across the UK and Europe, from startups through to unicorns to large enterprises as well. It includes two of the big four high street banks in the UK. Uh one of the top three telecoms in the UK and two of the top 10 retailers in the UK as well. I'm going to talk about some patterns in the data, share some insights and I will be crossing my fingers that it's useful to you. Um I should talk first about how we quantify the skills because that's that's the sort of baseline of how we find this data in the first place. There's a little bit of self self assessment involved in that. So we ask confidence based questions like uh how confident are you using agentic LLM systems to write code that kind of thing. But the core of it is people showing what they know uh demonstrating what they understand about how tokenization works for example. We want to test whether people know and can apply those ideas. And that's really important because people don't normally know what they don't know. That is a hard thing to do. And so self assessment on its own I think is kind of flawed. Um our philosophy when it comes to learning are that mental models are really the most important thing. The reason why is that mental models help you extrapolate to new situations to solve new unseen kinds of problems. Um, I want to give an example of this. There's a common bit of advice to Claude, which is where you ask it to format text in markdown because markdown is relatively token efficient. It uses fewer tokens than other forms of uh of formatting. Good advice. I'm not saying that's a bad idea, but I think it's less useful advice if you're running your LLM locally or maybe if you're not paying per token. Maybe your model of payment doesn't work for that. or maybe you just need a different format that doesn't map well to markdown. The tip and trick approach is kind of limited in how you can use it. Uh let's say you wanted to set a maximum token count in your uh prompt to an LLM and you ask for a response in XML. Do you know what would happen? Because if you have a good mental model for how the system works, the answer is kind of easy. Like maybe you would expect it to produce something very tur. It would write some very minimal response that satisfied that your request for XML but in very few tokens. Or maybe what you'd get is something truncated. You'd get the full response, full kind of beautiful XML, uh but it's truncated. It never actually finishes because of that max token count. If you have a good mental model for how the system works, you can apply it to this question. And the answer feels kind of obvious. Uh I'm not going to tell you the answer now. Come see me on the stand if you want to know. Um the first thing that we started with was asking developer experience type questions to probe people's perceived understanding of LLMs and how much they expected it to impact their own productivity. And what we saw from the results is a couple of peaks that represent both the skeptics who haven't used it much, people who are low in their understanding, but also don't expect to get much benefit from it. And we saw people who understood it deeply and expected big productivity gains as a result. And if you're wondering, yes, these things do correlate. Uh so higher understanding does go together with higher predicted productivity impact. But we also measured those abilities. We measured the level of people's understanding and the strength of their mental models. And what we saw was a pretty broad range of technical capabilities in teams. We also looked for some patterns in the data. Uh and this is this is maybe my my favorite thing. If you look at developers who haven't had formal learning around building these mental models for LLMs, there are still some areas where they tend to be strong and those are in AI tool usage and security. I found this very interesting and then I realized it should probably be intuitive because AI tool usage is really like AI doing function calls. It's the bit that should make sense to developers and security is something that should always have been close to the work of developers anyway that they are thinking about in a different way. If you look at common gaps, the most common areas of weakness for people who haven't been formally trained, uh we found that they were in customizing AI systems. So, building your own sort of LLM based bits of product, things like rag systems or knowing when to fine-tune LLMs and agentic workflows. Um, this isn't just consuming other people's agents. This is building your own systems. This feels like a shame because this stuff is the hotness. This is what we really want people to be doing most. And those are the the weaknesses that they they show. On the other hand, if you look at where people think they're weak and strong, um, you see very different gaps to what you measure in their abilities, the gaps that people feel are in their understanding of LLMs and their effectiveness at coding with them. People know that they don't know this stuff. So after the initial assessment, we then ran a learning intervention that looked like four or six learning sessions for different groups. And then we reassessed in a similar way to the initial assessment. And what we saw, yellow lines here are before, blue lines here are after, is that afterwards people perceive themselves as understanding better. They have more confidence in the productivity impact. We also measure a big increase in technical capabilities. So before the peak is around 40% score, afterwards it's around 70%. This is all nice. What we really want though is a shift in productivity. And so we looked into that with some help from DX. Uh we looked into two groups in the same organization that used PR throughput as their main measure of productivity. Both of these groups had had the tool-based training from an AI vendor. Um and if you compare the group we studied which is in blue here, they roughly doubled their PR throughput while the rest of the org slowly trailed downwards. Got slightly reduced PR throughput in the same time. Now, obviously this is not the whole story. As I said, we can never be confident of the difference between correlation and causation. Um, but hopefully food for thought. Anyway, at another larger organization, the team estimated how much their new skills in AI would save them in terms of time. Uh, and this is with one of the UK's largest retailers. After the training, they all took a DX survey about how much time AI was saving them, where 44% estimated saving 1 to three hours a week, 31% estimated 3 to 6 hours a week, and 17% estimated saving 10 or more hours a week. And the nice thing about DX is it extrapolates this out to the whole organization and comes up with an estimated savings measured as 51.6 million pounds. Um I should say this is considerably more than our costs which were less than.1% of that. What we do know is that adoption is not the same thing as gaining value. Like buying and using a saw does not turn you into a carpenter. And AI is very expensive. So if you're using it without skill, it's potentially just a waste of money without significant benefit. Um, DX put out this report where they found that companies that were adopting AI most rapidly saw a very minor change in PR throughput where 65% adoption increase in AI um led to 8% increase in throughput. And to me this all makes sense because I think knowing mental models makes us better as software developers. We use mental models all the time to inform what we do to make better decisions about how we build and the same is true when it comes to AI. We need mental models to get to better usage and to productivity gains. You can do that with us with skill well. We have a stand out there. Come chat to us. But however you do it, that is the missing key for people to be both confident and productive. Now I have to include this slide I'm afraid. Um, my co-founder Haley writes all these talks with me and she owns the budget for conference sponsorship. And so, while I know that you don't want to hear a sponsor pitch and I certainly don't want to give you one, maybe I can instead just say just come talk to us at the stand out there if you're interested. If you're interested in learning more generally, come talk to us about that. Uh, and I hope you all enjoy the rest of the conference. Thanks very much., AI Productivity at Enterprise Scale, LeadDev, 01m1kmwxq6s6c71p96h9fs5mkg.