Designing for AI Agents: McP—Jessica Kerr  ## Metadata - Author: [[O'Reilly]] - Full Title: Designing for AI Agents: McP—Jessica Kerr - Category: #articles - Document Note: --- tags: [MCP, AI Agents, Product Design] author: [Jessica Kerr] source: [YouTube] reference: [https://www.youtube.com/watch?v=yop6v0KxRwM] date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} --- # 🏷️ Title: Designing for AI Agents: MCP—Jessica Kerr ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - Video: Designing for AI Agents: MCP—Jessica Kerr. https://www.youtube.com/watch?v=yop6v0KxRwM - Honeycomb observability platform. https://www.honeycomb.io/ - Anthropic Model Context Protocol overview. https://modelcontextprotocol.io/ *(If inaccessible, use the YouTube context above.)* - Playwright testing framework. https://playwright.dev/ - Figma design platform. https://www.figma.com/ ## 📝 Summary - Jessica Kerr explains MCP’s role in AI agent workflows. [Video](https://www.youtube.com/watch?v=yop6v0KxRwM) - MCP connects agents to live context securely and flexibly. - Honeycomb uses MCP to bring production data to daily tools. - Stakeholders include designers, developers, sales, and data teams. - Risks include token bloat, weak error design, and tool sprawl. - Bottom line: Treat MCP as a third interface with living documentation. ## 📖 Main Content - MCP enables agents to fetch context from services and tools. - An MCP server exposes prompts and tools to the agent client. - The agent mediates auth, tool calls, and model interactions. [Video](https://www.youtube.com/watch?v=yop6v0KxRwM) - Fewer, clearer tools reduce tokens and improve reliability. - Opinionated responses guide models toward useful next steps. - Honeycomb teams saw better feedback loops across code and ops. [Honeycomb](https://www.honeycomb.io/) - Formats like ASCII tables can beat JSON for brevity and clarity. - Uncertainties remain around cross-model consistency and security policies. - Good evaluations track task success, time, calls, and token spend. ### 🗒️ Notes #### ⏱️ [00:06](https://www.youtube.com/watch?v=yop6v0KxRwM&t=6) Design excitement - Amanda Bernard values AI’s design impact. - Honeycomb needs clear UI for complex telemetry. #### ⏱️ [02:19](https://www.youtube.com/watch?v=yop6v0KxRwM&t=139) What MCP connects - MCP links agents, models, users, and tools. - Calendar, tasks, and Figma can integrate. #### ⏱️ [04:27](https://www.youtube.com/watch?v=yop6v0KxRwM&t=267) Not a classic API - LLMs accept changing text outputs. - MCP allows flexible, evolving responses. #### ⏱️ [06:41](https://www.youtube.com/watch?v=yop6v0KxRwM&t=401) Living documentation - Client authenticates and requests tools. - Server lists tools with instructions. #### ⏱️ [08:40](https://www.youtube.com/watch?v=yop6v0KxRwM&t=520) Minimize tools - Too many tools waste tokens. - Disable rarely used endpoints. #### ⏱️ [10:43](https://www.youtube.com/watch?v=yop6v0KxRwM&t=643) Opinionated outputs - Provide helpful guidance in responses. - Use compact formats like ASCII. #### ⏱️ [12:42](https://www.youtube.com/watch?v=yop6v0KxRwM&t=762) Feedback loops - Improve error messages with expertise. - Let production signals refine design. #### ⏱️ [14:37](https://www.youtube.com/watch?v=yop6v0KxRwM&t=877) Evaluation practice - Score success, latency, calls, and tokens. - Run nightly benchmarks across agents and models. ## 📚 References - [Designing for AI Agents: MCP—Jessica Kerr](https://www.youtube.com/watch?v=yop6v0KxRwM) - [Model Context Protocol](https://modelcontextprotocol.io/) - [Honeycomb](https://www.honeycomb.io/) - [Playwright](https://playwright.dev/) - [Figma](https://www.figma.com/) ## 🗣️ Paraphrase - MCP gives agents secure access to up-to-date context from many systems. - Teams should trim tool lists, craft guidance, and measure outcomes continuously. ## 📰 Newsletter - MCP is reshaping agent UX by centering real context, not screens. Watch Jessica Kerr’s talk, then evaluate your agent tool lists. Start small, write opinionated responses, and measure token costs. Watch: https://www.youtube.com/watch?v=yop6v0KxRwM • Docs: https://modelcontextprotocol.io/ • Honeycomb: https://www.honeycomb.io/ ## ✅ Action Items - DevRel - Record a 6-minute MCP primer for OSS maintainers. - Demo “fewer tools” vs “API dump” with metrics. - Developer advocacy: plans, talks, demos, and measurable outcomes. - Publish MCP design checklists with token budgets and examples. - Education: workshops, curricula, office hours, mentorship programs. - Create a lab: secure OAuth flow for an MCP server. - Developer experience: docs, SDKs, examples, templates, quickstarts. - Ship a minimal MCP starter with tests and tracing hooks. - Developer marketing: campaigns, content, funnels, and attribution. - Run a “Context over UI” campaign with case studies. - Community management: moderation, code of conduct ops, surveys. - Host a roundtable on MCP evaluation metrics. - Internal/external/cross-project: align goals, SLAs, and feedback loops. - Define SLAs for MCP error quality and latency. - OSPO - Policy & Governance: contribution policies, decision rights, charters. - Add MCP service contribution and review steps. - License Compliance: policies, scanning, review, and training cadence. - Audit MCP servers for third-party license obligations. - Security Management: vuln intake, embargo policy, signing, attestations. - Require signed server builds and rotated credentials. - Software Inventory: SBOM strategy, ownership, freshness SLOs. - Track MCP servers and dependencies in SBOMs. - Developer Support: training, tooling, legal guardrails, consultation hours. - Offer quarterly MCP security clinics. - Community Engagement: upstream funding, MoUs, operator agreements. - Sponsor reference MCP servers for common tools. - Strategic Alignment: KPIs, OKRs, executive reporting, initiative roadmaps. - Report agent success rates and token costs quarterly. - Community - Maintainers: roadmap clinics, governance templates, release hygiene. - Add MCP error-message standards to templates. - Contributors: onboarding paths, “first issue” rotations, pairing sessions. - Label “good first MCP server” issues. - Sponsors: transparent budgets, milestones, deliverables, impact reports. - Fund evaluation harnesses and datasets. - Ecosystem: cross-project incident drills and coordinated disclosures. - Practice MCP credential leak scenarios. - WordPress - Core: priorities, issue triage, performance budgets, release criteria. - Explore MCP-like adapters for .org infrastructure. - Core Performance: profiling plans, metrics dashboards, perf bounties. - Track agent-added script costs in wp-admin. - Core AI: responsible AI guardrails, evaluation sets, pilot projects. - Pilot agents for triage with strict privacy. - Core Program: coordination rituals, cross-team dependencies, retrospectives. - Share evaluation dashboards with teams. - Mobile: app backlog, telemetry review, offline UX improvements. - Evaluate agent tasks on devices with spotty networks. - Playground: browser labs, reproducible bugs, education modules. - Build an MCP sandbox for tutorials. - Community: meetup kits, WordCamp playbooks, speaker mentorship. - Provide an “Agents and MCP” talk kit. - Training: lesson plans, learning paths, certification pilots. - Create MCP server certification basics. - Marketing: campaigns, narratives, social editorial calendars. - Highlight accessible, low-token agent designs. - Support: response SLAs, knowledge base gaps, escalation paths. - Document common agent failure modes and - Summary: Designers may be hesitant to use AI (for good reason). But as Jessica Kerr, Honeycomb’s engineering manager of developer relations, highlights in this talk from AI Codecon, AI tools and protocols like MCP can extend human-created design in new ways, adding complementary value. However, “to use [AI] well,” Jessica points out, “we have to innovate in how we do design.” Watch her talk for examples, use cases, and insights gleaned from Honeycomb’s MCP MVP. Follow O'Reilly on: LinkedIn: https://www.linkedin.com/company/oreilly/ Facebook: http://facebook.com/OReilly Instagram: https://www.instagram.com/oreillymedia BlueSky: https://bsky.app/profile/oreilly.bsky.social - URL: https://www.youtube.com/watch?v=yop6v0KxRwM This is Amanda Bernard. She is the headof product design at Honeycomb. And shesays that AI is the most excitinginnovation in design since she startedin this career many years ago. Andthat's saying something because as aproduct designer, she has reasons to behesitant. At Honeycomb, we care a lotabout our product UI. Honeycomb is anobservability platform, and so we take ina bunch of data, and then we have topresent that data in ways that arecomprehensible so people can figure outwhat's happening in their software.That's really hard, which is why we hiregreat designers like Amanda. But AI andin particular, MCP ignores our preciousproduct UI. And yet, Amanda is excited about MCPbecause it brings that sweet productiondata right to where people are. Andthat's why MCP is an MVP.I'm illustrating this with ourinternal experiences of implementing anMCP very recently so that you canlearn what we've learned. For instance,internally, our salespeople could loginto Honeycomb to find out what peopleare doing in the product, but they don't.However, they will type in Claude,Desktop. Hey, I’m working on thisaccount. Can you tell me who from thisaccount logged into our product and whatthey did there? And with Honeycomb's MCP, Claude can tellthem, or in the IDE, developers canfinally correlate production data withcode, and that closes a feedback loopfrom develop to deploy tooperate, and then production shouldinfluence development. That’spart of our mission at Honeycomb. Soit’s really exciting that MCP can helpwith that. Okay, so it was important, but what isit?MCP stands for Model Context Protocoland, model, actually, it’s for agents whichalso talk to a model. Um, but MCPs letthe agent supply context to the model,and that’s context in the real presentworld. And then the protocol isimportant because, like Tim said, any AIagent can use these MCP servers, and anyservice, and any person can implementthem. So it connects a lot of dots. Itlets the AI pull data sources together,which is fun. Uh, soan MCP doesn’t connect directly to aThe model connects to an AI application,which is probably an agent, and then thatAI application mediates between theuser, the model, and any MCP servers andother tools. For example, I can hook up Cloud Desktop toHoneycomb, and a sauna, and some Googleintegrations. All of these tools giveit the ability to rifle through mytasks and even do things like makeappointments.So, it gives the model new powersin the real world, more useful than justchat. I can ask Claude, in this case, "Hey,what should I work on next? CheckGoogle Calendar, check my tasks, andlook for any production emergencies."It can do that with all these MCPsor in my IDE. It could be Cloud Codedoing this, or Cloud Code hookingup to Honeycomb and various other places.But Cloud Code has access to my filesystem, so I can ask it to run somemanual tests in the browser, and it canuse the Playwright MCP for that.Here we go. Um,I can ask it which endpoint is slowestin production, take me to that code, andit can do both of these things. Them together.Yeah. Another one we really like is theFigma MCP because it takes our beautifulproduct designs that Amanda's teamcreates and makes them available to theagent at development time. Uh, so itincludes things like the variables thateach different color is named and thenumber of pixels between components.This beats this not out of pasting in ascreenshot and hoping the agent caninterpret it. Uh, highly recommend. Okay, so when you look at it like this,you might say, okay, so MCP is just asoftware API. Look at those arrowsconnecting two pieces of software.It's different. It is different becausewith a software API we have verydeterministic softwareand it expects like classicalsoftware that you can predict what it'sgoing to do by reading the code. Itexpects something particular to comeback from that API and particularparameters to send, and to change that isa production that goes throughdocumentation and a bunch of emails anddeprecation and human beings to get codeupdated. That's a big pain. Whereas withMCP, I mean, yeah, there's some JSONWrapper a little bit, but most of whatthe MCP sends is text that's going to beinterpreted by an LLM. And if that textlooks completely different than what itlooked like yesterday, that LLM does notcare because every conversation is new. That gives us an amazing amount offlexibility and it's very different. Soyou might look at this and say, "Oh,it's a text-based UI."Not the same thing. Because if you havelike a CLI or something that's atext-based UI for humans, those humansused it yesterday and they wanted it to bethe same. They also wanted to do exactlywhat they told it and nothing else. Now you can look at this as a UI if youthink of the models themselves as theusers. So I need my MCP to returnresponses that ChatGPT-4 and 5 canunderstand that works with Claude Haikuand Claude Stonnet and Olama. And so I can test that, and also thesedozen users that I have for thisinterface are thrilled to sit down andhelp me test it and give me their mostlytwo complimentary opinions all day long.So we have unprecedented ability tochange this interface and to test itand that's because the MCP is its ownLiving documentation, and I'll showyou what I mean by that. Let's getinto a very high-level overview of theprotocol. Okay, so when we connect anMCP server to an AI application, theagent spins up a littleMCP client module or process to connectto the MCP server. Now this client isdeterministic code. It can be tested andit has the possibility of beingsecure, which is useful when we doauthorization. When we do the OAuthdance, it pops up in my browser, and I can loginto Honeycomb and authorize the agentto act on my behalf, and then thosecredentials are secure. They don't go tothe LLM. Next, the client says, "Hey,what you got for me?" And the MCP serversays, "Oh, I've got some prompts. I gotsome tools." There are a couple of otherthings, but tools are what matter. Toolsare the fingers. [clears throat]We also give it some instructionson how to use those tools, and theseinstructions are going to be interpretedby the LLM. This is one piece ofliving documentation. We get to say,hey, when you use this MCP, we want toCall get workspace context and then getsome data sets, get some columns, andthen run some queries. After it gets thoseinstructions, it’s like, "Oh, hey, yousaid you had some tools for me. What doyou have?" And the server supplies theentire list and complete definitions ofall the tools it offers, which can betoo many. Looking at you, Asana. Do not just wrap your software API andmake each REST endpoint into a tool. No,just every one of these is sending atool definition in the context of everysingle LLM request. That’s justrude. There are a lot of tokens involvedhere. So, hot tip, when an MCP doesthis, when they give you just a zilliontrillion tools, turn most of them off. I’m never going to ask it about goals ina sauna, whatever those are. But I doneed it to have tasks. The less you cansend to the LLM with every request, thesmarter it’s going to be and the moreable to focus on what the user wants. So for instance, here’s a tooldefinition. It’s got a description andit’s got parameters and there’s a wholebunch of curly braces and quotes andTherefore, tokens in this data are fewer. Tools are better; just make itsupport what the user needs.All right. So the application gathersthe prompts, which include the tiny bitof text the user sent, a bunch of systemprompts, all those instructions, and thenall those tool call definitions andships them off to the model.The model responds with, "Well, what comes next is clearly some tool calls."Uh, it gives a list of tool calls back.The agent intermediates there and passesour tool calls to our MCP,and then we supply some sort of textand then that gets back to the LLM alongwith the complete conversation historyand everything else. All right. So what does that look like?Uh, for instance, the LLM might say, "Run aquery. I want a count and a heat map," andwe respond with Askiart. [laughter]This is [snorts] copied directlyout of one of our responses.We tried JSON; it used a lot of tokens and wasn’tsuper communicative.I personally agree, it’s not super communicative.CSV is a little better, but actually, somehow the modelcan interpret and work with Askiart, andit’s more efficient. Send now in some cases where it works.Um, I love that. Uh, another example, ifthe LLM asks for a trace, we respondwithsome of what it asked for. It asked forall the spans in this distributed trace.Give me everything that happened in thisrequest. Uh, and we respond with aninterpretation of that and some of thedata. Uh, so we've got a little bit ofJSON and structured data here, but thenthe meat of it is in this text, and wetell it, "Hey, I know a large trace whenI see one. This is a large trace. Whatyou probably want to do is zoom in onthe slowest span. Here's the tool callfor that. And then below that is a CSVblock with the sum of the spans. So we'rebeing opinionated in what we send back. We're not just following instructions asa human would expect from a CLI.That's what I mean by MCP is its ownliving documentation. In ourinstructions and in our responses, we'reguiding the model with what it's likelyto want to do and directions it can go.Then the model can interpret thatwith respect to what the user said.And you can give it guidance in a waythat a human would find rude. You can beBeing bossy with these LLMs is annoying. Yeah, don’t do that to people.It’s annoying, but it’s very communicative and effective.So, MCP is really a third interface.It is not a human UI, and it is not a software API.It’s something else. Agents are something else, and that makes it fun. Some of the design principles we’ve started to learn are about using fewer tools and conserving tokens to make it smarter and giving it these instructions.There’s kind of a taste to actually telling it what to do; we’re not just supplying data.We’re supplying expertise in our responses from the MCP, and because we get this feedback loop, we can change it frequently because every conversation is new.We can learn these design principles, and please let’s share them with each other as fast as we can.The feedback loops are everything here. For instance, from production, we can see what the LLM is doing.Here’s an error that we had a lot for a while.This is a frustrating error; it basically says you’re wrong.It tells you to stop being wrong, but that's not useful.The model doesn’t know what is right or wrong. Operators are valid. We do, and we should tell it. So, here's abetter one that we move to. Um, hey, youused the same column name for these twoplaces. Here's why that doesn’t makesense. Here's what that really means.Here's what to do instead. Yes,expertise. Supply it, including in theerror messages. Or here's another one. I like this one.For a long time, it kept saying that weshould count some field, but we werelike, "No, no, no. It’s just count. Wedon’t count fields. We just count." Andit kept doing it. Eventually, wegot the message and somebody looked intoit, and our data store actually supportscounting only the rows with a value inthis field. We just never exposed it inthe UI. Well, now we have an interfacethat we can just change. So, we did. Wemade this work and then the errorstopped on July 28th, and a few weekslater we're like, hey, let’s give thisto people, too. And we went through thechange log and blah blah blah process toput it in our human UI. Uh, so the MCP has given us feedback that improves ourwhole product.Also, we notice that the errors in theTool calls are just part of theprocess. They’re not failures. Because ifwe want this tool call to work well, whatis work? Success is not at the singletool call level; it’s at the wholeconversation. We need to consider all ofthese pieces: the agents and the modeland whether we were able to satisfy thehuman’s request. So for that, we can’tjust test our MCP. We have to doevaluation where we pick an agent, we picka model, we hook it up to the honeycombMCP unknown data. We give it a promptthat it ought to be able to solve, andthen we evaluate whether itsucceeded. Um, when we evaluate, we give it apass or fail. Did it solve the problem? Weuse an LLM to judge that. Uh, how longdid it take, wall clock time? How manytool calls? And how much did we spend ontokens? And those are things that we canhope to improve over time. So this is atrace of our evaluations. Uh, we send itfive or six different prompts and trythings and then judge them with an LLM.We run this nightly. We can run it withdifferent agents and models if we want.Um, and of course we check, we keep aneye on production, looking for actual HTTPFailures in our tool calls, which wouldbe bad, and check out our errormessages. But this evaluation is superimportant. As Morgante points out, if ourevaluation was good enough, we wouldn'tneed to write the software. We couldhave the AI do it and respond to itsown feedback loop. We're not there yet.Not even close. Okay, so MCP is great. It's its owninterface. It brings that data right towhere people are. It puts the contextthat our service offers right wherepeople need it to combine with others.And we get to be bossy and that's fun.And those feedback loops are critical. Back to this quote, if AI is like themost innovative thing in design, then touse it, we have to innovate inhow we do design., Designing for AI Agents: McP—Jessica Kerr, O'Reilly, 01k6bq0vgczpfbzdwzew431s6t.