Creating Excellent Developer Docs and Content ![rw-book-cover](https://i.ytimg.com/vi/7KTGtGQM2UA/maxresdefault.jpg) ## Metadata - Author: [[Major League Hacking]] - Full Title: Creating Excellent Developer Docs and Content - Category: #articles - Document Note: --- tags: [DeveloperDocumentation, DocsBestPractices, AIInDocs] - DeveloperDocumentation - DocsBestPractices - AIInDocs Author: [John Gried, Panelists: Jen Lever, Lorna Mitchell, Andrew Jen] Source: [State of Developer Education, Major League Hacking, YouTube Video] Reference: [Link](https://www.youtube.com/watch?v=7KTGtGQM2UA) date created: 2024-09-14 19:05 --- # Title: Creating Excellent Developer Docs and Content ## Table of Contents: 1. [📝 Summary](#summary) 2. [📖 Main Content](#main-content) - [🗒️ Notes](#notes) 3. [📖 References](#references) 4. [🗣️ Paraphrase](#paraphrase) 5. [📰 Newsletter](#newsletter) 6. [✅ Action Items](#action-items) 7. [🧪 Hypotheses](#hypotheses) - [📊 Methodology](#methodology) - [❓ Questions](#questions) - [🔮 Future Research](#future-research) - [💡 Implications](#implications) 8. [🧠 Learning Styles Assessment](#learning-styles-assessment) 9. [📊 Accessibility Assessment](#accessibility-assessment) 10. [🌍 DEIB Assessment](#deib-assessment) 11. [✅ Logical Validity](#logical-validity) 12. [🔍 SEO Evaluation](#seo-evaluation) 13. [🎯 Conclusion](#conclusion) ### References: - John Gried, Jen Lever, Lorna Mitchell, Andrew Jen, [State of Developer Education, Major League Hacking](https://www.youtube.com/watch?v=7KTGtGQM2UA) --- ## 📝 Summary: The panel discussion focuses on creating excellent developer documentation and content, exploring best practices, challenges, and the evolving role of AI. Experts Jen Lever, Lorna Mitchell, and Andrew Jen share insights on balancing interactivity and simplicity, addressing developer needs, and ensuring quality through tools like Docs-as-Code and API-first designs. The discussion also touches on how AI may impact documentation, but the consensus is that good, clear, and accessible writing remains critical. --- ## 📖 Main Content: The panel emphasizes the importance of developer documentation as a key component of a successful developer platform. While there are trends towards more interactive and AI-enhanced documentation, the foundation of good docs is still clarity, ease of use, and accessibility. Jen Lever highlights how Twilio focuses on use-case-based documentation over product-based, ensuring users get the information they need. Lorna Mitchell underscores the importance of documentation being accessible, arguing that too much interactivity can be a barrier for many users. The discussion shifts to AI, with the panelists sharing skepticism about the current utility of AI in documentation. While AI tools like ChatGPT are increasingly used by developers, the panelists believe that AI is not yet reliable enough to replace human-written content, especially in terms of providing context and precision. However, AI can be useful in automating repetitive tasks and enhancing internal processes for content creation. ### 🗒️ Notes: - **Developer Journey**: Documentation must meet users at different stages of their learning or building journey. - **Docs-as-Code**: Leveraging git and other tools for collaborative documentation helps maintain quality and consistency. - **Design-First Approach**: API-first and design-first documentation ensures better user experience and fewer frustrations. - **Interactivity vs Simplicity**: Not all documentation needs interactivity; simplicity often works better, especially for accessibility. - **AI in Documentation**: AI can help automate and assist with documentation but is not yet reliable for full content creation. --- ## 📖 References: - [State of Developer Education, Major League Hacking](https://www.youtube.com/watch?v=7KTGtGQM2UA) --- ## 🗣️ Paraphrase: 1. Jen Lever discusses the importance of having documentation that supports users at all stages of their interaction with a product. Whether they're just starting or deeply involved, the documentation must provide clear, relevant content that addresses their specific needs at each touchpoint. 2. Lorna Mitchell argues that while interactive documentation has its place, overly complex and interactive elements can be detrimental. She emphasizes the importance of maintaining simplicity, particularly for accessibility purposes, and suggests that written documentation remains the most universally useful format. --- ## 📰 Newsletter: 🎉 **Exciting Updates in Developer Documentation** In this edition, we delve into the latest best practices for creating **developer docs and content**. Experts from Twilio, Redley, and Fern share insights into balancing interactivity with simplicity, adopting **Docs-as-Code**, and using **API-first designs**. Plus, we discuss the evolving role of **AI**—useful for automation, but not yet a replacement for human-written content. Check out their key takeaways and apply them to your next doc project! 📚 Read more at: [State of Developer Education](https://www.youtube.com/watch?v=7KTGtGQM2UA). 💻 --- ## ✅ Action Items: - **Optimize Documentation Processes**: Focus on use-case-driven documentation, ensuring users can easily navigate and find what they need. - **Leverage Docs-as-Code**: Ensure version control, collaboration, and easy updates to documentation by integrating it into your codebase. - **Evaluate AI Tools**: Look into AI solutions that can automate internal processes but don't rely on them for end-user content creation yet. --- ## 🧪 Hypotheses: - **Hypothesis 1**: AI can assist in documentation creation by automating mundane tasks and enhancing search functions but will not replace human writers in generating context-rich content. - **Hypothesis 2**: API-first design leads to more accurate, user-friendly documentation by ensuring that the API's functionality is clear from the beginning. ### 📊 Methodology: - **Surveys**: Conduct surveys to gauge developer satisfaction with AI-enhanced search vs. traditional search methods in documentation. - **Usability Testing**: Perform A/B testing on API-first vs. code-generated documentation to measure developer understanding and satisfaction. ### ❓ Questions: - How can AI be better integrated into documentation without sacrificing clarity and context? - What are the common frustrations developers face with interactive documentation, and how can they be addressed? ### 🔮 Future Research: - Explore the long-term impact of API-first designs on developer success and satisfaction. - Investigate the use of AI for real-time code snippets and troubleshooting within documentation. ### 💡 Implications: - Improved integration of AI could revolutionize how documentation supports developers, reducing search time and improving relevance. However, maintaining human oversight is crucial to ensure the content remains accurate and contextually rich. --- ## 🧠 Learning Styles Assessment: 1. **Visual Learners (Spatial)**: - **Present**: The panel doesn't focus on visuals, but interactive documentation could enhance learning. - **Improve**: Add more flowcharts or diagrams to visualize API workflows. 2. **Aural Learners (Auditory-Musical)**: - **Present**: No specific auditory content discussed. - **Improve**: Incorporate video tutorials or podcasts for auditory learners. 3. **Verbal Learners (Linguistic)**: - **Present**: Written documentation is highly valued and considered the most versatile format. - **Improve**: Expand with storytelling and case studies to enhance engagement. 4. **Physical Learners (Kinesthetic)**: - **Present**: Little hands-on content mentioned. - **Improve**: Create interactive tasks that allow users to practice implementing code directly in the documentation. 5. **Logical Learners (Mathematical)**: - **Present**: Logical, step-by-step guides are mentioned as valuable for engineers. - **Improve**: Include more code examples and decision trees for problem-solving. 6. **Social Learners (Interpersonal)**: - **Present**: Not directly addressed. - **Improve**: Encourage community contributions and peer review in documentation practices. 7. **Solitary Learners (Intrapersonal)**: - **Present**: Written docs cater well to solitary learners. - **Improve**: Add reflection prompts or journaling sections for deeper personal engagement. --- ## 📊 Accessibility - Document Tags: [[devdocs]] [[devrel]] - Summary: The text discusses creating excellent developer documentation and content, emphasizing the importance of collaboration between technical writers, engineers, and developers. It highlights the significance of up-to-date and user-friendly documentation to enhance the developer experience and product understanding. The text also touches on the role of user research and continuous improvement in shaping the future of developer documentation. - URL: https://youtube.com/watch?v=7KTGtGQM2UA&si=lqF5rAJXtj6swbBc [Music] welcome to the state of developer education season 2 brought to you by Major League hacking we bring together panels of experts on developer education developer relations and developer advocacy to talk about the best practices in a variety of topics so that you can improve how your business is working with developer communities whether you're a deell leader an advocate or a community Builder you're in the right place to explore Innovative approaches to Learning and Development in technology I'm John gried co-founder of Major League hacking and I'll be guiding you through this episode's discussion prepare to connect engage and transform the way we think about developer education hello everyone and welcome back to the state of developer education season 2 uh our panel for April 2024 is about developer docs in content and we have a fantastic group of panelists here with experience building docs and content at a wide variety of companies I'm really excited to hear everything they have to say uh and share some tips and best practices for you to improve your own docs and content strategy and create an excellent experience for your developer Community uh let's start off with a quick intro from each of the panelists and then we will Dive Right into questions Jen do you want to kick us off sure um so I'm Jen lever and I lead the developer Education team at twio right now so I've been here for a couple years uh before that I led the documentation team at GitHub um where I started as a a writer on the team there and before that I've just basically been a technical writer my entire career so um all I've ever known is documentation um got into the developer space when I joined GitHub though and that's kind of where I've I found my calling I guess and I just have really enjoyed writing for the developer audience so that is um my backstory and yeah happy to be here today two companies with fantastic documentation so I think we have you to thank for that oh no the team not me uh Lorna you want to go next hi I'm Lorna I'm based in the UK I live uh in the north of England so like living in the UK not living in London I hope that's not too strange for anybody I lead developer experience at redley which includes uh documentation and although I've had a quite a long career as a software engineer I've written more docs than the average software engineer um I have a thousand posts on my blog I've written some books described myself as an engineer with a writing problem um so after a few years in deal that's given me the opportunity to sort of bring engineering and writing together and I've worked on the documentation projects uh I I mean I own them at red doley and also previously at Ivan um and I worked on the Vonage docs when I worked there as well so this is like my happy place of taking a technical thing and helping people be successful with it love it and uh last but not least Andrew hi everyone my name is Andrew Jen I am a software engineer based in Brooklyn uh working at Fern which is a uh early stage startup uh building both SD generation and docs generation tools um and our bread and butter is uh taking open API specs and turning them into beautiful documentation and the bar that we tried to hold uh is getting people uh spun up with stripl like documentation with as little effort as possible I've only been on in this role for five months um and my previous role was a for deployment engineer app pounder and so I'm the newest and the most novice uh in this room when it comes to docs um but I have a few thoughts so I'm very happy to hear to be here and to talk to everyone here fantastic I I love the example of strip like docks it's kind of a good emotional Touchstone for developers maybe so I'd love to start off the question uh with sort of a a broad overview here of your philosophy um obviously like you've all mentioned some organizations that are known for having exceptional documentation what are some of the like core guiding principles that each of you uh you know hold yourselves and your teams to when you're thinking about docs and content at your organizations I I think for me it's uh it's really about like meeting the users where they are in their Journey that's really kind of when I go back to these like mission statements that I've I've uh worked on at different companies it is always about like like finding the right um moment for the developer and making sure they have content um for whatever they're trying to do basically so that's the mission statement is always kind of kind of a line around something with that um but it is about like okay a developer comes and they say what what am I doing what can I do with this you want to provide content at that moment when they're like oh I'm interested I want to build something well they need a different kind of content in that next moment so it is really about like understanding all of those kind of touch points throughout the developer journey and just making sure that they have content um along the way so that they don't feel frustrated so they don't give up and and leave your products so that's kind of the the mission that we have is just like enabling people making sure they're supported during all of their work of course there's a lot more that goes into it but that's kind of the the overview of how we think about it yeah I would definitely Echo that I think thinking about who you're writing for and especially if it's within one documentation platform you've got multiple audiences that can be quite a challenge um I red doly do uh developer portals API documentation so we have everybody from like super you know we have 800 microservices to what's an API again um and so trying to find ways to not just create that documentation but then also how to signpost people to what they need right now do you know about this thing are you learning something oh here's an explanation of it do you already know how to do it and you came for the commands to copy and paste okay well so then we have something that's a bit more like step-based um and trying to help people to both know that that exists and and expect it from us but also find it in their moment um because a lot of that content can be overlapping so that's a challenge I think from um my point of view um having been in organizations where writing docs in the first place is a challenge um getting our Engineers to be able to generate any sort of documentation is better than having none at all so we prioritize creating Tools around autogenerating uh documentation that is up to dat and our customers customers which are uh doc readers want to be able to go to their doc sites and copy paste functional code uh that's like I guess the bare minimum of having good documentation is having accurate code siit so you can take and uh paste into vs code or some compiler and be able to run with that information and start exploring the API or start exploring the SDK product on your own as a developer and once you hit a point where things sto working that's when Vel opers get really frustrated and we've heard of customers who like ended deals oh maybe that's a little dramatic but we've heard of a couple scenarios of that where they can't trust the the product because they can't trust the documentation I mean i' I've certainly churn from using a particular API because I ran into a uh unresolvable problem in the docs right I think that's actually a common experience for a lot of developers one of the things you touched on though is like this idea of getting Engineers to you know like leave comments and write docs um and and I'd love to hear from our panelists like who owns what part of that right like are Engineers responsible for self-documenting you know in like an open API spec or something in a way that can be generated is it up to a content and like technical writing team to translate that into good docs like how do you actually approach the distribution of responsibilities for docs comments and then maybe like the the user facing part of that so I've had this experience in a in a bunch of places um particularly because I haven't worked very many places that really had dedicated writers at all um so you're doing it with hopefully enthusiastic bystanders sometimes less enthusiastic bystanders but you have like other people who are not um really inside the documentarian discipline neither am I really I always tell people I'm not a writer um and the tactics that I've taken for that is to bit like open source contribution ions always say thank you um this stuff isn't hard um and to try to make sure that there are templates and guidelines and really clear review guidelines and you know I've run office hours to bring in new contributors and stuff like that in the past um this is very much about company culture though if you if you have a company culture that is like customer fanatical everybody will try and you're very unlikely to have lots of problems with secret product features that you didn't need to build because no one can use them anyway whereas in a company which only values code that's much more difficult and it's something that I'm not sure I've already cracked that one yet I'd agree with all of that that you just said um it's it's a shared responsibility I'd say too because um it's definitely up to I'd say like developers and Engineers to really be able to provide that information but it's up to you if you have a dedicated technical writing team or in our case a developer Education team to be able to take that information and curate it um in in a way that makes sense and I feel like that's kind of where to me developer education is going in a lot of ways is is taking um information provided uh either like from developers or or in a different way being able to curate a different experience um for folks but um I'd also say to me uh the docs are kind of a responsibility of everyone at the company which maybe is like me like trying to pass off my responsibility to people but um part of this is like these teams are so small to Laura's point it's uh you know like for for instance right now in Tulio we only have five folks creating all the content um for most of our communication based products which is like not enough um so what are we going to do here and yeah we definitely rely on uh Engineers being able to provide a lot of information like all of our error code documentation um those folks are that's coming from Engineers um that's not coming from us but at the end of the day we need like help from product managers to create content we need um Engineers to jump in and create content we need folks from support we need marketing we need all these people to align on you know what we're telling customers um what the the usefulness of features or new products are all these things so I think that's going back a little bit to like what Andrew was talking about too just in terms of like platform which is we need tooling that people can use that everyone can use because having something like you know just writing something in a Google doc and Publishing it on a CMS is just not not the way to get people to contribute to the documentation so you know having things that feel comfortable for folks uh having a dox's code environment um you know really being able to contribute in interesting ways and saying like you know we have a culture of documentation like we mentioned stripe earlier in the the the conversation it's because they have a culture of documentation at the company that's why they're so successful so if you can cultivate that that culture and have the Tooling in place you will have a better product you will have developers um really love what you're doing in the documentation but if you don't um you're just going to be kind of working to get there and working against things yeah I love that perspective um one thing I'd be curious about Jen you know you mentioned like Doc as code and you know I know we've been talking a little bit about generative docs so I want to dig into that a little later but you know twilio is also known for having a lot of really fantastic like tutorials and quick starts and blog posts and you know I worked at twilio I mean at this point more than a decade ago and I remember like one of my first tasks on the job was like going through all of these old WordPress posts and updating them to a current version of the SDK which like I'm sure you have a better process now but I'm curious like how you think about content more holistically than just generating the docs right like like where do all of these other learning resources uh fit into that picture I'll tell you a little bit of a secret here amongst all of our friends on this podcast right now which is that we actually didn't have a better process up until um now basically so when I was saying like oh you know it's not great to write in a Google doc and put it in a CMS that was our process up until like this year we're actually just finishing up a doc's code migration to get us onto a new platform so that we can do these things we do use open uh API for like our our um API documentation but just in terms of like understanding in um across like all of our content um we're really lucky because we have a larger team so I mentioned my team is like fairly small I do own the the engineering team as well that um does our doc platform and our doc site but we we're part of a larger part of an organization called the developer Network at tulo which is kind of uh if you're in the devil space you you may have heard of it because it it just works really well and they've got a great history of um being able to really bring excitement and playfulness into developer relations but there's a lot of different teams within there that create different types of content so create blog posts create videos all these things so we all have kind of different responsibilities my team really is just the documentation at this point um together we're really trying to to understand like what are those moments um that we can capitalize on so if there's something interesting happening in the industry right now which is like right now it's it's all AI it's all anyone can talk about so it's like what can we do there you know there things that we can do in blog posts do we need to create a video do we need to um you know understand what our products can do around AI do we need to just talk about it more in general have an event around it so um together the developer Network all these different teams can kind of come together and say hey together we can tell a more compelling story with these different types of content and understand what people need from us so for instance like when you're writing a blog post it is more about that like point in time interesting thing that you're trying to capture whereas docs are more like Evergreen you're like trying to tell the complete story of how you use the product all of the use cases all those things and you're hoping that it stays relevant and current and accurate forever which is not always the case but but the blog posts really are those point in time things videos are a little bit like that as well um so it is just understanding I guess um what story you're trying to tell in that moment and then figuring out the best medium to tell it in the thing that will be most engaging to people that's awesome um so uh I'd love to dig in a little bit more into this idea of like creating a pipeline and generative tools for docs right so Jen you kind of alluded to the fact that you all are rolling out a platform Lorna and Andrew I know that you both work at companies that build these you know kinds of platforms it feels like an interesting challenge to me to design a meta doc tool right to like really look at it not from the perspective of how do I make twilio's docs the best but what are the best practices for docs period um you know what are the the design principles there right like what are the things that are important you know regardless of what platform or API might be using these platforms I can start so one big thing that Jen mentioned that resonates with me is this migration uh from cms's into docis code we've heard from uh many prospects and customers that there's always this tension between uh maintaining docs in a g repository versus maintaining it in a CMS like WordPress and oftentimes we hear um folks who use this TMS sometimes regret that position later because it made it easy to write docs but you know writing docs as a point in time uh snapshot of your product uh means that that information is going to be outdated within an hour um often times uh your engineering team is moving very quickly and uh sometimes like having to maintain a platform where uh Engineers have to log in to a portal and figure out how to maintain information and keep things up to date um it's like a whole uh other part of the job description that I think Engineers don't often uh consider as like one of the the top priorities but it can be a really uh big issue and it can be a and should be a top priority for uh companies that want to deliver um a high quality experience for your customers especially if you're API company so I think the other side flip side of the coin is if you migrate everything into a code first uh uh mentality such as um maintaining all your docks and a code repo uh you immediately um make it hard for PMS and Technical writers to help contribute uh because it is a pretty large not un insurmountable but pretty large learning curve for people who don't spend all their time in get um so there really needs to be a middle ground between the two to really create that collaborative atmosphere between engineers and Technical writers I'd like to draw on a couple of those threads because there's a bunch of really interesting conversation going here everybody's mentioned docs as code yes if you're not doing it that way what are you doing I think if you need to ship any volume of documentation with any accuracy at any kind of speed like why G is the best collaboration tool around why would you do that differently I've been giving quite a lot of talks in the documentarian space to try to unlock the author experience for them I think we underestimate the sheer repertoire of skills that the average Tech writer has a lot of them are much more technical than you would ever imagine and they do all kinds of things with access to the internet and docs if you leave them unsupervised they can handle the most Arcane ridiculous workarounds in these Enterprise CMS tools like git and GitHub is nothing in comparison to that and I really feel that as developers we have a responsibility not just to build the best tools for ourselves but to then play that development or authoring experience forward it should be easy to create you should have a choice of tools you know there are very good goys and Integrations for git also it's really not that hard although do recommend that you don't allow your engineers to teach git to your writers many good writer oriented git resources exist and and it's you know Jen and I both spoke about the importance of knowing your audience this has gone wrong for me a bunch of times and when it's gone right it's because we've given that audience the information they need and the and the use cases that they need to succeed and not everything an engineer thinks you need to know um because I disagree it's like a funny like tieback to the whole idea of having Engineers write docs in the first place is like Engineers I think I I've observed similar to you that like Engineers uh often have trouble distilling down information into the requisite pieces uh and that can result in like a fire hose that's difficult to digest yeah turns out technical communication is a skill um I'm I'm trying to like direct the hose with templates and things but even the most enthusiastic developer can be too much so I think the author creation is really important markdown also Tech writers are incredibly smart it's not that hard um there are lots of gooey tools there are lots of great web editors uh we are overstating the difficulty of this but we could support them more for for sure and I think that's a really big thing so that's like one whole thing I'm going to park it but maybe we need to untangle some of those threads again um in terms of how to make that easier I don't think you shut out technical stakeholders but you do need good tooling so you do need immediate previews you do need those need to be public you need to be able to just put a page in front of somebody and that's important the other topic that I wanted to pick up was um I work a lot with open API that's that's my background um I'm on the open API steering committee I live and breathe this stuff um and I find it very interesting for us to talk about that as generative something that you only make from code because alongside the doc's code movement I'm seeing very much the design first movement where you absolutely don't generate from code because then you produce an open API description whether that destination is for docs or for sdks or for something else which is only as good as the information that you can maintain p in your codebase all we we might be generating from an open API but I don't I think it's a false assumption that you always want to generate the open API sometimes you want to write the open API make sure get everyone on board make sure it's up to standard and then have it implemented and passed Downstream to the other tools so that's something that yes there are some docs maybe config settings or something that you can rip out of a code base but for the most part that's not always the best way interesting so so what you're saying just so I make sure I understand here is that like a best practice you're seeing is that people design the API sort of like product and user experience as its own thing and then ensure that the code actually accomplishes that rather than simply taking a huge code base and generating an API spec off of it yeah and it works really well for changes as well because the open API is a contract you can easily generate docs from it so even just here's the open API here are the docs let's get everyone round the table everyone comp part as a p in this from using the open API Source you can also I like to spin up a mock server because that's is how my brain works sometimes I just need to curl a thing to reason about it um everybody can do the thing that works for them and then we've got the contract everybody knows exactly what's happened what the change will look like how the error messages will be you don't you're not asking an engineer to make a sign of endpoint and just hoping that she does it right yeah you know it's it's we've got the details nailed we can run the linting get the consistency there and hopefully the back end does exactly what you expect you can generate bits of it or the front-end client and the docs will be ready to go I'm also a big advocate of letting the um technical writers in the room the design time because open API I has like you you have to name things really well and they have little like short summary Fields this micro copy it's a it's an art and if you can get your communicators in the room we build the right thing we know what it's for we talk about it correctly the error messages make sense and so the design that's why the design first apis um are gaining so much mind share I'm not sure it's the easiest way to build an API but it might be the overall least effort honestly I will love to build off of what Lorna has mentioned here about design first or API spec first uh developments which is a tenant that um at paler um and my colleagues who' worked at AWS is a widespread um practice and it typically is widespread in big organizations where those organizations have uh invested in these tools internally and have kind of gotten everyone used to this culture of Designing specs before they built something so when I left paler I was actually quite shocked that in the real world there's not so much of that culture and it's um and there's not a lot of uh good tools out there that make this easy and so um what we've I've seen with a lot of customers is um the direction of generating open API specs from server as Lauren had mentioned and that's often times harder than going the opposite direction which is designing your spec and generating server code and generating docs and generating SD and all of the different things from one Central design spec and so uh one of the reasons why Fern was created was to take some of those like learnings from both AWS and app paler where perhaps we we uh believe that open API isn't the best spec for uh representing an API spec or maybe that's not true but rather it's hard to design an API from open API even though it's a very good spec because everyone uses it so I think some of the tools that need to be worked on in the future um and F is trying uh to do some of that as well is build better tools for developers to uh right specs very efficiently and make that a core part of your organization's culture so it doesn't matter how you get there uh it just matters that you have the right tools to start thinking from a design perspective and not just start writing codes immediately if you're inspired by what you're hearing and want to get more involved consider partnering with us at Major League hacking Major League hacking is the largest community of early career developers in the world we help leading devil teams connect with our community of over 150,000 developers each year to learn about their technology platforms and bring them with them into the industry to explore opportunities for collaboration visit sponsor. ml. we can help bring you to hacker community events like hackathons meetups workshops conferences and even open- Source contributor and training programs there are plenty of avenues for you to make an impact don't miss out on the chance to shape the future of developer education I feel like this is like a core uh uh character trait of engineering organizations which is that for any given job you want to accomplish there's uh perhaps an infinite number of tools to get to your desired end State like I'm thinking of like nodejs web Frameworks you know like like how many are there I don't know there's a new one every day but it's interesting to hear that there's kind of like differing perspectives and that are popping up out here um yeah I don't know I mean I'm curious like like before we move on if you have anything to add to that Laura I just want to really Echo that it doesn't matter how you get there like if you're currently in a state where your API is open API description is generated from the bits of string and chewing them that are sort of behind the curtain like that's real life and if you have systems that work for your business then who am I to tell you that you're doing it wrong clearly you're doing it wrong right um so I just think like that's a really important thing and I'm seeing more and more people who are generating code and then doing more of a transform step and looking at how can we enrich this you know adding extra information at to enhance the output of the sdks by giving them a bit more context or publishing just some end points hiding some private or experimental ones into the docks and seeing a more API transformation and there's things going on in the open API space like the overlays um initiative which lets you yeah update the descriptions and add and remove things between we generated this thing just going to massage it a bit okay and now we can move on so I think you know Andrew just said that we have more tools to build 100% And I'm excited about it so I I want to uh take the conversation in a slightly different direction from here uh one thing we haven't touched on yet is how you actually evaluate the quality of your docs and perhaps the ROI for the business right and I think we can all agree that having docs is sort of the bare minimum of being a developer platform like if developers are using your tool they need some documentation but how do you actually quantify are these docs good are they accomplishing the business goals that we've set out you know what what factors in to that sort of like look back on on where things are at and I'd love to start with you Jen because I feel like sounds like you've been through a pretty iterative process like actually like improving and building the docks at twio this is something I think I've struggled with my entire career um it's just it it's an impossible question um and I I've gone through a lot of different iterations at different companies about how you measure Roi what you're trying to look at and uh yeah when I started here at toio we went on this like fact finding Mission we went out to all these companies um like all the like top docs companies that you could think of and we were like what do you do what metrics do you look at you know like how are you measuring Roi and no one had great answers they were kind of like I don't know what do you do so it was it was really interesting to hear like this is not something that's solved if uh there's somebody listening who thinks this is a good idea to solve this I would love to invest in your product but I think the the short answer here is a little bit like it depends on what organization your team is in um what metrics they're reporting up and you have to kind of align to those so for instance right now my team is in the marketing org which means that we are aligning to marketing um metrics so it's it's about like how many signups are we getting through documentation that's one of those things like how long until somebody makes their first API call um how long till they spend this amount of money so it's like trying to understand where do the docs come into play like when were they last touched um in that the kind of customer Journey there's a lot of things you can look at you can look at like traffic and bounce rates and all these like Google analytics types of things as well but those don't really tell you anything um because sometimes you want people to stay on the page and sometimes you want them to leave and it's just kind of hard to know um what like you're interpreting within the data um we have a a feedback mechanism um on all of our Pages a thumbs up thumbs down as well as comments that actually tells us nothing as well because people aren't raing the docs page they're rating the experience they're having in the product at that moment so you get a lot of like thumbs down on like parts of the product that are confusing or that they're mad about we get comments all the time where people are just mad we had a flood of comments a couple weeks ago couldn't figure out why um people were really really mad at us and it turns out um they had sent out an email about something and inadvertently included a link to a docs page as the unsubscribed link instead so everyone's like we canot subscribe from here what are you doing well that makes it look like our docs are bad because if we're kind of compiling those ratings uh everybody was slams Downing that page so um I will say there is someone on my team who's our our product manager now on our platform and experience team Wade Christensen who came up with um what I think is probably the best shot at this which is like a docm metric framework that we're starting to work from that really calculates the health of particular areas of the documentation so it's saying like is this part of the docs healthy and that includes things like quality and accessibility and and lots of different things we can look for so different characteristics within there and then we can apply different um different metrics to those characteristics but it's a work in progress we'll see if it works uh but again it kind of goes back to what your part of the organization is is looking for it feels like a a simple thing like docs are necessary um they're important people should invest in them but the answer there is just not straightforward I mean I feel like I've put together enough Ikea furniture where it's like obviously I need instructions but there's a high variance between good and bad instructions yeah and there there's a lot of things that we can point to but it's pretty subjective honestly you know you really need like user research to understand like are these docs working but that's very point in time as well you know it's just just around a little area of the docs like ATO the the docs my team is responsible for we have about 5,000 Pages how are we going to measure the effective of all of those um it's it's just a hard question to answer and I think you know J Jen is so right like there there there are no good answers and I've done it from the point of view of devel which is the same thing I was like there's no really good answers I enjoyed seeing you know how many signups we have that convert that had seen docs like how those two things stay together but it's very very difficult um to really meas measure the value and I think a lot of lot of companies just don't realize that they could do a lot more with their documentation and crucially their users could do a lot more you know we're we're all building developer enablement tooling is literally what we do and we want them to go and be even more awesome even though they're already quite awesome um and and like did that happen are you just don't ever really know yeah and I just want to say too it's it's really interesting to me because again it's like it feels like such a an obvious thing especially when your your product is for developers um and you have to use the documentation to use the product like why would you not invest in the documentation but um I don't think companies see that a lot of times they think like oh it's it's working people there there's documentation but if you give you know a technical writing team a developer Education team just a little bit more a little bit more resources like think of what they could do like just the fact that we've been talking about developer tooling and yet I have worked on so few document teams that have dedicated engineering teams to support those tooling requests like there's something broken in the industry and and it's frustrating because you know I remember I I keep going back to this I don't know why I'm sure there's a more updated survey I could reference but the 2017 GitHub open source survey um talked about the most important thing for developers was documentation and yet what are we doing as an industry when we don't really have enough folks who understand what what it takes to create documentation they're not invested in it they're not thinking about it from the the holistic view of a company who's supporting documentation everyone agrees it's important but no one wants to actually work on it and this the ones who do the the folks in this room who care about it who understand how important it is um we just need like a little bit more a little bit more funding a little more resources a little more like care I think from from the rest of the folks in Tech to actually make a difference here so that's me getting on a soap box here but yeah it's always interesting to take things that from one perspective are a need to have and put them in the context of a business that is you know at certain levels very disconnected from that need right and I think that that happens all the time whether it's devil or docs or many other uh things in in this world um well one of the things you touched on Jen that I'd love to hear a little bit more about you know also from the other panelists is feedback and sort of like um uh qualitative like reactions from developers like user testing uh seeing people interact with your docs I'd love to to hear about like any examples you have where watching someone go through your docs led to a significant change or getting a particular feedback led to a you know big change and I think I want to go beyond like you know there was a typo or or something like that right like what actually changed your perspective on how Developers were interacting with your docs um I can give a fairly recent example um we had a lot of changes uh to our compliance product in the last year based off of carriers so not things we wanted to do things we had to do um and it was confusing and people were mad about it and a lot of that is is on the product you know it's not things that we can fix you know documentation can't fix product things at the end of the day um but we heard heard the frustration from developers and it it was very much like okay it's going to take this amount of time from the product team um on their road map to fix some of this these things so what can we do in the documentation like how can we make things clearer or easier so um we actually brought a contractor in um through a company that we work with stack Builders who is great if you ever need any engineering resources they're wonderful um we brought in a contractor just to focus on this type of our documentation for about six months to clean up everything that they could um they're contract has ended and so now our team is focusing on it a little bit more to clean it up according to like additional standards so it is really just making sure that if we hear that there's an area of the product that folks are having problems with we look at the documentation and say like what can we do here um you know at the end of the day one of the other things I I forgot to mention but just like ticket deflection to support you know that's one of the big things we want to try to do so if you see like support is getting slammed with some area um you know you want to try to clean up the docks as much as possible um there's other things too that I've done in the past like we used to at GitHub um there were these things called patchworks I don't know if anyone ever want any of those they were like super fun little events where you would get people together um just for a couple hours and you'd walk them through like getting started with GitHub and so my team was doing an offsite one time and we ran a patchwork and we had everybody like kind of come in um and we took that as an opportunity just for all of us to say like how are people getting started with GitHub what are they running into what questions do they have so very unofficial user research but seemed like you know good time to do it and I think that more than like I guess triggering some kind of big change in the docks I think that for us just gave us a little bit more empathy so to your point earlier Andrew Relic Tech writers like having problems with GitHub like we had all been working in it for so long that it seemed super easy but seeing you know from the point of view of someone who had never touched it who didn't know anything about git had never looked at GitHub it gave us A A Renewed sense of like okay we're we're doing this for a reason we are trying to make things easier it it kind of reminded us of why you create documentation in the first place so that was just a a nice moment I think for our team you know what you were saying Jen is very reflective of my experience in mh's community like when I go to our hackathons we're typically working with college students so they're relatively early in their careers and you know like as the mlh I get asked to debug all sorts of different random things and watching someone who like has never seen one of these platforms before go through it for the first time is this like incredibly enlightening experience like a lot of people you know in our community like it's their first time using GitHub it's their first time using stack Overflow it's their first time using an API and it really like gives you perspective of what we take for granted when we're designing developer platforms like what we think everyone knows already and you know whether someone's a student or whether they're a really experienced developer there's always gaps in knowledge that we don't predict and so I think that like the type of thing you're talking about like patchworks or even going to like meetups or hackathons or whatever else and watching people interact with a product is just like this incredibly valuable experience that can't really be replicated through automated you know tools yeah some of my staff were at write the Dos last week and they had they do a writing day like a hack but for writers and they came back with such a big list of things it was just great and I feel like they had that they created those light bulb moments for people who hopefully will go on and contribute more open source documentation themselves someday I think the only thing I I could add to this is that I think it's a very difficult problem to quantify the impact that docs can have on a business and um as Jen has alluded to sometimes it's not really worthwhile to spend too much time trying to quantify it and spend more time trying to uh spend that one one-on-one time with your customers and getting feedback I think uh every like design oriented company knows that getting to spend as much time as possible collecting feedback from your customers is invaluable because you develop that empathy you understand how they use their product why they want to use their products and we spend a lot of time developing that for the the customers of a product but not so much on the documentation part so that's a big gap that I think uh every company can spend a little bit extra resources into so uh you know we talked a little bit about feedback and sort of like how you evolve documentation over time well one thing I'd like to sort of like touch on here is evolution in docs as a format right like I remember remember back when I was starting out as a developer Ducks were very um uninteractive I guess is the best way to describe them like it was literally text on a page uh and Andrew I know you started off by talking about how you're building like stripelike docs right and when I think of stripe stocks I think highly interactive right highly sort of like adaptive to what you're trying to do I'd love to hear from each of you like what are the trends right now in modern docs like like what differentiates someone who's building docs in a 2024 way versus a 2010 way I think this is a super interesting question and my personal preference is like the 2010 way um because I just think sometimes you know I've worked in deal I've been a twitch streamer I'm a video course author and um the written word has great power um I have some accessibility needs myself so whatever your cool interactive thing is I probably can't use it so if you haven't written it down as well then I can't actually use your product either so locking me out of the docs is not really great but the dogs also can be put through text to speech can be made bigger can be translators can be studied in a way that's appropriate for everybody um and increasingly the machines can read them so that's becoming increasingly important as we get a little bit more you know the large language models are reading our languages having it written down is really really important important so that's my preference but I'm from a like quite an old school open source kind of you know I'm a Linux user whatever um and I think the trends do and should look different depending on your target audience um you I'm currently working at redly our target audience is somewhere between uh full-time documentarian practitioner and there are some Engineers um but they are more application developers probably whereas in my previous role um I worked for Ivan who are open source database as a service company our audience is really really experienced this adment right so they have different documentation preferences and I think it's really important that we don't as a documentation platform provider that we don't make assumptions that we don't give so much interactivity that our users are making terrible experiences um and we kind of give the tools but not anything more so that might be controversial that I really wish your docs were less interactive um and they didn't bump around when I was trying to read them but that's how I feel about it so so are Man pages like the platonic ideal of docs yeah I mean I don't think Man pages are all that useful because they they're just like a list of commands right and we create too much content like that uh we create too much here is a here is a reference page have fun you know a lot of the time you go to a read me and it shows you how to install the thing but there's no suggestion of what it actually does you know what what what problems does it solve under which criteria would I Choose Or Not choose this tool we never say what it is what it's for what it does um that's always missing I know how to install it and when I made the transition from full-time engineer into developer Advocate it meant that I work in I mean like you said it about being at hackathons you debug what's in front of you so you instantly become a poly Pro programmer by the necessity I'm a really experienced developer but there were some dependency managers that I wasn't familiar with yet and that makes you a beginner and people make assumptions so that's a definitely a tangent but um I think we we need to structure the information well who is the audience what do they know what do they need to know why are they here you know what would a typical what content type is this but does it need to be interactive or not that can depend a bit and it can depend a bit on your corporate voice as well like twilio is very fun and has ridiculous interactive things that wouldn't work for another company so it's very much about finding something that works for you and for your the majority of your audience I feel like you're in my brain a little bit with this answer that was like exactly what I was going to say as well I I'm always like a little embarrassed to admit but I don't really pay attention to a lot of like the big documentation Trends because they never feel relevant to my work in that moment because to Lara's point it is about understanding like what your particular users need so if I'm going to like throw something on the doc site that's just cool just for the sake of it being you know a trendy fun interesting new thing but it doesn't actually serve a purpose and makes it harder for people to digest the information and use it I would be doing a disservice to our users so for me it's really about getting the basics right and that's what we're focusing on right now is just like fundamental so again like we are moving to a new platform we are thinking through how to make our our content less product based more use case based these are the types of things that help people um and so I I think it is a little bit about moving away from this idea of of trying to to keep up with all of the latest trends and really just focusing on on what people need from us from our documentation that being said there's definitely some stuff that like we have to keep up with so again going back to like all the AI things um you know I'm I'm very fortunate to have some folks on my team who are able to keep up with that at the rate at which things are changing which I am not um so really looking through like what can we do within our platform like an obvious example would be um you know because some of the search things that folks are doing just um like our search right now and our site we know is not good do we need to bring an assistant in to help us with these things um maybe we'll see um we're also just thinking more about like um more like using AI within our platform so not even user facing things but what can we do within our platform to make things a little bit easier for people creating the content like that's that's really more where we're focused on some of these like Trends within documentation is understanding kind of behind the scenes what can we make faster easier more scalable because at the end of the day users aren't going to care if my team is only five people what they care is if they have the information they need so what can I do with less so that's really where we're focusing our time um is is understanding that and really investing in our platform I feel like that's a great segue to really the last big topic I wanted us to cover which is AI right like obviously it's the hot thing of the moment you know all three of you have touched on it in different ways you know how do you approach this sort of like perhaps very hyped up but also arguably very useful technology that's coming down the pipe actually relate to both Laura and Jen's perspective more than I relate to the stripe uh way of building docks even though I have said that uh we're inspired by stripe in a lot of ways but when I use stripe when I go through their docs I also skip all the interactivity portions and that's definitely not something that we prioritize building either just it's a lot of visual clutter and I do respect that there's um multiple modes of learning and that uh different people have different preferences and uh reading is one of them but as Lura said uh it is one of the most versatile format so investing in that first and foremost is very important and I'm a Avid Reader so I read a lot and I pay attention to having a good typography and having text within a certain width of the page and these uh details that really matter to reader because I think we've all like experienced opening a man page on Terminal it's like this block of text and there's no way anyone can uh like dive into that uh without putting a lot of like mental energy towards it when it comes to AI I think I take a somewhat similar uh approach here which is you probably donate AI in your docs um especially not user facing AI um I think one of the hardest challenges in information technology in the last century is uh information retrieval and figuring out how like how and when to uh surface information that's most relevant to the person looking for that information and AI could be helpful there uh but we've also seen AI in practice not being quite helpful I try to use gbt on a daily to like be the first place where I search for information and it's 50% of the way there in terms of finding the right information for you um and incorporating that into your docs might actually be counterproductive today until we have a better like technology behind surfacing documentation uh maybe a more refined or fine-tuned model that is specialized in documentation something like that uh that needs to come down the line before it's really valuable to users I mean I don't think even open apis docs have a chat bot on them at the moment which is maybe the case in point of what you're describing there uh Jen or Lorna do either of you have strong opinions about how AI is perhaps going to change the the practice of documentation or content maybe I'm naive but I actually don't think it's going to change a lot of things so like when Lura was talking about like um context like that's something that cannot be generated um by itself you know someone has to create that context someone has to write it somebody has to you know be working with product managers and Engineers to say this is what this thing is and this is why you'd use it and how you use it and then maybe you know then like yeah I can come in and say like oh now I'm going going to you know pull the information from many different places and spit something out that's even better but I think at the end of the day um people still need to be able to create the content to create the context and and it may look different like we're thinking about like what is the future of documentation look like will that be a a a doc site like we have right now I don't know maybe there will be a doc site plus something else maybe it'll be more um in product uh documentation that's in there and that to me is more like back to the the tooling and the platform like being able to support pull in your content into many places um but those are not necessarily related to AI um for me it's it's really about like how can I automate um the the things within our tooling or within our our processes right now that we don't need to do in the future and figuring that out um but from like a frontend kind of user facing point of view I don't think there's going to be a lot of changes in the next few years at least um that folks will will see AI making a big difference within formal documentation I guess to to push back on that a little bit before we go further like aren't we in kind of a situation now where we're in the AI large language model whether we like it or not like whether or not it's a Incorporated part of your docs your content people are going to chat GPT and asking it questions about how to use twilio so how do you deal with that sort of like reality even if it's not changing the the core discipline itself so we're looking into that I don't have a good answer for it right now but understanding like what do we need to do differently within our documentation are there things within like our metadata that we need to provide so it picks it up easier do we need to think differently about how we have you know some of our docs that we write kind of thing um but I don't think there will be a lot of big changes that we need to make necessarily I think actually going back it might be more like how can we still get attribution so when we're talking about metrics and stuff like oh no now no like they're not going to our site sometimes we're not going to see that last touch what do we need to do differently to understand like if the docs are useful and that part is actually scarier to me than anything else because all of those metrics that relates right back up to like funding um so it does make me a little nervous for like docs industry as a whole to understand like you know we're already having a hard time telling the story of the value of documentation will that get harder in the future because people aren't coming to the doc sites as much as they were so I think that to me is is the scarier part um and understanding how we can kind of work against that a little bit or still get some of those those clicks yeah I definitely would Echo that I think um the assistants have a lot to offer us the stuff they're really good at is patent matching right so they can do here are five bullet points write at me in the form of an academic essay this is going to be a talk I want it as an abstract please write this as a blog post make it more marketing make it less marketing make my talk sound friendly make me sound professional hey I'm British it turns down the superlatives try that I'm British turns down the superlatives amazing and that's very very subtle so in terms of enabling non-expert writers to turn out correct content presented in the expected style there's value there um you know I'm currently publishing templat but maybe we would just train something to be like no we don't do it like that we do it like this like you need to say what it is before you launch into it this is a terrible explanation try again so there there're like individual writing level things but like Jen I think that good docs are good docs and I think we have a new audience and that audience is going to misuse the content we need to be quite careful about how we write for it but even more careful about where we get our own information you know we will return to was this human written like organic food or like was it does it have preservatives in like was it written by a person do you know the person do you trust them I think that's going to become more of a problem because you know it can write our docs for us yeah of course it can and some of some people will go that route and will'll be able to spot their Docs they sound plausible but they're not actually going to get you to the task that you need so I think there are risks there are risks that people like oh my nephew says chap GPT can do it so I fired my Doc's team um and then we lose the expertise in the industry so there's a lot of risks here I think things might change but maybe less than the hype train would have you believe that's a great note to end on and I will keep an eye out at my local like Brooklyn farmers market for artisanal docs in the near future um you know handmade and all that but thank you all so much for for being on this panel uh Jen Lorna Andrew a lot of really fantastic uh discussion and best practices here for everyone listening um if you enjoyed it definitely subscribe for more in the future but uh thanks everyone and uh Happy hacking thank you for joining the state of developer education for our latest expert panel if today's discussion sparked new ideas or in interest in how to reach developer communities and teach them about your platform we encourage you to visit sponsor. mh. to learn more about how you can engage with this vibrant high impact community of developers remember to subscribe to our channels for updates on future panels and episodes as well as industry events like Dev Kong on behalf of Major League hacking I'm John gried we look forward to seeing you at our next panel for, Creating Excellent Developer Docs and Content, Major League Hacking, 01hz261h28j4da4mwpk2xvv70b.