Navigating AI Integration and Ethics in Open Source Communities  🔗 **Source:** [Navigating AI Integration and Ethics in Open Source Communities](https://share.snipd.com/episode/bc756463-a16a-47e5-b7e3-616e86a33d83) ## Highlights - **LLMs Are Just Big Corpora Reflecting Their Source Text** - Large language models are essentially corpora that reflect the makeup of their training text. - Kimberly Pace Becker explains models learn from scraped web text and will mirror whatever genres and biases that corpus contains. Transcript: Kimberly Pace Becker Yeah, I don't know that the first thing I noticed was the patterns. I think what I saw first was a couple things that just came from my training in corpus linguistics. A corpus is just a big database in case you don't know, you know, your listeners don't know what that is. It's a fancy word for like a big body of text that has been gathered from wherever. And so a large language model on the back end is really just a big database of writing that was scraped from the web and open source and not open source and copyrighted material and all Kinds of things. We don't have to get into that. And so the first thing I thought was in terms of how you make a corpus from the academic perspective is that you want it to represent whatever sample of language you are examining. And, you know, throw in some copyrighted materials, like, what are you really sampling? And then that's going to be reflected back to you, of course, in the output. So that was like kind of my first question, like, oh, my goodness, are we just going to get a bunch of like Reddit, repurposed Reddit material or, you know, blogs? So I was thinking about it just like on the back end, like, what does it look like? And then the other thing that I quickly noticed was people immediately started looking at it from a from the perspective of like the output and like it's writing for you, you know, like It's cheating. Education space is like, oh, it's like a new form of plagiarism or whatever. But I was thinking, no, like it does so much more than generate. Generative AI, I feel like always did it a disservice as a name because I knew that it could do feedback. That was like my first research project was like, if you gave it a paper that you knew was an A-graded paper, would it also give an A-grade? I wasn't looking for it to give me actual, you know, language that I might use as my own, but just to recognize patterns, because that's all it does. It's just a pattern recognition machine. So I was like, if it can recognize patterns and output something that seems like human language, then it can also recognize patterns in our human language and give us feedback on that. And so that is how I ended up founding an ed tech company was it was a feedback company. It wasn't about writing for anybody. It was about generating feedback. ([Time 0:02:14](https://share.snipd.com/snip/c4164051-a741-45d6-8fe0-26b684aaf298)) - **Use LLMs For Feedback Not Just Generation** - LLMs can do more than generate: they recognize patterns and can give feedback. - Becker used this pattern-recognition idea to found an edtech feedback product rather than a writing-for-you tool. Transcript: Kimberly Pace Becker Generative AI, I feel like always did it a disservice as a name because I knew that it could do feedback. That was like my first research project was like, if you gave it a paper that you knew was an A-graded paper, would it also give an A-grade? I wasn't looking for it to give me actual, you know, language that I might use as my own, but just to recognize patterns, because that's all it does. It's just a pattern recognition machine. So I was like, if it can recognize patterns and output something that seems like human language, then it can also recognize patterns in our human language and give us feedback on that. And so that is how I ended up founding an ed tech company was it was a feedback company. It wasn't about writing for anybody. It was about generating feedback. ([Time 0:04:11](https://share.snipd.com/snip/86474f72-697d-49f1-9413-57ea657002c9)) - **Scaling Dissertation Feedback With Agent Prompts** - Kimberly taught undergrads and grad students and used LLMs to scale dissertation feedback. - She and cofounder Jessica built prompts and frameworks so the tool only provided feedback, not disciplinary content, for graduate writers. Transcript: Kimberly Pace Becker So I was a lecturer in the English department at Iowa State when ChatGBT was released. I had just graduated with my PhD and gotten hired there, and I was mostly teaching business communications and freshman comp and like one graduate student research methods class. And I loved the grad students, not so much the undergrad. But I could see even just in the first year, I was like, I immediately had my students playing with it. They were like, what is this thing? Everyone's saying not to use it. It's cheating. Not the grad students. They weren't interested in it. At that point, it was like GPT-3, and it certainly wasn't capable of PhD-level communication. So they weren't, you know, curious or much at all. But the undergrads were really curious about it, and they were shocked that I was having them, you know, work with it. But Iowa State is a very tech forward English department. I mean, my degree is called, you know, linguistics and technology. And I mean, yours was, you were in the tech comm department, essentially. Derek Hanson Yeah, basically. And that's what got me where I am now with WordPress. Like that is like the bedrock of my journey. Kimberly Pace Becker Yeah, I mean, exactly. And the reason I chose Iowa State was because I was older coming back to graduate school. I'd already had a career teaching at a community college. I had, you know, two little kids. I knew that I would be in my 40s when I graduated. And so I wanted the tech to be part of because I thought it might give me an edge. And it totally has. I mean, I don't know that I'm, you know, Ray with in the big bucks because of it, but it's given me opportunities that I never would have had if I hadn't had that. Anyway, so yeah, that happened. And at the time, I was working on the side as a dissertation and graduate student research writing coach for a private company and had really hit it off with the owner of that company, Jessica Parker, who is now the co-host of Women Talking About AI. Graduate student writing, essentially, a company that had workshops, not just one-on consultations, which are really expensive because you need to hire a PhD coach and then you have To pay them PhD level. So we thought, well, we could make this, we could scale this easier if we had really small group meetings where each person paid, I don't know, something reasonable like $25 an hour in A four-person group, you know, then the coach is making $100 or whatever. This is just kind of roughly what we were thinking. And then we started getting curious about how AI might help them to give feedback because one of the problems in a lot of graduate programs is there's not enough. Well, the professors are, it's a very complicated and very long draft that you're giving to a teacher. It's not like an undergrad who's writing a couple pages and getting feedback on that. It's maybe, you know, a chapter of a dissertation. Yeah, it could be 25, 30 pages. A journal article is typically 30 pages in our field. And so that's no small thing. And so we thought, well, if it could just give some feedback, even if the feedback is not great, like if it's just better than average, could that work? And so we started playing around with that. And our clients loved it because then it was like, we weren't just getting chat GPT to give, you know, here's this paper, give us some feedback. It was like, using Swale's move step framework, please, you know, use that to give feedback on this introduction. Like, is this person accomplishing the communicative goals and using the strategies that a high level research writer would use? So we were like feeding in frameworks and using that as feedback and they loved it. They were like, this is like having you here with us at two in the morning when we're writing. So after that, we were like, we've got to lean into this. This is the future. We aren't going to be paying humans very long to do this. And I think we were really early. I think one of our problems was we were really early. We could see that, well, she could see as a business owner that like, you know, what's the future of a one-on consultation if a machine could do or even approximate that conversation? This is the future. And so she was sort of thinking, OK, well, how do we future proof a company? She's got the entrepreneurial business brain, but also is an academic. And I was the linguist who was like, yeah, we could absolutely bake in some linguistic frameworks behind this and, you know, give it some information about genre theory or, you know, Corpus linguistics principles, collocations, like what words come together in academia versus in other things to avoid, like all these things. It was just a really great synergy. And so we adopted a no-code platform that at the time, well, it has moved on now to agentic AI, but at the time, it basically allowed you to build in a prompt that the user would never see. And so there was like the chat GPT system prompt, or we were also using Claude at the time, the Claude system prompt, but then we were embedding another prompt that the user didn't see That said things like, don't write for the student, only give them feedback from this framework. Never, you know, never try to give them disciplinary content. Just look at the writing from the perspective of XYZ. And so that was kind of like the idea behind the company. ([Time 0:05:26](https://share.snipd.com/snip/d2f2b5f6-a61b-47e5-98f1-545c1255233e)) - **Build Hidden Guardrails And Use RAG To Prevent Hallucinations** - When building with LLMs, constrain their role via hidden system prompts and guardrails. - Becker's product used system prompts and RAG to prevent hallucinated citations and limit the model to specific feedback frameworks. Transcript: Kimberly Pace Becker And so there was like the chat GPT system prompt, or we were also using Claude at the time, the Claude system prompt, but then we were embedding another prompt that the user didn't see That said things like, don't write for the student, only give them feedback from this framework. Never, you know, never try to give them disciplinary content. Just look at the writing from the perspective of XYZ. And so that was kind of like the idea behind the company. And it did great for a little while until the frontier models like ChatGPT, Claw, Gemini got so good that without our expertise, they were able to give that really good feedback. Derek Hanson Yeah. And then we were. Yeah. You kind of touched on it a little bit. You mentioned agentic, right? Like anybody who's so like, you know, this audience predominantly are creators and developers. And we're all very familiar now with writing and using and sharing skills right so it almost sounds like what you were early on was creating the skills for these tools that were running In the background so that that's really cool to hear like essentially what has proven out to be working now, that's what you were doing early on with that really high level writing of Academic level writing. Kimberly Pace Becker Yeah, because in the background, what we had was an agent that the user didn't see. The user still was using a chatbot interface. But on the background of that, we had essentially an agent that was functioning behind the scenes that had specific guardrails of do's and don'ts. And then we even had some rag where we would give it, you know, a certain amount of information from what we were trying to do when we ended up closing was connect like a research database. Because, you know, one of the things all academics know is it will hallucinate citations, fake sources. And so we were having a real problem with that, even though every single starting prompt we gave our users was like, do not use Moxie to cite sources. Moxie does not cite sources. Number one question, two questions we would get. Number one, does Moxie evade AI detectors? And number two, why is Moxie hallucinating sources? We feel like, well, because all AIs hallucinate sources unless they're connected to a database. ([Time 0:11:03](https://share.snipd.com/snip/40c30f4d-4683-476b-a949-dc74a3706a2c)) - **Models Normalize Overconfidence In Language** - LLM outputs strip human uncertainty and sound overconfident, increasing risk of misinformation. - Becker warns that confident-sounding text removes natural doubt and can mislead noncritical users. Transcript: Kimberly Pace Becker I think main problem with embedding AI from a linguist's perspective, maybe not necessarily from a former AI business owner, is that it is so confident sounding. And when you extrude text that has all the doubt stripped out of it, which a human would naturally put in there, well, most humans, I mean, if you ever taught freshman composition writers, You know, they're very sure about everything. They're very certain. You know, that's a developmental thing for people who are that age. They're very certain they haven't had enough life experience to know, oh, that really strong intuition I had was actually totally wrong. But, you know, I have a concern about how the doubt and the certainty really get totally stripped out. And so it's like, I don't, I mean, I guess the best phrase I could give it is like normalized overconfidence. And again, like going back to the corpus that it's trained on, it's like blogs, corporate speak, you know, Reddit posts, social media posts, and people position themselves very strongly. Whereas experts, you know, not your everyday layperson, but an expert will tend to say, well, maybe, or it's possible that this might happen. And you know kind of where they stand on that. But these machines are just like extruding this very confident sounding text, like the certainty is gone. And so if they're not surfacing that to a user who, I mean, I'm not sure what kinds of things you would even be outputting, but any language that is stripped of uncertainty is sure to eventually Lead to misinformation because certainty has to be baked into everything we do or uncertainty. It's on a, you know, on the spectrum. But that's what I'm mostly concerned about is how sure they sound, confidently wrong they sound. And people who don't have a critical eye see their accuracy, see their fluency as the be all end all. And it's really not. ([Time 0:15:41](https://share.snipd.com/snip/89c3d97a-9c91-4dcc-ae4c-47c87ac161be)) - **AI Code Is Self-Testing But Docs Need Human Review** - Code generated by AI is easier to validate because it either runs or fails. - Becker notes documentation and readme files produced by AI need careful editing since they guide human implementation. Transcript: Kimberly Pace Becker Think it's less dangerous with code because when you're writing code, like you'll see immediately if it doesn't work, right? Like it'll either do the thing you need it to do or not. There's like this very public measure of success. But if you're just giving, like I'm thinking about readme files. Like when you're writing a readme file, that is directions to a human who's going to be implementing something. ([Time 0:19:30](https://share.snipd.com/snip/d7e7adf6-bc2c-446d-9b4b-3c5366eda7ed)) - **Audit Outputs Not Just Code In Open Source** - Open source excels at code auditability but may lack auditability of human-facing outputs. - Becker urges attention to who reads AI outputs, audience assumptions, and biases from training data and annotators. Transcript: Kimberly Pace Becker Well, when I think about the open source community, I think what they're really good at is like... Auditability of code, you know, like, are you able to validate what happened in the code, make it public and audit it? And then it's like this crowdsourced ability to, you know, make sure that it's working right and then it's ethical. And I think probably what, what they may be less practiced at, and I'm no expert, but this is just me kind of riffing, is maybe auditability of the outputs. And so the human facing, you know, not just the thing the machine needs to read and do, but the thing that a human needs to read and do. Because, you know, you can publish like all your infrastructure details, your settings. And if the user doesn't know that these are inherently machines with bias baked in, and there's nothing we can do about that. We cannot take that corpus that these huge companies built and fix it. We can't make it representative. We can't scrape out the bias because humans are biased. I mean, these machines learned bias from us. And of course, bias is relative, like who whose bias is it and whatever. But I just keep coming back to like rhetoric. And this will appeal to you because that's I know you've studied that language. Yeah. Like, assuming that transparency at the level of infrastructure is equal to transparency at the impact level, at the point where the user gets involved, like, who is reading what this Produces and in what context? And here's the rhetoric, right? Like, author, audience, purpose. Like, what are their assumptions? What are they coming to the table with? Those questions require a really different kind of contribution or contributor than the ones who showed up to, like, just fix the bug. So I'm always asking questions like, okay, who is benefiting from this? And who is this disabling? Or where has this come from? And what were the biases? Because it's not just the biases baked into like whatever data and text they scraped, but also annotators. So annotating that code, the human reinforcement learning part of machine learning, which I didn't know anything about at all until I just am like fascinated by these language machines. And I started looking at what humans are doing on the back end. Well, now that we're getting models that are really like discipline specific or specialized, you know, who is on the, who's the human on the back end doing that data? Because probably it's someone that's getting paid pennies for by the word or like Mechanical Turk. Amazon's got, you know, Mechanical Turk where you hire people to annotate data. But these companies scale AI and now it's happening in academic spaces with like Outlier, that company. You know, they're not, they're not quality wages for the kind of work that needs to be done. And that is going to be reflected in the lack of quality in the output, you know, not just from what it started with, but what the annotators did. ([Time 0:22:12](https://share.snipd.com/snip/f27fa103-415d-4840-bd32-483b4b4085eb)) - **Include Humanities And Diverse Ages In AI Teams** - Invite interdisciplinary contributors to AI decisions, not just engineers. - Becker recommends linguists, rhetoricians, sociologists, historians, seniors and youth to spot bias, scams, and rhetorical harms. Transcript: Kimberly Pace Becker Big tech jobs recently from Google or, you know, walked away from Anthropic and look at, you know, I'm thinking specifically about like Timnit Gabru who left Google. She published a very famous paper with Emily Bender, who's a linguist. It's that stochastic parrot paper. I don't know if you've heard of it, but it's about how language models mirror biases. And they were very concerned from the beginning. This was before any of the rest of us knew about a language model. They were working with Google and they were essentially whistleblowers. They were saying, this is moving too fast. This is going to pollute the information, the world of information in a way that we cannot backtrack. This is going to make the Internet worse than it already is. And Timnit Gebru quit. She walked away. So, I mean, there are a lot of, I think, first of all, there aren't that many women in tech. So you're not going to have as many women in the room. And I don't know how to fix that. You know, I mean, I just I don't I don't know how to fix that problem. Certainly people of color. But, you know, I work with seniors now and in a nonprofit that caters to senior adults and they are at high risk of being scammed. And so I think a range of ages need to be in the room. My teenager the other day, he was trying to find a reel to show me on Instagram. He's like, Mom, you'll think this is funny. And you know how you're just trying to find it and you're scrolling through again. And finally, he passed one. And I was like, wait, I want to see that. And he goes, no, that's just AI. And I said, how do you know? I mean, we had seen 1.5 seconds of it. And he goes, I can just tell. And I was like, how do you know? I don't understand. It was a picture of a man picking up a woman and throwing her down a bowling alley lane, which could happen. I mean, it wasn't, I mean, she didn't roll like a ball. She just fell. She just, he dumped her on the ground. She rolled over into the gutter and he immediately knew it was. And I was like, but how did you know? And he couldn't put his finger on it. But I think it's not just the obvious things like, you know, like women and people of color. Like I'm thinking age is important too. And people from a range of different backgrounds, because mostly it's educated people in the STEM background. So we need humanities people in there. We need linguists, rhetoricians. We need tech comm folks, sociologists, philosophers, historians, you know, on and on and on. And how do you get those people in the room? ([Time 0:27:12](https://share.snipd.com/snip/2bedbb36-923f-4388-a60d-e4020a8cd56a)) - **Ship AI Incrementally With Clear User Benefit** - Build AI features incrementally and sustainably rather than for novelty. - Becker cites Volker Hegelheimer's guidance: tech needs a clear user benefit and must be modular to change without breaking everything. Transcript: Kimberly Pace Becker It needs to have a real purpose. It needs to have a clear benefit to more than just the developer of it, the user of it. And it needs to be incremental and sustainable. And those two values, incremental and sustainable, really were, I mean, we talked about that every day when Jessica and I would meet. Like, if we make this change, is it going to overwhelm our users? Will they be able to understand it? Will they be able to work with it, flow with it immediately? Or is it an overhaul that is going to really set them back? And then if we implement it, can we sustain it over time? I just don't think you can go wrong if that's your, and now I know that's what agile means because we hired developers, you know, and I learned what that, that there's this whole framework Called agile. And first I was like, what are you even saying? But that's really all it is. It's just, you know, this idea that things are modular and you build slowly and you don't want to build it all because you may have to go back and change this. ([Time 0:32:23](https://share.snipd.com/snip/c2adf8cf-588c-46c1-baa5-eabd8f51b876)) - **Ask How Your AI Surfaces Uncertainty Before Shipping** - Before shipping an AI feature, decide how it handles uncertainty. - Becker's litmus: if you can't explain whether the tool surfaces, hides, or eliminates uncertainty, don't ship it into user infrastructure. Transcript: Kimberly Pace Becker I think it would be something about what does this tool do with uncertainty? Like in the face of uncertainty, in the code, in what the user is going to input, does it surface the uncertainty? Does it hide it? Does it eliminate it? Like, if you can't answer that question, if you don't know what it's going to do when it inevitably faces a question of uncertainty, then you're not ready to ship it. That is my big concern with AI is that it doesn't, you know, as a linguist, it's just not able to handle a lot of uncertainty. It is very confidently wrong. And you do not want that baked into the infrastructure of a WordPress site. ([Time 0:38:59](https://share.snipd.com/snip/ff81591a-305c-4425-b296-f26a33247eb8)) - **Introduce Friction To Preserve Rigor And Care** - Slow down and add friction to AI workflows to preserve rigor and care. - Becker required contextual questions before use at Moxie and argues friction is necessary for real-world integrity and safety. Transcript: Kimberly Pace Becker Think my answer is about the same as it was when a graduate student or a Moxie user would ask me, like, is this going to pass muster? Is my professor going to use AI to detect this? And my question would be to them, like, well, can you orally defend it? Like, despite what the written word says, what the code says, what the, can you stand up and can you orally, meaning with your voice and no notes, can you defend, not just the accuracy Of it, but the complexity and the integrity of it. And then ask, you know, am I being more confident than I should? Am I, you know, in terms of that uncertainty, like, does this sound like me? Would I sign my name to this? You know, or do I need to build in some sort of review steps specifically to calibrate that uncertainty that I might have? Because, you know, we have this idea in tech, especially, of friction. Friction is bad. Friction is always, you know, a step the user doesn't want to take. But we've got to slow down and pause. Care means slow down, pause. That means you may have to introduce some friction. It's not going to kill us to be uncomfortable. And I mean, that's something we struggled with. We would ask all these contextual questions before you ever even got started using Moxie. You had to come to the table with these answers and people hated it. People hated it. And our investors would be like, you cannot build so much friction into your tool. And we would say, sorry, but this is what academic research requires. And I mean, I would argue this is what real life requires. There's not a fasting. Rigor. Yeah. You cannot get past. Friction is everywhere. We can't run, you know, it's like that whole, the trope about you can't go around it. You got to go through it. Like you, it's just not possible to avoid discomfort and friction. ([Time 0:41:24](https://share.snipd.com/snip/77e0abdd-c651-4b11-ac9f-f9d162f42bd6)) - **AI Raises Ethical And Spiritual Questions Humans Must Carry** - The human role remains essential because AI lacks 'soul' and can't solve deeper ethical or spiritual questions. - Becker suggests AI challenges will ultimately be spiritual and require human discomfort and judgment. Transcript: Derek Hanson Like we're creating files for agents called soul and D, right? Like we as humans need to maintain and bring the soul to everything we do. Kimberly Pace Becker It's, well, we don't want to get too woo-woo, but I think it's ultimately going to be a spiritual problem. It is not a problem for tech to solve. Ultimately, this is going to be a spiritual problem for humanity. ([Time 0:43:58](https://share.snipd.com/snip/5908f2e5-0d97-4c22-8aad-4a8228f38380))