CHAOSS AI Detection Project Meeting, Feb 17 ![rw-book-cover](https://img.youtube.com/vi/lU2NKub8XwU/maxresdefault.jpg) 🔗 **Source:** [CHAOSS AI Detection Project Meeting, Feb 17](https://www.youtube.com/watch?v=lU2NKub8XwU) do that. Sweet. So, welcome to the first sort of trial meeting of I guess the chaos AI detection action. I don't know if you want to call it a working group because it's probably not that big, but it's like side side projects that sort of assists like three other working groups. I don't know. Um, yeah, it seems like I think this effort essentially came together just from like this idea. I mean, it seems like this idea I've kind of had this idea for a little while, but I think having talked about it in Chaos, it seemed like it piqu peaked various people's interests from from Andrew with ecosystems and Schllo's PhD research and just chaos augur grimmore lab like being able to use it. So, I think I think this is a kind of neat little like example of working upstream and working together. Um, do we want to kind of go around and do some introductions briefly? >> Yeah, sounds good. Wait, I I can start. So, I'm I'm Adrian. I'm I my most of my work up until now has been I mean, even still now uh in the Augur working group. I'm one of the sort of most act more active maintainers of Augur. Work closely with Sean and mostly other augur core team. I'm pretty active in that channel. Um just helping kind of unteify and and improve the Augur codebase. Um my day job is working at Red Hat's open source programs office. Um, I'm I'm kind of on the data team with with Cali and and Kaio and some of the other people helping kind of we run an auger instance internally for for Red Hat stuff, but the majority of my work is all just upstream community in chaos basically. Um, and obviously every every company under the sun is now interested in AI and like how do we measure how do we evaluate how do we you like use more of it or evaluate how it's being used. And I think there's also been a lot of stuff going on in the the open source community that scared people in terms of AI. So seems like there's a lot of interest here. Anyone else want to go up? >> I think I can go next. So yeah. Hi, I'm Schllo. I'm a PhD candidate at the University of Florida. Uh my research mainly uh work revolves around how sort of communication dynamics within open source repositories affect the security practices and repositories. Uh and yeah I've I've been involved with Augair recently. I think it's been what four months now. uh uh so yeah it's been it's been a good experience and uh yeah Adrian and I were discussing about uh you know AI guidelines and stuff like that for augur and that's also sort of uh ran into the so can we you know sort of built a heristic based detection tool or something for AI detection and uh that's how uh I got involved in this mouthful group that we are now calling data AI detection I completely forgot the name. >> Yeah, but >> I already had a repo called AI detection in the same code folder as well. So, it's like I have to kind of like force tab my way through to get to it. >> The data AI detection. Yeah. Yeah. So, that's me. >> Hi everyone. I'm Andrew. Uh I'm in the UK. I work on ecosystems primarily uh and hang out in various chaos working groups. Uh the main one I'm involved in is the package manager metadata group. Um I had kind of already been picking up a number of bits of signal in ecosystems. uh before Christmas I added in the detection for the kind of like co-authors and signed off um messages in uh commit messages and then also went through and pulled out a load of like labels that show up a lot that are created by you know like if you make a PR from the I guess it's called codec now but it used to be called chat GBT like code uh that would leave a label that said basically was it indication that it was made by that thing rather than using, you know, the bot account. There all kinds of different ways that these things work. Um, but all of them were just kind of like desperately built into different pieces of ecosystems. There's not like a nice standalone piece. Uh, I also got a project I've been working on this year called Git Packages, which basically is like ecosystems, but you run it locally and it analyzes the history of your dependencies for a given repo. And that is much more interesting to be able to go like, oh, I can run this analysis on a given repo locally. Well, this plugs straight into this detection thing. Um but also then if you want to do analysis of lots of packages locally then it's also very helpful to be able to go through and clone all of those the repos of each package and say okay well let's do local detection I don't need to to go anywhere else and then ecosystems naturally ends up being a place where it's like oh we can do some shared public good analysis where we run this code against public repos and then record the results so that people can just do a quick lookup rather than needing to wait for say a CI job to run on every on everything. If we're going to be analyzing it anyway, then just fling the um the results to a S3 bucket or similar and then cache them in Cloudflare. >> Awesome. >> Anyone else want to introduce themselves? >> Um hello everyone. I don't know. Okay. So, my name is Emanuel. I stay in Nigeria. I'm a junior software developer with um a Canadian startup, an IoT based Canadian startup. I'm a full stack web developer. So, I'm totally new to open source. I just joined the ARA group recently. So, I I just you know saw the meeting link. So, I decided to join and just get conversant with the ecosystem of the project. So yeah, it's been really nice getting to know everyone and understand what's really happening. Thank you. >> Awesome. We You were in the augur auger call just now, right? >> Yes, but I came quite late. >> Yeah, no worries. All the all the chaos meetings are generally kind of recorded and on YouTube if you want to go back and watch the last couple of them. I think it usually takes about a week or half a week for for I think Elizabeth to upload them. Um, but yeah, I'm happy to help help you navigate your way around chaos if >> if there's something in particular you're looking for. But glad to have you here. >> Thank you very much. I I'll thank you. I I'll check the videos out. Thank you very much. >> Yeah, no problem. Sorry, I wasn't sure if you sound like you're breaking up, but I got it. Yep. All right. So, yeah, I mean, I kind of mentioned earlier kind of what this is and how it came together. like it's kind of just like every everyone here kind of has at least one if not more than one sort of project that there there's sort of interest in there's kind of interest in this kind of like having a tool that kind of just is a effectively I guess an adapter layer between all the different sort of varying standardsish that are out there as far as how how to disclose like shared authorship or AI authorship of of code or other contributions. I think this this is sort of mostly focused on code, I guess, but like there's always all the like you could have like a a co-authored by on on a git commit. You could I think I feel like I saw a standard at one point or standard like people using like generated by or assisted by um to sort of >> in the description of the PR or the or sometimes in the commit message as well. >> There's like a commit flag like a extra sign off or something. Yeah. But >> yeah, I mean like people people disclose it. Like people say, "Oh, I used Gemini to do this in in PR comments or like there's so many different ways that people sort of site their sources as far as like, hey, I used AI for this." U and I think especially for the projects that don't outright ban it, but are maybe like, "Hey, we should we want you to just like Augur, I think the direction that I think the Augur policy is going is like we're not going to ban it. We just want to make sure that all the contributions AI or not are held to a certain standard of of quality and if you're using AI, you should be disclosing that like just like you would site your sources if you're copying code from Stack Overflow or whatever. Um just and also just to help us understand how AI is being used. So I think I think that's valuable too is like helping projects understand how AI is being used in their contributions and feeding into tools like Augur. Am I missing anything? as other people have. Oh, I mean the like the use case that I kind of thought might be interesting but also most people wouldn't look to it is once you're collecting it on mass then you can also go back to the AI tools and go like here is a large collection of data of stuff that has been produced by your tool uh for you know more and kind of quantitive analysis. uh but obviously that requires mining a lot of things uh to be able to to to give that over effectively. >> So like kind of providing data back to these these AI companies. >> Yeah. Like I mean they're probably doing it already. Um but you can also then be able to say like oh look here's Claude has contributed to 10% of the top like most critical Python packages or something like that. uh that like kind of ability to detect it uh is helpful and that's a good and or bad thing like depends on your like >> your stances but it's a fact that can be or at least you know like for the ones that we can detect because obviously you don't have to put any kind of indication in that you're using any so uh it's basically going on like whatever the signals are available. Yeah, that's actually really interesting because I I saw so one of the I don't know this probably wasn't super inspiring for this project, but I I've seen before other open source communities using tools mostly sort of more for like I guess social metrics, but they like uh I think I think it was the Universal Blue community who built Bazite, the sort of gaming Fedora based operating system. Um they were using a tool to kind of keep track of like their Discord and I think Reddit and stuff too, but I think mostly Discord for this. And that I guess that that metric was basically like it looked for people asking questions and then somebody responds to the question and then the question asker responds with like a thank you and it sort of track it like counted that and so it's like hey how many how many sort of positive like help interactions are we getting in this community. I almost wonder if that could be interesting to kind of tie into like providing that data back to AI companies as far as like hey yeah maybe Claude was used in these top repos or like but then maybe that could tie into some of like Schllo's sentiment >> you could definitely feed that into like sentiment analysis around the conversation as well and go like oh it turns out in these in this ecosystem everyone hates this so probably your like your PRs are bad uh for you know they train and test a lot more on Python than they do on Erlang for example uh to be able to go here's more quantitive data that suggests that no one is merging Claude Erlang uh contributions in open source at least you know ecosystems only picks up stuff on public projects >> yeah or you could even like hey we're like these PRs like we're responsible for the largest share of the bugs or the smallest share of the bugs or like so imagine I imagine a lot like in the self-driving car world it's always like how good is it compared to a human driver? And it's like it's sort of maybe harder maybe not hard to measure, but like I imagine you could use something similar with open source is like does does this code have more or less bugs than the human authored code before it? Like there's there's a ton of room for interesting analysis. I mean that's probably more like data science working group territory but like >> yeah just having the raw data though to be able to then pass over to other groups like right now and it we can end up with just almost like a dictionary of here are the different ways that you can do some level of detection because obviously if you're like oh actually like I only care about this bot and this bot only writes in this method I don't need to use the whole tool I just need to like troll through in a certain kind of way or I can use this API to do this rather than like I mean in some cases you might see it in the GitHub archive data and then you're like great I don't need to fetch anything other than I just troll through the GitHub archive data to find this particular string one brilliant but in other cases you're like oh I can't get there's no git ref data in GitHub archive so uh I have to go and you know fetch every repo to go and look inside its ref's database. >> Yeah, that's a really interesting angle that I'm excited to kind of explore more. I I didn't know really much about git refs until now, but there's been some interesting issue. I think the the issue you found was issue number two on the repo like that. There's some cool links in there. >> Yeah, that basically came from where I was spelunking around over Christmas in the ways that you can store extra data in inside of Git. Uh, and yeah, it turns out that other people use that as well. JJ uses it as well. If you're using JJ, it makes like a JJ name space in the refs folder and stores extra stuff there because when you push to GitHub, that data goes with it. So, it's actually then everyone pulls it back down. It moves around rather than like a hook which is only installed locally on your repo that you've checked out. >> Yeah. Uh, I did open and update that PR. Thanks for the review. Um, I fixed three of the four comments. The fourth one being like the confidence levels and I'm basically like I I don't know. We we up for us to decide what we want to do. Like for a CI check, you want a pass or a fail kind of thing. So then the configuration is like right. Well, how do we decide? Some of them are very easy to be like, yes, definite or not, but others are going to be pretty fluffy. Um, so maybe we need more data to be able to give us a bit of a test case or, you know, you started a thread of like here are some examples. Maybe we should put some more together that we can turn into like here's the JSON you would get from the GitHub API. uh can't well yeah can't even do it with that necessarily because you need to also check against the git repo but some test harness that we can then go like okay well we can make sure that we're covering all of these things and also we're not sure about this one like or it's like very difficult to test against >> yeah I almost I almost wonder like because it seems like right now we sort of just have like a high medium low sort of like I guess enum whatever whatever the structure is called um for like confidence scoring. Like I almost wonder if it'd be better. Like I feel like I've heard of more pointsbased things being used for like email filtering like spam assassin and stuff where it's like oh there's all these different heristics we look at to try and determine if this email is legit or spam and like each of them like either adds or subtracts points and like >> yeah did in liio was you would you would gain things for good behavior and lose points for bad behavior. you know, if your if your version numbers weren't exactly SER, then you're going to lose a point. Uh, for example, um, so we could definitely do that. It's basically just a case of coming up with what, you know, some decision on how we like what things we are going to score good and bad. Uh, I also maybe would against going like into the reputation of the user a little bit because that data can be huge. Uh but also some people might get very uh funny about having like oh this script then goes and you know analyzes how long has this person been on GitHub for for example is it's also kind of losy because I could sell a GitHub account to someone else uh if they were really trying to hide that >> as soon as they know the metric exists it's going to get gained. >> Yes. Yeah. Exactly. >> And I I also want to try and maybe move a little bit away from calling it good and bad. Like I think we could maybe still do the numeric scoring and just have it be like here is sort of how how likely this is to be AI. We're like hey positive points are like more likely to be AI and negative points are like hey here's more indicators of it being human. Um, like even maybe we could even do like run a spell check on the like any one metric, but like >> maybe run a spell check on the frequency of like known unusual words that >> chatbots like to say a lot dashes like >> like used a lot of headings in your uh in your issue. I mean, we joke, but there are definitely some huristics like that that like people get that kind of feel for it, even if it's not explicit. They're like this, it says some things that just setting off my trigger alerts. Of course, the problem is that uh a lot of these bots are trained on all the existing content. So, they do end up like all the emojis in readmes would just come from the JavaScript community uh having emojis in all their readmes. So it kind of you know you towards the average I guess of all open source if you if you go too far into the like the feels of a issue. I mean you could literally do embeddings and say like how close is this to the embeddings of all known uh AI generated PR descriptions for example and you can say oh this looks a lot like it except for that you know the shorter your PR description is the worse your embedding is going to be. Yeah, that was one of the ideas I sort of had part of the kind of original idea for this was like as sort of a side project. I almost wonder if it would be interesting to create because I know like there's all these AI detectors on the internet and they're varying degrees of not great um as far as like hey put an image or put put a paragraph in here like academic in the academic world like hey upload an essay and see if a student's been using AI right um and like we sort of have like virus total for like the security world where like you upload a binary and it will just scan it with every antivirus tool and it'll let you know like oh only one or two of these flagged it might be fine or like all of them are flagging it, then it's probably malicious. Um, and so like even if you've got a whole pile of unreliable tools, I almost wonder if you could kind of build something like that and then we could kind of use that as part of this to like >> maybe as maybe as like a sort of future. >> Yeah. I mean, you could using AI and not disclosing it. >> We could train a little BERT model uh once you have enough detected data, right? And then we just embed the BERT model in the um in the binary and it would be like oh yeah high confidence low confidence. Obviously, it's much harder then to kind of reverse engineer that and go like, "Oh, this is exactly why this is low confidence because it's like, well, we we smooshed like a million uh PR descriptions and titles and things into this uh into this model and came out with a a tiny binary on the other side. compared to the very like discrete steps of like oh yeah we did these reaxes and uh you matched on three of them like the debug there is much easier and like we could even have like if it's going to leave a PR comment it could put like a a hidden block of YAML that's like here is the very like verbose description of all the different checks that it did >> uh so that you can see once you get into the kind of like, oh, we we have a spam model essentially. Uh, it's like, well, for one thing, maybe some people are going to be like, I want my own custom hidden spam model. Uh, or it's going to be like, well, this is entirely opaque and we can't tell you why exactly this uh isn't working. Of course, I guess the further down that you go, you also end up in a place where you're like, do we we could use AI to try and detect if it's AI, but >> that's a very expensive way to do it. >> Yeah. And I think one of the things I like about this too is like I want to make sure that whatever we're doing like if we're going to be doing the B model or things like that, like >> make sure it's configurable. So like if if a maintainer is like I just want all the deterministic stuff. I don't I don't want any variability in whether or not this is AI. just like show me the deterministic ones. Maybe there's a flag like we pick a sensible default and then just let them customize it. Um because then like everyone's going to have probably a different label they apply or whatever. But I think even just having that be explicitly defined like you you you set up this action on GitHub or wherever and you you say, "Hey, I want AI things to be labeled with this label." I'm sure we could write a a small piece of code in Augur to check if this action is installed, read that config line and then we'll just dynamically use that detect like >> even just the existence of the config. But maybe maybe >> there's other ways to do it too as far like I've seen >> I think there's like the AI there's like some AI attribution spec I think I linked in one of my issues like maybe maybe it could post a comment with like a string that meets that spec and so we can just scan for that or what like whatever. I don't quite know exactly how we'd be making this data available to like other analysis like broader analysis tools like Augur that would look across whole orgs or whole sets of repos. >> But there there's options and I think I think that'll sort of solidify as we play with this. >> Uh so what's the next steps then? >> I'm not sure. I know um I mean it seems like we've got that PR. I think that's a really good like I mean I wasn't expecting such a big PR but like I mean it's new project I guess I guess that's fine for now. Yeah, I just took a lot of what I'd learned um on get working on git packages and took all the CLI stuff like you can do some cool stuff with Cobra to basically go like oh I can generate you a markdown contents for a website like if you wanted a >> oh wow >> a Hugo generated website out of this um like I show you the example uh in git dash packages where's the website gone Um this has a command reference which is entirely generated from um from the CLI as like it just goes and vomits out uh markdown files into the correct location and then Hugo just goes oh here's here's that whole thing uh as like individual commands with all the flags. Obviously like what we have right now is much smaller than the amount of different commands in git packages, but the it also gives you like in easy to generate man pages and the automatic help flags as well. Um, which is quite nice to just be able to kind of like keep all of that working as people would expect. So the tool is easy to use. Um, and then I also cut it up into different pieces rather than trying to have one big detector file. It's just like, oh, we'll just have lots of different ones so it's easy for someone to add in one without having to kind of just stare at a honking crate file of like every different possible thing in one place. >> Yeah. It seems like the one the last thing on this PR is the confidence scoring then, right? I think everything else has been addressed. >> Yes. Yes. >> Do we want to maybe just kick that and just say like merge this as it is because it's kind of our MVP and then make a new issue of like hey >> scoring based system and sort of discuss that further. >> Okay, >> that would be a good idea. Yeah. Uh I can also take the same um go release builder uh and plug that in if we like so that it just starts whenever we tag a release then it will build and attach the binaries to the release if you will then >> that sounds it will make the action faster as well because it's like oh I just download this from this release >> yeah I don't have to do the like go install you can do. Obviously, it depends on how much you trust. But the at least with Go, there's no package manager in the middle. It just like is all pulled straight from um the Go proxy and then built back into GitHub releases. It's all pretty transparent. Uh I can Yeah, if we make an issue for that, I can just basically copy what I did before fairly easily. That also makes the release notes basically automatic as well. >> Yeah, I mean I'm I'm like Yeah, I think definitely like once now that we've got kind of MVP, I think maybe give people a little bit of time to kind of test it, make sure it works, and make sure this is like, hey, this is something we're kind of confident releasing as like a V0. >> We can turn it on for the repo itself and see if it labels AI contributions as well. I need to jump in a couple minutes. Um but uh yeah, once we got the project to a good starting place, we can definitely share this a bit wider, I think, cuz people will be interested in and maybe even just as a call to um like give us more examples of things that we could do with uh flagging or testing against even if some of them we're going to be like, "Oh yeah, we're like we've already got good tests for that." But just getting like a people dumping some examples in would be very helpful. And then obviously anyone kicking the tires, people are going to try and run it in forjo and be like, "Oh, it doesn't quite work like this." And >> yeah, >> we can try and shake out some of those bits. Some of them might be might be difficult. I don't know. We'll see. Uh but at least the binary itself would be like, "Oh, you can you can download this and you can run it in as long as you have access to the git repo. Yeah, I just I merged your PR by the way. I I think I just made a new issue from that one comment I had on system. And >> that's cool. >> So yeah, I think we're I think people can start testing. I think we can yeah start doing some release maybe release automation or some of the some of the docs automation that I think you were talking about. um just get get some stuff there so that like as we sort of develop this and as we start writing content we can kind of give people >> Adrian by the way I shared a GitHub link uh it's an open source project it's a research study that a couple of researchers published last year where they developed their own data set of AI generated pull requests I think it has some minor >> I'll put that in the meeting notes thank thanks for that Um, yeah. And I think I think just generally if we've got ideas of like, hey, we should we should be like if there if there's an idea someone has of like, we don't know when we're going to do this, but let's put on the >> Yeah. Get it in the issue tracker. People can see it. >> Yeah. Yeah. Yeah, there's I think there's already I made a project board for it. Um, so as long as you just put in an issue or or something like that and we can kind of track it and start kind of labeling stuff. So at least the road map is kind of clear and like I don't know if we want to start doing more like a sprint style like assign people stuff or not. But I guess we'll see how this how this shakes out. >> Uh yeah, sorry. I don't know what happened. I don't know. I just automatically disconnected. But yeah, I think that data I I'll try to take a look at the data set sometime soon to see if we can you know get some insights from there. If we can get some more sort of markers or something um some more deterministic thing rather than you know just some non-deterministic LLM calling an LLM to see whether this was written by an LM that just doesn't make much sense. I'm gonna make a detection signal label so that we can kind of like if there's an issue like oh we should add this as a signal or that as a signal we can we can kind of label those and just >> Y that sounds awesome perfect yeah >> great are we going to do another call at some point I guess we can just arrange that in the Slack see how we go >> I'm my plan is sort of just to you re like do this like so the the Augur 8 not call is every other week it sort of alternates with the data science working group um different link but same time slot I think. Um so I think maybe we just like do like same thing as this every other week. Is that >> Yeah, that works. Um and if we can do it async and without any like if everything seems to be already in the issue tracker then that's good too. >> Yeah. Sound good? >> Thanks everyone. Have a good week. Yeah. Have a good day. Thanks guys. Cheers. Bye., CHAOSS AI Detection Project Meeting, Feb 17, YouTube, 01kp8hkssfycqd201k6bczzfq1.