Evaluating and Improving Replit Agent at Scale

Evaluating and Improving Replit Agent at Scale ![rw-book-cover](https://img.youtube.com/vi/snroDwX1-JU/maxresdefault.jpg) 🔗 **Source:** [Evaluating and Improving Replit Agent at Scale](https://www.youtube.com/watch?v=snroDwX1-JU) > [!note]+ Document Notes > tags: #ai-evaluation #agent-development #continuous-integration #software-benchmarks #developer-tools > author: Mikael Cata > source: YouTube > reference: https://www.youtube.com/watch?v=snroDwX1-JU > date created: 2024-12-19 10:30 > > # 🏷️ Title: Evaluating and improving Replit Agent at scale >  ## 🧰 Action Inbox - [H1] Implement continuous evaluation pipeline — move beyond single-score evals to production trace analysis #ai-evaluation - [H2] Build automated clustering system for failure analysis — group semantic failures for systematic debugging #agent-development - [H3] Establish AB testing infrastructure for agent improvements — critical for steady progress validation #continuous-integration ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [ByBench.ai](https://bybench.ai) - [SweetBench Benchmark](https://sweetbench.com) ## 📝 Summary - Replit has developed a continuous evaluation system for their AI coding agent that processes millions of user traces daily - They launched ByBench, an open-source benchmark for end-to-end AI coding evaluation that tests functional app correctness rather than just code patches - Their two-pillar approach combines offline benchmarks with online production trace analysis through automated clustering and AB testing - The system identifies failure patterns automatically and generates code fixes through their internal "Telescope" tool - Human judgment remains critical for prioritizing fixes and making product philosophy decisions based on mixed AB test results ## 📖 Main Content Mikael Cata, President and Head of AI at Replit, presents their comprehensive approach to evaluating and improving AI coding agents at scale. Unlike traditional software development tools, Replit Agent starts from natural language specifications with no framework constraints, requiring users to go from prompt to working application. The core challenge is that models and requirements change constantly, with multiple daily releases reaching millions of users. This necessitates moving beyond traditional single-score evaluations to continuous assessment systems. Replit's solution involves two main pillars: offline benchmarks for pre-release gatekeeping, and online production analysis for post-release improvement. The offline component includes their newly launched ByBench, which addresses limitations in existing benchmarks like SweetBench by testing functional app correctness rather than just code patch application. The online component processes millions of daily traces through automated clustering to identify failure patterns. Their "Telescope" system discovers problems, generates automated code changes, evaluates them against benchmarks, and runs AB tests when needed. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/snroDwX1-JU?t=0) — [H1] Introduction to Replit Agent evaluation challenges - Key idea: Replit enables prompt-to-app development with no technical constraints specified by users - Evidence: "The user expects to go from a prompt to a working application, but they don't tell us what kind of framework they want to use" #### ⏱️ [02:15](https://youtu.be/snroDwX1-JU?t=135) — [H2] Why traditional evaluation approaches are insufficient - Key idea: Constant model changes and daily releases require fundamentally different evaluation approaches - Evidence: "Models are changing almost constantly and if anything the pace of evolution is even faster than it was a year ago" #### ⏱️ [03:30](https://youtu.be/snroDwX1-JU?t=210) — [H3] Moving from single-score to continuous evaluations - Key idea: Production traces contain valuable signal that can be extracted automatically - Evidence: "We have a combination of fine evals... as well as online evaluations based on the system usage" #### ⏱️ [05:45](https://youtu.be/snroDwX1-JU?t=345) — [H1] Limitations of existing benchmarks like SweetBench - Key idea: Current benchmarks don't capture end-to-end app building from scratch - Evidence: "Users are not writing the tests. They often start from a completely empty code base" #### ⏱️ [07:00](https://youtu.be/snroDwX1-JU?t=420) — [H2] Introducing ByBench for end-to-end AI coding evaluation - Key idea: ByBench tests functional app correctness using automated evaluators - Evidence: "Rather than crafting them from synthetic data, we pick 20 real-world traces from usage of Replit" #### ⏱️ [09:15](https://youtu.be/snroDwX1-JU?t=555) — [H3] ByBench evaluation methodology using browser automation - Key idea: Evaluator agent opens browser and tests apps step-by-step with natural language test plans - Evidence: "The actions are like open the admin dashboard and log in with a certain account and click on this toggle" #### ⏱️ [11:30](https://youtu.be/snroDwX1-JU?t=690) — [H2] ByBench results show significant model performance gaps - Key idea: 2x gap between frontier and open-weight models, with self-extension being most challenging - Evidence: "Most models do a worse job when extending their own code" #### ⏱️ [13:00](https://youtu.be/snroDwX1-JU?t=780) — [H1] Online evaluation through production trace analysis - Key idea: Millions of daily sessions provide unscripted real user behavior data - Evidence: "We're collecting millions of sessions every day and they're very valuable because they capture what our users actually do" #### ⏱️ [14:45](https://youtu.be/snroDwX1-JU?t=885) — [H2] AB testing infrastructure for agent improvements - Key idea: AB tests provide mixed signals requiring human judgment and product philosophy - Evidence: "They never give you a crystal clear signal of what you should be doing" #### ⏱️ [16:30](https://youtu.be/snroDwX1-JU?t=990) — [H3] Automated trace clustering for failure pattern discovery - Key idea: Semantic clustering identifies failure patterns that regex-based approaches miss - Evidence: "We cluster them by type so that we find out the different failures that are happening at a certain point in time" #### ⏱️ [18:00](https://youtu.be/snroDwX1-JU?t=1080) — [H1] Telescope system for automated problem discovery and fixing - Key idea: End-to-end pipeline from problem discovery to automated PR generation and testing - Evidence: "We basically cut PRs with a coding agent based on the information we got from the trace" #### ⏱️ [20:15](https://youtu.be/snroDwX1-JU?t=1215) — [H2] Real-world example of environment setup timing issues - Key idea: Semantic trace clustering revealed subtle timing problems invisible in logs - Evidence: "Agent went on a tangent and started to try to fix the environment on the spot" #### ⏱️ [22:00](https://youtu.be/snroDwX1-JU?t=1320) — [H3] Human judgment remains critical for hypothesis formation and prioritization - Key idea: Humans must formulate hypotheses and guide agent-generated solutions - Evidence: "We always focus on formulating a hypothesis on what could be possibly go wrong there" #### ⏱️ [24:30](https://youtu.be/snroDwX1-JU?t=1470) — [H2] Q&A on open-source approach and building taste - Key idea: Evaluation should be collaborative, and taste develops through user feedback alignment - Evidence: "I don't believe in competing on evaluations... It helps everyone to create better products" ## 📚 References - [ByBench Open Source Benchmark](https://bybench.ai) - [SweetBench Benchmark](https://sweetbench.com) - [HumanEval Benchmark](https://github.com/openai/human-eval) - [Replit Platform](https://replit.com) ## 🗣️ Paraphrase - Replit has built a continuous evaluation system that processes millions of daily user interactions to automatically identify and fix AI agent problems - They created ByBench, an open-source benchmark that tests whether AI-generated apps actually work functionally, not just whether code compiles - Their approach combines traditional offline benchmarks with real-time production analysis, using semantic clustering to find failure patterns and automated systems to generate fixes ## 📰 Newsletter **Replit Launches Revolutionary AI Agent Evaluation System** The team at Replit has cracked a major challenge in AI development: how to continuously improve coding agents at scale. Their new approach processes millions of user interactions daily to automatically spot problems and generate fixes. Key breakthrough: ByBench, now open-source at bybench.ai, tests whether AI-built apps actually work end-to-end, not just whether code compiles. This addresses a major gap in current benchmarks that don't reflect real-world AI coding scenarios. Their "Telescope" system automatically clusters failure patterns from production traces and generates code fixes, though human judgment remains crucial for prioritization and product decisions. **Why this matters:** As AI coding tools become mainstream, evaluation systems must evolve beyond simple test-passing metrics to measure real user value and functional correctness. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Present ByBench as new standard for AI coding evaluation at developer conferences - Education: Create workshops on implementing continuous evaluation pipelines for AI agents - Developer experience: Build documentation comparing traditional vs continuous evaluation approaches - Developer marketing: Promote open-source evaluation methodology to attract AI engineering community - Community management: Foster contributions to ByBench benchmark expansion #### OSPO - Policy & Governance: Establish guidelines for open-sourcing internal evaluation tools and benchmarks - License Compliance: Review ByBench licensing for enterprise adoption considerations - Security Management: Assess security implications of automated PR generation from production traces - Software Inventory: Catalog evaluation infrastructure components for compliance tracking - Developer Support: Provide training on implementing production trace analysis systems - Community Engagement: Support academic collaborations on AI evaluation research - Strategic Alignment: Align evaluation methodology sharing with broader open source strategy #### Community - Maintainers: Encourage adoption of continuous evaluation practices in open source AI projects - Contributors: Create pathways for community contributions to evaluation benchmarks - Sponsors: Support funding for open evaluation infrastructure development - Ecosystem: Coordinate standards for AI agent evaluation across projects ## 🧪 Hypotheses - Continuous evaluation systems that combine offline benchmarks with production trace analysis will become the standard for AI agent development - Open-source evaluation benchmarks like ByBench will drive faster improvement in AI coding capabilities across both commercial and open-source models - Semantic clustering of failure traces provides more actionable insights than traditional log analysis for AI agent debugging - Human judgment and product taste remain irreplaceable in AI agent development despite increasing automation ### 📊 Methodology - Study design: Compare improvement rates between teams using traditional vs continuous evaluation approaches - Data sources: Track ByBench adoption and performance improvements across different model families - Experiments: A/B test automated fix generation vs manual debugging for common agent failure patterns ### ❓ Questions - How can the WordPress community adapt these continuous evaluation principles for content management and site building agents? - What specific metrics beyond functional correctness should AI coding evaluations capture for different use cases? - How do you balance automation with human oversight in production AI agent improvement cycles? ### 🔮 Future Research - Investigate applying continuous evaluation methodologies to other AI agent domains beyond coding - Explore federated evaluation approaches where multiple organizations contribute to shared benchmarks - Research optimal human-in-the-loop integration points for AI agent improvement pipelines ### 💡 Implications - Practice: AI teams must invest in production trace analysis infrastructure alongside traditional benchmarks - Policy: Evaluation methodology transparency becomes competitive advantage when shared openly - Security: Automated fix generation from production data requires careful access controls and validation - Education: AI engineering curricula must include continuous evaluation and AB testing methodologies - Funding: Evaluation infrastructure requires dedicated engineering investment beyond model development ### 🧠 Learning Styles Assessment Reading level estimate: 12th Grade (technical presentation with specialized terminology) #### 👀 Visual learners - Present: Architecture diagrams showing two-pillar evaluation system, clustering visualizations - Improve: Add flow charts of Telescope system, ByBench evaluation pipeline diagrams #### 🔊 Aural learners - Present: Video presentation with clear explanations and Q&A discussion - Improve: Provide podcast-style deep dive on implementation details #### 🗣️ Verbal learners - Present: Technical terminology explained with concrete examples and analogies - Improve: Glossary of evaluation terminology, comparison tables between approaches #### 🧩 Kinesthetic learners - Present: Open-source ByBench allows hands-on experimentation - Improve: Step-by-step implementation guides for trace clustering, interactive demos #### 🧠 Logical learners - Present: Clear system architecture, measurable results, systematic methodology - Improve: Decision trees for evaluation approach selection, formal algorithms #### 🧑‍🤝‍🧑 Social learners - Present: Open source collaboration invitation, community contribution pathways - Improve: Discussion forums for evaluation methodology, peer review processes #### 🧘‍♀️ Solitary learners - Present: Comprehensive documentation and self-contained examples - Improve: Self-assessment rubrics for evaluation system maturity #### 🌿 Naturalistic learners - Present: Evolution of evaluation systems presented as natural progression - Improve: Ecosystem thinking about evaluation infrastructure interconnections ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive link text in references - Improve: Add alt text for any system architecture diagrams #### Keyboard navigation - Present: Video player supports keyboard controls - Improve: Ensure timestamp links work with keyboard-only navigation #### Contrast and readability - Present: High contrast text in transcript and notes - Improve: Verify technical diagrams meet contrast requirements #### Text resizing and reflow - Present: Text-based content responsive to zoom - Improve: Ensure code examples remain readable at 200% zoom #### Media and captions - Present: Full transcript provided for video content - Improve: Add closed captions to original video if not present #### Forms and interactions - Present: Standard web interactions for links and navigation - Improve: Ensure any interactive demos include proper labels #### Standards reference - Meets WCAG 2.2 AA for text content, may need review for any visual diagrams ### 🌍 DEIB Assessment #### Cultural breadth - Present: Technical presentation accessible to global developer community - Improve: Include examples from diverse geographical contexts #### Inclusive language - Present: Gender-neutral technical terminology throughout - Improve: Define specialized AI/ML jargon for broader accessibility #### Neurodiversity support - Present: Well-structured content with clear headings and sections - Improve: Add content warnings for rapidly changing technical concepts #### Representation - Present: Technical focus allows broad participation regardless of background - Improve: Highlight diverse contributors to open source evaluation tools #### Psychological safety - Present: Collaborative tone encouraging community participation - Improve: Explicitly welcome newcomers to AI evaluation field ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Strong logical progression from evaluation challenges to systematic solutions - Inductive strength: Good use of specific examples to support general principles about continuous evaluation - Causal reasoning: Clear causal links between production trace analysis and agent improvement - Analogical strength: Effective comparison between traditional and continuous evaluation approaches #### Fallacies - Appeal to authority: Minor reliance on "frontier models" without defining criteria, but supported by empirical results - Hasty generalization: Some claims about "all AI teams" based on Replit's specific experience, but clearly contextualized #### Tone of voice - Overall tone: Technical but accessible, collaborative and open to community input, confident but not dismissive of challenges ### 🛡️ --- ## Full Transcript [00:21] All right. Hi everyone. I'm Mikael Cata [00:24] president and head of AI at Replit. And [00:27] today I'm going to be talking about our [00:29] both evaluating and improving on a daily [00:31] basis Replit agent at scale. [00:34] And as you know, Replit is a back-coding [00:36] platform for knowledge workers. We're [00:38] one of the top players in the space and [00:40] we've been literally battling with this [00:42] problem for the last you know almost 2 [00:44] years. [00:45] The key difference between I would say [00:47] back-coding has a very broad definition. [00:50] It ranges all the way from being used by [00:52] software developers. But in our specific [00:55] case is even more of an extreme [00:56] definition where you start from just a [00:59] natural language specification of what [01:01] the user wants and literally nothing [01:03] else. So the user expects to go from a [01:06] prompt to a working application, but [01:08] they don't tell us what kind of [01:10] framework they want to use. They don't [01:12] write tests. They just expect things to [01:15] work after you know our the agent run. [01:18] So it means that a lot of the [01:20] evaluations and a lot of the ways in [01:22] which we've been building agents in the [01:24] past especially for software developers [01:26] has to fundamentally change. [01:29] Now, [01:30] how do we track internally the fact that [01:33] the agent gets better on a daily basis [01:34] and is actually aligned with all with [01:36] what our users want? [01:38] This is especially challenging because [01:40] if you think about it, models are [01:42] changing almost constantly and if [01:44] anything the pace of evolution is even [01:46] faster than it was a year ago. So we are [01:48] always sit on shifting grounds. Models [01:51] change fast. We have to change our [01:54] prompts, system prompts, user prompts [01:56] very often, you know, to comply to the [01:58] new features that we're building in [01:59] product. Of course, the tools [02:01] definitions are changing as well. We're [02:03] adding more of them, we're simplifying, [02:05] and we're shipping constantly. Every [02:08] single day we have multiple releases of [02:10] wrap agent, which we put in front of, [02:11] you know, millions of users. [02:15] So, my argument today for this talk will [02:17] be that we have to fundamentally rethink [02:19] how we do evaluations. [02:21] On the left side, you can see how evals [02:25] look like for basically, you know, since [02:26] since we've been working on AI. Um they [02:30] you run your evaluation, you they [02:33] produce a single score, and that score [02:35] is produced by a human. And what the [02:37] human decides based on the score is, my [02:39] agent harnesses better than it was [02:40] before or it regress, or my model is [02:43] better than before, or we should be [02:44] doing, you know, more post training. [02:46] That's how how we've been doing this, [02:47] you know, for a for a long time. [02:49] I argue that we should move more and [02:51] more towards a continuous setting of [02:53] evaluations, where especially when you [02:56] actually have something runs in [02:57] production, and in our case it generates [02:59] millions of traces every day, there is a [03:01] lot of hot fat to be extracted from the [03:03] traces themselves. [03:05] So, what we do is we have a combination [03:07] of fine evals, so very similar to sweet [03:10] bench and other benchmarks that are very [03:11] famous in the space, as well as online [03:14] evaluations based on the on the system [03:16] usage. [03:17] What do we want our evals to capture? [03:19] Fundamentally, first of all, they need [03:21] to optimize what our users care about, [03:24] you know, rather than just telling us if [03:26] the agent is editing code correctly. [03:29] They should also highlight immediately [03:30] what is breaking, and they should inform [03:33] us where what we should be shipping next [03:36] in terms of improvement for our agent. [03:39] What goes into this [03:41] box, this black box? So, we're building [03:43] a new system that allows us to do these [03:45] constant improvements. [03:47] The system looks in the following way. [03:49] We have two pillars. On one end, we have [03:53] the standard old-school benchmarks. They [03:55] are usually employed before shipping a [03:59] new version of the agent. Think of them [04:00] sort of as a boolean flag. It's a [04:02] gatekeeper before deciding if you should [04:04] be launching a new version or not. So, [04:06] if there is a major regression on an [04:08] offline benchmark, of course you stop [04:10] your release. If you see no changes or [04:13] positive changes, you know that you can [04:14] go on. [04:15] But the part that is more intriguing and [04:17] is what is helping us to make constant [04:19] progress is the fact that we do a lot of [04:22] baby testing and we also cluster all the [04:25] traces that we obtain in production and [04:27] we try to gain insights from them. So, [04:29] in this case, [04:31] the online pillar actually works after [04:33] you ship a version of the agent and it [04:35] forces you to react as fast as possible. [04:39] And the loop here is, once I have these [04:41] two pillars in place, you reflect on the [04:43] results that both of them are giving [04:44] you, you likely change the code once [04:46] again, run your eval, rinse and repeat. [04:49] And this is the cycle that we go we are [04:51] going through every single day at [04:52] Replit. [04:55] I mentioned before SweetBench and I want [04:58] to give you a quick understanding why [05:00] we work on something, you know, more [05:02] powerful than SweetBench or at least [05:04] like more tailored towards our use case. [05:07] Um something as SweetBench, HumanEval, [05:09] they played a key role in our space, you [05:12] know, to allow us to air climb them and [05:14] make models more powerful. But [05:16] eventually, they all follow the same [05:17] protocol. [05:18] Uh you make sure that the code that is [05:20] generated complies with the user prompt, [05:22] then you apply the patch on your [05:24] repository, you run the test. If the [05:26] tests are passing, you know, you you [05:28] have a higher score on the benchmark. [05:30] This does not reflect what happens in by [05:32] coding. As I was mentioning before, [05:34] users are not writing the tests. They [05:36] often start from a completely empty code [05:38] base. So, there is not a scenario where [05:39] you can just apply patches. You're [05:41] building things from the ground up. [05:43] So, what we need to capture in instead [05:46] is does the app do what the user asked? [05:48] So, there is a functional correctness [05:51] gap between what SweetBench provides and [05:53] what we were working on. [05:55] And today on stage, I'm launching [05:59] ByBench. It's a new public benchmark for [06:02] back-coding end-to-end that we worked on [06:05] at Raptic for several months. I'm going [06:07] to give you all the details after the [06:08] URL and so forth, but let's start from [06:10] the setup of the benchmark. The input is [06:14] just a PRD. It's literally like a long [06:16] prompt that describes how to build an [06:18] application. [06:19] Rather than crafting them from synthetic [06:22] data, we pick 20 real-world traces from, [06:26] you know, usage of Raptic, just plain [06:28] English specifications [06:30] without any implementation constraints. [06:32] Then, we have an harness that builds the [06:35] application end-to-end. So, it goes from [06:38] an empty repo into something functional. [06:40] And then, the key insight is rather than [06:43] stopping the benchmark here and then [06:45] having humans evaluating the output of [06:47] those PRDs, we built automated [06:49] evaluators. This is what allows you to [06:52] go from a benchmark that you run maybe [06:54] on a weekly basis into something that [06:56] you can run every single time you [06:57] literally have a new PR merge in your [06:59] repository. So, of course, as you can [07:01] imagine, we have AI running all the [07:03] evaluations on our behalf. [07:06] The [07:07] the setup is the following. There is one [07:10] core, and the protocol is you have an [07:13] you have a input, you have an [07:15] implementation strategy, and then you [07:17] have a set of our core, sorry, our wired [07:19] evaluations that you run on your app. [07:21] And we came up, you know, with five [07:23] different pairings, but in reality, [07:26] there are a lot more pairings that you [07:27] could come up with yourself. The most [07:29] basic pairing is the input is the PRD, [07:32] and we build the application single shot [07:35] zero to one. [07:36] Then, maybe in order, you know, to go [07:39] from uh less complicated to more [07:41] advanced, we have something called by [07:43] bond ref or like the reference [07:44] implementation where we start from [07:46] something is already working and we [07:48] build a feature on top of that. [07:50] Let's make it more advanced. We have by [07:52] bond vibe where we start from an agent [07:55] on MVP and then we build a new feature [07:57] on top of that. Maybe this is something [08:00] that in the X lingo you would call slop [08:02] on slop. So if we have something that's [08:04] been built by the agent not verified and [08:06] then you had more agent code written on [08:08] top of that and then you run the [08:10] evaluations to see if it's working or [08:11] not. You can keep making this [08:14] arbitrarily more complex. Uh for example [08:16] on in agent four that we launched a [08:18] couple of months ago, we completely hide [08:20] the complexity of doing task [08:22] decomposition and running agents in [08:24] parallel and then merging all the [08:26] patches together. So [08:28] we can also take as an input the PRD, we [08:30] do the task decomposition, we run all of [08:33] that in parallel and then we run the [08:34] valves after that. As you can imagine [08:36] the parallel plus merge scenario is far [08:39] more uh you know, challenging for a [08:41] standard coding agent. [08:43] I give you here some more ideas we [08:44] describing in the paper, you know, but [08:46] you can even start from a buggy [08:47] application, build a feature on top of [08:49] that and see how much your agent [08:50] struggles, you know, to actually make it [08:52] make it work correctly. [08:55] So the the key question or the [08:57] complexity here is how do you actually [08:59] do the grading? And that's what I told [09:01] you before, you know, was where we spent [09:02] most of our effort. It turns out that [09:05] when we're working on our previous [09:06] version of the agent, we put quite a lot [09:08] of effort in building an automated, you [09:10] know, app testing feature which [09:12] literally doesn't know anything about [09:14] the app per se. Cuz if you think about [09:16] it, when it comes to web coding, you we [09:19] don't give any level of [09:21] uh you know, guardrails to our users. [09:23] They can decide the language they want [09:25] to use. We can decide a different [09:26] framework. So the evaluator has to be [09:29] totally agnostic on how the [09:30] implementation looks like. [09:32] And what our evaluator [09:34] evaluator agent does is it reads the [09:36] code base, it then opens a browser and [09:39] points it to the application that our [09:41] agent has built and then step-by-step [09:44] goes through our testing plan. And even [09:46] the testing plan is expressed in natural [09:48] language and the actions are like open [09:51] the admin dashboard and log in with a [09:52] certain account and click on this [09:54] toggle. And if any of those steps fails, [09:57] then you know, we collect all of them [09:58] together and we generate a score. So, we [10:01] go through all these test plan and then [10:03] we decide ultimately if the score is [10:05] good or not. So, this is the key [10:08] complexity. When it comes to SweetBench, [10:11] we always had a fixed surface. We know [10:13] the repositories, we know the task [10:14] hardness, we know exactly how to make [10:16] this work. In ByBench, the surface is [10:18] completely green field and that's why it [10:21] took us several months of work to get [10:23] there. [10:24] Now, as I mentioned, we're announcing it [10:27] today. Here's the QR code. You can find [10:30] the entire benchmark available open [10:31] source on bybench.ai. [10:34] I'll let you to dig into the paper. You [10:36] know, we [10:37] it's going to be presented a couple of [10:38] weeks from today at the conference on AI [10:40] and energetic systems. [10:42] Peter, the you know, lead of the project [10:44] and first author of the paper is sitting [10:46] here. So, after my talk, please come and [10:48] have a chat with him if you want more [10:49] details. [10:50] I'm just going to give us sneak peek of, [10:52] you know, the most notable results [10:53] there. [10:54] First of all, we are noticing almost a [10:57] 2x gap between the frontier models and [10:59] the open weights models. [11:01] And [11:02] the reason why I want you to pay [11:03] attention there is how I think it [11:05] because we know that [11:07] every single model player as every [11:09] single model builder is hit climbing [11:11] specific benchmarks. That's why we [11:13] release this open source. I want the [11:15] entire community to embrace it across [11:17] open weights and closed weights and make [11:19] sure that you know, we make progress [11:21] overall on by coding as well. [11:23] The other one, possibly not very [11:25] surprising is most models do a worse job [11:28] when extending their own code. So, the [11:30] slop on slop or by bomb vibe scenario [11:33] that I was describing to you before is [11:35] by far the most challenging. And that's [11:37] the reason why I always advocate to have [11:39] a testing step in between every time you [11:41] create a new feature, otherwise you keep [11:43] building on shaking foundations and [11:45] eventually your vibe coding application [11:47] is going to fail. [11:50] I'm going to give you back the QR code [11:51] later on. But, you know, let let's move [11:53] to the to the next step. [11:55] We talk about offline eval. I talked to [11:58] you about vibe bench, but as you recall [12:00] there were like two main pillars that we [12:01] care about. And the reason why we also [12:04] care about the online one is because [12:07] there is a great difference in terms of [12:09] volume of signal that we can collect. [12:12] So, on the offline or right now we're [12:14] working roughly 20 applications. The [12:16] reason, by the way, why we made it open [12:18] is because we are more than welcoming [12:21] contributions from the community. So, [12:23] more applications, harder prompts, we [12:25] can keep making this benchmark arbitrary [12:28] harder so that we make our traffic's [12:30] life harder to actually come with over [12:31] time. [12:32] But, [12:33] when it comes to Rabbit Engine instead, [12:35] you know, we're we're collecting [12:37] millions of sessions every day and [12:38] they're very valuable because they [12:40] capture what our users actually do on [12:42] the platform. And they're completely [12:44] unscripted, the agent is always running. [12:46] So, how do we distill useful information [12:49] out of them? [12:50] Well, it turns out that we run a lot of [12:54] AB tests. It's basically our way to keep [12:56] ourselves honest because vibe bench only [12:58] tells us part of the story. [13:01] And I invite every agent builder out [13:03] there to start to invest on their AB [13:04] testing infrastructure as soon as [13:06] possible. It's the best way in which you [13:08] can make steady progress. [13:10] Now, we ask a wide variety of questions [13:13] for ourselves. So, we we built a lot of [13:15] different instrumentation and metric in [13:17] our agent so that for instance, we can [13:19] find out is there a normal span ongoing [13:22] or is the agent running for longer than [13:23] expected? Or we constantly keep track of [13:27] the user sentiment. It's it's a very [13:29] easy thing of to collect because every [13:30] time you receive a prompt, you can do [13:32] sentiment analysis on it and then [13:34] receive the user is getting frustrated [13:36] or not. And last but not least, this is [13:39] very idiosyncratic to Rapid. Our users [13:41] can also publish their application on [13:43] our product. So, it's a very strong [13:46] positive signal if they decide that [13:47] whatever they built is worth to share [13:50] with their colleagues or to put it in [13:51] public in front of everyone. So, all [13:53] these signals clustered together gives [13:56] us something like this. If you have ever [13:58] done an AB test in your life, I'm sure [14:00] this looks familiar. [14:02] And [14:03] it might look familiar especially [14:04] because not everything is either green [14:07] or red. This is the, you know, hard [14:09] harsh truth about running an AB test. [14:12] They never give you a crystal clear [14:14] signal of what you should be doing. So, [14:16] in this case for example, uh the average [14:19] run duration of our agent, you know, [14:20] went up by 7% but conversely is 8% [14:24] cheaper and we saw the fluctuation in [14:26] terms of positive and negative [14:28] sentiment. [14:29] What shall we do in this case? This is [14:31] where human taste and product philosophy [14:34] still plays a key role. You will never [14:37] get or very rarely do we get like a [14:39] crystal clear configuration of the [14:41] results of your AB test. [14:43] And [14:45] to make our life a bit easier on how to [14:47] generate these AB test candidates, what [14:49] we do is we cluster all the traces that [14:52] we receive every single day and night as [14:54] you can imagine and we try to identify [14:57] first of all, which cluster are [15:00] capturing normal nominal behavior of the [15:03] agent. And as you can imagine, there are [15:04] plenty of them because the vast majority [15:06] of times the application is actually [15:08] successful but of course there is a long [15:10] tail of problem that it happens to [15:12] surface. [15:13] In case we find a problematic cluster, [15:15] then what do we do? We embed all the [15:17] failure summaries. [15:19] We [15:21] we cluster them by type so that we find [15:23] out the different failures that are [15:24] happening at a certain point in time. [15:26] Then we pass them through an LLM to [15:28] classify exactly what is going on. And [15:31] the fact that we do this [15:33] not based on regex on logs or like very [15:35] deterministic techniques is what [15:37] actually makes a difference because you [15:39] will be able to cluster things that are [15:41] semantically close to each other even [15:43] though the agent doesn't give you always [15:45] the same type of output. [15:47] And you know, once once we find these [15:50] uh cluster configurations, we are forced [15:52] to retrain them every single night [15:54] because we are running several versions [15:56] of the engine in parallel. We ship many [15:58] versions every single day, so we can't [16:00] live with a fixed cluster configuration. [16:05] And aside, you know, from also helping [16:07] us to identify the failure modes, [16:09] another interesting feature is the fact [16:11] that once we retrain those clusters, we [16:14] can go back and see if after we believe [16:17] we fixed a problem, a certain cluster is [16:19] actually disappeared. So, say you have a [16:21] certain tool failure that is happening [16:23] only a limited amount of time. So, not [16:26] something that will show up in your [16:27] DataDog dashboard, for example, not a [16:29] failure that happens 50% of the times, [16:32] but maybe it was happening 1% of the [16:34] times in certain specific conditions. [16:37] And from here, you will start to [16:38] realize, okay, I shipped my new PR and [16:41] that cluster is disappeared. So, you [16:43] start to have some evidence that you [16:44] mitigated the problem. [16:48] The [16:50] The technology that we built internally [16:53] was called Telescope and it's a fairly [16:56] simple loop, I hope, after I explain [16:57] everything. Uh first of all, we start by [17:00] discovering the problems. And this is [17:02] based on the trace clustering that I was [17:05] talking about before. [17:06] Then, based on that, we create code [17:09] changes. So, it's completely automated. [17:11] We basically cut PRs with a coding agent [17:14] based on the information we got from the [17:16] trace, based on information we collected [17:18] from the logs, from all the different [17:19] different dashboards that we had. [17:21] Then we evaluate if the change we [17:24] generated, first of all, is is is a [17:26] breaking change. So, we rerun by bench. [17:29] By bench is kind of like a litmus test. [17:31] If the score drops by 10 points, we [17:33] decide that the change is bad. [17:35] If it's a controversial change that we [17:38] believe could affect also negatively our [17:40] agent, we run an AB test. So, we put it [17:43] in front of the users and we try to find [17:44] out what are the trade-offs. If it's a [17:47] clear sure shot, then we just ship it in [17:49] production. [17:51] And [17:52] few times when we believe that the [17:54] hypothesis was correct and maybe the PR [17:57] was not perfect, we keep iterating on [17:59] the problem. Maybe maybe we run another [18:01] AB test until we are in a good [18:03] configuration and then we actually ship [18:05] it. [18:06] If you think about it, this is very [18:07] similar to the work you do as an AI [18:09] engineer all the time, but 90% of it is [18:12] now aided by an agent in the loop, you [18:15] know, rather than you having to sweat [18:16] about every single step of the process. [18:20] Let me give you all an example of, you [18:21] know, what we actually experienced, you [18:23] know, once in production. [18:25] Um [18:26] Rabbit Agent immediately starts an [18:28] execution environment the first time you [18:30] submit a prompt. And of course, to set [18:32] up everything, especially with the level [18:34] of complexity we built, requires, you [18:35] know, quite a few seconds. [18:37] We had a degradation in the long tail [18:39] where our, you know, setup time was [18:41] longer than expected and our agent was [18:44] ready to roll before the environment was [18:47] fully set up. [18:48] As you know, agents have become really [18:49] eager to fix problems. So, Rabbit Agent [18:52] went on a tangent and started to try to [18:54] fix the environment, you know, on the [18:56] spot. Um [18:57] it turns out that by the nature of [18:59] agents not being deterministic, every [19:02] single debugging session looked a bit [19:04] different. So, if it tried to find out [19:06] this long tail problem just by grabbing [19:08] the logs, it would have almost not [19:10] appeared. [19:11] But the moment we cluster all the [19:13] traces, you know, with basically with we [19:15] we through the semantic layer, then we [19:17] started to figure out that this problem [19:19] was happening quite often, and we [19:21] immediately able to create a patch and [19:24] fix the issue. In this case, we didn't [19:26] have to run an AB test because it was a [19:28] pure regression case. But in case it was [19:31] a problem that required testing, of [19:32] course, we would have put an additional [19:34] step in between. [19:37] And even though I'm a big fan of [19:40] trying to optimize as much as possible [19:42] the life of an AI engineer with agents, [19:45] I do think that there is a lot of [19:48] intellectual work to be done by humans [19:50] still in the kind of loop that I [19:51] presented to you. [19:53] For example, whenever we find one of [19:55] these clusters that I was talking about [19:57] before, as a team, we always focus on [20:00] formulating an hypothesis on what could [20:02] be possibly go wrong there. Cuz the [20:04] truth is when running in production, [20:06] you're going to get a really a large [20:08] amount of candidates of what you could [20:10] be fixing. So, you need to prioritize [20:12] where you put effort, where you put [20:13] resources. So, if the hypothesis is [20:16] intriguing enough, then we move forward, [20:18] and we don't just let the agent write a [20:21] PR without any kind of supervision. We [20:23] actually try to feed some of our [20:24] understanding of what is wrong. So, we [20:26] give it more guidance. [20:28] And ultimately, if you think about it, [20:31] every single time we decide of working [20:34] on a PR and doing an AB test, we're sort [20:36] of like shaping the hill that we try to [20:38] optimize. Cuz if we only care about, [20:40] say, making the product more affordable, [20:43] we're going to go and try to fix all the [20:45] anomalous spend problems, and then we're [20:46] going to do our best to optimize that as [20:49] much as possible. So, [20:51] all these choices really determine the [20:53] product philosophy of what you want to [20:55] put in front of our users. And then, [20:57] last but not least, as I was showing you [20:59] before on the AB test dashboard, if [21:02] there is not a clear result, ultimately, [21:05] if you decide to launch or not, is still [21:07] a choice that a human does. Often times, [21:10] that's that's on me in the rabbit. And [21:12] again, that determines really what's [21:14] going to be the ultimate product [21:15] experience. [21:18] So, my In closing, what I want you to [21:20] have as a takeaway today is don't think [21:22] of evaluation just as this last check [21:25] before shipping. It shouldn't be just a [21:27] boolean flag, but rather think of this [21:29] as an engine that allows you to ship a [21:31] better agent every single day. [21:34] Thanks, everyone. And [21:36] I'm going to invite Hannah on stage so [21:37] we can have a chat about how we've been [21:38] working together in the last few months [21:40] to go. Thank you. [21:42] >> [applause] [21:47] >> Awesome. So, my name is Hannah. I'm part [21:48] of the Applied AI team at Anthropic, and [21:51] I work with Michele [21:52] on the Replit agent and on everything [21:54] that they build with Claude. So, I want [21:56] to ask you a few questions about what [21:58] you built here. My first question is [22:00] Vibez is something that we've seen a [22:02] couple times. You've used it with us to [22:04] give us feedback on research models that [22:06] we're testing. But, what was was your [22:08] vision from the beginning to make [22:10] something that you were going to open [22:11] source and make available to the [22:13] community? I think a lot of people may [22:15] be trying to build evals like this for [22:16] themselves where they see holes in the [22:18] public benchmarks, but I think this is [22:20] kind of a unique approach, and I'm just [22:22] wondering if you could talk about that a [22:23] little bit. [22:24] Yeah, I've been asked this often, why [22:26] don't you keep this as a private appeal [22:28] of your company? Fundamentally, I [22:30] believe that [22:31] we should try to give back as much as [22:33] possible to the community. And we all [22:35] have to benefit from creating public [22:37] evals. It helps you to make better [22:39] models. It helps us to make a better [22:40] agent. It helps everyone to create [22:42] better products. And I I don't believe [22:45] in competing on evaluations. I come from [22:47] a research background where everything [22:49] should be open. [22:50] So, we'll we'll keep doing this for as [22:52] long as we can in Replit. And And again, [22:54] I I really invite people to collaborate [22:57] on Vibez. I I tried to instill this idea [22:59] in the community several times when I [23:01] was giving talks in the past and at a [23:03] certain point I realized we have to [23:04] build it ourselves rather than just [23:06] trying to invite others to do it. Mhm. [23:08] Well, I'm really excited to see what [23:10] people do with that. Um [23:12] I also want to ask you about the second [23:13] pillar, telescope. This is um you know, [23:17] a pretty sophisticated system that [23:19] you've built. It sounds incredibly [23:20] useful. It sounds like something you've [23:22] tuned quite a lot over time. [23:24] For people in the room who might be [23:26] trying to build something similar like [23:28] this, what lessons did you learn along [23:31] the way? What kind of tips could you [23:32] give them about how they can build [23:34] something useful like this? [23:38] So, first of all, if you try to do this [23:40] even, I don't know, 6 months ago and you [23:42] got discouraged because you didn't get [23:44] like a return on your investment, uh [23:46] definitely try it try it again. The same [23:50] inflection point that you experienced [23:51] with coding agents back say in November [23:53] with Opus 4.5 and and similar frontier [23:56] technology, uh it also reflects to what [23:59] I just talked about now. So, the the [24:01] fact that long context and models that [24:04] are really capable of reasoning on a lot [24:06] of content means that you can [24:08] practically inject an entire trace of [24:10] your agent into Opus and get fairly [24:13] sophisticated feedback about it. Um so, [24:17] you at the same time you should really [24:19] be investing in collecting all the [24:21] signal that you can from your agent and [24:23] as well as you know, the environment [24:25] where you're executing it. So, I know [24:26] that I I had to run fast during the [24:28] talk, you know, but I couldn't go in [24:30] depth on all the different signals that [24:31] we bring into telescope. It is not [24:33] purely [24:35] what's coming from the trace, it's also [24:37] the feedback that we collect in product. [24:39] Like we have a feedback form, so every [24:40] time the agent doesn't behave correctly [24:43] and the user complains, we have the user [24:45] point of view, we have the trace of the [24:47] agent, we have everything we instrument [24:49] in the platform in DataDog. [24:52] Unsurprisingly, all of that together, [24:54] put, you know, in context, helps even [24:56] more to pinpoint where where the issues. [24:58] Yeah, I think that idea of building the [25:00] clusters, like you were saying that a [25:02] single trace, it's very hard to debug [25:04] what might be going wrong there, but [25:05] when you are able to group them all [25:07] together and get these other signals in, [25:09] really reveals a lot more information [25:11] and something actionable. Makes a ton of [25:12] sense. Yeah, it does and I think it also [25:15] makes the life of an agent builder less [25:17] overwhelming because it really is when [25:19] you start to get a lot of usage, the [25:22] amount of feedback you receive is [25:23] actually overwhelming. And it's a good [25:26] It's a good signal. It means that people [25:27] care about what you're doing, but at the [25:29] same time it's hard to prioritize what's [25:31] actually important versus what doesn't [25:33] move the needle. [25:34] What I showed today is also a way for us [25:36] to try to understand, okay, this cluster [25:39] keeps showing on a daily basis and it's [25:41] fairly large and it has a lot of volume, [25:43] so we should actually fix that first of [25:45] all and then there is the long tail of [25:46] problems to fix will always be there [25:48] with agents given the fact that they're [25:49] non-deterministic. Yeah, that really [25:52] makes me think about the third thing I [25:53] wanted to ask you, which is what you [25:54] said about taste. [25:56] Uh, I think this is really interesting [25:57] point um that the taste of the AI [26:00] engineer is very important and could you [26:02] talk about how your team develops that [26:05] taste? Have you always had this taste or [26:08] have you improved your taste over time? [26:10] Like what's the process of building that [26:12] and developing that for people who might [26:13] be trying to build their own taste? [26:16] I think we all brewed it over time [26:18] because you know, when we launched [26:20] these, you know, was more than a year [26:21] and a half ago, agents were way less [26:24] powerful than they are today. [26:25] Uh, maybe not much taste to be applied. [26:27] It was more like a survival game just to [26:29] make sure it was barely functional. And [26:32] now that they're becoming so powerful [26:34] and you have such a wide variety of [26:36] choices you can make, um then you you [26:38] start to develop a taste that should [26:40] really be in tune with your actual user [26:42] base. Like if if you're building an [26:44] agent for software developers, maybe 80% [26:48] of the choices that we make every day in [26:49] the team will be almost the polar [26:51] opposite of what we do. [26:53] And you always want to keep in mind that [26:56] it's being used by personal likeness [26:57] different from you. Especially in our [26:59] case, like everyone at Rabbit is very [27:00] technical and we are giving the product [27:02] to knowledge workers who have never [27:04] written a single line of code. [27:06] So, it's a it's a very good trade-off to [27:08] to learn. [27:09] Awesome. Thank you for sharing that. [27:11] I'm sure people are going to have lots [27:12] of questions for you and for Peter, who [27:14] is one of the authors of Vibrent. So, [27:16] everyone please come find them and thank [27:17] you so much for coming for [27:19] >> Thank you. [27:20] Have a good one.  📄 **Full text:** ![[Full Document Contents/Articles/Evaluating and Improving Replit Agent at Scale.md|Evaluating and Improving Replit Agent at Scale by YouTube]] ## Highlights - how do we track internally the fact that the agent gets better on a daily basis and is actually aligned with all with what our users want? This is especially challenging because if you think about it, models are changing almost constantly and if anything the pace of evolution is even faster than it was a year ago. So we are always sit on shifting grounds. Models change fast. We have to change our prompts, system prompts, user prompts very often, you know, to comply to the new features that we're building in product. Of course, the tools ([View Highlight](https://read.readwise.io/read/01kr6v66ma21htp8bnp8j93pqm)) - Tags: 👻 ai highlighted - you run your evaluation, you they produce a single score, and that score is produced by a human. And what the human decides based on the score is, my agent harnesses better than it was before or it regress, or my model is better than before, or we should be doing, you know, more post training. That's how how we've been doing this, you know, for a for a long time. I argue that we should move more and more towards a continuous setting of evaluations, where especially when you actually have something runs in production, and in our case it generates millions of traces every day, there is a ([View Highlight](https://read.readwise.io/read/01kr6v66mmcjph7p92hj1vnsdh)) - Tags: 👻 ai highlighted - on slop. So if we have something that's been built by the agent not verified and then you had more agent code written on top of that and then you run the evaluations to see if it's working or not. You can keep making this arbitrarily more complex. Uh for example on in agent four that we launched a couple of months ago, we completely hide the complexity of doing task decomposition and running agents in parallel and then merging all the patches together. So we can also take as an input the PRD, we ([View Highlight](https://read.readwise.io/read/01kr6v66ntmfd07319gdzpbp51)) - Tags: 👻 ai highlighted - And I invite every agent builder out there to start to invest on their AB testing infrastructure as soon as possible. It's the best way in which you can make steady progress. Now, we ask a wide variety of questions for ourselves. So, we we built a lot of different instrumentation and metric in our agent so that for instance, we can find out is there a normal span ongoing or is the agent running for longer than expected? Or we constantly keep track of the user sentiment. It's it's a very easy thing of to collect because every ([View Highlight](https://read.readwise.io/read/01kr6v66pxqymsmgcv8sh70hem)) - Tags: 👻 ai highlighted - AB test until we are in a good configuration and then we actually ship it. If you think about it, this is very similar to the work you do as an AI engineer all the time, but 90% of it is now aided by an agent in the loop, you know, rather than you having to sweat about every single step of the process. Let me give you all an example of, you know, what we actually experienced, you know, once in production. Um Rabbit Agent immediately starts an execution environment the first time you ([View Highlight](https://read.readwise.io/read/01kr6v66rd0r7gfb3qmbycx187)) - Tags: 👻 ai highlighted