Rahul Agarwal – Validating AI Outputs at Live Events #InnerSourceSummit2025

Rahul Agarwal – Validating AI Outputs at Live Events #InnerSourceSummit2025 🔗 **Source:** [Rahul Agarwal – Validating AI Outputs at Live Events #InnerSourceSummit2025](https://www.youtube.com/watch?v=ozy10H8MeVs) > [!note]+ Document Notes > tags: [AI-validation, content-generation, event-technology] > author: Rahul Agarwal > source: InnerSource Summit 2025 > reference: https://www.youtube.com/watch?v=ozy10H8MeVs > date created: 2025-01-16 10:30 > > # 🏷️ Title: Validating AI Outputs at Live Events >  ## 🧰 Action Inbox - [H1] Implement chain-of-thought reasoning in AI content generation — adds "explain your steps" to prompts for better outputs #AI - [H2] Create quantitative content evaluation system — convert qualitative requirements to measurable parameters #validation - [H3] Build iterative feedback loops for content improvement — use LLM-as-judge with defined thresholds #automation ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [InnerSource Summit 2025](https://www.youtube.com/watch?v=ozy10H8MeVs) - [Chain-of-Thought Prompting](https://arxiv.org/abs/2201.11903) ## 📝 Summary - **What happened**: Rahul Agarwal shared IBM Events team's approach to making AI content generation reliable for live sports events, moving from 4-hour manual processes to automated systems - **Why it matters**: AI output is non-deterministic unlike traditional software, creating reliability challenges for autonomous content generation systems that need consistent quality - **Key stakeholders**: Event organizers, content editors, sports media companies, AI developers working on production systems - **Risks and tradeoffs**: Balance between creativity/novelty and factual accuracy, potential for hallucination in live event coverage, complexity of defining quality metrics - **Bottom line**: Three-phase approach using chain-of-thought reasoning, quantitative evaluation, and iterative feedback can significantly improve AI content reliability for time-sensitive applications ## 📖 Main Content Agarwal addresses a critical challenge in AI deployment: the non-deterministic nature of AI systems compared to traditional deterministic computing. While conventional software produces predictable outputs, AI agents can generate different results from identical inputs, making them unreliable for autonomous content generation. The presentation focuses on sports event content generation, where traditional editorial processes take 3-4 hours to produce match summaries. This delay conflicts with social media expectations for immediate content availability after events conclude. The solution involves three phases: enhanced content generation using chain-of-thought prompting, quantitative evaluation through LLM-as-judge systems, and iterative improvement through automated editing cycles. The approach aims to reduce human intervention while maintaining content quality standards. Chain-of-thought reasoning forces the AI to explain its steps, improving both output quality and debuggability. The evaluation system converts subjective editorial preferences into measurable parameters like factualness, novelty, relevance, and repetitiveness. The final phase uses historical editorial decisions to train an automated feedback system. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/ozy10H8MeVs?t=0) — [H1] AI reliability challenges in production - Key idea: AI agents produce non-deterministic outputs unlike traditional deterministic systems - Evidence: "Doing the same thing over and over again and expecting different results - that is what we are doing with AI" #### ⏱️ [01:45](https://youtu.be/ozy10H8MeVs?t=105) — [H2] Content generation pipeline requirements - Key idea: Autonomous systems need reliability without human intervention for scalable content creation - Evidence: "For that particular engine to fit into an autonomous system... it is not at all reliable" #### ⏱️ [03:20](https://youtu.be/ozy10H8MeVs?t=200) — [H1] Human-in-the-loop definition - Key idea: Human validation ensures content meets brand standards and accountability requirements - Evidence: "A human who is going to be taking accountability of whether this output is good enough to put my brand name on it" #### ⏱️ [05:10](https://youtu.be/ozy10H8MeVs?t=310) — [H2] Speed requirements for event content - Key idea: Traditional 3-4 hour editorial process too slow for modern social media expectations - Evidence: "For an average fan that's like four five hours that is history today... we don't want to wait five hours" #### ⏱️ [07:15](https://youtu.be/ozy10H8MeVs?t=435) — [H1] Chain-of-thought prompting methodology - Key idea: Adding "explain your steps" to prompts improves output quality and provides explainability - Evidence: "Very subtle change but very impactful change... you get by far a much better output and explainability" #### ⏱️ [09:30](https://youtu.be/ozy10H8MeVs?t=570) — [H3] Machine thinking for prompt design - Key idea: Designers should think like machines when creating prompts to provide sufficient information - Evidence: "When you're designing AI systems... it's important to think that you are a machine" #### ⏱️ [10:45](https://youtu.be/ozy10H8MeVs?t=645) — [H1] Quantitative content evaluation framework - Key idea: Convert qualitative editorial requirements into measurable parameters for automated assessment - Evidence: "No editor and no human is able to tell what they want... convert qualitative aspects into quantitative way" #### ⏱️ [12:20](https://youtu.be/ozy10H8MeVs?t=740) — [H2] Four evaluation parameters - Key idea: Factualness, novelty, relevance, and repetitiveness provide comprehensive content assessment - Evidence: "We want facts to be right... we like creativity... sticking to facts... nobody wants to read repetitive stuff" #### ⏱️ [15:30](https://youtu.be/ozy10H8MeVs?t=930) — [H2] Historical data for baseline creation - Key idea: Use past editorial decisions to establish acceptable quality thresholds rather than asking humans to articulate preferences - Evidence: "We take all the generated content for 2023... ask them to review it, approve it or edit it" #### ⏱️ [17:10](https://youtu.be/ozy10H8MeVs?t=1030) — [H3] Iterative improvement process - Key idea: Automated editing cycles use feedback to converge on acceptable content within defined thresholds - Evidence: "Run it 21 times... keep doing it until convergence is reached" #### ⏱️ [18:45](https://youtu.be/ozy10H8MeVs?t=1125) — [H2] Convergence patterns and trade-offs - Key idea: Parameters like factualness and novelty are not mutually exclusive, creating iteration challenges - Evidence: "Very difficult to come to a balance... these things are not mutually exclusive of each other" #### ⏱️ [19:50](https://youtu.be/ozy10H8MeVs?t=1190) — [H3] Learning through multiple iterations - Key idea: AI systems learn from back-and-forth editing cycles, similar to human editorial processes - Evidence: "Try bunch of things and then pick the best one... mix and match" ## 📚 References - [Original Presentation](https://www.youtube.com/watch?v=ozy10H8MeVs) - [InnerSource Summit 2025](https://innersourcecommons.org/) - [Chain-of-Thought Research](https://arxiv.org/abs/2201.11903) ## 🗣️ Paraphrase - AI systems require new validation approaches because they produce unpredictable outputs unlike traditional deterministic software, making autonomous content generation challenging for production environments - Sports content generation faces time pressure where traditional 3-4 hour editorial processes conflict with social media's demand for immediate post-event content availability - A three-phase validation system using chain-of-thought reasoning, quantitative evaluation metrics, and iterative feedback loops can improve AI content reliability while reducing human intervention requirements ## 📰 Newsletter **Making AI Content Generation Reliable for Live Events** AI-powered content generation promises faster turnaround times, but reliability remains a major challenge for production deployment. Rahul Agarwal from IBM's Events team shared practical solutions for validating AI outputs at live sports events. **Key Innovation**: A three-phase approach combining chain-of-thought prompting (adding "explain your steps"), quantitative evaluation metrics, and automated feedback loops reduced content generation time from 4+ hours to minutes while maintaining quality. **Practical Application**: The system converts subjective editorial preferences into measurable parameters (factualness, novelty, relevance, repetitiveness) and uses historical editorial decisions to establish quality thresholds. **Why This Matters**: As AI becomes central to content workflows, systematic validation approaches enable autonomous operation while preserving brand standards and editorial quality. Resources: [AI Content Strategy](https://www.ibm.com/ai), [Event Technology Solutions](https://developer.ibm.com/), [Content Automation](https://www.ibm.com/watson) ## ✅ Action Items #### DevRel - Developer advocacy: Create case studies on AI validation patterns, present at ML conferences - Education: Develop workshops on prompt engineering and LLM-as-judge implementations - Developer experience: Build SDKs for content evaluation and iterative improvement workflows - Developer marketing: Highlight reliability solutions for AI production deployments - Community management: Foster discussions on AI validation best practices #### OSPO - Policy & Governance: Establish guidelines for AI system validation and quality assurance - License Compliance: Review AI model licensing for content generation applications - Security Management: Address AI-specific vulnerabilities in automated content systems - Software Inventory: Track AI models and evaluation frameworks in use - Developer Support: Provide training on responsible AI deployment practices - Community Engagement: Support open source AI validation tools and frameworks #### Community - Maintainers: Develop governance models for AI-assisted content workflows - Contributors: Create evaluation datasets and benchmark tools for content quality - Sponsors: Fund development of open validation frameworks and tools - Ecosystem: Coordinate standards for AI content evaluation across platforms #### WordPress - Core: Explore AI content assistance with validation safeguards - Core AI: Implement evaluation frameworks for AI-generated content features - Community: Develop guidelines for AI-assisted community content - Training: Create curriculum on AI content validation and quality assurance - Documentation: Establish standards for AI-generated documentation review ## 🧪 Hypotheses - Chain-of-thought prompting with explicit step explanation improves AI content quality by 20-40% compared to direct generation - Quantitative evaluation parameters can reliably predict human editorial approval rates with 85%+ accuracy - Iterative feedback loops converge to acceptable quality thresholds within 5-10 iterations for most content types - Historical editorial decision data provides better quality baselines than explicit human preference articulation ### 📊 Methodology - A/B testing comparing chain-of-thought vs direct prompting across content types and domains - Correlation analysis between quantitative metrics and human editorial approval rates - Time-series analysis of iteration convergence patterns across different content categories - Comparative evaluation of baseline establishment methods using historical vs survey data ### ❓ Questions - How do evaluation parameters need to be adjusted for different content domains beyond sports? - What are the computational costs and latency implications of running 20+ iteration cycles? - How can the system handle conflicting stakeholder preferences for content style and tone? ### 🔮 Future Research - Cross-domain validation of the three-phase approach beyond sports content - Integration patterns with existing content management and editorial workflows - Scalability analysis for high-volume content generation across multiple simultaneous events ### 💡 Implications - **Practice**: Content teams need new skills in prompt engineering and AI evaluation system design - **Policy**: Organizations require governance frameworks for AI content quality and brand compliance - **Security**: Automated content systems need safeguards against hallucination and misinformation propagation - **Education**: Editorial and content creation curricula must include AI validation methodologies - **Funding**: Investment needed in evaluation framework development and human-AI workflow optimization ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (Flesch-Kincaid) #### 👀 Visual learners - Present: Process flow diagrams, parameter scoring tables, iteration convergence charts - Improve: System architecture diagrams, content quality visualization dashboards #### 🔊 Aural learners - Present: Video presentation with clear explanations and examples - Improve: Podcast version focusing on implementation challenges and solutions #### 🗣️ Verbal learners - Present: Clear definitions of technical terms, analogies to traditional editing - Improve: Glossary of AI validation terminology, comparison frameworks #### 🧩 Kinesthetic learners - Present: Step-by-step implementation examples, iteration process walkthrough - Improve: Hands-on workshops, interactive evaluation parameter tuning exercises #### 🧠 Logical learners - Present: Systematic three-phase approach, quantitative metrics, convergence analysis - Improve: Decision trees for parameter weighting, algorithmic flowcharts #### 🧑‍🤝‍🧑 Social learners - Present: Team collaboration on editorial standards, stakeholder alignment processes - Improve: Group exercises on evaluation criteria definition, peer review workflows #### 🧘‍♀️ Solitary learners - Present: Technical implementation details, individual parameter analysis - Improve: Self-assessment tools for content quality, personal learning exercises #### 🌿 Naturalistic learners - Present: Evolution from traditional to AI-assisted editorial workflows - Improve: Ecosystem thinking about content creation lifecycles ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading hierarchy, descriptive technical explanations - Improve: Alt text for process diagrams, table headers for metric data #### Keyboard navigation - Present: Video controls accessible, clear focus indicators - Improve: Transcript navigation, interactive demo keyboard access #### Contrast and readability - Present: High contrast text, readable font sizes in presentation - Improve: Enhanced contrast for diagram elements, larger text in code examples #### Text resizing and reflow - Present: Responsive video player, scalable content layout - Improve: Flexible diagram scaling, mobile-optimized code samples #### Media and captions - Present: Full transcript provided for video content - Improve: Synchronized captions with technical term highlighting #### Forms and interactions - Present: Simple video interaction model - Improve: Accessible parameter adjustment interfaces for demos ### 🌍 DEIB Assessment #### Cultural breadth - Present: Technology solutions applicable across global event contexts - Improve: Examples beyond Western sports, multilingual content validation #### Inclusive language - Present: Professional technical terminology, gender-neutral examples - Improve: Plain language summaries, cultural context for sports references #### Neurodiversity support - Present: Structured presentation format, clear section divisions - Improve: Content warnings for fast-paced technical concepts, processing time considerations #### Representation - Present: Focus on technical solutions rather than demographic representation - Improve: Diverse use case examples, varied industry applications #### Psychological safety - Present: Acknowledges learning challenges, "hard way" experiences - Improve: Normalize iteration and failure as part of AI development process ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear logical progression from problem identification to solution implementation - Inductive strength: Generalizes from sports content to broader AI validation challenges - Causal reasoning: Establishes clear cause-effect relationships between validation steps and quality improvement - Analogical strength: Effective comparison between human editorial processes and AI iteration cycles #### Fallacies - Appeal to authority: References Steve Jobs quote without contextual justification for AI domain applicability - False dichotomy: Presents human vs AI as binary when hybrid approaches may be optimal #### Tone of voice - Overall tone: Professional and technical with practical focus, acknowledges challenges honestly while maintaining solution-oriented approach ### 🛡️ DARVO --- ## Full Transcript [00:05] Hello everyone. Um thank you thank you [00:07] so much for having me here. Uh today I'm [00:10] going to talk about a topic which um [00:13] which is really important in terms of [00:16] being able to adopt AI solutions in uh [00:19] the real world right and u this is [00:22] something I would say uh we learned uh [00:25] as a team at events team uh very um I [00:30] would say the hard way right um so I [00:33] would ask uh to the folks present here [00:35] right like how many of you all have [00:38] actually worked worked with AI agents or [00:40] try to deploy them in production. [00:44] So you guys know how much of a nightmare [00:47] it is to get a reliable output. Do you [00:51] guys agree with that? Right. And u we [00:55] come from like prei we come from a world [00:59] which is very deterministic. Right. And [01:01] what does deterministic mean is it's a [01:04] computer and what I say to do that is [01:08] what it does. Right? So everybody [01:10] including your non-coders to your coders [01:12] are very used to that sort of paradigm. [01:17] But now with AI agents and AI kind of [01:20] like taking a lot of the center stage in [01:22] big applications, it has become very [01:25] non-deterministic. [01:26] Uh there's a very popular uh [01:30] definition of insanity, right? And it [01:33] says that doing the same thing over and [01:37] over again and expecting a different [01:39] results that is commonly referred to as [01:42] insanity. But friends, that is what we [01:45] are doing with AI. [01:48] Right? So [01:50] the task that we wanted to solve for is [01:54] content generation and in content [01:56] generation uh it's a big aspect where [02:00] everybody thinks oh wow LLM can generate [02:03] content it can generate blog post it can [02:05] generate social media post but what uh [02:08] nobody realizes this is for that [02:10] particular engine to fit into an [02:14] autonomous system a system that is going [02:16] to be running u without any [02:19] interventions to be running again and [02:21] again on multiple and generating like a [02:24] pipeline and sort of a factory of [02:27] content creation. It is not at all [02:29] reliable and that is primarily because [02:33] you don't know what it's going to [02:34] create, right? And once it has created [02:37] it, you always want a human in the loop. [02:41] So what does human in the loop mean? [02:43] Right? I feel like uh there there was a [02:44] question here uh today about what is [02:46] human in the loop. So human in the room [02:49] commonly refers to as a human who is [02:51] going to be taking a decision of an [02:54] accountability of sorts of whether this [02:57] output is good enough to put my brand [02:59] name on it. Right? So whether it is for [03:01] personal use, whether it's for IBM, [03:03] whether it's for any uh big event such [03:05] as Wimbledon or US Open, if they are [03:08] posting any content, it is on them and [03:11] their accountability and their rel uh [03:15] expectation of that level of event. they [03:19] are that what they are posting is [03:22] accurate is uh uh something that is [03:25] relevant and something that everybody [03:27] wants to know about and something that [03:29] they are willing to easily be able to [03:31] put forth. Right? So that is where a [03:34] human in the loop comes into picture [03:36] where if there's an autonomous system [03:38] that is generating this content this [03:40] human in the loop is going to validate [03:43] this read through this sort of like an [03:45] editor where his he or she is uh [03:48] modifying the content making sure it is [03:50] good enough and if it is not good enough [03:52] rejecting it right this is not good [03:55] enough quality this is not the brand [03:57] this is not the uh brand aesthetics that [03:59] we want to go with so we do we will not [04:01] be posting this Right? Something that [04:04] sort of [04:06] wall of human intervention [04:10] that is what our uh human in the loop is [04:13] going to be doing. Now the problem [04:16] statement that we are working with [04:19] starts there where we want the AI to be [04:22] more and more reliable to be able to [04:25] generate content that does not need so [04:28] many iterations does not need uh to be [04:32] modified and be like oh no no no this is [04:34] not what I meant please generate again [04:37] or remove this point this is not good [04:39] tell me some other ideas you know so [04:41] this sort of uh back and forth is [04:44] something that you are you we do a lot [04:47] in on charge GPT on Plexity and all the [04:50] other chat assistants but in an [04:52] autonomous system where we are writing [04:54] code for an agent to be able to do this [04:57] we want that to happen autonomously as [04:59] well. Now uh so what we want right what [05:04] we want that we want to get to the [05:06] required output with as many uh less [05:09] tries as we can. The next thing we want [05:12] is we know nothing is perfect right like [05:15] how muchever we say okay do it again do [05:17] it again we still be expecting that [05:20] there would might be need where be a [05:22] human there might need to be any manual [05:24] edits required for it to be of quality [05:26] content but we want it to be minimal [05:30] right we want it to decrease the amount [05:31] of effort it takes and then why are we [05:34] trying to do this in the first place one [05:36] of the reasons why we do this is faster [05:39] content generation right like imagine is [05:41] a tennis match. Now this uh Yanik Sar [05:44] and Alcarez right they have gone and uh [05:48] hashed it out on the court. Everybody is [05:50] engrossed. It's been like six hours and [05:52] the the match has ended with a [05:54] phenomenal win for either one of the [05:56] players. Now there are people who and [05:58] fans who love tennis who are able to [06:01] come together and watch it live, enjoy [06:03] that action. But there are a lot of [06:06] other fans who for whatever reasons [06:08] probably attending this summit today are [06:10] not able to watch the match live right. [06:13] So we so the organizations and uh media [06:17] houses generate blog post and different [06:19] types of content. [06:22] A typical uh time it takes to generate a [06:25] content post any match is 3 to four [06:28] hours. That is because the editors and [06:31] the journalists are going through the [06:32] whole match trying to identify okay what [06:35] was nice what can we talk about and [06:37] because it's not going to be a huge [06:39] novel it's going to be a small piece [06:42] where they want to highlight the key [06:44] moments the key times in which the match [06:46] shifted and at the same time they want [06:48] to make it engaging enough for uh fans [06:51] to be able to read it. So that is what [06:54] takes like I would say 3 to four hours [06:56] and then you'll take another 30 to 45 [06:58] minutes where they go through an editor [07:00] cycle and then the editor the chief [07:02] editor edits it and makes it available. [07:05] So for an average fan that's like four [07:07] five hours that is history today right [07:10] like in social media we are used to [07:12] things coming in a snap so we don't want [07:14] to wait five hours after the match has [07:16] ended to know what actually happened. [07:19] So this is where we want to gen that is [07:22] why we want this AI system and that is [07:24] where this AI system will be helpful. [07:27] Now how do we solve for it? Right? The [07:30] way we are solving for it is I would [07:33] split it in three phases. Right? First [07:35] is a content generation part where we [07:37] want to make sure that the first content [07:40] that we create is actually good enough. [07:43] Right? And for this we'll be leveraging [07:45] something called a chain of thought uh [07:47] reasoning. And uh to give you an example [07:51] of what does chain of thought mean [07:53] right? So if you give any large language [07:55] model a prompt something like this where [07:57] you're saying write a two sentence [07:59] product that describes a smartwatch for [08:02] kids right now this particular prompt is [08:06] good but it's not good enough that the [08:10] LLM may or may not work with the with [08:13] giving the desired output right it would [08:15] give something very generic something [08:17] like this is a smart watch it's colorful [08:19] fun for kids to use it use it you know [08:23] But next what we're going to do is we're [08:25] going to add something called uh chain [08:28] of thought where we force the LLM to [08:31] explain your steps. So that is the only [08:33] change we are going to do. It's a very [08:35] subtle change but a very impactful [08:37] change that can be done to any of the [08:40] prompts that you're trying to use where [08:42] you add this thing called explain your [08:44] steps. Now in this steps when LLM is [08:46] trying to generate and predict the next [08:48] word it is creating a reasoning and in [08:52] turn making the content better for [08:54] itself. So when it generates like okay [08:57] so we want to generate a product the [08:59] product is smart watch for kids and what [09:01] are the key requirements we want it to [09:03] be focused on safety we want to make it [09:05] sound fun. Now what does safety mean? [09:08] Right? Safety means GPS tracking, [09:10] parental alerts and what does fun means? [09:14] Games, color uh colorful uh colors. That [09:16] is something that it attributes as fun. [09:19] And then using all of these, it [09:21] generates a much better output. Now [09:23] there are two things that happens with [09:25] such a explanation. Number one is you [09:29] get by far a much better output. But [09:32] number two is you get explanability. you [09:35] understand what is going on and why this [09:38] particular result has come. So for [09:40] example uh if we do not like this [09:42] output, we know what to change. We want [09:44] to describe what fun means. So if we say [09:46] oh he's saying the LLM is considering [09:48] fun as games and colorful design but [09:50] that is not fun is fun is probably [09:52] something else based on the definition [09:55] of the product itself or we want to [09:58] introduce characters or designs of sorts [10:00] which are more specific to the watch and [10:02] the product. So we know to go back on [10:05] the prompt and modify it in such a way [10:07] that we can get a more reliable output. [10:09] So it is it works in two ways where it [10:11] helps in creating better content but at [10:13] the same time it also helps you debug [10:15] and make uh create a better prompt [10:18] right. So um one of the things that uh [10:21] we were talking about earlier in one of [10:23] the sessions is uh [10:26] um how do we uh so that we should not we [10:29] are not machines, humans are not [10:31] machines, right? And uh I agree with [10:33] that completely. No, no wonder, right? [10:37] But I will tell this where there's a [10:39] caveat to it is that when you're [10:41] designing AI systems and when you're [10:43] designing prompts such as this, it's [10:45] important to think that you are a [10:47] machine, right? Uh I'm definitely [10:49] contradicting here but it's just to [10:51] understand that for an LLM if you think [10:54] if you are an LLM and you are given this [10:57] prompt to think about what to create [10:59] will you be able to do that with that [11:01] limited information if not then add more [11:03] information to the prompt that is going [11:05] to help you create the desired output [11:08] that is going to make uh help you use [11:11] this technology so much more better and [11:13] easier. [11:15] So the next part is uh after so now we [11:18] have generated the content right now we [11:20] are uh to some extent much better from [11:22] where we started for content generation. [11:24] The next thing is we want to evaluate [11:26] the content. Now content evaluation is [11:30] again a really important aspect. Now if [11:33] I go and ask to the stakeholders to the [11:35] editors what are you looking for in [11:37] content they'll say XYZ. But when I I [11:40] give that prompt to the LLM like hey [11:43] generate a content that satisfied XYZ I [11:45] take it back to them like uh you know [11:48] it's not it's not right maybe we can do [11:50] this maybe we can do that. So what I'm [11:53] trying to say is that no editor and no [11:57] human is able to tell what they want. [12:00] Like one of the things that Steve Jobs [12:02] says is uh customer does not know what [12:05] is right. Right. So that is primarily [12:09] coming from the fact that it's not not [12:11] very easy to articulate what people want [12:14] and that is where we ended up um trying [12:18] to create a judge that is going to [12:20] convert a qualitative aspects of what is [12:24] important for content generation into [12:26] quantitative way such that we can make [12:28] decisions on top of it. So we came up [12:31] with a prompt that looks like this where [12:34] we are defining parameters in terms of [12:36] what is it that we're looking for. We [12:38] want the facts to be right. We know LLM [12:41] tend to hallucinate. We want to make [12:42] sure that the output that we are [12:45] producing is factually correct. If the [12:47] if Alcarz has won the game, we don't [12:50] want LLM to create content about sinner [12:52] winning the game. Right? And then the [12:55] next part is novelty score. Now this is [12:58] uh the reason we separate this fact [13:00] factualness and novelty is we we like [13:03] creativity. We like when a editor or [13:06] when a journalist comes up with their [13:08] own way of explaining something that has [13:11] happened. So we like novelty. But what [13:13] we don't want it is to it it to come at [13:16] the cost of factualness where you are [13:18] now misrepresenting the facts. So that [13:21] is where we use something called as the [13:23] novelty score. And in this novelty score [13:25] we want to make sure that the content [13:27] that is produced is actually uh is new [13:31] but then at a certain threshold where it [13:34] is new to be creative enough but not as [13:37] new that we don't know what is happening [13:39] now. Now next part next two is relevance [13:42] and repetitive. Now relevance is where [13:45] we want to make sure that it is sticking [13:47] to facts. that is sticking to the [13:49] information that is actually available [13:51] and uh relevant to the game of the [13:54] tennis uh in this situation and it can [13:56] be applied to anything that you're [13:57] generating repetitiveness. This is [14:00] something that we had to add primarily [14:02] for long form content. So when we so we [14:05] we did this activity for more than one [14:08] type of content generation. So when we [14:09] did for small uh content where we're [14:11] creating three bullet points of sorts, [14:14] it was much easier and it was uh giving [14:16] a really good output. But when we moved [14:18] on to a more long form content where it [14:21] we asked it to generate like paragraphs, [14:23] uh it started to repeat the same thing [14:25] like trying to make sure that the [14:27] factual score is correct and there's [14:29] like it's more relevant. It just said [14:31] sinner served the best sinner served at [14:34] a max average speed of 160 km/h. And [14:37] then in the next slide uh 160 km hour [14:40] sinner serve speed was had a good great [14:43] advantage against crash. Now this is [14:45] where it starts to repeat the same facts [14:48] and everybody nobody wants to read [14:50] repetitive stuff. So this is again [14:53] something that we had to add more uh I [14:55] would say uh empirically than [14:58] theoretically. [15:00] And this is what the output looks like [15:02] right. So in this the context that is [15:05] the uh individual facts that we gave it [15:09] uh which which is more of a templatized [15:11] structure of what uh the output uh and [15:15] what has happened in the match and we [15:17] used a lot of our um maths and [15:21] statistics to identify what is relevant [15:23] and what what we would import be uh [15:26] calling it as important enough to be uh [15:28] referred to in a blog post right and not [15:31] create a big amount of noise for the [15:33] large language model to choose from and [15:35] then the paragraph is what it generated [15:38] right then finally the LLM as a judge [15:40] was able to output hey this is factually [15:43] correct so I'll give it 100% 100 score [15:46] and then it explains why each and every [15:48] score is given and what is the reason it [15:51] has deducted any score if it did now [15:55] this helps us tune this prompt and tune [15:57] the definition of parameters over here [16:00] such that it is actually reflecting what [16:02] we want to see [16:05] next. Now, okay. So, now we have this [16:08] content. We have this judge who has told [16:10] us, hey, this is not right. This is [16:12] right and things like that. Now, what do [16:14] we do from there? So, this is where we [16:17] start to use uh some of the theory of [16:21] mind prompting as we want to call it. [16:23] And what does this mean is we take [16:27] historical uh editors. So we ask the our [16:31] human editors and our human human in the [16:33] loop uh folks to assist us in this in [16:37] this aspect where we take all the [16:40] generated content for 2023 right of US [16:44] Open and then we ask them to review it, [16:46] approve it or edit it in terms of what [16:49] they seem fit to be presentable to the [16:52] website and that is and then we use that [16:55] output as our basis to create some sort [16:58] of a memory mind of co of of sorts where [17:02] we are not u asking the humans what you [17:06] expect what you do not expect but [17:08] instead using our LLM as a judge to [17:11] create baselines to create expectation [17:14] values that are acceptable to be [17:17] approved by the uh and and and uh human [17:22] in the loop. So what does that do that [17:25] means? So once uh your AI content is [17:28] created, we have our judge evaluating it [17:31] and then next part is where we want to [17:34] do an editor correction here and make [17:37] sure that it is good and ready for human [17:39] approval. Now how do we do that? So then [17:43] this is what we use as a next step in [17:45] our AI agent where we ask it to uh be [17:49] have as an editor and use the same [17:52] parameters to eval to edit it but now we [17:55] are giving it a feedback say hey this is [17:57] the paragraph you give but it is not [18:00] good in terms of factualness. It is not [18:01] good in terms of novelty. We we want it [18:04] to change uh you we want it to to change [18:06] the relevance or topic alignment. So [18:08] that is where we use this feedback to be [18:12] able to ask the LLM to edit this [18:14] response. [18:16] Now many of you folks might be wondering [18:19] like so you're saying you're saying 5% [18:21] increase the factual list by 20% [18:23] decrease the factual list by 20% what [18:25] does that mean right like how will [18:27] anybody interpret it now the reason why [18:30] this works is because of the definition [18:33] of parameters and we have this this is [18:37] what we get where we are able to [18:41] identify and the way we ran it we run it [18:43] is we run it 21 times If we have a max [18:46] iteration of 20 times, we want it to [18:48] keep doing it, keep doing it, keep doing [18:50] it until and unless a convergence is [18:52] reached. What is convergence? It needs [18:53] to be in the required threshold of [18:56] expectations. Okay, this is good enough [18:58] content to be used later. The first uh [19:02] column over here talks about initial [19:04] assessment. That is where we started [19:06] with the content generation. And this uh [19:09] I do not have the numbers of before we [19:11] used chain of thought prompting but it [19:14] was much worse where the factualness was [19:15] off the charts, the repetitiveness was [19:17] off the charts but then with initial [19:19] assessment uh with with chain of thought [19:21] prompting we started off from a really [19:23] good baseline where we have like very [19:26] low factual but we have very high [19:28] novelty right and that was something [19:31] that was not acceptable to uh a lot of [19:34] the stakeholders. So we run our twominut [19:38] uh within 2 minutes run the 20 [19:40] iterations. Uh this was all done on the [19:43] what's the next platform and with that [19:46] we were able to get uh some of them were [19:49] converging where we it meet exactly the [19:52] expectations and hence we have uh 0.0 [19:55] zero uh [19:57] difference in all these parameters [20:00] versus some of them were not and the [20:03] average scores are reflective of what [20:06] the output is and even when things are [20:09] not converging what we observed in our [20:11] data is that it would go back and forth. [20:14] So it would edit uh it's very difficult [20:17] to sort of come to a balance between [20:19] factualness and novelty and some of the [20:22] times even relevance because these [20:24] things are not mutually exclusive of [20:26] each other. You cannot decrease one [20:28] without compromising the other. So that [20:30] is where uh LLM would go into a loop [20:32] where it would edit something and then [20:34] come back and add the same thing in the [20:36] next iteration. So it would keep doing [20:38] this back and forth uh for a couple of [20:40] times and then uh suddenly in the fifth [20:43] try it would do something completely [20:45] novel and uh learn from all its multiple [20:48] iteration steps and then use uh get [20:51] suddenly converge to the expectations. [20:53] So what we observed was uh it would [20:55] create multiple uh new uh iterations and [20:59] then immediately use all of that to [21:02] create an output that would satisfy [21:04] everything. That is that is how I would [21:07] say a lot of human editors would work [21:10] right like you try bunch or bunch bunch [21:12] of things and then you pick the best one [21:14] out of all the iterations mix and match. [21:16] So that is what we ended up doing. So um [21:20] yep that is uh what I have for today. Uh [21:25] any questions?