AI Should Not Replace Well-Built Data Fundamentals by Shane Glass

AI Should Not Replace Well-Built Data Fundamentals by Shane Glass 🔗 **Source:** [AI Should Not Replace Well-Built Data Fundamentals by Shane Glass](https://www.youtube.com/watch?v=y4Hp5mEtukg) > [!note]+ Document Notes > tags: [data-fundamentals, artificial-intelligence, open-source] > author: Shane Glass > source: YouTube > reference: https://www.youtube.com/watch?v=y4Hp5mEtukg > date created: 2024-12-28 12:00 > > # 🏷️ Title: AI Should Not Replace Well-Built Data Fundamentals >  ## 🧰 Action Inbox - [H1] Evaluate current data pipeline architecture for scalability requirements — 02:45 #infrastructure - [H2] Implement Apache Iceberg for storage interoperability — 08:30 #storage - [H3] Integrate AI coding assistants into data engineering workflow — 06:15 #productivity ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Apache Kafka](https://kafka.apache.org/) - [Apache Beam](https://beam.apache.org/) - [Apache Iceberg](https://iceberg.apache.org/) ## 📝 Summary - Google's Shane Glass argues AI should enhance, not replace, solid data fundamentals - Core data fundamentals remain critical: storage, movement, and transformation - Modern data infrastructure must be scalable, flexible, and interoperable - Open source tools like Kafka, Beam, Iceberg, and Spark provide strong foundations - AI serves best as a productivity multiplier for building robust data systems ## 📖 Main Content Shane Glass, a Developer Relations Engineer at Google's Open Source Program Office, challenges the notion that AI eliminates the need for strong data fundamentals. He argues that while AI innovation accelerates rapidly, the core requirements for data infrastructure remain unchanged: reliable data storage, efficient data movement, and robust data transformation. The presentation emphasizes three key requirements for modern data fundamentals: scalability (supporting varying workloads), flexibility (handling diverse data structures), and interoperability (working with multiple downstream tools). Glass uses a dishwasher metaphor to explain how AI coding assistants can accelerate open source development without replacing the underlying infrastructure needs. For data movement, he recommends Apache Kafka and Beam for streaming and batch processing, noting that modern users demand continuous updates with minimal latency. Storage solutions should avoid vendor lock-in through tools like Apache Iceberg, which combines data lake flexibility with data warehouse structure. For transformation, Apache Spark remains the standard, especially with growing AI workload demands requiring sophisticated feature engineering. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=0) — [H1] Opening and Core Thesis - Key idea: AI might solve data problems, but open source solutions already exist - Evidence: "AI should not replace well-built data fundamentals" #### ⏱️ [00:45](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=45) — [H2] Common Data Problems - Key idea: Weak data fundamentals cause tangible business problems - Evidence: "Dashboard that incorrectly reported key metrics and wasted time and money" #### ⏱️ [01:30](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=90) — [H1] AI Innovation Pace Warning - Key idea: Rapid AI development doesn't eliminate fundamental data needs - Evidence: "In three weeks someone will come along and prove me 100% wrong" #### ⏱️ [02:15](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=135) — [H2] Reframing AI's Role - Key idea: AI should build upon, not replace, data fundamentals - Evidence: "More precise conversation around how to use AI to help you build your data fundamentals" #### ⏱️ [02:45](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=165) — [H1] Three Core Data Fundamentals - Key idea: Focus on data storage, movement, and transformation - Evidence: "Unlikely that AI will abstract these away, even long term" #### ⏱️ [03:30](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=210) — [H1] Modern Infrastructure Requirements - Key idea: Data fundamentals must be scalable, flexible, and interoperable - Evidence: "Tools must support workloads of varying size and throughput" #### ⏱️ [04:15](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=255) — [H2] Dishwasher Metaphor Introduction - Key idea: AI tools are like dishwashers - helpful automation, not replacement - Evidence: "Building with open source tools directly is like handwashing dishes" #### ⏱️ [05:00](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=300) — [H1] Data Movement Challenges - Key idea: Modern users demand continuous updates with minimal latency - Evidence: "Right data written to right geographic and technical locations at right time" #### ⏱️ [05:45](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=345) — [H2] Apache Kafka and Beam Solutions - Key idea: These tools capture and stream data with minimal latency - Evidence: "AI pipeline that detects credit card fraud to block fraudulent transactions" #### ⏱️ [06:15](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=375) — [H3] AI Coding Assistants as Dishwashers - Key idea: Gemini CLI and code assist simplify Beam development - Evidence: "Make it easier to write code for Beam's programming framework" #### ⏱️ [07:00](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=420) — [H2] When Proprietary Tools Make Sense - Key idea: SaaS connectors change rapidly, making maintenance challenging - Evidence: "Pipelines to Slack, Asana, Salesforce - specs change so rapidly" #### ⏱️ [07:30](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=450) — [H1] Storage and Tooling Lock Risk - Key idea: Tooling lock-in is greater risk than vendor lock-in - Evidence: "What if your storage locks you into specific analytics tool" #### ⏱️ [08:30](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=510) — [H1] Apache Iceberg Solution - Key idea: Combines data lake flexibility with data warehouse structure - Evidence: "Brings flexibility of data lakes and combines with structure of data warehouses" #### ⏱️ [09:15](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=555) — [H2] AI-Powered Storage Optimization - Key idea: AI tools can optimize partitioning and identify unused data - Evidence: "Gemini in BigQuery with intelligent recommendations for partitioning" #### ⏱️ [10:00](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=600) — [H1] Transformation and Feature Engineering - Key idea: AI workloads increase demand for sophisticated feature engineering - Evidence: "Demand for feature engineering growing rapidly with AI workloads" #### ⏱️ [10:45](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=645) — [H2] Apache Spark Recommendation - Key idea: Spark meets modern transformation requirements exceptionally well - Evidence: "Spark ML has incredibly robust support for feature engineering" #### ⏱️ [11:30](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=690) — [H3] Advanced AI Transformation Tools - Key idea: Natural language interfaces can build data pipelines - Evidence: "BigQuery data engineering agent builds pipelines based on natural language inputs" #### ⏱️ [12:00](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=720) — [H1] AI as Architecture Assistant - Key idea: AI helps focus on architecture rather than implementation details - Evidence: "Help me focus on how to architect something rather than execute specific architecture" #### ⏱️ [12:45](https://www.youtube.com/watch?v=y4Hp5mEtukg&t=765) — [H1] Future Data Practitioner Skills - Key idea: Best practitioners understand both fundamentals and AI tools - Evidence: "Strong understanding of data fundamentals and open source tooling" ## 📚 References - [Apache Kafka](https://kafka.apache.org/) - [Apache Beam](https://beam.apache.org/) - [Apache Iceberg](https://iceberg.apache.org/) - [Apache Spark](https://spark.apache.org/) - [Google Cloud Dataflow](https://cloud.google.com/dataflow) ## 🗣️ Paraphrase - AI tools can accelerate data engineering work but cannot replace the need for solid foundational infrastructure in storage, movement, and transformation - Modern data systems must handle diverse workloads, data types, and downstream applications while avoiding vendor lock-in through open source solutions - The most effective approach combines open source data fundamentals with AI-powered development assistance to build scalable, maintainable data infrastructure ## 📰 Newsletter Shane Glass from Google's OSPO delivered a timely reminder that while AI tools are transforming development workflows, they work best when built upon solid data foundations. His key insight: treat AI like a dishwasher - incredibly useful for automating routine tasks, but you still need clean dishes (data fundamentals) to start with. For WordPress developers working with data-heavy sites, this translates to ensuring your database architecture, caching strategies, and content workflows are solid before layering on AI features. Tools like WP-CLI for data migration and the REST API for flexible data access become even more valuable when enhanced with AI coding assistants. Resources: [WP-CLI](https://wp-cli.org/), [WordPress REST API](https://developer.wordpress.org/rest-api/), [Database Optimization](https://developer.wordpress.org/advanced-administration/performance/optimization/) ## ✅ Action Items #### OSPO - Policy & Governance: Develop contribution guidelines for AI-enhanced open source data tools - Software Inventory: Catalog open source data infrastructure dependencies in AI projects - Developer Support: Create training materials combining traditional data engineering with AI tooling - Community Engagement: Support Apache projects mentioned (Kafka, Beam, Iceberg, Spark) - Strategic Alignment: Define KPIs for measuring AI productivity gains in data infrastructure #### Community - Maintainers: Document AI-assisted development workflows for data infrastructure projects - Contributors: Create first-issue programs for AI + open source data tooling integration - Sponsors: Fund development of AI coding assistants for open source data tools #### WordPress - Core: Evaluate database query optimization opportunities using AI analysis - Core Performance: Profile data access patterns with AI-powered monitoring - Meta: Implement intelligent caching strategies for .org infrastructure - CLI: Develop AI-assisted commands for database migration and optimization - Hosting: Create cross-host performance benchmarking with AI analysis ## 🧪 Hypotheses - Organizations using AI tools with strong open source data foundations achieve better scalability than those replacing fundamentals entirely - Data practitioners who understand both traditional data engineering and AI tooling deliver more maintainable solutions - Open source data infrastructure combined with AI coding assistants reduces time-to-market without sacrificing reliability - Teams avoiding vendor lock-in through tools like Apache Iceberg adapt faster to new AI workload requirements ### 📊 Methodology - Compare performance metrics between AI-first vs. fundamentals-first data architectures - Survey data practitioners on productivity gains from AI coding assistants vs. infrastructure improvements - Measure technical debt accumulation in projects using different AI integration approaches ### ❓ Questions - How do we measure the ROI of AI coding assistants versus infrastructure investments? - What specific data engineering tasks benefit most from AI automation versus human expertise? - How will the role of data engineers evolve as AI tools become more sophisticated? ### 🔮 Future Research - Long-term sustainability of AI-enhanced open source data infrastructure - Integration patterns between proprietary AI services and open source data foundations - Skills development pathways for data practitioners in an AI-augmented world ### 💡 Implications - Practice: Data engineers should learn AI coding assistants while maintaining infrastructure expertise - Policy: Organizations need guidelines for AI tool procurement that complement open source strategies - Education: Training programs must cover both traditional data fundamentals and AI-enhanced development - Funding: Investment in open source data infrastructure remains critical despite AI advances ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Speaker slides, dishwasher metaphor visualization - Improve: Architecture diagrams showing AI + open source integration patterns #### 🔊 Aural learners - Present: Conference presentation format with clear narration - Improve: Podcast-style discussion of key concepts #### 🗣️ Verbal learners - Present: Clear analogies (dishwasher metaphor), technical definitions - Improve: More real-world case study narratives #### 🧩 Kinesthetic learners - Present: Hands-on tool recommendations with specific implementations - Improve: Step-by-step tutorials for implementing suggested architecture #### 🧠 Logical learners - Present: Three-part framework (storage, movement, transformation) - Improve: Decision trees for choosing between open source and proprietary solutions #### 🧑‍🤝‍🧑 Social learners - Present: Conference setting, invitation to visit Google lounge - Improve: Discussion prompts for team architecture planning #### 🧘‍♀️ Solitary learners - Present: Individual reflection on current architecture choices - Improve: Self-assessment checklist for data infrastructure requirements #### 🌿 Naturalistic learners - Present: Ecosystem thinking about data infrastructure - Improve: Sustainability implications of different architectural choices ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive technology names - Improve: Alt text for any architecture diagrams, expanded acronyms #### Keyboard navigation - Present: Standard video player controls - Improve: Timestamped navigation links, transcript with clickable sections #### Contrast and readability - Present: High contrast slides, clear fonts - Improve: Ensure all diagrams meet contrast requirements #### Media and captions - Present: Video format with audio - Improve: Professional captions, downloadable transcript ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global perspective on data sovereignty requirements - Improve: Examples from multiple geographic regions and industries #### Inclusive language - Present: Technical terms clearly defined - Improve: Avoid sports analogies that may not resonate globally #### Neurodiversity support - Present: Clear structure with numbered points - Improve: Visual summaries, consistent terminology throughout #### Representation - Present: Single speaker perspective - Improve: Include diverse voices on data engineering practices ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Strong premise that data fundamentals remain necessary - Inductive strength: Good examples from weather companies, fraud detection - Analogical strength: Effective dishwasher metaphor for AI tool relationship #### Fallacies - Appeal to authority: Minimal, mostly relies on technical merit - False dichotomy: Avoided by positioning AI as complementary, not replacement #### Tone of voice - Overall tone: Professional but approachable, uses humor appropriately ### 🛡️