YouTube Video rzWatCmeG9c 🔗 **Source:** [YouTube Video rzWatCmeG9c](https://www.youtube.com/watch?v=rzWatCmeG9c) > [!note]+ Document Notes > tags: [apache-iceberg, software-design, data-architecture] > author: Brian Lyles > source: Open Source Summit North America 2024 > reference: https://www.youtube.com/watch?v=rzWatCmeG9c > date created: 2024-12-19 14:30 > > # 🏷️ Title: Developer Lessons from Apache Iceberg: Lessons in Structure, Scale, and Software Design > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Implement stable IDs in schema design — prevents data loss during column renaming #schema-evolution - [H2] Design versioning as first-class feature — enables rollbacks and time travel queries #versioning - [H3] Build maintenance into APIs from start — treat operability as product feature #maintenance ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Apache Iceberg](https://iceberg.apache.org/) - [Open Table Formats](https://delta.io/blog/open-table-formats/) ## 📝 Summary - Senior AWS engineer Brian Lyles presents Apache Iceberg as a masterclass in software design patterns - Uses library evolution metaphor (Alexandria to card catalogs to ISBNs to search) to explain data architecture - Eight key developer lessons: stable IDs, schema evolution, versioning, atomic commits, predicate pushdown, maintenance APIs, observability, and clean boundaries - Emphasizes designing systems for change rather than assuming static requirements - Bottom line: Apache Iceberg demonstrates how to build resilient, scalable systems through proper abstraction layers ## 📖 Main Content Apache Iceberg is an open table format that provides a standardized way to organize and query large datasets stored in object storage like S3. Rather than treating files and folders as the source of truth, it creates a catalog-first approach with metadata layers that enable powerful features like time travel, schema evolution, and efficient querying. Lyles argues that the core innovation isn't the data technology itself, but the software design patterns that make it work. These patterns include: using stable internal IDs separate from human-readable names, building versioning into the system architecture, implementing atomic commits for consistency, and creating clean boundaries between storage and compute layers. The presentation uses the evolution of libraries as an analogy - from the Library of Alexandria's physical scrolls to modern searchable catalogs - to illustrate how data systems should prioritize discoverability and structure over raw storage. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=rzWatCmeG9c?t=0) — [H1] Opening and Context - Key idea: Speaking about Apache Iceberg instead of AI or infrastructure - Evidence: "I don't want to talk about AI because you're gonna I looked on the schedule, all the talks are about AI" #### ⏱️ [02:15](https://www.youtube.com/watch?v=rzWatCmeG9c?t=135) — [H1] Speaker Background - Key idea: 32 years in open source, started with Linux sound card patch in high school - Evidence: "My first patch, my first open source...was when I was in 11th grade when I was in high school and my sound card didn't work" #### ⏱️ [04:30](https://www.youtube.com/watch?v=rzWatCmeG9c?t=270) — [H1] Library of Alexandria Analogy - Key idea: Physical scrolls in one location created single point of failure - Evidence: "They stole all the scrolls off the ships...when it came time whenever the wars came and the fire came, guess what? We had all these scrolls and we couldn't find anything" #### ⏱️ [06:45](https://www.youtube.com/watch?v=rzWatCmeG9c?t=405) — [H2] Evolution of Cataloging Systems - Key idea: Card catalogs and ISBN systems created universal standards - Evidence: "Every book, magazine, journal, periodical gets assigned this number that says it was here in the world" #### ⏱️ [09:20](https://www.youtube.com/watch?v=rzWatCmeG9c?t=560) — [H1] Catalog First, Storage Second - Key idea: Don't just put data in storage without proper cataloging - Evidence: "So libraries evolved from shelves to systems. We invented catalogs, ISBNs...Catalog first. Don't just put your scrolls on the shelves" #### ⏱️ [11:15](https://www.youtube.com/watch?v=rzWatCmeG9c?t=675) — [H1] Apache Iceberg Overview - Key idea: Open table format that creates standards for data in object storage - Evidence: "What these do is they are ways of describing files in object storage in a way that makes it easier for you to access them in the future" #### ⏱️ [13:40](https://www.youtube.com/watch?v=rzWatCmeG9c?t=820) — [H2] Iceberg Architecture Layers - Key idea: Hierarchical structure from data files to manifest files to metadata to catalog - Evidence: "From data files...to manifest files that list these data files...to manifest list that lists the manifest files...to metadata files...to catalog" #### ⏱️ [15:55](https://www.youtube.com/watch?v=rzWatCmeG9c?t=955) — [H1] One Table, Many Engines - Key idea: Iceberg provides abstraction layer allowing multiple query engines - Evidence: "We have one table, many engines, and one spec to rule them all" #### ⏱️ [17:30](https://www.youtube.com/watch?v=rzWatCmeG9c?t=1050) — [H1] Stable IDs Pattern - Key idea: Use internal IDs separate from column names to enable safe renaming - Evidence: "Because it has stable ids, it actually doesn't use the name. It uses a pointer to another ID" #### ⏱️ [20:10](https://www.youtube.com/watch?v=rzWatCmeG9c?t=1210) — [H2] Schema Evolution Without Rewriting - Key idea: Decouple logical schema from physical storage to enable changes - Evidence: "You need to decouple the logical schema...from where it's stored" #### ⏱️ [22:45](https://www.youtube.com/watch?v=rzWatCmeG9c?t=1365) — [H1] Designing for Change - Key idea: Systems that don't allow for change create technical debt - Evidence: "When we write code and we say that there will never be any change...What we're actually doing is compounding that debt early" #### ⏱️ [25:20](https://www.youtube.com/watch?v=rzWatCmeG9c?t=1520) — [H1] Versioning as First-Class Feature - Key idea: Built-in snapshotting enables time travel and rollback capabilities - Evidence: "Every time that we change data with iceberg, we get a snapshot, which means that I can actually go back" #### ⏱️ [28:35](https://www.youtube.com/watch?v=rzWatCmeG9c?t=1715) — [H2] Optimistic Concurrency - Key idea: Snapshot-based atomic commits prevent conflicts in distributed systems - Evidence: "I do my commit and whenever I want to commit again I know what the current version of the database is" #### ⏱️ [31:15](https://www.youtube.com/watch?v=rzWatCmeG9c?t=1875) — [H2] Predicate Pushdown - Key idea: Skip work by filtering at metadata level before scanning data - Evidence: "The fastest thing to do is to not do anything...don't walk every shelf" #### ⏱️ [34:40](https://www.youtube.com/watch?v=rzWatCmeG9c?t=2080) — [H1] Maintenance as Product Feature - Key idea: Build compaction and cleanup into the system rather than as afterthought - Evidence: "Treat maintenance as a feature, not an afterthought" #### ⏱️ [37:25](https://www.youtube.com/watch?v=rzWatCmeG9c?t=2245) — [H2] Observability Through Metadata - Key idea: System tables provide queryable metadata about data lineage and operations - Evidence: "You can query not only your data, but there's system tables where you can query about data of your system" #### ⏱️ [40:10](https://www.youtube.com/watch?v=rzWatCmeG9c?t=2410) — [H1] Clean Boundaries - Key idea: Separate storage and compute through REST catalog API - Evidence: "Apache icebergs exposes this thing called the rest catalog API and it separates basically the table metadata from the engines" #### ⏱️ [42:30](https://www.youtube.com/watch?v=rzWatCmeG9c?t=2550) — [H1] Core Design Principles - Key idea: Names for humans, IDs for systems; design for rollback; operations as APIs - Evidence: "If roll back is scary, you built a trap, not a platform" ## 📚 References - [Apache Iceberg Documentation](https://iceberg.apache.org/) - [Delta Lake](https://delta.io/) - [Apache Hudi](https://hudi.apache.org/) - [Trino](https://trino.io/) ## 🗣️ Paraphrase - Apache Iceberg demonstrates excellent software design patterns through its data architecture, particularly around stable identifiers, versioning, and clean API boundaries - The evolution from physical libraries to digital catalogs parallels how data systems should prioritize discoverability and metadata over raw storage - Building systems that assume change rather than stasis prevents technical debt accumulation and enables long-term maintainability ## 📰 Newsletter This week's spotlight is on Apache Iceberg - not just as a data technology, but as a masterclass in software design patterns that WordPress developers can apply to any project. **Key Learning**: Brian Lyles from AWS shares eight crucial design patterns from Iceberg that make systems resilient: stable IDs (separate internal identifiers from display names), first-class versioning (built-in rollback capabilities), and maintenance-as-a-feature (not an afterthought). **Practical Application**: These patterns apply beyond data systems. When building WordPress plugins or themes, consider: How do you handle schema changes? Can users easily rollback problematic updates? Are your APIs designed for observability? **Community Connection**: The talk emphasizes "preserve knowledge, make it discoverable, and design it for generations" - core values that align perfectly with WordPress's open source mission. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create talks showing design patterns from successful OSS projects like Iceberg - Education: Workshop on "Designing for Change" using real-world examples - Developer experience: Audit APIs for rollback safety and versioning support - Developer marketing: Content series on software architecture lessons from data systems - Community management: Facilitate discussions on system design patterns #### Community - Maintainers: Template governance docs that separate logical from physical organization - Contributors: First-issue programs focused on observability and maintenance features - Sponsors: Fund architectural documentation and design pattern libraries #### WordPress - Core: Implement stable IDs for database schema changes, improve rollback mechanisms - Core Performance: Predicate pushdown patterns for query optimization - Core Program Team: Establish versioning standards across all WordPress APIs - Documentation: Create design pattern library showing WordPress architectural decisions - Meta: API versioning strategy for .org services ## 🧪 Hypotheses - Systems designed with stable internal IDs will have fewer breaking changes during evolution - First-class versioning reduces deployment fear and increases release velocity - Maintenance APIs reduce operational overhead compared to runbook-based approaches - Clean boundaries between components enable easier testing and replacement ### 📊 Methodology - Compare schema change impact between systems with/without stable IDs - Measure deployment rollback frequency and success rates across different versioning approaches - Survey operational overhead for API-driven vs manual maintenance procedures ### ❓ Questions - How can WordPress core implement stable IDs for database schema evolution? - What would first-class versioning look like for WordPress themes and plugins? - How might WordPress separate logical content structure from physical storage implementation? ### 🔮 Future Research - Design patterns from other successful open source projects applicable to WordPress - API design principles that enable long-term backward compatibility - Observability patterns for distributed WordPress installations ### 💡 Implications - Practice: Apply catalog-first thinking to any data organization project - Policy: Establish versioning requirements for all public APIs - Security: Stable IDs and atomic commits reduce attack surface during updates - Education: Teach system design through successful open source examples - Funding: Invest in architectural documentation and design pattern libraries ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Library evolution diagrams, Iceberg architecture layers - Improve: More system architecture diagrams, data flow visualizations #### 🔊 Aural learners - Present: Full video presentation with clear narration - Improve: Audio-only podcast version highlighting key quotes #### 🗣️ Verbal learners - Present: Strong analogies (libraries, catalogs, ISBNs), clear definitions - Improve: More real-world examples, case study comparisons #### 🧩 Kinesthetic learners - Present: Live demos of Iceberg features, Python code examples - Improve: Hands-on workshop materials, step-by-step implementation guides #### 🧠 Logical learners - Present: Clear progression from problems to solutions, numbered principles - Improve: Decision trees for choosing patterns, cause-effect diagrams #### 🧑‍🤝‍🧑 Social learners - Present: Conference presentation format, community examples - Improve: Discussion guides for team architecture reviews #### 🧘‍♀️ Solitary learners - Present: Comprehensive notes, quotable principles for reflection - Improve: Self-assessment checklists for system design #### 🌿 Naturalistic learners - Present: Evolution metaphor from libraries to modern systems - Improve: Ecosystem thinking about software dependencies ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive section titles - Improve: Alt text for architectural diagrams, code snippet descriptions #### Keyboard navigation - Present: Standard video player controls - Improve: Timestamped navigation, chapter markers #### Contrast and readability - Present: High contrast slides, readable fonts - Improve: Ensure code examples meet contrast requirements #### Text resizing and reflow - Present: Scalable presentation format - Improve: Responsive note layout for mobile reading #### Media and captions - Present: Full video with transcript - Improve: Professional captions with speaker identification #### Forms and interactions - Present: Simple video interface - Improve: Interactive timeline, searchable transcript ### 🌍 DEIB Assessment #### Cultural breadth - Present: Universal library metaphor, global technology examples - Improve: Examples from different cultural approaches to information organization #### Inclusive language - Present: Technical terms clearly defined, avoided unnecessary jargon - Improve: More accessible explanations of data concepts #### Neurodiversity support - Present: Clear structure with numbered acts, consistent formatting - Improve: Visual section breaks, summary bullets #### Representation - Present: Open source community values, collaborative examples - Improve: Diverse contributor examples, varied use cases #### Psychological safety - <!-- deep-read:sections end -->