Deploying Presto in AWS with Ahana Cloud 🔗 **Source:** [Deploying Presto in AWS with Ahana Cloud](https://www.youtube.com/watch?v=Hk3wZbbmujc) > [!note]+ Document Notes > tags: [apache-hoodie, data-lakehouse, aws-deployment] > author: Nadine > source: YouTube > reference: https://www.youtube.com/watch?v=Hk3wZbbmujc > date created: 2024-12-19 15:30 > > # 🏷️ Title: Deploying Presto in AWS with Ahana Cloud - Apache Hoodie and Data Lakehouse > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Set up hoodie clustering service scheduling — operational efficiency #hoodie - [H2] Implement incremental ETL using hoodie queries — compute optimization #performance - [H3] Configure hoodie indexing for read/write optimization — query performance #indexing ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Apache Hoodie Documentation](https://hudi.apache.org/) - [Ahana Cloud Platform](https://ahana.io/) ## 📝 Summary - Apache Hoodie presentation covering data lakehouse architecture and deployment with Presto/Ahana - Demonstrates how Hoodie enables transactional guarantees and streaming data ingestion at scale - Covers Copy-on-Write vs Merge-on-Read storage types with practical e-commerce examples - Shows integration between Hoodie tables in S3 and Presto query engine via Glue metastore - Key benefit: enables record-level updates/deletes on immutable storage like S3 with ACID properties ## 📖 Main Content Apache Hoodie originated at Uber in 2016 to solve inefficiencies in ETL pipelines during hypergrowth, handling petabytes of data with frequent updates from ride lifecycle events. Before Hoodie, large Spark jobs periodically rewrote entire datasets in HDFS to absorb upstream changes - an inefficient and hard-to-scale approach given HDFS's immutable nature. Hoodie brings transactional guarantees to data lakes and handles streaming sources with bursty writes or late-arriving data. It's not just a table format but a complete platform offering table services like clustering, cleaning, and indexing. The platform supports record-level inserts, updates, and deletes on S3, enabling compliance with data privacy laws while consuming real-time streams and CDC data. Two main storage types exist: Copy-on-Write (COW) for batch workloads with higher write amplification but lower read amplification, and Merge-on-Read (MOR) for streaming scenarios. COW creates new parquet file versions on each update through merge operations, while supporting both snapshot queries (current table state) and incremental queries (changes between commits). ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=0) — [H1] Introduction to Apache Hoodie Origins - Key idea: Hoodie started at Uber in 2016 to address ETL inefficiencies during hypergrowth - Evidence: "dealing with petabyte of data" during Uber's ride lifecycle management #### ⏱️ [02:30](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=150) — [H2] Pre-Hoodie Challenges with HDFS - Key idea: Large Spark jobs periodically rewrote entire datasets in immutable HDFS - Evidence: "process was not only inefficient but also very hard to scale" #### ⏱️ [04:15](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=255) — [H1] Hoodie's Core Value Propositions - Key idea: Brings transactional guarantees and streaming data handling to data lakes - Evidence: "effectively work with streaming sources where you can handle bursty writes or late arriving data" #### ⏱️ [06:00](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=360) — [H2] High-Level Architecture Overview - Key idea: Hoodie ingests from any source and connects to any supported query engine - Evidence: "ingest data from particularly any source whether it's an OLTP database" #### ⏱️ [08:30](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=510) — [H2] Hoodie as Platform vs Table Format - Key idea: Hoodie provides table services beyond just formatting (clustering, cleaning, indexing) - Evidence: "hoodie is not just a table format but it's also a platform within itself" #### ⏱️ [10:45](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=645) — [H1] Record-Level Operations on S3 - Key idea: Enables inserts, updates, deletes on immutable storage for compliance - Evidence: "perform record-level inserts updates and deletes on S3 allowing you to comply for example with data privacy laws" #### ⏱️ [13:20](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=800) — [H2] Built-in Integrations and Storage Formats - Key idea: Uses Apache Parquet and Avro with Spark, Hive, Presto integrations - Evidence: "hoodie uses Apache parquet and also Apache Avro for data storage" #### ⏱️ [16:00](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=960) — [H1] Workshop Architecture with E-commerce Example - Key idea: Demonstrates cart operations (view, add, remove) flowing through S3 to Presto - Evidence: "e-commerce example where people are checking their cart or viewing items" #### ⏱️ [18:45](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=1125) — [H2] Copy-on-Write Storage Type Explained - Key idea: Higher write amplification, lower read amplification, good for batch workloads - Evidence: "every time you ingest data there's like kind of this merge operation happening" #### ⏱️ [22:15](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=1335) — [H1] Snapshot vs Incremental Queries - Key idea: Snapshot shows current table state, incremental shows changes between commits - Evidence: "snapshot query takes a current view of your whole table at that moment" #### ⏱️ [25:30](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=1530) — [H2] File Versioning and Update Mechanics - Key idea: Updates create new parquet versions while unchanged data keeps original files - Evidence: "when there updates the parquet version changes to a new version" #### ⏱️ [28:00](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=1680) — [H2] Incremental ETL Benefits - Key idea: Query only changes between commits to save compute resources - Evidence: "if you can just query the changes this is how you save on compute resources" #### ⏱️ [30:45](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=1845) — [H1] Presto Integration Value - Key idea: Without Hoodie, Presto works with files directly without change optimization - Evidence: "without hoodie then Presto is just working with files directly it has no concept of what are the changes" ## 📚 References - [Apache Hoodie Official Site](https://hudi.apache.org/) - [Apache Parquet Documentation](https://parquet.apache.org/) - [AWS Glue Metastore](https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html) ## 🗣️ Paraphrase - Apache Hoodie emerged from Uber's need to efficiently handle massive data updates in their ride-sharing platform without rewriting entire datasets - The platform provides ACID transactions on data lakes while supporting both batch and streaming ingestion patterns with built-in optimization services - Copy-on-Write storage offers simplified operations for batch scenarios by merging updates into new file versions, enabling both full snapshots and incremental change queries ## 📰 Newsletter Apache Hoodie brings database-like capabilities to your data lake! This powerful platform originated at Uber to handle petabyte-scale updates efficiently. Unlike traditional approaches that rewrite entire datasets, Hoodie enables record-level operations on immutable storage like S3. Key benefits include transactional guarantees, streaming data support, and incremental query capabilities that dramatically reduce compute costs. The Copy-on-Write storage type works great for batch workloads, creating optimized parquet files while supporting both snapshot and incremental queries. Perfect for compliance requirements (GDPR deletions), real-time analytics, and building efficient ETL pipelines. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create Hoodie integration demos, present at data engineering conferences, track adoption metrics - Education: Develop lakehouse architecture workshops, office hours for Hoodie questions - Developer experience: Improve Hoodie documentation, create starter templates and SDKs - Developer marketing: Launch incremental query campaign, create comparison content vs competitors - Community management: Moderate Hoodie Slack, survey user pain points, collect integration feedback #### OSPO - License Compliance: Review Hoodie Apache 2.0 compliance, scan dependencies - Security Management: Monitor Hoodie CVEs, establish security response procedures - Software Inventory: Document Hoodie SBOM, track transitive dependencies - Developer Support: Train teams on Hoodie best practices, provide integration tooling - Community Engagement: Fund upstream Hoodie development, establish contributor agreements - Strategic Alignment: Define Hoodie adoption KPIs, measure lakehouse transformation progress ## 🧪 Hypotheses - Copy-on-Write storage provides better performance for batch analytical workloads compared to frequent streaming updates - Incremental queries reduce compute costs by 60-80% compared to full table scans in ETL pipelines - Hoodie's transactional guarantees enable reliable CDC ingestion without data loss or duplication - Integration with Presto/query engines provides better performance than direct file access for complex analytics ### 📊 Methodology - Benchmark COW vs MOR performance across different workload patterns and data volumes - Measure compute cost savings using incremental vs snapshot queries in production ETL pipelines - Test CDC reliability under various failure scenarios with transaction rollback capabilities ### ❓ Questions - How does Hoodie's clustering service impact query performance compared to manual partitioning strategies? - What are the optimal file sizing and commit frequency configurations for different use cases? - How does Hoodie perform with very wide tables (hundreds of columns) in terms of update efficiency? ### 🔮 Future Research - Performance comparison between Hoodie, Delta Lake, and Iceberg across diverse workloads - Integration patterns with modern streaming frameworks beyond Spark and Flink - Cost optimization strategies for Hoodie deployments in multi-cloud environments ### 💡 Implications - Practice: Teams can implement incremental ETL patterns reducing infrastructure costs significantly - Policy: Data governance frameworks need updates to handle lakehouse ACID properties and time travel - Security: New attack vectors around metadata manipulation require monitoring and access controls - Education: Data engineering curricula should include lakehouse architectures alongside traditional warehouses - Funding: Budget allocation shifts from ETL infrastructure to query optimization and governance tooling ### 🧠 Learning Styles Assessment Reading level estimate: Grade 12 (technical domain knowledge required) #### 👀 Visual learners - Present: Architecture diagrams, file versioning flowcharts, commit timeline visualizations - Improve: Interactive demos of COW operations, S3 bucket structure diagrams #### 🔊 Aural learners - Present: Video presentation with clear narration and examples - Improve: Audio-only summary of key concepts, podcast-style case study discussions #### 🗣️ Verbal learners - Present: Clear definitions of COW/MOR, analogies to database concepts - Improve: Glossary of lakehouse terms, real-world story comparisons #### 🧩 Kinesthetic learners - Present: Hands-on notebook exercises, CLI demonstrations - Improve: Step-by-step setup guides, troubleshooting checklists #### 🧠 Logical learners - Present: Performance comparisons, cost analysis, architectural decisions - Improve: Decision trees for storage type selection, ROI calculators #### 🧑‍🤝‍🧑 Social learners - Present: Community Slack mentions, collaborative workshops - Improve: Peer review processes, team implementation guides #### 🧘‍♀️ Solitary learners - Present: Individual notebook exercises, self-paced learning - Improve: Reflection questions on architecture decisions, personal project ideas #### 🌿 Naturalistic learners - Present: Data lifecycle analogies, ecosystem thinking - Improve: Environmental impact of compute optimization, sustainability metrics ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive slide content - Improve: Alt text for architectural diagrams, structured table descriptions #### Keyboard navigation - Present: Standard video controls available - Improve: Slide-by-slide navigation, skip-to-section links #### Contrast and readability - Present: Good text contrast in presentation slides - Improve: High contrast mode for diagrams, larger code font sizes #### Text resizing and reflow - Present: Video scales appropriately - Improve: Transcript with responsive text layout #### Media and captions - Present: Video with audio narration - Improve: Complete captions, downloadable transcript, audio descriptions #### Standards reference - Address WCAG 2.2 AA requirements for time-based media and complex images ### 🌍 DEIB Assessment #### Cultural breadth - Present: Technical examples applicable globally - Improve: Multi-region deployment examples, diverse use case scenarios #### Inclusive language - Present: Technical terminology with definitions provided - Improve: Plain-language summaries, avoid unnecessarily complex jargon #### Neurodiversity support - Present: Structured presentation with clear sections - Improve: Content warnings for rapid technical demonstrations, consistent pacing #### Representation - Present: Female presenter in technical role - Improve: Diverse speaker lineup, varied industry examples #### Psychological safety - Present: Encouraging tone about learning complex concepts - Improve: Explicit welcome for questions, beginner-friendly alternatives ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear progression from problem (HDFS inefficiency) to solution (Hoodie features) - Inductive strength: Uber case study generalizes to other high-scale scenarios effectively - Causal reasoning: Demonstrates how file versioning enables incremental queries and cost savings - Analogical strength: Database transaction concepts map well to data lake operations #### Fallacies - Appeal to authority: Uber's success doesn't guarantee applicability to all use cases - needs qualification - Hasty generalization: Cost savings claims need quantified evidence rather than theoretical benefits #### Tone of voice - Overall tone: Educational and confident with practical examples, appropriate technical depth for audience ### 🛡️ DARVO Analysis - Deny: No significant denial patterns observed - Attack: No attacks on competing approaches or critics - Reverse victim/offender: Not applicable to technical presentation - Emotional cues and rhetoric: Positive framing focused on problem-solving rather than competitive positioning ## 🎯 Conclusion Apache Hoodie represents a significant advancement in data lake capabilities by bringing transact <!-- deep-read:sections end -->