Deploying Presto in AWS with Ahana Cloud  ## Metadata - Author: [[YouTube]] - Full Title: Deploying Presto in AWS with Ahana Cloud - Category: #articles - Document Note: --- tags: [apache-hoodie, data-lakehouse, aws-deployment] author: Nadine source: YouTube reference: https://www.youtube.com/watch?v=Hk3wZbbmujc date created: 2024-12-19 15:30 --- # 🏷️ Title: Deploying Presto in AWS with Ahana Cloud - Apache Hoodie and Data Lakehouse ## 🧰 Action Inbox - [H1] Set up hoodie clustering service scheduling — operational efficiency #hoodie - [H2] Implement incremental ETL using hoodie queries — compute optimization #performance - [H3] Configure hoodie indexing for read/write optimization — query performance #indexing ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Apache Hoodie Documentation](https://hudi.apache.org/) - [Ahana Cloud Platform](https://ahana.io/) ## 📝 Summary - Apache Hoodie presentation covering data lakehouse architecture and deployment with Presto/Ahana - Demonstrates how Hoodie enables transactional guarantees and streaming data ingestion at scale - Covers Copy-on-Write vs Merge-on-Read storage types with practical e-commerce examples - Shows integration between Hoodie tables in S3 and Presto query engine via Glue metastore - Key benefit: enables record-level updates/deletes on immutable storage like S3 with ACID properties ## 📖 Main Content Apache Hoodie originated at Uber in 2016 to solve inefficiencies in ETL pipelines during hypergrowth, handling petabytes of data with frequent updates from ride lifecycle events. Before Hoodie, large Spark jobs periodically rewrote entire datasets in HDFS to absorb upstream changes - an inefficient and hard-to-scale approach given HDFS's immutable nature. Hoodie brings transactional guarantees to data lakes and handles streaming sources with bursty writes or late-arriving data. It's not just a table format but a complete platform offering table services like clustering, cleaning, and indexing. The platform supports record-level inserts, updates, and deletes on S3, enabling compliance with data privacy laws while consuming real-time streams and CDC data. Two main storage types exist: Copy-on-Write (COW) for batch workloads with higher write amplification but lower read amplification, and Merge-on-Read (MOR) for streaming scenarios. COW creates new parquet file versions on each update through merge operations, while supporting both snapshot queries (current table state) and incremental queries (changes between commits). ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=0) — [H1] Introduction to Apache Hoodie Origins - Key idea: Hoodie started at Uber in 2016 to address ETL inefficiencies during hypergrowth - Evidence: "dealing with petabyte of data" during Uber's ride lifecycle management #### ⏱️ [02:30](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=150) — [H2] Pre-Hoodie Challenges with HDFS - Key idea: Large Spark jobs periodically rewrote entire datasets in immutable HDFS - Evidence: "process was not only inefficient but also very hard to scale" #### ⏱️ [04:15](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=255) — [H1] Hoodie's Core Value Propositions - Key idea: Brings transactional guarantees and streaming data handling to data lakes - Evidence: "effectively work with streaming sources where you can handle bursty writes or late arriving data" #### ⏱️ [06:00](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=360) — [H2] High-Level Architecture Overview - Key idea: Hoodie ingests from any source and connects to any supported query engine - Evidence: "ingest data from particularly any source whether it's an OLTP database" #### ⏱️ [08:30](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=510) — [H2] Hoodie as Platform vs Table Format - Key idea: Hoodie provides table services beyond just formatting (clustering, cleaning, indexing) - Evidence: "hoodie is not just a table format but it's also a platform within itself" #### ⏱️ [10:45](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=645) — [H1] Record-Level Operations on S3 - Key idea: Enables inserts, updates, deletes on immutable storage for compliance - Evidence: "perform record-level inserts updates and deletes on S3 allowing you to comply for example with data privacy laws" #### ⏱️ [13:20](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=800) — [H2] Built-in Integrations and Storage Formats - Key idea: Uses Apache Parquet and Avro with Spark, Hive, Presto integrations - Evidence: "hoodie uses Apache parquet and also Apache Avro for data storage" #### ⏱️ [16:00](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=960) — [H1] Workshop Architecture with E-commerce Example - Key idea: Demonstrates cart operations (view, add, remove) flowing through S3 to Presto - Evidence: "e-commerce example where people are checking their cart or viewing items" #### ⏱️ [18:45](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=1125) — [H2] Copy-on-Write Storage Type Explained - Key idea: Higher write amplification, lower read amplification, good for batch workloads - Evidence: "every time you ingest data there's like kind of this merge operation happening" #### ⏱️ [22:15](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=1335) — [H1] Snapshot vs Incremental Queries - Key idea: Snapshot shows current table state, incremental shows changes between commits - Evidence: "snapshot query takes a current view of your whole table at that moment" #### ⏱️ [25:30](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=1530) — [H2] File Versioning and Update Mechanics - Key idea: Updates create new parquet versions while unchanged data keeps original files - Evidence: "when there updates the parquet version changes to a new version" #### ⏱️ [28:00](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=1680) — [H2] Incremental ETL Benefits - Key idea: Query only changes between commits to save compute resources - Evidence: "if you can just query the changes this is how you save on compute resources" #### ⏱️ [30:45](https://www.youtube.com/watch?v=Hk3wZbbmujc?t=1845) — [H1] Presto Integration Value - Key idea: Without Hoodie, Presto works with files directly without change optimization - Evidence: "without hoodie then Presto is just working with files directly it has no concept of what are the changes" ## 📚 References - [Apache Hoodie Official Site](https://hudi.apache.org/) - [Apache Parquet Documentation](https://parquet.apache.org/) - [AWS Glue Metastore](https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html) ## 🗣️ Paraphrase - Apache Hoodie emerged from Uber's need to efficiently handle massive data updates in their ride-sharing platform without rewriting entire datasets - The platform provides ACID transactions on data lakes while supporting both batch and streaming ingestion patterns with built-in optimization services - Copy-on-Write storage offers simplified operations for batch scenarios by merging updates into new file versions, enabling both full snapshots and incremental change queries ## 📰 Newsletter Apache Hoodie brings database-like capabilities to your data lake! This powerful platform originated at Uber to handle petabyte-scale updates efficiently. Unlike traditional approaches that rewrite entire datasets, Hoodie enables record-level operations on immutable storage like S3. Key benefits include transactional guarantees, streaming data support, and incremental query capabilities that dramatically reduce compute costs. The Copy-on-Write storage type works great for batch workloads, creating optimized parquet files while supporting both snapshot and incremental queries. Perfect for compliance requirements (GDPR deletions), real-time analytics, and building efficient ETL pipelines. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create Hoodie integration demos, present at data engineering conferences, track adoption metrics - Education: Develop lakehouse architecture workshops, office hours for Hoodie questions - Developer experience: Improve Hoodie documentation, create starter templates and SDKs - Developer marketing: Launch incremental query campaign, create comparison content vs competitors - Community management: Moderate Hoodie Slack, survey user pain points, collect integration feedback #### OSPO - License Compliance: Review Hoodie Apache 2.0 compliance, scan dependencies - Security Management: Monitor Hoodie CVEs, establish security response procedures - Software Inventory: Document Hoodie SBOM, track transitive dependencies - Developer Support: Train teams on Hoodie best practices, provide integration tooling - Community Engagement: Fund upstream Hoodie development, establish contributor agreements - Strategic Alignment: Define Hoodie adoption KPIs, measure lakehouse transformation progress ## 🧪 Hypotheses - Copy-on-Write storage provides better performance for batch analytical workloads compared to frequent streaming updates - Incremental queries reduce compute costs by 60-80% compared to full table scans in ETL pipelines - Hoodie's transactional guarantees enable reliable CDC ingestion without data loss or duplication - Integration with Presto/query engines provides better performance than direct file access for complex analytics ### 📊 Methodology - Benchmark COW vs MOR performance across different workload patterns and data volumes - Measure compute cost savings using incremental vs snapshot queries in production ETL pipelines - Test CDC reliability under various failure scenarios with transaction rollback capabilities ### ❓ Questions - How does Hoodie's clustering service impact query performance compared to manual partitioning strategies? - What are the optimal file sizing and commit frequency configurations for different use cases? - How does Hoodie perform with very wide tables (hundreds of columns) in terms of update efficiency? ### 🔮 Future Research - Performance comparison between Hoodie, Delta Lake, and Iceberg across diverse workloads - Integration patterns with modern streaming frameworks beyond Spark and Flink - Cost optimization strategies for Hoodie deployments in multi-cloud environments ### 💡 Implications - Practice: Teams can implement incremental ETL patterns reducing infrastructure costs significantly - Policy: Data governance frameworks need updates to handle lakehouse ACID properties and time travel - Security: New attack vectors around metadata manipulation require monitoring and access controls - Education: Data engineering curricula should include lakehouse architectures alongside traditional warehouses - Funding: Budget allocation shifts from ETL infrastructure to query optimization and governance tooling ### 🧠 Learning Styles Assessment Reading level estimate: Grade 12 (technical domain knowledge required) #### 👀 Visual learners - Present: Architecture diagrams, file versioning flowcharts, commit timeline visualizations - Improve: Interactive demos of COW operations, S3 bucket structure diagrams #### 🔊 Aural learners - Present: Video presentation with clear narration and examples - Improve: Audio-only summary of key concepts, podcast-style case study discussions #### 🗣️ Verbal learners - Present: Clear definitions of COW/MOR, analogies to database concepts - Improve: Glossary of lakehouse terms, real-world story comparisons #### 🧩 Kinesthetic learners - Present: Hands-on notebook exercises, CLI demonstrations - Improve: Step-by-step setup guides, troubleshooting checklists #### 🧠 Logical learners - Present: Performance comparisons, cost analysis, architectural decisions - Improve: Decision trees for storage type selection, ROI calculators #### 🧑‍🤝‍🧑 Social learners - Present: Community Slack mentions, collaborative workshops - Improve: Peer review processes, team implementation guides #### 🧘‍♀️ Solitary learners - Present: Individual notebook exercises, self-paced learning - Improve: Reflection questions on architecture decisions, personal project ideas #### 🌿 Naturalistic learners - Present: Data lifecycle analogies, ecosystem thinking - Improve: Environmental impact of compute optimization, sustainability metrics ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive slide content - Improve: Alt text for architectural diagrams, structured table descriptions #### Keyboard navigation - Present: Standard video controls available - Improve: Slide-by-slide navigation, skip-to-section links #### Contrast and readability - Present: Good text contrast in presentation slides - Improve: High contrast mode for diagrams, larger code font sizes #### Text resizing and reflow - Present: Video scales appropriately - Improve: Transcript with responsive text layout #### Media and captions - Present: Video with audio narration - Improve: Complete captions, downloadable transcript, audio descriptions #### Standards reference - Address WCAG 2.2 AA requirements for time-based media and complex images ### 🌍 DEIB Assessment #### Cultural breadth - Present: Technical examples applicable globally - Improve: Multi-region deployment examples, diverse use case scenarios #### Inclusive language - Present: Technical terminology with definitions provided - Improve: Plain-language summaries, avoid unnecessarily complex jargon #### Neurodiversity support - Present: Structured presentation with clear sections - Improve: Content warnings for rapid technical demonstrations, consistent pacing #### Representation - Present: Female presenter in technical role - Improve: Diverse speaker lineup, varied industry examples #### Psychological safety - Present: Encouraging tone about learning complex concepts - Improve: Explicit welcome for questions, beginner-friendly alternatives ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear progression from problem (HDFS inefficiency) to solution (Hoodie features) - Inductive strength: Uber case study generalizes to other high-scale scenarios effectively - Causal reasoning: Demonstrates how file versioning enables incremental queries and cost savings - Analogical strength: Database transaction concepts map well to data lake operations #### Fallacies - Appeal to authority: Uber's success doesn't guarantee applicability to all use cases - needs qualification - Hasty generalization: Cost savings claims need quantified evidence rather than theoretical benefits #### Tone of voice - Overall tone: Educational and confident with practical examples, appropriate technical depth for audience ### 🛡️ DARVO Analysis - Deny: No significant denial patterns observed - Attack: No attacks on competing approaches or critics - Reverse victim/offender: Not applicable to technical presentation - Emotional cues and rhetoric: Positive framing focused on problem-solving rather than competitive positioning ## 🎯 Conclusion Apache Hoodie represents a significant advancement in data lake capabilities by bringing transact - Document Tags: [[hoodie]] [[indexing]] [[performance]] - Summary: Deploying Presto in AWS with Ahana Cloud - URL: https://www.youtube.com/watch?v=Hk3wZbbmujc all right let's get started with thesecond portionum I'm talking about Apache hoodie andhow it plays a role in a data lake houseso I'm Nadine I am an Apache hoodiecontributor and I'm coming in from onehouse and I'm working with Wen today onshowing you what you can do with thelake housesoum let's go ahead and get startedso let's talk about the origins ofApache hoodie so hoodie started at Uber in 2016 to kind of addressinefficiencies across ingestion and ETLpipelines during its hyper growth stageso during this time hoodie was dealingwith petabyte of data and if we walkthrough an example of an Uber ride thereis a life cycle of a writer so a writeryou know starts a ride it goes throughthe trip and then it finally ends itends the writer ends its ride so thereare multiple updates that can happen so before hoodie there were large sparkjobsthat were used to periodically rewriteentire data sets in Apache hdfs toabsorb Upstream online table inserts andupdates and deletesum reflecting changes of the trip statusso if we remember hdsf is an immutablestore right so this process was not onlyinefficient but also very hard to scalpso what does hoodie bring to the dataLake well we kind of talked about this a little earlier but one of the mainthings it brings is transactionalguarantees to the data Lake and this isreally important when you're workingwith big data so one of the other thingsthat hoodie really Prides itself in andhow it was natively built from the geckowas to effectively work with stringsources where you can handle burstyrights or late arriving data and so onso this was literally the inspirationabout uh with hoodie soum so we talk about hoodie when we get into the connectors and stuff we'llshowcase this a little bit moreand this is this part here so if we takea look at this kind of overviewhigh-level architecture diagramhoodie can ingest data from particularlyany Source whether it's an oltp databaseyou know any types of data lakes or datawarehouses and streaming sources so onceyou ingest data from there hoodie we'llmanage your hoodie tables within thedata Lake like for example today we're going to be working with S3 to performspecific sort of services to eitheringest or manage the dataum so once you you know for example youclean the old data or you want toperform clustering uh to be able to youknow maintain your ingestions as well asmaintain your query performance and sothat hoodie can help do that within umwithin its platform and from there youcan connect to any query engine of yourchoice that hoodie supports to be able to get the data and then performanalytics or build some sort of dataapplication that you want that you wantto build and so with hoodie you canbuild near real-time applications forexample you can build a personalizationapp so you can build a customer 360 appand so on and so forthso we talked a lot about like thedifferent table formats and it's good tomention that hey that hoodie is not just a table format but it's also a platformwithin itself where there are manyservices to ensure that you get writeand read optimizationsum so one of Hoodie has a set of tableservices for clustering cleaningum and so on so for example if we talkabout the clustering you can schedule aclustering service and then you cancreate your clustering plan and then youcan execute the service on top of it wewon't explore clustering today but theseare the various examples of what hoodieuh what hoodie brings we also I think there was a questionabout indexing hoodie has indexes thatsupport right optimizations as read asas well as read optimization so they'rea whole Suite of indexes that if we wantto get into we can talk aboutso who using hoodie you can performrecord level inserts updates and deleteson S3 allowing you to comply for examplewith data privacy laws or you canconsume real-time streams or you canwork with change data captures or CDC you could re you can reinstate latearriving data and track history androllbacks Hoodie has a concept of atimeline and a timeline you can seeeverything that's happening to yourtable so you create data sets and tableshoodie manages the underlying formathoodie uses Apache parquet which whentouched about and also Apache Avro fordata storage and includes built-inIntegrations with spark Hive and Prestoand this allows you to query any hoodiedata set using some tools that you're familiar with and love to use todayum so in hudi we're going to coverconsistent snapshots point in timetravel incremental changes later on inthe workshop so we'll cover we'll goover some of the table storages thathoodie offers which is copy on write andmerge on read and we'll go one of thethings we'll cover is how to doincremental updates on hitting and toquery those updates to see what thosechanges areum so if you're interested in learning more and want to get involved withApache hoodie Community you can followus on slack I'm usually pretty active onslack and there's also you know blogsthat are on the the hoodie site as wellas videos on getting started and andmuch moreso this is kind of a quickum overview of hoodie and then from herewe can get started on the actual labportionso let me go aheadand let me go ahead and switch my I have anotherdoc really quickand then let me go ahead and switch thatoverokaysookay let's do this so before we get into the actualnotebooks I want to go ahead and talkabout some Concepts really quick andthen we'll get into the actual theactual lab itselfso let me go ahead and put this onpresenter Viewum I think that's rightokayso we're going to talk about thedifferent storage types that hoodie has and then we're going to walk through alittle bit of the workshopum and then we're going to go intolooking into the future with theconnection with with integration betweenPrestone hoodieso this is kind of a high levelarchitect overview of what we'rearchitecture diagram of what we're goingto go over so what we have in the dataset today is we have some sort of likee-commerce example where people arechecking their card or viewing itemsadding item to cart things like this sowe have a lot of updates inserts and kind of uh maybe things people removingthings from the items from their cart sodeletes happening and it's all happeningand getting started into the S3 dataLake from there we're gonna on thehoodie side we're going to write thedata and we're going to query the dataand then from and we're going to writethe data to the hoodie table and thenfrom there we're going to use Presto toquery the data and we're going to usethe glue the glue the glue Hivemetastore and that's the that's howwe're going to query the S uh that's three data so in the first part we'reactually in the jupyter notebook we'reactually going to query through sparkand then at the end we're going to showyou how you can query through the PrestoCLI which is what one showed you whenyou were in when you SSH into the Prestoumuh ec2 instanceso I already talked about thisokayso the couple of things I want to coveris a different storage types that areavailable on on hoodie so hoodie has what we call a copy on write storagetype which is if you work with the datawarehouse it's exactly the same thingand then the other storage type thathoodie offers is what we called merge onread or Morand we're going to talk about a lot alittle bit about thisso a copy on Right storage is reallygood if you want to havea higher right amplification by a verylow read optification and it'soperationally on the hoodie side less uhless complex so what I mean is if you have if you're working with batch datasets and let's say you're you know every15 minutes or every hour you'reingesting data you would probably wouldwant to use a copy on right table forfor that in the copy on right table usesApache parqueand then what happens though is everytime you ingest data as you're ingestingdata there's like kind of this mergeoperation happening where it's combiningthe new data with the data that alreadyexists on the parquet file and itcreates a new version of that parquetfile because of this merge operation it has a high slightly higher writeamplification when it comes to the queryside it has a lower amplification forreading because you can just readdirectly in the parquet files and sothis is what a copy on right is we'lltalk about what this difference isbetween a copy and write and a merge onread table is later on but let's talkabout what this what this screen isshowing you so with hoodie you have twodifferent types of queries you can runon a copy on write or cow table andthey're called a snapshot query and an incremental querya snapshot query takes a it's basicallylike a current view of your whole tableat what as it exists in that moment anincremental query is basically justshowing you the updates that happenbetween let's say 0.2 or 0.3 or committo commit three or commit for a commitfive or commit to and commit five itdoesn't give you the whole snapshot butit gives you the just the changes thathappen when you were ingesting data soif we look at the blue boxes we have an insertion of a b c d e and a commit timeof zero and if you seeum the the prefixes like file oneunderscore t0.4k file two t0 dot part Kand file three underscore t0 dot parquetwe can see that these are basically ABCDis in searching these are the parkingfiles that it's being insertion so ifyou did a snapshot query it's going tobe the same thing A B C D E and if youdid an incremental query you're going tosee the whole uh blue boxes of what is because it's commit time zero and therehasn't been any changes and it's justthe first insertion of datanow if we go to the orange boxes now yousee if you look at the orange boxes thatcommit time one you see a goes to aprime and B doesn't change so a gotupdated basically B nothing happenedthere's no update somebody if you go ona second uh the file two you seeum see the C uh the C data didn't changebut D went to D Prime so there was anupdate on um on on the D data set and then e there was nothing happenedthere's no update so it didn't there wasno new version of the parquet so if yousee when there updates the parquetversion changes to a new version andthis is where kind of the merges happenwhere you get the new data plus theexisting data and there's like somemerge logic that happens and then thethe new version of the file gets createdwith e there's nothing happen so it'sjust still file file three underscore t0because there is no update happening butif you ran a snapshot queryon this you would see the current state of uh of the table basically so you'llsee a prime b c d Prime e because aprime and D Prime got updated but therest stayed the same but if you justwanted to run an incremental query andyou wanted to see okay well what changedbetween commit time 0 and commit timeone well really a prime got updated andD Prime got updated so you can justquery what you want to what you want tosee change and this is actually how youbuild an incremental ETL you can you canjust to get those changes if we think about a compute efficiency where if youhave to rescan the whole data thatcauses increased compute resources to dothat but if you can just query thechanges this is how you save on computeresources so this is what hoodie offersis incremental updates incrementalqueries things like this now if we go tothe red box or a red orange box if youwillum we have an update where a prime goesto a double Prime B didn't change CDPrime didn't change but we have a newinsert well egot e got updated to ePrime and then we have a new insert with f so now if you ran a snapshot query onCommit time 2 you can see that thecurrent state of uh of the table is adouble Primeum b c that didn't change D Prime ePrime and F because f is a new insertionand then if you wanted to run anincremental query on this you get thechanges between commit one and commit 2.so you get a double Prime e Prime andthen we have a new insertion which is fyes just want to make a comment to connect some thoughts is a great slideso I I there's a lot to follow here Ithink but um I think that main takeawayis without hoodie then Presto is justworking with files directly it has noconcept ofuh what are the changes it has noconcept of how to optimize the file sothat I can regret so that's kind of thelayer that hoodie is really adding hereand the way that it does that it hasother metadata that it knows how to workwith that was kind of the hoodie filethat I showed you or directory inside the data Lakeso that's kind of the high level thing Iwant to make sure you take away andshe'll go with other table types butthat's so you notice that they're allparquet but now the engine knows how tointerpret these different parquet filesand can do a lot more advanced kind ofcapabilities because of this uh formatand if we talk about these changes rightthese are updates that are happeningright um these are like changes that arehappening so this is what's happening onthe data Lake but he provides thattransactional guarantees, Deploying Presto in AWS with Ahana Cloud, YouTube, 01ka7nryzxzc7j3tbvxs5jp29f.