YouTube Video rzWatCmeG9c ![rw-book-cover](https://img.youtube.com/vi/rzWatCmeG9c/maxresdefault.jpg) ## Metadata - Author: [[YouTube]] - Full Title: YouTube Video rzWatCmeG9c - Category: #articles - Document Tags: [[maintenance]] [[schema]] [[versioning]] - Summary: YouTube Video rzWatCmeG9c - URL: https://www.youtube.com/watch?v=rzWatCmeG9c So to get started um first thing is I am not used to speaking to a room where I can see all the people. This is odd. Hi. Yeah. Hi. Hi in the back. Um first of all I want to thank you all for coming. Um, I have been on the talk circuit for a long time, decades, and um, but since the pandemic, I've spoken once, and I thought I retired. Um, but Nitia Ruff, who won the award, the big board today, she's like, "Brian, you should come do this." So, I'm like, "I'll put together a talk." And I don't want to talk about AI because you're gonna I looked on the schedule, all the talks are about AI. So, I'm not talking about AI today. And and usually I talk about infrastructure for many years, infrastructure, programming languages, and I'm not talking about any of those. I wanted to talk about Apache iceberg, open source, one of the neatest pieces of software I've ever seen. So I submitted this talk. They were like, "Hey, Brian, what do you want to talk about?" And I said, "I'm going to talk about from files to tables, and we're going to rethink how we store and query data." That seems pretty exciting. But um as it always happens, um the conference reached out and they're like, "Hey, Brian." And I'm like, "What's up?" And then they were like, "Uh, we want to move you to the dev check." And I'm like, "For real?" And then in typical Brian fashion, I'm like, "Say less." So we get a new talk type. We got a new title for this talk. And now it's developer lessons from Apache iceberg lessons in structure, scale, and software design. I'm Brian Lyles. I am a senior principal engineer for Amazon Web Services. I am the tech lead for our entire data analytics stack. So software like Glue, EMR, Athena. Yeah, that's somewhere in my purview. Um I am a three I've hit three decades of working for money for code. I have no stress so you see it not in my face. Um but um I am really true to the open source community. That's how I got my started. My first patch, my first open source, I didn't even call it that at the time was when I was in 11th grade when I was in high school and my video or my sound card didn't work. So I'm like, "Oh, I downloaded Linux. Let me go do some C." And I stumbled and I got it working and I submitted it. And I did not know what I was doing at the time, but back in 1993, that started my journey. So I am 32 years in of doing open source. And I'm weird about my talks. Um, older almost 50. Brian is like, we should give talks that not only talk about tech, but should talk about the world. And we should do it in a way that is very approachable to everyone. So, I said I'm going to split this talk into three acts and I'm going to talk about something that happened in the history um the library of Alexandria. Have y'all heard about this thing? >> All right. This is interesting though. So, over a couple thousand years ago in Alexandria, not Alexandria, Virginia, which is near where I live, but Alexandria in northern Africa, Egypt, um this is what they said. They said, "What we're going to do is when all the ships come in, we're going to take all the original scrolls out, we're going to copy them and we're going to save the original scrolls in our library and that way that all this knowledge is being preserved." And I'm like, "That's a great story." But then I thought about it. I'm like, "Wait a second. They stole all the scrolls off the the ships. That's not what this talk is about." And they said, "You know what we're going to do? We're going to put all of these scrolls in one place." And you know what ended up happening? That was fine. Um the library grew and it's actually was one of the winners of the world. But um a problem happened. It kind of caught on fire and now we don't even know what it looks like. But here's something interesting that the cataloging was inconsistent and these knowledge was stuck in these physical scrolls. And when it came time whenever the wars came and the fire came, guess what? We had all these scrolls and we couldn't find anything. So, we thought about this for a while, hundreds of years, more hundreds of years, and we invented this thing. Do you all know what this is? Um, I went to school in the 1900s, as my daughters say. And this is where we went to find books. It is a card catalog. And it was um and the version I learned used the Dewey decimal system which was some numbers which told you what shelf or what row and what shelf and where to find this book. And I'm like oh my god this is amazing because I love libraries. And then we invented this. Do you all know what this is? It's an ISBN. And what it breaks down to is that every book, magazine, journal, periodical, periodic um gets assigned this number that says um it was here in the world and it was this company and um it was this release. Multiple editions get multiple numbers. And then the the first the last part that four is a check digit. And I'm a nerd. I just like math. I'm like someone actually said that we could check this to make sure that it was correct. That's pretty cool. Do you all know what this is? Um, this is what changed the world. And I actually will say that this is what got a lot of us here today. It was a Google search bar. You know what I can go do? I can go search for anything. And years ago, you could find anything. Now, now you know, you know, things happened. And and maybe that is not as true as it used to be, but um that's not the end of our evolution. And now instead of Google, I use Plexity. Actually, I did say this is not an AI talk. Um, this is the only AI in this talk. U I use Plexity instead of Google now is because not only can it search, it searches on the context. And I think this is where the history is. So I'm only bringing this up is that um we really had some evolutions here. So libraries evolved from shelves to systems. We invented cataloges, ISBNs, and you know what? We started creating standards, universal ways to talk about our data. And so what did we learn? Well, we learned that there's a catalog first. Don't just put your scrolls on the shelves. You'll never be able to find them. Catalog them where they are. You know, some little information about it, and then do storage. So this talk is about old school card cataloges to new school code. And you know what? Don't read the code. Just know that it's JSON. Does not really matter what's in this right here, but this is kind of important. And this is where I tie this soft this talk to software. So we've been we've spent decades treating files and folders as truth. I mean, I can show you on my desktop right now. Um, I have this folder here called secret files. And what I do is when my desktop gets too messy, I create a folder with the date. I put every file in there. And then I put that file in the secret files folder. That's me doing classification. But this is what this talk is about. It's about how do we at scale do these great things? And remember, this is a dev talk. So there's some more dev stuff coming as well. So, here's our transition. We don't shove things by guesswork. And I did a little bit of research on libraries and and I'm going to use shelf in my in my lexicon now. I'm going to say let's shove that. We should shove together. Um I don't understand. I just like words with V's in them. And shove is just a really good one. So, we're not to the talk yet. It's a three-act talk and we're on act zero. So, this is just, you know, the music or the orchestra. um warming up and once again we're not to the talk yet because what I really want to talk about today is Apache iceberg. Who here is familiar with the term Apache iceberg? Who here has committed code to Apache iceberg? All right, good. I can say whatever I want and no one will know. So, I do want to level set because I know we go to these talks sometimes, especially dev talks, and they're like, I'm going to talk about whisbang fu. And then they could talk about it, and you're like, wait a second, you lost me at Whiz, and I didn't even get to the bang or the foo. So, I'm going to give you a highle primer on this amazing piece of software called Apache iceberg. So, there's this thing called open table formats. And if you listen to the keynote this morning, the presenter from Google actually mentioned some of the popular ones. There's one called Apache Iceberg, there's Hoodie, and then there is Delta Lake. And what these do is they are they're ways of describing files in object storage in a way that makes it easier for you to access them in the future. Now, why would you want to do that? Well, everyone we're past big data and we can't call it like supreme big huge data, but everyone takes all their data and they put it in S3 or Azure storage or Google storage or minio locally. And what ends up happening is you can't find any of these things. So what Apache iceberg does is it actually puts some standards on top of these things. So we'll start from the top of this image. So we go from data files you know all the stuff and notice it's not just a file it's parquet or it's orc or it's arrow and these are formats where we can store data and then we can get some information about the data we store and then there's manifest files that list these data files and then there's manifest list that lists the manifest files and then there's metadata files that just gives you a little bit of more information about what's here and then there's this idea of a catalog Remember I said catalog first, shelf second. And this catalog is well where do I find this data? It's super important. So why does this matter? So really what it comes down to is that there is no winner. There is no best way that stands the test of time. And we can say that and you know we can like no Brian this is actually the best thing. And actually I will give you an example of something that I think is the winner. Um Amazon S3 that's a winner. Uh it's a winner because of the we talk about the S3 protocol and everyone implements S3 protocol. You know I would say that is the winner for object storage get push um head delete that's that's a winner but not a lot of other winners and we should just keep in mind that things will change. We will create new database engines. We will create new file formats and we should embrace change right up front. So I'm not one for reading slides. The slides are for you to read whenever you get tired of talking or me talking. Um but think about this. We have one table, many engines, and one spec to rule them all. So we're in the developer track and developers love good abstractions. and iceberg really when you boil down to it, it's just an it's an abstraction for data. This is one step above. We spent so many years talking about infrastructure engineering and things like that. I spent many years in the Kubernetes space and years before that in other spaces, but now I'm talking about abstractions for data. So let's get on with the talk act one. So the idea of this talk is to use Apache iceberg as an explanation on how you can become a good developer. But if then you use Apache iceberg, you can take advantage of all these great patterns. It's kind of meta in this way. So what I wanted to do is I think I have like eight examples of just amazing patterns in Apache iceberg that I think will make you a better developer. So the first one is um spec first contracts plus stable ids. That's a that's that's that's a that's a lot of words, but really what it comes down to is um in libraries in ISBN changes when a new edition is printed and it's specific to the physical manifestation of that book, but the catalog record, the entry in the library system stays the same. So, how can we actually take advantage of that in our dev life? And what I wanted to do is um I actually wanted to say, well, how can I do Apache iceberg? Um, okay. How can I do Apache iceberg in a talk with live demos with, you know, who knows if I even have internet? So what I ended up doing was I run I wrote a whole system to run Apache iceberg locally to do all these explanations for me and then I'm going to show you like a Python um cheater at the end just so you can actually see what I did here. So this first one is an example about renaming columns. Who here has had a database before? Anyone here had a database before? Right. Yeah. Who here has dropped a database in production? That's not this talk. you should never telling yourself. Um but what it really what happens is is that we create a schema and this schema is really simple. We have a user ID, we have email address, um we have full name and a created at. And then we put some data in there. And then you know the PHP says, well, hold on. Uh oh. Okay. We can see that there's data in here. But then someone's like, well, hold on. Email address is really long. Why don't we rename it email? And also, you know, full name is super long. Once again, we're antiore. We're going to rename this to name. So what happens in your typical databases like Postgress or MySQL if you do this? Well, there's an issue is that um we don't think about stable ids a lot of time when we're thinking about our relational databases. So what ends up happening is um we lose those changes in the past. But what ends up happening with Apache iceberg is because it has stable ids, it actually doesn't use the name. It uses a pointer to another ID. Um, what ends up happening is now when I change names and I have data over many files, you know, I could have literally terabytes of of data here, um, I don't lose any of this. So I can see in my query three that I can select all the columns showing the new schema and I can reorder columns if I would like to reorder columns. And let's see what else we have here. And that no longer becomes a problem. And you're like Brian why were you we really talking about this? This is a dev talk. Well here's the reason why. And I'm going to this is the this is the cheater in Python. I have a schema and if I rename some columns, notice I use a dict for the schema instead of actually having the names. I can actually rename but when I go to read those records in the future, I don't have to worry about that anymore. So that's a simple one. I mean that's actually that that's almost trivial. But another problem that we have with big data especially data in data lakes is over time our partitions need and our our schemas and our partitions change and so how do we do that without rewriting data so um the example that I'll use here is I have data and I'm a small startup and I'm partitioning on daily or weekly and then I become a big huge uh success some kind of viral success And I need to think about well what if I want my queries to be partitioned by hourly or minute because it's all streaming in now. So what does the library do here? Well the library when a new addition comes in it doesn't break the catalog. We just add another record to the card catalog system. And so what we need the lesson that we have to learn here is that you need to decouple the logical schema. you know where the catalog knows the data from from where it's stored. So um no demo for this one um but I'll just show you this code here that ex explains this and why this is important. When we write code and we say that there will never be any change because we didn't actually allow for change. What we're actually doing is compounding that debt early and we're making it very difficult for us to change our code. And when you make it very difficult for you people to change your code um progress slows down and people don't want to work on that code. So instead of making that code better we create another whole system and now you have your second system and then you get your third system and then you get your fifth system and then someone comes in and says why do we have all these systems and it was just because you did not design your code for change. So here's another important one is versioning as a first class feature. You need ways to experiment. You need ways to roll back. And why do we need that? Well, my library example here is that we check out and return history. Every change is logged. If you're old enough, you can go in the back of the book, pull the card out, and you can see who checked this book out from 10 years ago. Um, but now inside of Apache Iceberg, we can actually get this system for free. We can go backwards in time. We can go forwards in time. And guess what we did not do? We didn't change any of the data. So, I'm actually am going to show you this demo. And in this demo, um, what we're going to do is we have a bookstore and we have inventory with some stock in there. And you'll see that we have some current inventory. So now what we want to do is we're going to have a sale and we're going to change some prices of things. And when we change the prices of things, you know, that's where we are. But you'll notice that every time that we change data with iceberg, we get a snapshot, which means that I can actually go back. So whenever I update my demo or update the prices again after the sale, um, I'm still getting snapshots. So what does this mean? What this means is what this means is that I can actually say, well, what did my inventory look like on Monday before the sale? What did my inventory look like on Tuesday during the sale? And what did my inventory look like on Wednesday after the sale? You had to create no other data structures. Um, iceberg actually provides that snapshotting for you built in. And why do we want to do this? And actually, I'll give you another example of something that's important for versioning. Um, anyone here used Kubernetes? Like really use Kubernetes? Like have written a spec file like written manifest yourself by hand with Vim or Ed? You've written with Ed? >> All right. No ED editors here. Um, so here's why that's important. If you go to every single what's the first line of every single manifest line >> what's the second line of every single >> huh >> yes so you have an API version and you have a kind and why is the API versions important as we've when we learned early in Kubernetes whenever we had um v1 betas and then we and then we realized oh this is ready for production you know people could move with us and you could stick on your v1 beta forever ever or your v1 beta 1 forever. But now because it's built into the system, we didn't have to think about across Kubernetes of how to make changes and bring people along with us. And the obligatory um Python example here is look here. I'm actually building and versioning. I'm as I do my commits, I just have a snapshots and that could be a right-headed log or anything else. But um for this code to fit on this slide, it's it's actually just an it's just a list. And whenever I write there, I can always see my snapshots and I can always go back. So let's move on to um act two correctness at scale and at scale is the most important part here. So the issue that I'm going to talk about now is optimistic concurrency and atomic commits and as we know um distributed systems are super messy especially because whenever you introduce that second writer there's a problem here and now we have to think about ways to do locking so that we can actually um get data in but what snapshot I mean what iceberg does here is pretty neat because we have snapshot based um atomic commits what we can actually do is without doing any work is I do my commit and whenever I want to I want to commit again I know what the current version of the database is and when it's not that version anymore I just I um I get the current version and then I can write when I know that the data hasn't changed from underneath me super simple thing but you get this for almost free inside of iceberg so we should always build for retries we should always assume assume that there's going to be failure and especially at scale things are going to fail. Matter of fact, if you work at scale right now, something is failing right this second and you should just assume that if it's not failing, you didn't measure or your monitoring is broken. Something is always going to fail. So assume because it's always going to fail that we will code failure detection and things to get around that. We'll design for that at the beginning. So here's another important part thing for working at scale. Um skipping work and at the end of this talk what I what I have is a whole bunch of quotes and one of my favorite ones is you know the easiest work is the work you didn't do and you kind of get this in big SQL systems like iceberg is. So what we want to do is we have all this data. So I have all the data for all the hits to my website for all times. instead of looking for I'm want to find if someone tried to go to a certain page. So, what do I do? Well, from from my point of view, well, I can just go search for that that file. I can search for that um part of the URL and I'll see if I get it. But I also know that I didn't add that until, you know, uh September of last month. And so, what I should do is never scan before last month. And really what this in is called is called predicate push down. And so what we should do when we're building these big systems is the fastest thing to do is to not do anything. And I like my library quote here is um don't walk every shelf. If I'm looking for a book in the library, I go to the card catalog. It tells me about where in the library it is and then I can see what shelf it's going to be on. I don't have to walk down every single row of the library to find a book. So once again, simple code here. And what I'm doing here is I'm just looking at my min and my max. And I'm realizing that my code is a little bit off here, but I'm filtering out rows that I don't need to look. So I don't need to scan those things. So let's go to act three. And this is my favorite one because I am forever an infrastructure person and I like to talk about infra related things. So what we're going to talk about in this section over I think two or three is operability as a product. So the first thing is um and you probably caught this as I was saying it earlier is that because Apache iceberg can do a lot of snapshots. I'm sure there's a lot of files that get created that hold all my snapshots. And what can we do there? Well, there is something built into Apache iceberg called compaction. And what it does is instead of having to scan thousands of files, it's a process that runs out of band and then creates a new snapshot with all those things combined. So now instead of having to uh do all of those uh scans, I have to do one and and actually I will show a demo for this one. Let me um I think this is demo five. Nope, it's demo six. So in my demo here I have all these files all these parket files and I actually do need to search through them to to see the data but what I can do is I can as I'm doing my query notice the time there it took me you know 0.2 two to two to almost um a third of a second 03 of a second to do all this scan because I have to scan 20 small files. But now um I can compact and instead of going from you know a whole bunch of files I can go down to one or two. And now when I do my query you'll see here that um it goes from 0299 to you know an order of magnitude faster to um do this because there's less files to scan through. And I apologize. I assumed that I could see these things. And I I'm looking down on this monitor right here and I should turn around. Look at this one. It's much bigger. Yes. Um but so see that using compaction instead of scanning every like a whole bunch of files, you know, try to move that down to to one. And the and the example here is actually the more important part is that and when we build these big systems, we do maintenance as an afterthought. We clean up big things when they become a problem. We um clean we know that there's there's demons and devils over here in this corner and we just stay away from it until we have to. So really you should be more proactive instead of reactive and treat maintenance as a feature, not an afterthought. Now two more. So this next one is um observability through metadata tables. So this is the best part about Apache iceberg. And remember before I was saying that there is manifest list, manifest files and then there's metadata. There's so much data in the system that describes the system working itself. In the library in the library, you have the card catalog and that's kind of what it does. You can actually see data about who the author was, when it was published, and if you really cared, what its ISBN number was, where it is in the library. So in iceberg, they have something very similar where you can query not only your data, but there's system tables where you can query about data of your system. And there's some really interesting data in this system as well. So one of the things you can find is anyone here familiar with lineage of your data. So because you can actually query the metadata inside of Apache iceberg, you can build lineage of data. So you had raw data that was streamed in from Flink and it was stored in partition buckets and then you had a data engineer clean it up to make sure that the data quality was right. And then you actually had an analyst or a science do experiments on this data to produce this other data that was managed that was merged with another piece of data and now you have this one datam here at the end. Now using Apache iceberg you can actually determine where that data came from over time. And the examples here is because I have all these snapshots, I can actually do this. And and because I have the snapshot, I can actually determine when this data was moved, when this when this data was what came into the system, when this data might have been um changed through an aggregation or an ETL or something like that and at the end I can actually say, hey, this is why I have this is where this data came from. So the final part of act three yes the final part of act three is clean boundaries and this is what made Apache iceberg the winner is that it disagregated storage and compute before like you're thinking about your um your OOLTP databases your Postgress your MySQL your Oracles you we had tight coupling between the storage and to compute. Generally, the data was stored where you did the query and then we got we got more complex um we got more complex implementations where we were now writing to certain places and reading from other places but still um it was kind of messy. One thing that you can do with Apache Iceberg is you can actually pull out the query engine entirely and you can use different types of query frontends whether it be Trino or Spark or other ones in some cases. And the reason this is possible is because um Apache icebergs exposes this thing called the rest catalog API and it separates basically the table metadata from the engines. So, I am standing between you all in lunchtime and and what I want to do is um give you all a little bit of a treat. And what this is is um if you want to take anything from this talk, I distilled it into a set of quotes. And so, let me show you the set of quotes in a few slides. So, the first thing to remember is the shelf doesn't change. Oh, the shelf will change, but the catalog must not. So, you must have a canonical source of where everything is. That's the most important thing. Iceberg, uh, as we look at Apache iceberg, it looks like a data project, but if you go read the source code and actually go look at the descriptions of how it's put together, it's just about design. And you could take those lessons from you could take those lessons from this data project and use them on pretty much any of your other projects. So starting with my um the next slide if you want to if you want to get pictures is the next slide. So the core takeaways here are APIs contracts you know schema evolution things will change versioning you will make a mistake and people will want to stay where they are so make that make that a first class system of your first class component of your system. Um atomic commits realize that when I commit something it's there we know it's there. Um, don't do work that you don't have to do. Um, build maintenance into your APIs, whatever you're building. Also, make sure that you can look at your system and observe it and then set boundaries and be purpose with those purposeful with those boundaries. Set those contracts early. So, couple ideas and these could all be LinkedIn post. Um, and they probably will be if you follow me on LinkedIn, you probably know that I just post about random stuff. So, names are for humans. Remember that any nice name is for humans. They are not for computers. They are for systems. I mean names are for computers. IDs are for systems. Here's my favorite one. If roll back is scary, you built a trap, not a platform. If you do deploy and they're like, "Let's roll back." And someone says, "Oh, I don't know about that." That's a problem. You should go fix that. Another favorite one op should be a API not a runbook. Now this is aspirational for a lot of teams but really think about it your operations at scale should be through an API and it shouldn't take a you know a person who had to wake up at 4 a.m. to read this runbook and then go type something in on a production system. It should be an API. The fastest job is one you never run. And then another one here is when knowledge is not stored. When knowledge is stored but not structured, it is lost. Don't create data heaps with just stuff in them. You know, put some structure around them. Don't shelf by guesswork. Shelf by spec. Find out what your card catalog is and use that. The shelf will change. The catalog must not. And one of my favorite ones here, and this is probably my favorite um slide in the whole talk, is preserve knowledge, make it discoverable, and design it for generations to come because you will not be the last person to touch this code unless it's the code that you wrote for your home rack at home. So remember that, let's do good things for the people who come behind us. And I will tie this back into the open source um open source um theme of this conference is the reason why I contributed that change when I was in 11th grade back in 1992 93ish is because I wanted no one to have the problems to use their little Sound Blaster card and their Linux server. So that's it. Thank you. Go have lunch., YouTube Video rzWatCmeG9c, YouTube, 01kabx4rg3wthmxa34zkqcgy5p.