Open Source, on-Disk Vector Search With LanceDB  ## Metadata - Author: [[Changelog Master Feed]] - Full Title: Open Source, on-Disk Vector Search With LanceDB - Category: #podcasts - URL: https://share.snipd.com/episode/db907bb0-1908-40a8-8c31-46a595ce963b ## Document Notes ## Highlights - **LanceDB's Origin Story** - LanceDB originally started as a data infrastructure solution for computer vision projects, not as a vector database. - The vector search capabilities were later separated and popularized for generative AI use cases by the open source community. Transcript: Speaker 1 Absolutely. And first, I wanted to also give a big shout out to Prashant as well. As you were saying, there's a lot of hype and noise in this area. There are a lot of different choices. And for users and developers who are building generative AI tooling and applications, it's always kind of confusing, like which one is good? And should you listen to the marketing from one tool versus another? So it's great to see someone with an engineering background who can write so well to actually take the time and just try out a ton of different tools and interview a bunch of different Companies and come to his own conclusions. I'm super happy and excited that he's a fan of LandCV and we hope to make that better for him and also all of our users. So, you know, back to LandCV, I think, so we started the company two years ago at this point, we didn't start out as a vector database company, actually. Because I think if you kind of remember, ChatGPT is barely one year old. Speaker 2 Yeah, the dawn of AI. Yes, exactly. Speaker 1 And so the original motivation was actually serving companies building computer vision and building new data infrastructure for computer vision. So I had been working in this space for a long time. I'd been building data and machine learning tooling for about almost two decades at this point. I started out my career as a financial quant, and then I became involved in Python Open Source. I was one of the original co-authors of the Pandas library. And that really got me sort of excited about open source, about Python and building tools for data scientists and machine learning engineers. And so at the time, this was in 2020 and 2021, what I observed was that the company I was working for, 2BTV, it was a streaming company. So we dealt with both machine learning problems for tabular data and also for unstructured data like images and the video assets and things like that. And what I had noticed was that anytime a project touched this multimodal data for AI, from images to like the text for, you know, let's say subtitles or summaries to the poster images. These projects always took a lot longer. They were much harder to maintain, and it was difficult to actually put into production. At the same time, so my co-founder, Leigh, who I had met during my days at Cloudera, he was working at Cruise and sort of dealing with the same issues. And so we put our heads together and our conclusion was that, hey, it's not the sort of top application or workflow layer or orchestration layer that's the problem. It's the underlying data infrastructure. If you look at sort of what's been out there, like, you know, Parquet and Org has been around and they've been great for tabular data, But they really suck for managing unstructured data. And so we essentially said, hey, what would it take to build a single source of truth where we can toss in the tabular data plus the unstructured data and give much better performance At a much lower cost, a total cost of ownership, an easier foundation to build on top of for companies dealing with but a lot of vision data. And so this comes in handy when you want to explore your large vision data sets for, let's say, autonomous driving. This comes in really handy for things like recommender systems and things like that. So we started out building out that layer, that storage layer in the open source. And that took about a year's worth of effort to really get to a shape that is usable, kind of like Parquet or Ork and other formats in these tools. And that was when generative AI burst onto the scene and became sort of a revolutionary technology. And what happened at the time was we had originally built in vector index for our computer vision users to say, hey, let's deduplicate a bunch of images or let's find the most relevant Samples for training, for active learning and things like that. And it was sort of that open source community that discovered, hey, this can be really good for genitive AI as well. That's when we sort of separated out another repo to say, hey, this is a vector database. And it's much easier to communicate with the community than to say, hey, you're looking for a vector search, ([Time 0:03:19](https://share.snipd.com/snip/b5e91d57-c6ff-4832-bdd6-2e1d7e527830)) - **Getting Started with LanceDB** - To start with LanceDB, install via pip or npm and load your data as a pandas or polars dataframe. - Configure embedding models within LanceDB and perform fast vector search returning results as dataframes, integrating seamlessly into ML workflows. Transcript: Speaker 1 There's a couple of sort of prototypical workflows that we see from our users. I think at the smaller scale for LanceDB, you're installing it via like PIP or NPM or something like that. And in general, you get some input data that comes in as like a Pandas data frame or maybe a Polar's data frame. And then you interface with an embedding model. You can do that yourself or you can actually configure the LAN-CV table and say, hey, use OpenAI embeddings or hey, use these hugging face embeddings. LAN-CV can actually take care of all that. So that's a pretty quick sort of data frame to LandCV. And then you can search it. And then that comes out as data frames or Python dicks or things like that. That plugs into the rest of your workflow that are likely data frame or pedantic or Python dicks. So that's number one. And then kind of number two is really these large-scale use cases where some of our users have anywhere from like 100 million to multiple billions of vectors in one table. And that's a much bigger production deployment. And typically, what makes LanceDB stands out in that area is, one, it's very easy for them to process the data using a distributed engine like Spark. And they can write concurrently and get that done really quickly. I think we're one of the few that offers GPU acceleration in terms of indexing. So even for those really large data sets, you can index pretty quickly. And then number three is because we're able to actually separate the compute and storage, even at that large vector size, you don't really need that many query nodes. Like you can actually just have one or two like fairly average and commodity query nodes that runs on your storage of choice, depending on what latency requirements you want, and then Just have a very simple architecture. For these types of architectures, the query nodes are stateless and they don't need to talk to each other. So when you need to scale up or when a node drops out, it has to come back in. There's no sort of leader election, there's no coordination. It ([Time 0:15:53](https://share.snipd.com/snip/509e0016-c9e0-471e-9fa8-c26d8fa3e997)) - **Separation of Compute and Storage** - LanceDB separates compute and storage, enabling scalable vector search on commodity compute. - This architecture allows queries on large datasets stored on cloud storage with efficient resource use and simpler deployment. Transcript: Speaker 1 Yeah, absolutely. So this goes back to what we talked about earlier with separation of compute and storage. And if you've been sort of steeped in like data warehousing, data engineering land, this has been a big arc of data warehouse innovation in the past decade by allowing us to scale up the Storage versus the compute separately. This is the thing that makes the system seem magical where you can you can process a huge amounts of data on what seems like you know pretty commodity or pretty weak compute and so the analogy That i like to make with these situation is kind of like a lot of us are familiar with let's say like duck db demos or videos and you could see instances where DDoC DB is processing hundreds Of gigabytes of data on just a laptop and in a very fast amount of time. And they are able to spit out results almost interactively. And there are companies from MotherDuck to there's a new company called ValPlan that is looking to essentially distribute DuckDB queries on AWS Lambdas. It's basically the same thing. It's all about the separation of compute and storage. And that's only possible if you have the right underlying data architecture for storing vectors and the data itself. ([Time 0:20:45](https://share.snipd.com/snip/e10c907b-e0c1-4118-a512-67a37e77e338)) - **Innovative Data Architecture** - LanceDB uses a custom columnar data format that allows fast random access unlike Parquet. - Coupling this format with disk-based vector indices enables interactive querying and efficient handling of wide tables for large-scale AI workflows. Transcript: Speaker 1 Yeah, so it's two things. One is the one is the columnar format. So typically, you know, from Gen AI to machine learning, you can have very wide tables, but typically a single query only needs a couple of columns. So columnar format allows you to only have to fetch and look at a very small subset of that data. Number two is that columnar format needs to be paired with an index, like the vector index in this particular scenario. And that vector index, in order to give this separation of compute and storage, has to be based on disk. So you have to be stored the data on disk, not force the user hold everything into memory, and then be able to access that very quickly. And then number three is how to connect that index with the columnar format. So a columnar format like Parquet does not give you the ability to do fast random access. So even if you had that good index using Parquet, you would not be able to get interactive performance in terms of queries. And it's only by having a new columnar format like LANDS that can give you fast random access and fast scans that you can successfully put these two together and deliver the thing. ([Time 0:22:22](https://share.snipd.com/snip/4d35aaaa-3ef3-4c09-83e7-b932cab74da1)) - **Rust Boosts Development Safety** - LanceDB's core components are written in Rust, enabling safety and rapid development. - Rust's safety significantly reduced bugs compared to prior C++ code, allowing frequent releases and confident iteration. Transcript: Speaker 1 Yeah, absolutely. I love talking about Rust. The Rust package is actually not a client, but so on the core of both the data format and the vector database is actually in Rust. So the Rust crate that we have is actually the database or the embedded database. And so, and we actually build, for example, the JavaScript, again, the same thing with JavaScript. It's not just a client, but it's also an embedded database in JavaScript. So that is actually based on top of the Rust crate and kind of like you have in, say, like Polars or something like that. You have like a Rust core, and then you connect that into JavaScript. So we had actually started out in 2022 writing in C++ because Parquet is written in C++, you know, like serious data people and database people write in C++, right? Speaker 3 Until they find Rust, of course. Speaker 1 Right. And it was sort of a hack project during Christmas time in 2022, at the end of 2022, where we had to get a hack project for a customer, actually, and where we had to actually reimplement Partially the repath for Lance format. And what we found was just, it was so good that we decided to just actually rewrite everything in Rust. I think biggest things were we were a lot more productive. We rewrote roughly six months of solid C++ development in about three weeks with Rust. And this was like us learning Rust as beginners as we went along. A lot of that initial Rust code has again been rewritten over the past year, but it just made us feel a lot more productive. And then number two is the safety that Rust offers you has been amazing. With C++, every release, it just didn't have a good feeling. It was almost like, where's that next set fault going to come from? Whereas with Rust, you know, we felt very confident making multiple releases per week with, you know, major features. And we did not see anywhere near the sort of issues that we saw with C++, right? So everything has been really great. I know that like Rust has become really popular now for actually even with vector databases like Quadrant, I think is Rust. Pinecone, they're not open source, but in turn, they publicly said that they've written their whole stack in Rust as well. ([Time 0:24:30](https://share.snipd.com/snip/3bb32c15-6c5b-49f9-a96f-e7991738591b)) - **Diverse Real-World Applications** - LanceDB's users build varied applications from personal productivity to legal and healthcare domain chatbots with retrieval-augmented generation (RAG). - Unique features like table versioning and time travel allow querying vector data snapshots, useful in code analysis and issue tracking. Transcript: Speaker 1 Use cases for LAN CB in the community that I see falls into three or four large buckets. One is, of course, generative AI, RAG, and things like that. And I think it's not so much the use of LAN CB that I think is really cool, but it's the applications that people build with it that is really cool and amazing. And I think a lot of the applications that people build that is cool, that really takes advantage of LAN CB is things where you need RAG to be very agile and that you need it to be really sort Of tightly bundled with your application, you can sort of call this RAG from anywhere and have it return pretty quickly and without too much complexity. And so this is where I see a lot of folks from like your standard, like chatbots and chat with documentation things like productivity tools, where they build things that help people Organize their daily schedules to much more high stakes things in production and code generation or healthcare and legal and things like that. And so there, I think typically you see vector data set sizes from the tens of thousands up to single-digit millions of vectors typically. And so production means you really scale up both the number of data sets that you have and then the number of vectors that you have. And one of the cool things that I've seen that takes advantage of LANs-EV and LANs format uniquely is there's a code analysis tool that sort of analyzes your GitHub repository and plugs It into a RAG, like customer success sort of tool. And what they want to be able to do is say, query the state of the database, like this today versus yesterday versus a week ago, to say, hey, was this issue fixed or not? And like, what's still outstanding? And so LanCB uniquely gives you this ability to version your table and also do time travel. So you can say any data vector database can do like, give me the 10 most similar things to this input uniquely. But what Lance CB gives you the ability to do is say, give me the 10 most similar as of yesterday or as of a week ago. And we do that sort of automatically for you. Yeah. ([Time 0:29:13](https://share.snipd.com/snip/6b8945d3-8ef4-4e25-acde-bf32442cca8f)) - **Integrate LanceDB with DuckDB** - Use LanceDB integration with DuckDB to enable SQL querying over vector search results. - This allows natural language powered SQL generation and enables seamless workflows combining vector DB with relational data. Transcript: Speaker 1 The things that's really cool about DuckDV is its extension mechanism. And so I think they've also published like a extension framework for Rust-based extensions. And so we have sort of a basic integration going there. And I think in New Year, what you can expect from us is actually we're going to be spending a little more time to make that integration be more rich, meaning our goal is for you to be able To say to write like a duck db udf to do vector search and then the results come back as as like a duck db table where you can then run additional query like duck db queries on top of that and So and sort of the same the same thing with like polars right so you can, and the goal is to essentially make it so that like vector database is no longer a thing that you even have to think About. It's people are generally more familiar with like DuckDB or Polars as the sort of that tool that just stitches together the workflow. So ([Time 0:34:03](https://share.snipd.com/snip/40442305-fee1-43e4-8dae-aede8290d361)) - **Future of AI: Personalized & Domain-Specific** - The next wave will bring explosion of retrieval and personalized AI applications, not just generic chatbots. - Domain-specific AI agents will democratize expertise in areas like legal and healthcare, creating magical user experiences. Transcript: Speaker 2 Because that's where you're living. What excites you about, you know, whatever it is, the next six months, the next year, and what you think is kind of coming as this tooling rolls out there further and further, people learn To apply it better and better. What's exciting for you? That's a great question. Speaker 1 I think there are, there are lots of things that I think holds a lot of promise in the next six to 12 months. I think we'll see one is this explosion of retrieval kind of information retrieval tools. So we already see a lot of companies that are adding like generative AI in like customer success management and like documentation and things like that. And so I think we'll see a lot applications providing value that can be also personalized and not just like chat GPT style answers, but actually personalized to their own data or their Own cases or things like that. And then number two is I see a lot of successes in very domain specific agents that are able to dive deep into legal or healthcare or some domain very specifically and build things that Seem sort of magical, whether it's compliance or driving better outcomes or creating things that would democratize a lot of these sort of like very deep expertise type of domains. And then I think a little bit further out are generalized like low code and no code tools for you to build, you know, very sophisticated applications using generative AI through code Generation and sort of creative, let's say creative interfaces and things like that. ([Time 0:38:02](https://share.snipd.com/snip/94890be4-3713-4336-b07c-c75d0899f20a))