OpenSearch: How the Project Went From Fork to Foundation  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: OpenSearch: How the Project Went From Fork to Foundation - Category: #podcasts - URL: https://share.snipd.com/episode/2745d75c-a308-45a9-b8c3-b47071255bf8 ## Document Notes ## Highlights - **OpenSearch Origin** - OpenSearch originated from Elasticsearch in early 2021 due to Elasticsearch's restrictive relicensing. - The decision to fork was community-driven, focusing on an Apache 2.0 licensed alternative. Transcript: Anandhi Bumstead I can give you a little bit of a backstory of what OpenSearch is. Yes, please. OpenSearch originated from Elastic, Elastic, Elastic Search about early 2021. And the decision to fork from Elastic Search was done because of a relicensing change from Elastic. It was changed to a more restrictive source available license. One of the things OpenSearch did was, and I was not part of that journey, but when the decision was made, it was really made from the customers and the community. And we listened to them. And we also wanted to really focus on bringing in an alternative open source version of OpenSearch under the Apache 2.0 license. And so here we are. And I joined when we were sort of in the mode of innovating in OpenSearch. And so here I ([Time 0:01:38](https://share.snipd.com/snip/c69e3950-3b89-4937-8613-d110c35137ee)) - **Foundation Move** - OpenSearch's move to the Linux Foundation aims to foster neutral governance and broader community involvement. - The Linux Foundation's established presence will also boost OpenSearch's brand awareness. Transcript: Heather Joslin Am. Excellent. And you mentioned that Elasticsearch went behind a more restrictive licensing. A lot of that going around these days. That's right. For sure. Can you give us sort of a sense of what went into the decision to donate OpenSearch to the Linux Foundation? Yeah. Anandhi Bumstead So, you know, when I joined a little bit of what I'll say, like we really were doing a lot of things in the open, being transparent and open, but where we really looked at a neutral foundation Is for neutral governance and also to bring in a broader community. And if you look at foundations, like most foundations, they really facilitate neutral governance and also enabling companies to collaborate in open source. And Linux Foundation has been very successful. I mean, tons of open source projects Linux Foundation has been governing. So we really wanted to lean into that. Another aspect of Linux Foundation I just want to say is they're really good about facilitating things like eventing and marketing, training and certification. And I do think OpenSearch is in a journey where we do need to increase its brand awareness. We do need to bring in more training and documentation so people can use OpenSearch more. So I think it was like the best match. And we announced the transfer in September of 2024, ([Time 0:02:30](https://share.snipd.com/snip/dc48a323-7a26-4004-8ad4-9f4f109e41fe)) - **OpenSearch Use Cases** - OpenSearch is a versatile data ingestion and analytics engine with diverse applications. - Use cases range from observability and log analytics to security, alerting, and semantic/hybrid search, especially with generative AI. Transcript: Heather Joslin Just recently. Right. For people that are not as familiar with OpenSearch, when I was doing research for this, I came across a quote from Carl Meadows, director of product for the OpenSearch project. He called it a Swiss Army knife. Can you elaborate on what might be meant by that? What are some of the use cases that you're seeing? I mean, we know it's an analytics engine. Anandhi Bumstead Yeah. So OpenSearch is a flexible data ingestion system. It builds on, like you said, the core search engine analytics, and also it has a visualization out of the box. And some of the use cases for OpenSearch, and that sort of leads to the analogy of OpenSearch being a Swiss Army knife, people widely use it for observability, log analytics. They use it for security analytics, for alert detection. And they're heavily used, especially with the Gen AI Momentum, as a vector database. And so OpenSearch is also used for a lot of search scenarios. We have what is called semantic search. And we have this thing called hybrid search, where you can do a keyword search and you can do a semantic search. And all of these capabilities make OpenSearch widely used across a variety of these analytics scenarios. ([Time 0:03:51](https://share.snipd.com/snip/b0ed22fd-7598-41b2-a0de-7b41fb532bc1)) - **Performance Improvements** - OpenSearch has addressed initial criticisms about slower indexing speeds compared to Elasticsearch. - The latest release boasts significant performance improvements, including 6.5x faster queries and 25% better indexing. Transcript: Heather Joslin So vector, that's obviously an area that's growing rapidly because of Gen.AI. Yes. OpenSearch is sometimes criticized for being slower than Elasticsearch in terms of indexing speed and handling complex queries. Are those criticisms valid? And if so, is there anything on the roadmap to close that gap? Anandhi Bumstead Some of the criticisms are valid earlier on. Like I said earlier, we were focused on stabilizing and bringing the community on board. We were focused on that. But soon after, you know, our primary focus continues to be performance and we spend a lot of time tuning performance. I'm just going to talk about a few things we're doing in performance just to reiterate this. So one of the things we did last year was we released this thing called OpenSearch Benchmarks, which is a way for transparently in OpenSource, you can measure the performance of OpenSearch And you can bring in various workloads into that tool. So you can look at the query, how the query is doing, how is indexing doing, and all of the measurement, we can do it in Open. So if you fast forward now, we just did an OpenSearch 2.17 release, which is the latest release. And the performance of the most complex queries is 6.5x faster than our first OpenSearch release, which is 1.3. So it's a significant shift we've done in terms of the query performance. And from an indexing perspective, you know, last year we did this thing called segment replication. And we have about 25% throughput on indexing improvement compared to a default like document replication. And then the most interesting thing I want to say is because we value a lot of multi-stakeholder and community contribution, we collaborated with Intel on Z-Standard, and compression Is one of the most important things for OpenSearch for performance and storage efficiency, and we've achieved about 30% performance improvement. And our roadmap, you know, I'll just briefly talk about our roadmap. Our roadmap has, across indexing, search, and storage, and vector capabilities, we're doing a ton of work with performance and cost optimization in mind. So we're excited about that. ([Time 0:05:03](https://share.snipd.com/snip/ce345fdb-3911-4fe8-9a99-3789654bd5ed)) - **Generative AI Support** - OpenSearch supports generative AI by providing efficient indexing, fast querying, and vector database capabilities. - It powers AI applications, chatbots, content generation, and semantic/neural search. Transcript: Heather Joslin Yeah, it does seem like there's more concern about resource optimization with Gen AI workloads requiring so much resources. Speaking of Gen AI and its needs, how has that shaped OpenSearch? Anandhi Bumstead So we've been together with this Gen AI momentum, and one of the things you see with this AI innovations is it's transforming search experience for all skill levels of all users, right? And as you look at the AI applications built these days, a lot of these AI applications need efficient indexing. They need really fast way to query these large language models and they need knowledge bases and so on. And OpenSearch is really in the heart of it. It provides all of these capabilities with providing these ML capabilities. We also allow you to build chatbots, AI assistants, generate content. And I talked about semantic and neural search, which is natural language to query search. And so we do all of that in OpenSearch. And then for builders who are very more advanced, we enable them to do things like search re-ranking, or you can rewrite your queries, and you can build these RAC-based applications, Which is retrieval augmented generation AI applications. And all of this really requires a vector database. In most AI-based applications, they require a vector database. And OpenSearch is the vector database solution. And the last thing I'll say is OpenSearch is a popular vector solution today in this day and age. But in 2019, OpenSearch pioneered what is called the KNN, nearest neighbor search, in the search engine. And all our vectors today, our vector database is powered by these KNN indexes. So we're really in the heart of this whole era of Gen AI. Right, right. ([Time 0:07:23](https://share.snipd.com/snip/a739024c-e773-45ad-95a0-1e3b2528e34c)) - **Open Source and Trust** - Trust and open collaboration are essential for open-source success, while restrictive licensing erodes trust. - Companies carefully evaluate legal and technical trade-offs when considering projects with restrictive licenses. Transcript: Heather Joslin And, you know, the cutting edge of that. Yeah. So I'm excited about that too. Since we're at all things open, open source is on everyone's mind. As we mentioned, OpenSearch was created as a fork and as a reaction to Elasticearch going behind a more restrictive license. We've seen the rise of that. I mentioned this, a lot of that going around. We've seen OpenTofu because of Terraform. We've seen Valky because of Redis in response to those fully open source projects going behind a more restrictive licensing. What do you think of this trend that we're seeing? Do you think going to be more of this? Okay. Anandhi Bumstead So everything I've believed in, like in my 30-year career in this industry and I'm working in open source, I do think trust is the heart of open source. Like, you know, you have to build trust. Re-licensing, you know, maybe it is motivated by monetization or whatever it is. It does erode trust. And also companies, a lot of companies, including AWS, they are looking at trade-offs. They're looking at legal, technical trade-offs. They're looking at, you know, whether this is a risk they would like to take in a project which is with restrictive licensing and stuff. So I would say I hope that this trend doesn't continue. I hope that we continue to innovate in open source and we build a more open model for collaboration. That's what I would say. So ([Time 0:09:20](https://share.snipd.com/snip/c5a143f5-fb48-4909-b37a-b13626f0c851)) - **OpenSearch Growth** - OpenSearch needs contributors to share innovative ideas, raise awareness, and sponsor employee contributions. - Collaboration, like Uber and Slack integrating their Lucene engine systems, is key for future innovation. Transcript: Heather Joslin Open source is part of the Linux Foundation now. What do you feel are the things it needs from the maintainers to have it grow in a way that it needs to to meet the moment? Anandhi Bumstead Yeah, great question. You know, as part of transferring to the Linux Foundation, we have created a technical steering committee, which is comprised of contributors and maintainers of OpenSearch. Chairing committee, we have about 15 members there across representing many organizations, including ByteDance, RN, Oracle, Uber, Slack, and more, including AWS. What we really need from the contributors and the community is bringing in their ideas and innovation and also raising awareness about open search. We talked about Gen.AI. We talked about how search is reaching a different level of evolution. And we really want the companies also sponsoring their employees to contribute to OpenSearch. And the last thing I'll say is we recently are working with Uber and Slack. And they have a Lucene engine system for their search. And what they've seen is the goodness of collaborating in OpenSearch, which is also a Lucene engine-based system, and bringing their ideas and their innovations into OpenSearch. And that's exactly what I want to see. That's the future I see for OpenSearch. More innovations. ([Time 0:10:45](https://share.snipd.com/snip/22200a9c-7532-4468-8e3f-16a77e9fa7a7)) - **Benefits of Contributing** - Contributing to open source benefits companies by fostering diverse innovation and solving customer problems collaboratively. - This approach ultimately drives business growth, as seen with projects like Linux, Kubernetes, and PyTorch. Transcript: Heather Joslin That's an issue in open source more generally. There are a lot of companies, it sounds like these companies are very involved in it, but there are definitely many companies that might be using OpenSearch and not contributing back. What would you say to companies? I mean, this seems like a constant problem that OpenSource, just about every organization in the world uses OpenSource software, but only a small percentage contribute upstream. Anandhi Bumstead Actually, you know, this is something, you know, like it's a really cultural change. And like I was part of the cultural change in AWS. I was earlier on in my other organizations I used to work with. And what people don't realize is it's basically, it's a flywheel which gives back to you. So, you know, when you innovate and you're bringing in your innovations with a diverse set of stakeholders, you're actually solving customer problems. Yeah. You would not have done it in your own little, like, you know, controlled space or whatever organization, right? Right. So I do think that bringing more, like you look at any Linux foundation, Linux, you look at Kubernetes, you know, you look at PyTorch, you look at like so many of these examples. I do think diverse innovation actually brings in more value and actually it helps the company to grow its business based on the value we're getting from these innovations. So I am a big believer, and I hope we can bring more believers into this ecosystem. How ([Time 0:12:11](https://share.snipd.com/snip/84821146-b923-446e-8fd2-b0b07da26e76)) - **Getting Involved** - Get involved with OpenSearch through foundation.opensearch or opensearch.org. - Contribute by raising brand awareness, giving technical talks, or improving documentation. Transcript: Heather Joslin Can someone get involved in the OpenSearch project? Anandhi Bumstead So you can start with going to foundation.opensearch You can go to opensearch.org as well. And it's a simple way to first learn about it. You can reach us on public Slack channels and you can reach us in our GitHub repo and we have all of the maintainers and the contributors actively engaged. Find any way to contribute. Even doing brand awareness is contribution. Doing technical talks, documentation is contribution. So I welcome any sort of contribution into OpenSearch. ([Time 0:13:38](https://share.snipd.com/snip/c3d16493-992f-4562-bf74-b473a299999d)) - **Exciting Innovations** - Anandhi Bumstead is particularly enthusiastic about OpenSearch's vector innovations and cost/storage efficiency improvements. - Ongoing work includes disk-optimized storage in the vector space, with opportunities for collaboration. Transcript: Heather Joslin We'll include, with the article with this story, we'll include some links so people can do that. Yeah, definitely. One more question. We talked about some of the use cases. Is there a use case that you think is particularly cool or innovative or that you know could work, but you haven't seen anybody try yet? Anandhi Bumstead Wow, that's a good one. So my sort of things, which I'm really, I get excited a lot about performance. We talked about it. I also get a lot of excitement around the vector innovations we're doing. And one of the recent work we're doing is around, you know, how do we be more efficient in cost and storage? And so the thing which really excites me is some of the work we're doing in that area. We're doing work around disk-optimized storage in the vector space. And we would love to collaborate with more people to innovate in this area and really focus on vector performance and cost efficiency. So that's something I'm always excited about. Excellent. Well, ([Time 0:15:00](https://share.snipd.com/snip/720a7795-1365-48b3-8452-905d6249cf4f))