# OpenSearch  ## Episode metadata - Episode title: OpenSearch - Show: The Open Source Way - Owner / Host: SAP SE - Guests: [Pallavi Priyadarshini](https://share.snipd.com/person/927d862f-68f7-4179-96f3-fa3ea8925796), [Karsten Schnitter](https://share.snipd.com/person/e2cc0604-d9e3-412b-a587-54a1d740b089) - Episode publish date: 2025-10-16 - Episode AI description: Pallavi Priyadarshini, Director of Search for the OpenSearch Project and Amazon's OpenSearch Service, alongside Karsten Schnitter, a software architect in SAP's observability tech, share insights on OpenSearch. They discuss its evolution from Elasticsearch, SAP's strategic shift to this open-source platform, and their significant contributions. Key topics include the integration of AI for advanced search capabilities, community collaboration, and future developments like observability enhancements. They also explore how AI can revolutionize incident response and offer guidance for newcomers. - Duration: 38:32 - Episode URL: [Open in Snipd](https://share.snipd.com/episode/49079160-d655-4772-afba-69c2bbaa140b) - Show URL: [Open in Snipd](https://share.snipd.com/show/b1b29f2a-adba-4a31-a243-ef177053789c) - Export date: 2026-02-17T23:25:34 ## Snips ### [OpenSearch Is A Community-Led Platform](https://share.snipd.com/snip/1a1f6d90-39f7-42eb-983a-e848ba84f356) 🎧 02:41 - 03:45 (01:03) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/11241712-7522-42f9-ba39-6d37bbfdf4fa" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - OpenSearch is a community-driven open-source platform for search, analytics, observability, and vector DB under Apache 2.0. - The project began as an Elasticsearch fork and now has its own identity and roadmap. #### 💬 Quote > OpenSearch started as a fork of Elasticsearch in early 2021 after Elastic changed to a more restrictive license. > — Pallavi Priyadarshini Pallavi Priyadarshini on OpenSearch history and identity #### 📚 Transcript **Pallavi Priyadarshini:** thank you again for having me and as a representative of open search so for those of you who aren't familiar with open search we are a community-driven open source search analytics and vector database platform with integrated tools for observability, security, visualization, and AI-powered applications. And all of these are available under the 2.0 Apache license. And a little bit of history. OpenSearch started as a fork of Elasticsearch in early 2021 after Elastic changed to a more restrictive license. Four years later, OpenSearch project has created its own path and it has its own identity and a trajectory of innovation driven by the community where a community of over 3,500 members and OpenSearch was transitioned to the Linux Foundation in September of 2024, which accelerated the growth of the community and the technical innovation. --- ### [SAP Adopted OpenSearch After Elastic License Shift](https://share.snipd.com/snip/f3383a50-a254-4460-a3b0-c86bfb701f4c) 🎧 06:24 - 07:08 (00:43) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/b499b211-1db8-4744-ba07-b68425fede67" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - SAP moved from ELK to OpenSearch after Elastic changed its license, adopting the fork early for service continuity. - That adoption ensured the logging service could be offered to customers without licensing risk. #### 💬 Quote > So at that time, because we were selling the service to customers, that was a huge shock because it prohibited us from doing that with the newer versions. > — Karsten Schnitter Karsten Schnitter on why SAP adopted OpenSearch after Elastic's license change #### 📚 Transcript **Karsten Schnitter:** Yeah, so when I joined the logging services eight and a half years ago, we were running like services based on the Elkstack, so Elasticsearch, Logstash, Kibana. And that is what we used to build cloud logging in the first place. We were using like Open Distro from AWS when it came out because it made our lives easier. But then Elastic basically changed the license. So at that time, because we were selling the service to customers, that was a huge shock because it prohibited us from doing that with the newer versions. So it was quite clear, okay, when the fork for OpenSearch was announced that we adopted and now also contribute to it because of the setup in a way where it cannot be taken away anymore. --- ### [Governance Shift Grew Diverse Contributions](https://share.snipd.com/snip/0f7ad4a1-7f7c-4d18-8a63-31c9b1871218) 🎧 07:35 - 10:24 (02:48) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/a829cc34-525c-4a17-90e6-08f29f0e7007" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - OpenSearch moved to the Linux Foundation and formed the OpenSearch Software Foundation to broaden governance and contributions. - Multiple companies now contribute major features and the Technical Steering Committee guides, not controls, the roadmap. #### 💬 Quote > OpenSearch project now is governed by the OpenSearch Software Foundation under the Linux Foundation. > — Pallavi Priyadarshini Pallavi Priyadarshini on governance and community growth #### 📚 Transcript **Pallavi Priyadarshini:** OpenSearch project now is governed by the OpenSearch Software Foundation under the Linux Foundation. And it is, there are 16 companies that are part of the OpenSearch Software Foundation, including SAP, which is a premier member of the foundation. And to govern the foundation, there is a governing board and there is also a technical steering committee, which is comprised 15 technology leaders from 10 different companies. So what we really want to enable is open collaboration. Ever since OpenSearch project moved to the Linux Foundation in September 2024, we have really seen the growth of that community and also the collaboration that comes along. And that was the whole point of AWS moving OpenSearch project to the Linux Foundation in that AWS did not want it to be a single vendor project. So now what has happened is the community has grown. We had, like I mentioned earlier, we have more than 3,500 contributors from 400 different organizations. So these are not just contributors, but they also use OpenSearch for the different use cases in search, analytics, observability, or as a vector database. And we really promote the model of open collaboration to give a few examples. Our most recent major version release was OpenSearch 3.0 that happened in May 2025. And in that release, there are significant innovations coming from non-AWS contributors such as Uber. So Uber has really transformed the OpenSearch architecture into a cloud-native architecture. ByteDance contributed significant performance improvements to the core architecture in terms of the segment replication. It also contributed conversational search. SAP was instrumental in contributing back FIPS compliance to the software and has been contributing to the security plugin as well. And there are others like DTEX, which has made significant contributions to the dashboards, which is powering the observability. NVIDIA contributed back GPU acceleration for vector index bills. So we are seeing innovations coming in from many different companies, which is the true power of OpenSearch. And because of the technical steering committee, the roadmap is also community-driven. The technical steering committee gives the guidance on the processes that run the roadmap. --- ### [SAP Contributed OpenTelemetry Support](https://share.snipd.com/snip/31432b20-06ec-41c9-9b56-ae4c02662255) 🎧 12:37 - 14:05 (01:27) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/c8d00a80-94fd-433d-bae9-db22abcd75a2" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - SAP contributed OpenTelemetry support to OpenSearch to enable logs and metrics ingestion, expanding beyond traces. - That work led Karsten Schnitter to become a maintainer after collaborating with existing project maintainers. #### 💬 Quote > So support for traces was always there, but metrics and logs were lagging. And that was the first contribution. > — Karsten Schnitter Karsten Schnitter on SAP's initial technical contributions to OpenSearch #### 📚 Transcript **Karsten Schnitter:** actually we've started with a different focus so the FIPS is the latest thing that we do to this is really high grade enterprise feature like having FIPS compliance this is like in U.S. governmental standard that you have to fulfill for that they're non-military appliances so if you want to have government to contracts but showcasing that you can do that also enables digital sovereignty because basically it's a set of algorithms for security that you need. And if you can configure them, then basically also for EU use cases, this is really important. But we actually started with observability where Jung and I are at home. And this is when we saw the emergence of open telemetry. SAP ran an internal architecture board where it was decided, okay, that this is the future action that we would be going to. So this is where I was also a member of. And then I said, okay, if we now want to enable open telemetry for all SAP teams, we better have a service that can take in the data. And this was our first or one of the first major contributions that we did through the project. So for traces was always there, but metrics and logs were lagging. And that was the first contribution. I think that was really a big contribution that we did to the project. the data proper ingestion tool has learned by our contributions to speak open telemetry for metrics and logs. And that's also how I became a maintainer of that project, actually. --- ### [OpenSearch As An AI Foundation](https://share.snipd.com/snip/fd8f0bb4-e6ca-4588-96bd-9ee5afe96aa4) 🎧 16:57 - 19:50 (02:53) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/c23bc887-de17-4722-80f1-42a01b3981c1" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - OpenSearch is positioning itself as an AI foundation combining vector DB, search, and observability for agentic AI use cases. - Community contributions are accelerating multimodal, semantic, and conversational search features. #### 💬 Quote > OpenSearch is in a unique position with its vector database and its AI search capabilities, which is pretty important for the agentic AI use cases. > — Pallavi Priyadarshini Pallavi Priyadarshini on OpenSearch's AI and search roadmap #### 📚 Transcript **Pallavi Priyadarshini:** on search and observability, we are seeing some exciting innovations come in. The way OpenSearch is positioned today, it's really becoming the AI foundation to power all these different use cases. So on search side, what we are seeing is it is evolving or it has already evolved from a keyword-based search. It went to semantic search, hybrid search, multimodal, where you can do images, unstructured data, audio, all kinds of data together, right? And the most recent trend is around conversational search, and it's also becoming the foundation of this agentic AI use cases, bringing in a new way of interacting with LLMs, etc. And OpenSearch is in a unique position with its vector database and its AI search capabilities, which is pretty important for the agentic AI use cases. And it is only because of open source that we are able to move so fast. Like on the search side, we are seeing innovations come in from our partners, such as I had mentioned earlier, ByteDance is contributing to the conversational search. Similarly, IBM Data Stacks, it contributed a new vector engine, JVector, to OpenSearch. Now on observability as well, we are seeing that OpenSearch is becoming the central place to drive your analytics slash logs, metrics, traces, experiences. And again, we are seeing contributions coming in. Like we already talked a lot about SAP is critical partner for the OpenSearch project. It is a founding member of the OpenSearch Software Foundation since day one. And other companies also look up to what SAP is doing. In fact, the case study that SAP published in terms of its use with OpenSearch. It was very popular with the other set of partners. We had a conference last week in San Jose, OpenSearchCon, where Hari from SAP, he presented the keynote, which was very well received. And we also saw other partners which are coming together to make observability side of things more robust. In fact, we had a discussion last week with contributors from Apple who are interested in the pipe processing language upgrades to OpenSearch. And we just had core maintainer announced from Apple as well because they have started contributing to the code base. So the bottom line is OpenSearch is becoming the AI foundational platform to power a bunch of these use cases on search, observability and analytics. --- ### [Vectorizing Logs Needs Aggregation](https://share.snipd.com/snip/531e355e-3e85-4cfe-876d-61992d36d527) 🎧 20:03 - 21:40 (01:37) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/f0d1c760-3ee7-4b06-a3a0-5af16e1da1bc" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Applying vectors to high-rate, small observability records is costly and often impractical without aggregation. - Effective ML on logs requires grouping records into larger blocks before embedding to keep storage and compute feasible. #### 💬 Quote > If you want to do machine learning on it and create a vector, you multiply the data. > — Karsten Schnitter Karsten Schnitter on challenges of applying ML/vectorization to observability data #### 📚 Transcript **Karsten Schnitter:** Let me comment on that. So it is really great to see all the AI and vector search use cases of OpenSearch. And now we come to the holy grail for observability. And this is to somehow enable this kind of features for observability. So the problem is observability data each and every set, each log message is usually relatively small. And if you want to do machine learning on it and create a vector, you multiply the data. Like if you have a 1k log message that's already pretty long, but the only thing that is bigger is stack traces. But if you have the usual log message that's relatively small and now you create a vector that is 4k, and that is not a good idea. So in the end, it comes down to how to combine your data into larger blocks, and then apply the machine learning on top of that. And that is something that is really interesting, something that can be done with open search. This is the PPL support, for example, that Paobaki mentioned and others. And this will become very, very interesting. I want to give a quick shout out to our SAP colleagues, because internally I've already seen a POC that is a REC-based approach that is looking inside your observability data, mainly the logs, and then giving recommendations how to reconfigure your application or what to fix because you are experiencing crashes. That's at the moment on POC level, but I hope it becomes a product soon. And that already showcases like how much you can do the AI features of OpenSearch and how much you could integrate it, given that there's also an MCP server in OpenSearch 3, for example. So there's a huge potential, but we have to figure out how to apply it for our use cases. --- ### [POC Shows Automated Fix Recommendations](https://share.snipd.com/snip/804cc60c-75c1-4a0d-a46c-158206f0ebe1) 🎧 20:49 - 21:33 (00:44) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/8341c33c-f307-4a02-aad3-9362cf59bb6c" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - SAP ran a POC that analyzes logs and recommends configuration or fixes, demonstrating early AI-enabled observability value. - The POC shows potential for automated remediation and developer guidance but isn't production yet. #### 💬 Quote > I've already seen a POC that is a REC-based approach that is looking inside your observability data, mainly the logs, and then giving recommendations how to reconfigure your application or what to fix because you are experiencing crashes. > — Karsten Schnitter Karsten Schnitter describing an SAP POC for AI-driven observability guidance #### 📚 Transcript **Karsten Schnitter:** And that is something that is really interesting, something that can be done with open search. This is the PPL support, for example, that Paobaki mentioned and others. And this will become very, very interesting. I want to give a quick shout out to our SAP colleagues, because internally I've already seen a POC that is a REC-based approach that is looking inside your observability data, mainly the logs, and then giving recommendations how to reconfigure your application or what to fix because you are experiencing crashes. That's at the moment on POC level, but I hope it becomes a product soon. And that already showcases like how much you can do the AI features of OpenSearch and how much you could integrate it, given that there's also an MCP server in OpenSearch 3, for example. --- ### [Use PPL And MCP To Simplify AI Workflows](https://share.snipd.com/snip/a09910a9-ad95-4ed3-84aa-c431601e2355) 🎧 22:04 - 23:28 (01:23) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/5cc26b8c-053b-4a75-ad83-30fc55dd6cbb" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Use OpenSearch's PPL and natural language features to build observability and analytics workflows without forcing users to learn query languages. - Leverage MCP server and agentic search blueprints to simplify connecting models and building agentic experiences. #### 💬 Quote > The MCP server, the model context protocol server support was unveiled in OpenSearch 3.0 and in OpenSearch 3.1 and 3.2. > — Pallavi Priyadarshini Pallavi Priyadarshini on MCP server and agentic support #### 📚 Transcript **Karsten Schnitter:** Yeah, **Pallavi Priyadarshini:** I think you covered it, Karsten, but basically PPL is a powerful language, pilot processing language, that allows you to discover more insights from a lot of right? So it provides these rich syntax with which you're able to power a lot of your observability and analytics experiences. So what's happening is with every release of OpenSearch, we are upgrading the PPL language. And then along with the PPL language itself, are also focused on the natural language experiences. I think that's the most recent trend we are seeing where the users don't really want to learn one or the other language and plus these agentic search use cases. So what OpenSearch is doing is it's focusing on the natural language experiences where OpenSearch does the language conversion for you. And again, going back to OpenSearch as an AI foundation or a platform, it is rolling out features that allows builders to build those experiences, right? I think Karsten mentioned that the MCP server, the model context protocol server support was unveiled in OpenSearch 3.0 and in OpenSearch 3.1 and 3.2. And we have a very tight eight-week release cadence. --- ### [Join Community Channels To Start Contributing](https://share.snipd.com/snip/91e7ce39-59db-47d0-a0c2-d4b03fe7e3a3) 🎧 31:36 - 33:20 (01:44) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/64064f72-8e21-4804-a379-08dca3a287a6" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - To contribute, join OpenSearch's Slack, forum, and follow easy-first issues and RFCs to propose work. - Reach out to maintainers in community meetings to get guidance and onboard contributions quickly. #### 💬 Quote > There is a community Slack, there is a community forum, there is an events page, we have a documentation. > — Pallavi Priyadarshini Pallavi Priyadarshini on how new contributors can get started #### 📚 Transcript **Pallavi Priyadarshini:** we are seeing that multiple organizations are adopting open search and these are not just big organizations. I do want to take this opportunity to say that it's also the smaller open search really values any contribution and we're seeing contributions not just from the big names that we talked about, but also a lot of individual contributors and smaller organizations like universities across the globe. So what's happening is in the community, we have built the mechanisms to easily onboard contributors, right? There is a community Slack, there is a community forum, there is an events page, we have a documentation. Then obviously the community members, different teams like search team, storage, observability, indexing, they all set up their community triage roadmap meetings and it's all decentralized, right? Although the overall processes are governed by the technical steering committee, but the maintainers and contributors operate independently to discuss their respective, we call, we have something called RFC, request for comment, where you can pitch a new idea and the maintainers come together in the community meetings, discuss that. And if they are on board, that this is a good thing to have, it gets folded into a release, you know, into an upcoming release. So there are multiple ways to get involved. Obviously, there's the OpenSearch project website that is like the entry point for everything I mentioned, the Slack, the forum. And we also have easy first issues that we tag for new contributors to pick up. --- ### [Shared Foundation Multiplies Innovation](https://share.snipd.com/snip/93e08766-64f6-4ad3-a9a4-162fb2c0f706) 🎧 36:20 - 37:35 (01:15) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/5caa538b-3006-469f-80e2-dfa0dac89b85" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - OpenSearch's momentum comes from many organizations sharing foundational pieces and building differentiating features on top. - Contributing back accelerates innovation and helps others reuse common building blocks instead of reinventing them. #### 💬 Quote > Rather than building everything on your own, why don't we benefit from what the community has built and then build the functionality you don't see and contribute it back? > — Pallavi Priyadarshini Pallavi Priyadarshini on the benefits of community-driven development #### 📚 Transcript **Juergen Walter:** share everything. And otherwise, I would have sharing, getting in the same mood. But so the key takeaway, SAP is running substantial amount of OpenSearch clusters for observability. We're really big, we're petabyte scale, and we're standing in the shoulder of giants. All right, Karsten. Yeah, **Karsten Schnitter:** so adding to that, so SAP is a really active member in the OpenSearch community, sharing our insights, contributing code and other content to the project, answering questions, pointing out documentation issues, and we just invite everybody to join us. **Karsten Hohage:** Okay, great. And then key takeaway, Pallavi. Yeah, **Pallavi Priyadarshini:** we are all really excited about open search and in general about open source and different organizations wanting to standardize on a common open source platform for observability, search, and analytics and build on top of that solid foundation. Rather than building everything on your own, why don't we benefit from what the community has built and then build the functionality you don't see and contribute it back? That's the open source And that's also the open search way. --- Created with [Snipd](https://www.snipd.com) | Highlight & Take Notes from Podcasts