OpenSearch | Feat. Pallavi Priyadarshini  ## Metadata - Author: [[The Open Source Way]] - Full Title: OpenSearch | Feat. Pallavi Priyadarshini - Category: #podcasts - URL: https://share.snipd.com/episode/dae8bc6c-5284-4424-9e3d-9b1706427ee5 ## Document Notes ## Highlights - **OpenSearch's Community-Led Origin** - OpenSearch is a community-driven, Apache-2.0 licensed search, analytics and vector platform born from an Elasticsearch fork in 2021. - It moved to the Linux Foundation in 2024 and now has thousands of contributors from hundreds of organizations. Transcript: Pallavi Priyadarshini Thank you again for having me and as a representative of open search so for those of you who aren't familiar with open search we are a community-driven open source search analytics and Vector database platform with integrated tools for observability, security, visualization, and AI-powered applications. And all of these are available under the 2.0 Apache license. And a little bit of history. OpenSearch started as a fork of Elasticsearch in early 2021 after Elastic changed to a more restrictive license. Four years later, OpenSearch project has created its own path and it has its own identity and a trajectory of innovation driven by the community where a community of over 3,500 members And OpenSearch was transitioned to the Linux Foundation in September of 2024, which accelerated the growth of the community and the technical innovation. ([Time 0:02:41](https://share.snipd.com/snip/78ddec08-c60d-48b1-b9a5-343e7b36f149)) - **SAP's Large-Scale Observability Deployment** - SAP uses OpenSearch primarily for observability via its Cloud Logging service supporting logs, metrics, and traces. - The deployment already runs thousands of clusters and reached petabyte scale with millions of gigabytes ingested monthly. Transcript: Juergen Walter A big project with a lot of focused topics, but the major use case within SAP is observability at the moment. And Carsten Schnitter and I are working on a service called Cloud Logging, which is providing observability for SAP BTP, the SAP Business Technology Platform, and it supports logs, Metrics, and traces. To better understand what this service really provides, it's, I think, good to understand the environment we operate in, which is mainly the business technology platform, which Has different environments, such as Cloud Foundry and Kima, by the way, both also open source. And I think also Kima was once in one of your podcast episodes. And there are environments where applications can be deployed. So where then, for example, extensions for setups of customers and SAP internal are deployed. And these applications need to be observed. And these environments come with an opinionated way of providing observability data. And we have built on top of OpenSearch, some tailored integrations using parsing and analytical dashboards. So they get an opinionated all-inclusive package for observability. Like we're not using all of, on providing all of the features, but it's mostly an open search as a service labored for observability. And when I just said major use case so that you get some impressions, we're currently running more than 11,000, maybe 12,000. I don't check it every day. Clusters, we had huge external growth. ([Time 0:04:00](https://share.snipd.com/snip/208938f1-e7e8-48f1-abbe-f6707be9a263)) - **Migration From ELK To OpenSearch At SAP** - SAP migrated from the ELK stack to OpenSearch after Elastic relicensed, adopting and contributing to the fork. - That move ensured SAP control and continuity for their customer-facing logging services. Transcript: Karsten Schnitter Yeah, so when I joined the logging services eight and a half years ago, we were running like services based on the Elkstack, so Elasticsearch, Logstash, Kibana. And that is what we used to build cloud logging in the first place. We were using like Open Distro from AWS when it came out because it made our lives easier. But then Elastic basically changed the license. So at that time, because we were selling the service to customers, that was a huge shock because it prohibited us from doing that with the newer versions. So it was quite clear, okay, when the fork for OpenSearch was announced that we adopted and now also contribute to it because of the setup in a way where it cannot be taken away anymore. So that's really important. ([Time 0:06:24](https://share.snipd.com/snip/45481de6-4b50-4537-84c1-ebdd90860f5e)) - **Multivendor Governance Drives Innovation** - The OpenSearch Software Foundation under the Linux Foundation governs the project with multi-company participation and a 15-member TSC. - Major innovations and roadmap items now come from diverse contributors like Uber, ByteDance, SAP, NVIDIA and others. Transcript: Pallavi Priyadarshini OpenSearch project now is governed by the OpenSearch Software Foundation under the Linux Foundation. And it is, there are 16 companies that are part of the OpenSearch Software Foundation, including SAP, which is a premier member of the foundation. And to govern the foundation, there is a governing board and there is also a technical steering committee, which is comprised of 15 technology leaders from 10 different companies. So what we really want to enable is open collaboration. Ever since OpenSearch project moved to the Linux Foundation in September 2024, we have really seen the growth of that community and also the collaboration that comes along. And that was the whole point of AWS moving OpenSearch project to the Linux Foundation in that AWS did not want it to be a single vendor project. So now what has happened is the community has grown. We had, like I mentioned earlier, we have more than 3,500 contributors from 400 different organizations. So these are not just contributors, but they also use OpenSearch for the different use cases in search, analytics, observability, or as a vector database. And we really promote the model of open collaboration to give a few examples. Our most recent major version release was OpenSearch 3.0 that happened in May 2025. And in that release, there are significant innovations coming from non-AWS contributors such as Uber. So Uber has really transformed the OpenSearch architecture into a cloud-native architecture. ByteDance contributed significant performance improvements to the core architecture in terms of the segment replication. It also contributed conversational search. SAP was instrumental in contributing back FIPS compliance to the software and has been contributing to the security plugin as well. And there are others like DTEX, which has made significant contributions to the dashboards, which is powering the observability. NVIDIA contributed back GPU acceleration for vector index bills. So we are seeing innovations coming in from many different companies, which is the true power of OpenSearch. And because of the technical steering committee, the roadmap is also community-driven. ([Time 0:07:35](https://share.snipd.com/snip/7e94e9cf-7c91-49a0-8b73-fe041414a049)) - **Standardize Ingestion With OpenTelemetry** - Use OpenTelemetry as a standardized ingestion path to simplify observability across environments. - Map data into OpenSearch with provided integrations and the OpenSearch catalog for ready-made visualizations. Transcript: Karsten Schnitter I think that was really a big contribution that we did to the project. So the data proper ingestion tool has learned by our contributions to speak open telemetry for metrics and logs. And that's also how I became a maintainer of that project, actually. All right. Karsten Hohage Okay. That was, at the time, that was not even your task yet. And you did this as one of the, I don't know, free roaming contributors? Or because you said that's how you got to the project? Karsten Schnitter Yeah, we just contributed like we contacted one of the maintainers, David, which we have a really good context to. I talked to him a lot in Amsterdam on the last OpenSearchCon and this is how the contribution to that particular project started. ([Time 0:13:49](https://share.snipd.com/snip/e72c11f5-20c5-4598-82cb-e6bb17de15a4)) - **OpenSearch As An AI Foundation** - OpenSearch positions itself as an AI foundation combining vector search, observability and analytics for agentic and conversational use cases. - Contributions from companies like ByteDance, IBM and SAP accelerate AI search and vector capabilities. Transcript: Pallavi Priyadarshini On search and observability, we are seeing some exciting innovations come in. The way OpenSearch is positioned today, it's really becoming the AI foundation to power all these different use cases. So on search side, what we are seeing is it is evolving or it has already evolved from a keyword-based search. It went to semantic search, hybrid search, multimodal, where you can do images, unstructured data, audio, all kinds of data together, right? And the most recent trend is around conversational search, and it's also becoming the foundation of this agentic AI use cases, bringing in a new way of interacting with LLMs, etc. And OpenSearch is in a unique position with its vector database and its AI search capabilities, which is pretty important for the agentic AI use cases. And it is only because of open source that we are able to move so fast. Like on the search side, we are seeing innovations come in from our partners, such as I had mentioned earlier, ByteDance is contributing to the conversational search. Similarly, IBM Data Stacks, it contributed a new vector engine, JVector, to OpenSearch. Now on observability as well, we are seeing that OpenSearch is becoming the central place to drive your analytics slash logs, metrics, traces, experiences. And again, we are seeing contributions coming in. Like we already talked a lot about SAP is critical partner for the OpenSearch project. It is a founding member of the OpenSearch Software Foundation since day one. And other companies also look up to what SAP is doing. In fact, the case study that SAP published in terms of its use with OpenSearch. It was very popular with the other set of partners. We had a conference last week in San Jose, OpenSearchCon, where Hari from SAP, he presented the keynote, which was very well received. And we also saw other partners which are coming together to make observability side of things more robust. In fact, we had a discussion last week with contributors from Apple who are interested in the pipe processing language upgrades to OpenSearch. And we just had core maintainer announced from Apple as well because they have started contributing to the code base. ([Time 0:16:57](https://share.snipd.com/snip/34bfa0c1-7fa4-4066-bc0f-873a58057e60)) - **Embedding Challenges For High-Rate Logs** - Applying vector embeddings directly to high-rate, small observability messages is costly and inefficient. - Effective ML for observability requires grouping records into larger blocks before embedding or using lightweight/sparse models. Transcript: Karsten Schnitter Let me comment on that. So it is really great to see all the AI and vector search use cases of OpenSearch. And now we come to the holy grail for observability. And this is to somehow enable this kind of features for observability. So the problem is observability data each and every set, each log message is usually relatively small. And if you want to do machine learning on it and create a vector, you multiply the data. Like if you have a 1k log message that's already pretty long, but the only thing that is bigger is stack traces. But if you have the usual log message that's relatively small and now you create a vector that is 4k, and that is not a good idea. So in the end, it comes down to how to combine your data into larger blocks, and then apply the machine learning on top of that. And that is something that is really interesting, something that can be done with open search. This is the PPL support, for example, that Paobaki mentioned and others. And this will become very, very interesting. I want to give a quick shout out to our SAP colleagues, because internally I've already seen a POC that is a REC-based approach that is looking inside your observability data, mainly The logs, and then giving recommendations how to reconfigure your application or what to fix because you are experiencing crashes. That's at the moment on POC level, but I hope it becomes a product soon. And that already showcases like how much you can do the AI features of OpenSearch and how much you could integrate it, given that there's also an MCP server in OpenSearch 3, for example. ([Time 0:20:03](https://share.snipd.com/snip/1d5ff401-7179-44b0-b0ca-e42d045db7c4)) - **Leverage PPL For On-the-Fly Analysis** - Use PPL (Pipeline Processing Language) to transform and analyze observability data on the fly inside OpenSearch. - Combine PPL with semantic and agentic features to enrich observability workflows without external tooling. Transcript: Karsten Schnitter The pilot processing language that OpenSearch uses to analyze data on the fly and reformat it, transform it. It's similar to Splunk processing language, but it's a complete kind of its own. Maybe, Pallavi, can you explain it a little bit better? Pallavi Priyadarshini Yeah, I think you covered it, Karsten, but basically PPL is a powerful language, pilot processing language, that allows you to discover more insights from a lot of data, right? So it provides these rich syntax with which you're able to power a lot of your observability and analytics experiences. So what's happening is with every release of OpenSearch, we are upgrading the PPL language. ([Time 0:21:50](https://share.snipd.com/snip/3ece41b5-6d4c-4d98-94b5-0fcea547630b)) - **Connect Or Host Models With Blueprints** - Connect OpenSearch to external LLMs via ML Commons or host lightweight models locally using provided blueprints. - Build connectors quickly using templates to support new models as they appear. Transcript: Pallavi Priyadarshini So Karsten, there are many ways. Again, OpenSearch gives the foundational blocks, but OpenSearch has something called ML Commons, using which there are these different connectors to different models, right? So you can connect out to external models, or you can choose to host your model within your OpenSearch deployment. So it gives you the flexible options to either connect to any model or to host it within the OpenSearch deployment. And what the team is focused on is building the blueprints so that whenever these models come out, we are able to ship connectors as soon as possible. Give you an example, when DeepSeek came out, like we all woke up one day and we saw a lot of news around DeepSeek, right? So OpenSearch contributors were able to ship out a connector for DeepSeek in two days because there are these blueprints using which you can easily build connectors to any model that Comes out. ([Time 0:25:26](https://share.snipd.com/snip/97e47aaf-f34d-4928-825d-b97a81ab1385)) - **Towards Automated Incident Remediation** - Agentic workflows can combine observability data with code and knowledge contexts to automate diagnosis and remediation. - Early POCs show automation could detect deploy-caused outages and propose or create rollbacks. Transcript: Juergen Walter This could go. On the one hand, we have the observability data, but we also have then potentially other contexts like knowledge bases, code bases, which we then can all connect in an agentic workflow. And then the real beauty comes, there's like an outage and it's automatically detected that this is caused by a deployment of a change and that then the system could create a pull request To revert that change. And so this is where it's going. And first tests really show really promising results on this. Karsten Hohage Sorry, I don't know if I understand that correctly. Is the charming thing in that is that you have all the information from the logs and traces, and that's why you can probably infer where the error is? Karsten Schnitter We actually have more in SAP, so this is not like the common open platform, but there is always abstraction on top. So if you build an application on BTP, you're usually using some other kind of automation on top, like deploy with confidence, build code. So there are other frameworks that actually help you generate the applications in the first place or manage them. So there's external context even. So there are some application frameworks like the CAP framework that are used, and then there's this kind of deployment automation on top. And so we not done that yet but there's a huge potential for integration with these kind of frameworks to get more context. So maybe you get an exception and you know the line of the code. You know where this application lives. You know where the code lives. You know when it was deployed. You can reach out to those systems or these systems can reach out to us, getting the exception and then provide the developer with much more context. So like, look, your second to last version that you deployed yesterday in production that fails in production only. It was fine on your staging cluster, but now it fails. And probably the buck is around here. So go and do that. And in the meantime, here is how to roll back the change. So these are all automations that can be built on top of that. This is vision to some extent, but the POC that I mentioned goes, for example, into that direction. ([Time 0:29:00](https://share.snipd.com/snip/63a9eb17-8f64-43e6-b34a-5b778a56b856)) - **Start Contributing Through Community Channels** - Join the OpenSearch community via the project website, Slack, forums, and RFCs to contribute or adopt the stack. - Start with tagged easy-first issues and reach out to maintainers for onboarding and mentorship. Transcript: Pallavi Priyadarshini We are seeing that multiple organizations are adopting open search and these are not just big organizations. I do want to take this opportunity to say that it's also the smaller open search really values any contribution and we're seeing contributions not just from the big names that we talked About, but also a lot of individual contributors and smaller organizations like universities across the globe. So what's happening is in the community, we have built the mechanisms to easily onboard contributors, right? There is a community Slack, there is a community forum, there is an events page, we have a documentation. Then obviously the community members, different teams like search team, storage, observability, indexing, they all set up their community triage roadmap meetings and it's all Decentralized, right? Although the overall processes are governed by the technical steering committee, but the maintainers and contributors operate independently to discuss their respective, we call, We have something called RFC, request for comment, where you can pitch a new idea and the maintainers come together in the community meetings, discuss that. And if they are on board, that this is a good thing to have, it gets folded into a release, you know, into an upcoming release. So there are multiple ways to get involved. Obviously, there's the OpenSearch project website that is like the entry point for everything I mentioned, the Slack, the forum. And we also have easy first issues that we tag for new contributors to pick up. ([Time 0:31:36](https://share.snipd.com/snip/5ae6eac8-2d53-410a-88b5-95eaaf9cb311))