Microsoft Wants to Make Service Mesh Invisible  🔗 **Source:** [Microsoft Wants to Make Service Mesh Invisible](https://share.snipd.com/episode/5e622071-cdce-4946-8c2b-0d444e5498b4) ## Highlights - **Make Service Mesh Invisible With Network-As-Policy** - Service mesh should be invisible to users while delivering network security like mTLS and observability without app changes. - Mitch Connors frames success as users not knowing they're on a mesh while getting encryption, auth, traces, metrics, and traffic controls. Transcript: Mitch Connors Well, I'll spend the first 10 seconds saying that success for me looks like most people not knowing what a service mesh is, even though they're using one. Most people don't need to think about the details, but what a service mesh does is provide network security. Think encryption and authorization, as well as observability, think traces, metrics, logs, and network controls, traffic shifting, etc. Without any changes to the application layer. So your network can do that on your behalf. Effectively, you can program your network. ([Time 0:01:34](https://share.snipd.com/snip/d8145b1e-249f-4a63-9c70-b6065ba5c7d8)) - **From Sidecars To Node Proxies Solved Upgrade Pain** - Istio originally used sidecar Envoy proxies injected into every pod which solved routing but created heavy day-two operations like restarting apps for proxy upgrades. - Ambient mode moved encryption to a single lightweight Rust proxy per node and optional envoy waypoints to avoid app restarts. Transcript: Mitch Connors So we dynamically inject an envoy proxy everywhere. We call it the sidecar. And it did the job. It became a really standard kind of thing. All of the routing. But it created really substantial day two operational problems. You run Helm, upgrade Istio, of your proxies have upgraded. You need to go and recreate all your applications, allow rollouts to start all over again in order to take advantage of that upgrade. So Ambient, we re-evaluated those decisions. We found that we could safely move all of the encryption technology into a dedicated Rust-based proxy that is just one per node. Very lightweight. And that's actually a substantial fraction of our users that just need MTLS and some layer 4 policy. Okay. So for them, that's enough. They don't need to add anything else. For our users that have come to know and love our L7 features, telemetry, weighted load balancing, etc., they can actually just create an envoy deployment that upgrades independently Of applications. No need to restart anything and will dynamically redirect traffic to the waypoint. ([Time 0:04:35](https://share.snipd.com/snip/71fb8c1d-56e8-4677-a7d4-13d2b520585d)) - **Use Managed Mesh To Offload CVE And Upgrade Burden** - Move mesh operations into a managed product if your teams can't reliably track CVE patches and upgrades across many tools. - Mitch says Azure Kubernetes Application Network will handle upgrades, maintenance windows, and rollbacks for you. Transcript: Mitch Connors So this means you need to watch for security updates, like any other project. We have them about once a month. And when they come out, you need to upgrade what you've got. We give you the ability to upgrade one application at a time so you can run the versions in parallel in your cluster. If you find a problem, you can always roll that application back to the former version, but it's still acting as one mesh. Looking at the product side, we found that users were still feeling like it was a pretty heavy burden to watch for those CVE patches to make sure they're on top of them. Platform engineers are busy. They own dozens of tools, all of which are getting regular CVEs. So by moving this into a managed layer, we're able to say, hey, you just tell us what the network should do. You give us the instructions and programming. We'll make sure everything is always up to date on your behalf, but still give you that ability to roll back when you need to and control your upgrades, make sure that we're following Maintenance windows, et cetera. ([Time 0:06:04](https://share.snipd.com/snip/3930c1e9-5e24-40da-884f-eab2a429e7bf)) - **AI Forces Cost-Aware Routing Using Token Counts** - AI workloads break the assumption that all requests have equal cost, making traditional request-based routing insufficient. - Connors highlights token counts as a proxy for LLM request cost and the need for inference-aware routing and rate limits. Transcript: Mitch Connors Yeah, AI is definitely causing all of our users' network needs to really evolve and expand exponentially. If you think just about the simple task of LLM routing, oftentimes we do routing based on requests, HTTP requests, and we assume that each request has approximately the same difficulty Of serving, the same weight on the compute. Well, that's not even remotely true with an LLM, where one request might be the perpetual high message being sent to our LLM, and another request might be someone asking Copilot to explain Service mesh for them, and that's going to take a little bit. So we really need a way to make sure that we're evaluating how we distribute those requests based on the actual cost of serving them. And that's approximated with token counts. Right. But this presents a little bit of a problem. We've spent so long slowing the APIs down in Istio and saying they should be stable, they should be standard, they should be boring. AI is not boring. ([Time 0:10:00](https://share.snipd.com/snip/873b4d8e-079e-4c3f-958a-d494987c1455)) - **Two-Speed Mesh Model For Stable And Experimental AI** - Istio adopts a two-speed model: stable, production-safe Istio APIs and experimental integrations like Agent Gateway for rapidly evolving AI protocols. - This lets teams choose boring stability or alpha AI features without destabilizing core APIs. Transcript: Mitch Connors Way that we're doing that is by partnering with another cloud-native project or another Linux Foundation project. The name of the project is Agent Gateway. They're designed for agentic traffic from the ground up. And they're designed by engineers who understand Istio very well, have a long history with us. And so we're able to integrate that into your mesh as a native component. If you want boring, stable, super safe APIs for production, you use the Istio APIs, and you get relatively little AI functionality. If you want cutting-edge, A2A, MCP, and all of the other alphabet soup of AI, you can use Agent Gateway APIs, and it's just a clear signal that these are going to be an alpha experience. We can't promise you that they're going to be around in the same shape in two years, but you've expressed interest in using the bleeding edge of technology. And so we're going to support you in that as well. Got it. Frédéric Lardinois Yeah. I think Agent Gateway actually ([Time 0:11:14](https://share.snipd.com/snip/05f94221-adaa-4d27-94e4-c46c5854d2bb)) - **Estimate Tokens At The Edge To Preemptively Route Requests** - Use a tiny estimator model at the edge to score request complexity and pre-route expensive LLM calls while reporting token usage back for backend rate limiting. - Connors describes the Gateway API inference extension with an endpoint picker and token reporting in responses. Transcript: Mitch Connors So, with the Gateway API's inference extension, you can use the endpoint picker service to actually run a tiny LLM whose goal is only to score the question in terms of complexity. It's a token estimator and then can help you with your routing proactively. And then on the back end, as the request comes back from the LLM, it reports how many tokens you've used in the JSON payload. ([Time 0:12:40](https://share.snipd.com/snip/d699268a-f7b1-433e-b6ba-b68c117a302c)) - **Simplify Messaging To Win Back Mesh Adoption** - Many customers still lack service mesh or mTLS because Istio earned a reputation for complexity; simplifying the message to concrete features helps rebuild trust. - Connors notes 60% of users lack mesh and Microsoft frames AppNet around mTLS, multi-cluster, and traffic shifting instead of 'service mesh'. Transcript: Mitch Connors Don't have any sort of service mesh on their cluster. They don't have MTLS encryption. Okay. And something like 80% of.... Frédéric Lardinois Why is that? Mitch Connors Well, you know, I'll admit, I have to fess up. We earned the reputation for being very complicated. We were very complicated. And I don't blame a user who tried service mesh five years ago from saying, I don't want anything with the word Istio put on my cluster at any point in time. Trust is hard to earn back, but we're working to do that. And we're also working towards an audience that maybe doesn't have the cloud-native vocabulary to express their needs as a service mesh, but understands them much more from a tactile Business perspective. I need to make sure that I have in the U.S. FIPS-compatible encryption across all of my connections. So we'll talk to them about that rather than about the topic of service ([Time 0:14:06](https://share.snipd.com/snip/5601ed4c-6fe4-4394-99c3-a38e0b9df378)) - **Block Shadow AI By Inspecting Request Bodies For LLM Calls** - Enforce approved model endpoints by inspecting request bodies and blocking unapproved LLM requests, not just relying on DNS or IP-level controls. - Connors suggests body inspection to prevent shadow AI and ensure corporate compliance with approved AI providers. Transcript: Mitch Connors One thing that we're not able to accomplish today, but if we're looking to tomorrow that I hope to be able to serve customers in, is simply around making sure that only the approved AI Tools are being used in a given cluster. It's wonderful the platform engineering teams are saying, like, here, we'll provide Copilot or Foundry or OpenAI to you. But there's not a good way today to say, and by the way, if you're sending a message that looks like an LLM request and it's not to one of our approved servers, we need to block that. You can do things like DNS filtering, IP masking for that. But realistically, there's always ways around those tools. ([Time 0:16:33](https://share.snipd.com/snip/5df5324c-d17a-4f3f-8acf-019d93023e7a)) - **Multi-Cluster Ambient Extends Identity-Based Security** - Multi-cluster Ambient support lets meshes enforce cryptographic identities and policies across cluster boundaries so workloads can move without losing trust. - Connors argues identity (service account) plus cryptographic enforcement is key for multi-cluster traffic safety and workload mobility. Transcript: Mitch Connors I'll say probably the biggest development in the last year in Istio has been Ambient support for multi-cluster. We knew that before we could turn Ambient into a product for Microsoft, we needed to have a multi-cluster story. Why is that? Frédéric Lardinois Sorry, let me just back up. Well, because every customer has multiple clusters. Mitch Connors And when you don't have a consistent route of trust and a consistent policy engine for those clusters, any traffic that goes between clusters effectively just bypasses all of the network Safety that you've set up using your service mesh or your CNI. So within multi-cluster, we're able to say, we don't actually care where a workload runs per se. A workload has an identity, that's its service account, and all of its privileges depend on that identity, which is then going to be cryptographically enforced over the wire for all Traffic end-to in-cluster or multi-cluster. ([Time 0:17:54](https://share.snipd.com/snip/07b82ed0-babe-4ca7-85ea-a2e86b219ade))