Cloud Native On-Demand: From Chaos to Control in Enterprise AI/ML 🔗 **Source:** [Cloud Native On-Demand: From Chaos to Control in Enterprise AI/ML](https://www.youtube.com/watch?v=_zqOuG-h8d4&pp=0gcJCQMKAYcqIYzv) > [!note]+ Document Notes > tags: [kubernetes, mlops, enterprise-ai, container-registry, cncf] > author: Brad Mikley > source: YouTube > reference: https://www.youtube.com/watch?v=_zqOuG-h8d4 > date created: 2024-12-19 15:30 > > # 🏷️ Title: Cloud Native On-Demand: From Chaos to Control in Enterprise AI/ML > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Evaluate KitOps for ML artifact management — replaces fragmented tooling #mlops - [H2] Assess current container registry capabilities for ML artifacts — leverage existing infrastructure #infrastructure - [H3] Review AI/ML project versioning and security practices — address gaps in tracking #security ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [KitOps Project](https://kitops.ml/) - [Jozu Hub](https://jozu.ml/) ## 📝 Summary - Brad Mikley presents KitOps, a CNCF project for managing AI/ML artifacts using container registries - Traditional deployment practices break down with AI/ML due to large models, complex data dependencies, and fragmented tooling - KitOps packages ML artifacts as OCI artifacts, enabling secure versioning and deployment through existing container registries - Solution addresses enterprise needs for private ML catalogs, security scanning, and audit trails - Demonstrates significant cost savings and infrastructure reuse compared to fragmented storage solutions ## 📖 Main Content Enterprise AI/ML faces unique challenges that traditional application deployment practices cannot handle. Models exceed Git's capabilities, training data becomes unwieldy, and security practices often lag behind traditional software development. Brad Mikley, CEO of Jozu and maintainer of KitOps, argues that the solution lies in leveraging existing container registry infrastructure rather than building new specialized tools. The core problem stems from fragmentation: models stored in S3, data in data lakes, code in Git, and documentation in wikis, all with inconsistent versioning. This creates operational complexity and security gaps, particularly problematic in regulated industries requiring strict audit trails. KitOps addresses this by treating ML projects as OCI artifacts stored in container registries. This approach provides enterprise-grade security (SHA-256 verification, signing, immutability), leverages existing RBAC systems, and enables partial downloads of specific artifact layers. The solution maintains tight coupling between models and data while providing the versioning and security practices enterprises require. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=0) — [H1] Introduction and Problem Statement - Key idea: Traditional deployment practices fail for AI/ML due to artifact size and complexity - Evidence: "AI/ML really breaks everything. Models are too big for Git. Even Git LFS is awkward and difficult." #### ⏱️ [01:30](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=90) — [H2] Private AI/ML Project Catalog Requirements - Key idea: Security-sensitive environments need centralized, private catalogs for version control and auditing - Evidence: "For folks doing fine-tuning, especially in security sensitive environments, you really need to have a private AI/ML project catalog" #### ⏱️ [02:15](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=135) — [H2] Current Fragmented Tool Landscape - Key idea: Organizations use multiple disconnected tools for different artifact types - Evidence: "Multiple tools for housing different types of artifacts, models over here in S3, data in parquet files in a data lake" #### ⏱️ [03:00](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=180) — [H3] Object Store Limitations - Key idea: S3 and object stores lack proper versioning and metadata capabilities - Evidence: "The versioning is pretty poor. The metadata is terrible if it even exists at all" #### ⏱️ [04:00](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=240) — [H3] Git LFS and DVC Shortcomings - Key idea: Git-based solutions have high storage costs and security limitations - Evidence: "Git LFS storage cost is very high. It's not really tamperproof for the artifact itself, just the pointer" #### ⏱️ [05:00](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=300) — [H3] Experiment Tracker UI Problems - Key idea: ML-focused UIs create communication barriers between data science and DevOps teams - Evidence: "DevOps teams trying to go through MLflow and figure out what was the last good version before the one that just broke in production" #### ⏱️ [05:45](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=345) — [H1] Container Registry as Solution - Key idea: Reusing existing container registries provides battle-tested security and infrastructure - Evidence: "Every enterprise already has a container registry. It's battle tested with RBAC, scanning, signing, global distribution" #### ⏱️ [06:30](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=390) — [H1] KitOps Architecture Overview - Key idea: KitOps packages ML artifacts as OCI artifacts with layered storage - Evidence: "Each asset is stored as a separate layer, you can pull only what you need" #### ⏱️ [07:15](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=435) — [H2] Enterprise Security Benefits - Key idea: Inherits container registry security features including SHA-256 verification and signing - Evidence: "You get SHA 256 verification, cosign, notary, existing RBAC, immutability, and great audit trail" #### ⏱️ [08:00](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=480) — [H2] OCI Image vs OCI Artifact Distinction - Key idea: OCI artifacts optimize for storage rather than runtime environments - Evidence: "OCI image is optimized for running something. Artifact standard is about storing, layers separate different types of artifacts" #### ⏱️ [09:30](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=570) — [H2] Tight Coupling Requirements - Key idea: ML projects require tighter coupling between data and models than traditional microservices - Evidence: "Unlike microservices which are loosely coupled, the data and model weights are always very tightly coupled" #### ⏱️ [10:15](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=615) — [H1] Live Demo: Packaging Model Kit - Key idea: Simple CLI workflow packages diverse ML artifacts into versioned model kits - Evidence: Demonstrates packaging data, docs, MLflow artifacts, notebooks, and models with single command #### ⏱️ [13:45](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=825) — [H2] Repository Features and Collaboration - Key idea: Web interface provides artifact inspection, diffing, and sharing capabilities - Evidence: "Quick at-a-glance view of what's inside, detailed manifest information, easy sharing via Slack" #### ⏱️ [15:30](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=930) — [H2] Deployment Integration - Key idea: Auto-generates Kubernetes YAML and Docker deployment configurations - Evidence: "Autogenerate deployment specifics for DevOps teams, convert model portion to OCI image for Docker" #### ⏱️ [16:45](https://www.youtube.com/watch?v=_zqOuG-h8d4&t=1005) — [H3] Python Integration - Key idea: PyKitOps library enables programmatic integration with experiment trackers - Evidence: "Build model kit directly out of CubeFlow or MLflow or directly from a notebook" ## 📚 References - [KitOps Open Source Project](https://kitops.ml/) - [Jozu Model Registry](https://jozu.ml/) - [OCI Artifact Specification](https://github.com/opencontainers/artifacts) - [CNCF Projects](https://www.cncf.io/projects/) ## 🗣️ Paraphrase - KitOps solves ML artifact management chaos by packaging models, data, and code as OCI artifacts in existing container registries - Traditional deployment practices fail for AI/ML because models are too large for Git and organizations end up with fragmented tooling across different storage systems - Container registries provide ideal infrastructure for ML artifacts since enterprises already have battle-tested security, RBAC, and audit capabilities in place ## 📰 Newsletter **Transform Your ML Chaos into Container Registry Control** Managing AI/ML projects often feels like herding cats - models in S3, data in lakes, code in Git, docs in wikis. Brad Mikley's presentation on KitOps offers a compelling solution: treat your container registry as a secure model registry. **The KitOps Advantage:** - **Reuse existing infrastructure** - Your container registry already has enterprise security, RBAC, and audit trails - **Layered artifact storage** - Pull only what you need (model, data, or config) while maintaining tight coupling - **Enterprise-grade security** - SHA-256 verification, signing, and immutable versioning out of the box **Practical Impact:** Organizations report significant cost savings versus S3 egress fees and eliminate the operational complexity of managing multiple artifact storage systems. Ready to explore? Check out the [KitOps tutorial](https://kitops.ml/) for hands-on experience, and consider how your existing container registry could become your ML artifact hub. Resources: [KitOps Documentation](https://kitops.ml/), [Jozu Hub](https://jozu.ml/), [OCI Artifacts](https://github.com/opencontainers/artifacts) ## ✅ Action Items #### DevRel - Developer education: Create tutorials comparing KitOps workflow to traditional ML artifact management - Community engagement: Host workshops on container registry patterns for ML teams - Developer experience: Develop integration guides for popular ML frameworks (MLflow, KubeFlow) - Documentation: Build comparative analysis content showing cost and complexity benefits #### OSPO - License compliance: Review KitOps CNCF project licensing and enterprise compatibility - Security management: Evaluate OCI artifact security model against enterprise ML requirements - Strategic alignment: Assess container registry ML usage patterns for policy development - Community engagement: Participate in CNCF AI/ML working groups and artifact standards discussions #### Community - Maintainers: Engage with KitOps maintainers on enterprise feature requirements and roadmap input - Contributors: Identify opportunities to contribute ML-specific container registry patterns - Ecosystem: Build bridges between traditional DevOps tooling and emerging MLOps practices ## 🧪 Hypotheses - Organizations using container registries for ML artifacts will achieve 30-50% cost reduction versus object storage solutions - ML teams adopting unified artifact management reduce deployment errors by consolidating versioning systems - Container registry security models provide sufficient ML-specific compliance coverage for regulated industries - Tight coupling of ML artifacts improves reproducibility compared to loose reference-based approaches ### 📊 Methodology - Cost analysis: Compare total ownership costs of KitOps vs S3/object storage for various ML workload sizes - Error rate study: Measure deployment success rates before/after unified artifact management adoption - Security assessment: Evaluate container registry ML security posture against industry frameworks - Reproducibility testing: Measure experiment reproduction success rates with tightly-coupled vs loosely-coupled artifacts ### ❓ Questions - How do container registry storage costs scale compared to specialized ML artifact stores at enterprise volumes? - What are the performance implications of storing large ML models as OCI artifact layers? - How does KitOps integration complexity compare with existing MLOps toolchain investments? ### 🔮 Future Research - Multi-cloud container registry ML artifact replication patterns and performance characteristics - Integration patterns between KitOps and emerging ML governance frameworks - Cost optimization strategies for large-scale ML artifact management in container registries ### 💡 Implications - Practice: ML teams need to evaluate container registry capabilities for unified artifact management - Policy: Organizations should assess ML artifact governance requirements against container registry security models - Security: Container registry ML usage requires review of access patterns and audit trail requirements - Education: DevOps teams need training on ML artifact lifecycle management and deployment patterns - Funding: Budget planning should consider container registry scaling costs for ML workloads versus specialized tooling ### 🧠 Learning Styles Assessment Reading level estimate: 12.5 (Technical/Professional) #### 👀 Visual learners - Present: Architecture diagrams, CLI screenshots, web interface walkthrough - Improve: Add data flow diagrams showing artifact relationships, cost comparison charts #### 🔊 Aural learners - Present: Full video presentation with clear narration and live demonstration - Improve: Add chapter markers, key concept audio summaries #### 🗣️ Verbal learners - Present: Technical analogies (hub of wheel), clear problem/solution framing - Improve: Add glossary of MLOps vs DevOps terminology, more real-world examples #### 🧩 Kinesthetic learners - Present: Live CLI demonstration, hands-on tutorial links, step-by-step workflows - Improve: Interactive cost calculator, sandbox environment for testing #### 🧠 Logical learners - Present: Problem statement with clear cause/effect, structured solution comparison - Improve: Decision matrix comparing solutions, ROI calculation framework #### 🧑‍🤝‍🧑 Social learners - Present: References to enterprise collaboration, DevOps/ML team communication challenges - Improve: Case study discussions, community forum links, team adoption guides #### 🧘‍♀️ Solitary learners - Present: Self-paced demo, independent tutorial resources, documentation links - Improve: Self-assessment checklist, individual learning path recommendations #### 🌿 Naturalistic learners - Present: Ecosystem thinking about tool integration, infrastructure reuse patterns - Improve: Sustainability impact of infrastructure consolidation, lifecycle thinking ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive link text in references - Improve: Alt text for demo screenshots, ARIA labels for technical diagrams #### Keyboard navigation - Present: Standard video controls, accessible reference links - Improve: Keyboard shortcuts for video chapters, skip-to-demo option #### Contrast and readability - Present: High contrast video presentation, clear terminal demonstrations - Improve: Ensure code examples meet 4.5:1 contrast ratio, larger demo text #### Text resizing and reflow - Present: Responsive text layout, scalable content structure - Improve: Ensure technical diagrams remain readable at 200% zoom #### Media and captions - Present: Full transcript provided, clear audio narration - Improve: Add chapter-based captions, downloadable transcript with timestamps #### Forms and interactions - Present: Standard video playback controls - Improve: Accessible demo interaction examples, form labels in tutorials ### 🌍 DEIB Assessment #### Cultural breadth - Present: Enterprise-focused examples, cloud-agnostic approach - Improve: Include examples from different industry verticals, global deployment patterns #### Inclusive language - Present: Professional technical language, avoided unnecessary jargon - Improve: Define specialized MLOps terms, provide plain-language summaries <!-- deep-read:sections end --> ## 📺 Video  --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=_zqOuG-h8d4)* **[00:02]** Hi, my name is Brad Mikley. Um, today we're going to talk about how to go from a chaos to control in your enterprise a IML. I'm the CEO and founder of Josu and one of the maintainers of the Kitops open source project which is of course one of the CNCF projects. Uh, Jose is actually my second startup. Um, I've held positions at AWS, Red Hat and Code in the past. So, let's jump in. Um, for most people on uh watching this, you probably know how to deploy applications safely. You've got your code in Git, **[00:34]** you've got your Shaw digest, you've got versions, you've got environment configs uh separated and tracked, deployment manifests are versioned, automated security scan, the whole nine yards. But AML really breaks everything. Models are too big for Git. Even get LFS is awkward and and difficult. Uh, training data, you end up with something crazy. I'll get to that in a second. often environment configs are not really tracked. They tend to be kind of embedded sometimes. Uh you've got security scans that are not automated and not consistent. They may not in fact **[01:04]** be done at all. A lot of times people feel like, well, the base model was security scanned, so I'm going to fine-tune it. I don't need to do anything. It's not always that simple. And so I want to start with the project catalog, which is definitely not the most exciting part of a IML, but it is in a lot of cases the hub of the wheel, the most important thing that holds it all together. And for folks doing fine-tuning, especially in security sensitive um environments, you really need to have a private E IML project catalog because that's where you're going to do your version control. You're **[01:35]** going to basically have a point of logging or reporting for your security. You're going to need policy based polls and deployments that connect to that. You want to have collaboration so the teams can still move quickly. Um, and you it's typically the place where you're going to also do your auditability, your your kind of single source of truth. You don't want to build this on a shaky foundation. Um, right now in the market, there are a lot of fragmented tools. There's inconsistent versioning. A lot of organizations we speak to actually **[02:06]** have multiple tools for housing different types of artifacts, models over here. Let's say in S3, data in paret files, maybe that's in a data lake. You've got your code of course in git docs in a wiki. They're all separately versioned maybe. Um but you often have spreadsheets or just tribal knowledge that knows what goes with what. So what are some of the things that people do use today? Well, you've got S3 or object stores that are nice and scalable and of course they're familiar to everybody. But the **[02:36]** versioning is pretty poor. The metadata is terrible if it even exists at all. and they can get expensive really fast, especially if you have large models, large data sets, or things that are changing a lot. This is a good example. Um, this is a set of files. These are data files, but really other than the date, which is the obvious part, there's really not much more I can tell about these files unless I pull them down, unzip them, and actually start interrogating them. That's rough because if I need to go through 20 of these to **[03:07]** find what I'm looking for, i.e. Maybe there was a piece of PII data added to the database by accident and I need to figure out when was it added. How do I go back? Which snapshot, you know, was the most recent that didn't have it, for example. Could mean pulling down these a lot. Otherwise, git LFS, we talked about git is nice and familiar. We love git. You love git, but it storage cost is very high. It's not really tamperproof um for the the artifact itself, just the pointer. DVC is ML specific, but it's **[03:38]** also not tamperproof. It's honestly kind of the same as Git. Um, except not even as good because it doesn't use Shaw. It uses MD5. Experiment trackers. Everybody's got those and they are necessary. They're great. Um, but whether you're using MLflow or CubeFlow, weights and biases, you take your pick. You have some amount of lock in. You typically have a UI that makes a lot of sense for data science teams, but can be very confusing for DevOps teams. And in the middle of a production incident, that is not when you want to have broken communication at **[04:09]** 3:00 a.m. and DevOps teams trying to go through ML flow, for example, and figure out what was the last good version before the one that just broke in production. Now, one of the things I learned from my time at large organizations like Red Hat and uh even more so AWS is the secret to security and speed really is reuse. You don't want more components than you need. Um, and the reality is that every enterprise already has a container registry. It's battle tested. It's got our back, scanning, signing, global **[04:40]** distribution, caching, dduping, compliance, and audit trails are there. Integration with dev tools and production environments. So, in a lot of ways, it's kind of the ideal hub of this wheel. So, the question is, can we store ML artifacts alongside container images in a useful way? So we created kit ops which basically is secure packaging for a IML projects and lives on that a OCI artifact standard. You can put models in it of any format any type data sets whether for training validation etc. Uh **[05:11]** you can drop your code in there notebooks uh maybe your test scripts for for ML flow um git repos pointers to those things if you don't want to put the actual items in there themselves. uh configuration prompts, base prompts, system prompts, template prompts, documentation, and metadata. Now, because each asset is stored as a separate layer, you can a pull only what you need, i.e. give me only the model in the code. I don't need the data set today or give me just the configuration and the prompts because I've got a running model. I just want to see how it **[05:41]** works. The second thing about being able to put it in layers is of course you can take advantage of that container um uh optimized storage that that registries have where they'll look at the layers and see if they changed and if they didn't they won't actually restore it. So this brings enterprisegrade security out of the box. Not only do you get that shaw 256 verification but you can use cosign notary or whatever you like for signing. you've got the existing arbback so you don't need to worry about aligning off Z between your registry your production system your devops system etc else which is always a pain **[06:14]** um you have immutability and you have a great audit trail now for most people uh watching this you probably understand the difference between an OCI image and an OCI artifact but just in case let's quickly go over it so an OCI image is optimized for running something it is about creating that running environment you use a docker file which is kind of a recipe for that repeatable environment. And it builds everything in sequence from top to bottom. So you can't just run a Docker file from the middle down. **[06:44]** It always has to be top to bottom. When you look at KIDOPS though, it's using the artifact standard, which is different than the OCI image. Now the artifact standard is about storing. And so it still uses layers, for example, but those layers have a different semantic purpose. They're really just there to separate out different types of artifacts. Now the nice thing is you can store everything you need in one place. You pull only what you need. Now nice thing is you can put your models as a layer. Those can be separate from the base model and stay store only what you **[07:15]** need. Data sets can go in storing a snapshot of your training and evaluated evaluation data will help you in an audit situation or when you need to do diagnostics for a production environment. So this really ties your data and your model together. And that's important because unlike microservices which are loosely coupled, the data and the model um weights are always very tightly coupled. You can also add code context and that's really important as well for the same reason. Um you could have your notebooks **[07:46]** in there. You can have your ML uh flow uh experiment runs as I talked about it dependencies that you need. It keeps that code context tied to a specific version of the model to a specific version of the data set. And that specific specificity and interconnectedness is one of the key um important parts of being able to reproduce an a IML project. It's a little bit different than reproducing just an AI a nonAI micros service. This by the way when I speak about a IML projects I'm talking not just about models but about agents um and anything **[08:18]** else that you build with AI as part of it. So the KOPS project is really about turning your existing container registry into a secure secure model registry. And by the way, it can save you money, especially if you're looking at it versus something like an S3 and egress costs there. All right, so let's take a look at a quick live demo. Okay, so now I'm in my terminal. I'm just going to show you how quick and easy it is to actually package up something as a model kit. So here's my particular directory. You can see **[08:49]** I've got my data, my docs, my MLFlow artifacts, my MLFlow run notebook, uh my pickle file serialized model, my requirements text, and um my my notebook. Everything I need is there. Now, of course, uh you can store everything as the artifacts themselves in a single KitOps model kit. And if you're in a highly regulated industry, that's the best thing to do because it ensures that you have no references that can get corrupted later. However, if **[09:19]** you're not in a regulated industry or you don't need that quite that level of rigor, you can actually just put in references here. And so you could say, well, I've got versioning already happy in git. I don't need to add my ML flow run, my requirements, or my notebook. I just need to point to the repo that has those things and the commit that is associated with this particular model kit. And that all works fine, too. Let's take a look at that kit file because that's really the outline of what's here. Um I'm just going to scroll **[09:49]** back up a little bit and you can see this looks exactly like what we saw um earlier. You've got a package description here. You've got your model. I've dropped in the parameters from MLFlow. Um so that's going to be things like what were the best hyperparameters we found? What was the metrics when we used those hyperparameters? And then in case I need to go back to that exact run, I have actually got things like the run ID. Um, skip forward and down. You see I've got my data sets in here and I've got a bunch of docs, a lot of, you **[10:20]** know, licenses and things like that that are also important. So now I'm going to pack this. So when I pack it, that means I'm going to take all these different items and put them into create that model kit. And it's going to use the kit file to know how to find everything and what's there. So I'm going to go ahead and do that. Um, it's already saved a bunch of these layers. So, as you can see, it only needed to save a couple things differentially, which is great. Now, I'm going to do a kit list. You can see I've got a number of these in my environment already. Uh, **[10:51]** I did a bunch of work on this about a week and a half ago. So, you can see those older versions here. And here's the one from today. Excellent. So, it saved it all perfectly locally. Now, let's push that from my loco version to my um my uh repository. In this case, I'm going to use the Jozu repository, but you can push it to any repository you want. All right, and that's it. Okay, so now here we are in the Jozu hub. Um, as you can see, I've saved this model kit here. **[11:22]** Uh, you can see this is the one I just created. If I wanted to share it, of course, I can just pull the tag that's going to copy this command. I can send that to anybody over Slack or whatever have you and they're going to get the exact same model kit. I get a quick at a glance view of what's inside this model kit up here. Or I can look in more detail down here. And you can see that a lot of this is the same information I had from the kit file, but now expanded because I have also the information from the manifest. So now I have the Shaw, I have the size, I have a number of other things. It's formatted a much easier way. These little pulls also make it **[11:55]** easy for me to copy the kit command. So I can pull just a section of this model kit rather than the whole thing. I can also do a quick diff if I was let's say looking at um an older version. I may come back and look at the March 5th version and see what changed there. I've added a lot to the code section. As you can see I added some uh license information to the data set. Um but the data set itself didn't change. And you can see I've added a number of things to the documentation as well. So you know **[12:25]** the the data set and model in this case haven't changed uh but some of the metadata has and so that may or may not justify a redeploy. Of course the other thing I want to do is deploy this. So at some point I want to deploy this either um probably to Kubernetes. So in this case um this is not a uh an LLM. So I'm just going to use a basic container. Um this is going to give my team everything that they need for the specifics of this model. Now, this obviously isn't the full deployment YAML. There's a lot of **[12:57]** deployment YAML uh I'd want around it. Uh that was specific to my organization, specific to the project, specific to the environment, all those kinds of things. We're not trying to replace that. We know that in general, the DevOps team has a good handle on that part. We're just trying to provide the little chunk that is needed to them that they may not know, that they may not be able to find as easily. And so, we autogenerate that for them. The other possibility is of course you might just want to deploy it to Docker and run it either locally or on a shared instance. In this case, you just execute this command and it'll **[13:29]** actually pull it down as an image. We automatically convert the model portion of the model kit to an OCI image and then that can be run inside Docker. So that's how I get quickly from having a bunch of artifacts on my local machine to having a model kit that I can share that's stored in my enterprise registry in a secure fashion potentially security scanned and even deployed to my Kubernetes environment all quite simply. Now the question is, can I do that automatically without having to use the CLI? Because not everybody likes to use **[14:00]** a CLI. And the good news is that yes, we have something called Pyit Ops, which is our library for Python that allows you to do all these things through code. Super helpful if you want to build your model kit directly out of an experiment tracker like CubeFlow or MLFlow or directly from a notebook. So you can see that. I'm not going to go through this in great detail, but basically you just import in the kit ops packages, define your credentials, initialize the manager, log in to your registry, initialize it using your um **[14:32]** your kit file, and then pack and push the same thing I just did in the CLI. If you want to learn more about kit ops, there is a tutorial there. You can uh you can use your phone to jump into that. Or if you want to learn more about Jozu, you can do that as well. Thank you very much for your time and I hope this was helpful.