Comprehensive Approach on SONiC Quality  ## Metadata - Author: [[YouTube]] - Full Title: Comprehensive Approach on SONiC Quality - Category: #articles - Document Note: --- tags: [sonic, quality, testing, microsoft, network-operating-system] author: Anil panala source: YouTube reference: https://www.youtube.com/watch?v=sDQQgV_8DSI date created: 2024-12-19 10:30 --- # 🏷️ Title: Comprehensive Approach on SONiC Quality ## 🧰 Action Inbox - [H1] Establish community test case sharing protocols — standardize upstream contributions #testing - [H2] Implement security patch disclosure process — TSC discussion ongoing #security - [H3] Expand image qualification coverage — increase multi-platform testing #quality ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [SONiC GitHub](https://github.com/sonic-net/SONiC) - [Linux Foundation Networking](https://lfnetworking.org/) ## 📝 Summary - Microsoft's Anil Panala presents SONiC quality assurance processes and community collaboration needs - Why it matters: SONiC has grown from simple beginnings to powering Azure's cloud infrastructure at massive scale - Key stakeholders: Network operators, hardware vendors, open source contributors, Azure cloud infrastructure - Risks and tradeoffs: Quality vs speed of delivery, security patch disclosure vs operational security - Bottom line: SONiC requires robust community-driven quality processes to maintain reliability at cloud scale ## 📖 Main Content SONiC (Software for Open Networking in the Cloud) is Microsoft's open-source network operating system that disaggregates hardware and software in networking infrastructure. The presentation outlines Microsoft's comprehensive quality assurance approach as SONiC has evolved from a simple solution to a complex system powering Azure's multi-tier network architecture. The quality process involves multiple stages: pull request reviews with unit testing, image builds, comprehensive image qualification including integration testing across multiple physical platforms, and staged deployment with feedback loops. Microsoft seeks community support in sharing upstream test cases, methodologies, and contributing to control plane testing to enhance SONiC's reliability across diverse deployment scenarios. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=0) — [H1] Introduction and Presentation Goals - Key idea: Microsoft wants to share insights on SONiC qualification processes with the community - Evidence: "We just wanted to give a little glimpse on the insights of how the SONiC qualification is done when any change request comes in" #### ⏱️ [02:30](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=150) — [H2] SONiC Architecture Overview - Key idea: SONiC disaggregates network hardware and software using containerized components - Evidence: "It disaggregates the hardware Network hardware and the software which is very very important at the industry level" #### ⏱️ [04:45](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=285) — [H2] SONiC in Azure Cloud Infrastructure - Key idea: SONiC powers multiple tiers of Azure's network infrastructure with ongoing expansion - Evidence: "Most of the t0 and tier one we already enabled the SONiC tier 2 we are actually currently in progress" #### ⏱️ [06:20](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=380) — [H1] Why Quality is Critical Now - Key idea: SONiC has grown in complexity, features, and deployment scale requiring enhanced quality focus - Evidence: "We started with a small and we are in the very big stage at this point and everything what we are doing is quality it makes very very important" #### ⏱️ [07:15](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=435) — [H2] Change Request Process - Key idea: All changes go through high-level design review, test planning, and issue triaging - Evidence: "Any feature requests will go through the high level design review and the test plan review" #### ⏱️ [08:00](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=480) — [H1] Quality Gate Process - Key idea: Multi-stage validation from unit tests through image qualification before deployment - Evidence: "Before it's merged with the code there is series of unit tests being done" #### ⏱️ [09:30](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=570) — [H2] Image Qualification Stage - Key idea: Comprehensive testing across multiple platforms with integration and control plane validation - Evidence: "We have integration testing we do the data and the control plane testing and we also make sure that it's been tested across the multiple physical appliances" #### ⏱️ [11:00](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=660) — [H2] Current Working Processes - Key idea: Community support, comprehensive gatekeepers, and growing test case coverage - Evidence: "We have full support from community and it we have a lot of comprehensive quality Gatekeepers" #### ⏱️ [12:15](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=735) — [H1] Community Support Needs - Key idea: Microsoft seeks upstream test cases, methodologies, and expanded testing contributions - Evidence: "We need support from you guys to share and any of the Upstream test cases you've been running across your organizations" #### ⏱️ [14:30](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=870) — [H3] Image Release Strategy Question - Key idea: Generic images for broad SKU support with specific images for targeted hardware - Evidence: "When we do the publicly available any images we have a generic images specifically which can be deployed into any of the skus" #### ⏱️ [16:45](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=1005) — [H2] Security Patch Management Discussion - Key idea: Security patches don't align with release cycles and require careful disclosure processes - Evidence: "Security patch is not something that always go with the release right just vulnerability are getting uncovered at any random time" #### ⏱️ [18:20](https://www.youtube.com/watch?v=sDQQgV_8DSI?t=1100) — [H3] TSC Security Process Development - Key idea: Technical Steering Committee developing structured approach for security vulnerability handling - Evidence: "The TSC is discovering discussing of how do we want to handle that collectively because we have a lot of operators" ## 📚 References - [SONiC Official Website](https://sonic-net.github.io/SONiC/) - [SONiC GitHub Repository](https://github.com/sonic-net/SONiC) - [Linux Foundation Networking](https://lfnetworking.org/) - [Azure Networking Documentation](https://docs.microsoft.com/en-us/azure/networking/) ## 🗣️ Paraphrase - SONiC has evolved from a simple network operating system to a complex platform powering Azure's massive cloud infrastructure across multiple network tiers - Microsoft employs a rigorous multi-stage quality process involving unit testing, pull request reviews, comprehensive image qualification, and deployment validation to maintain reliability - The community's involvement is crucial for scaling quality assurance through shared test cases, methodologies, and expanded testing coverage across diverse hardware platforms and use cases ## 📰 Newsletter Microsoft shared insights into their comprehensive quality assurance approach for SONiC, the open-source network operating system powering Azure's cloud infrastructure. As SONiC has grown from simple beginnings to supporting complex multi-tier networks, maintaining quality has become critical. The key learning: Quality must be "shifted left" - finding bugs early saves significant time and costs. Microsoft's process includes rigorous PR reviews, unit testing, comprehensive image qualification across multiple platforms, and staged deployment with feedback loops. **Community Opportunity**: Microsoft actively seeks community contributions in test cases, methodologies, and control plane testing. This presents an excellent opportunity for network operators and vendors to influence SONiC's quality standards while benefiting from collective improvements. Resources: [SONiC Documentation](https://sonic-net.github.io/SONiC/), [GitHub Repository](https://github.com/sonic-net/SONiC), [Community Forum](https://groups.google.com/g/sonicproject) ## ✅ Action Items #### DevRel - Developer advocacy: Create SONiC quality contribution guides and testing workshops for community onboarding - Education: Develop testing methodology curriculum for network operators adopting SONiC - Developer experience: Improve documentation for test case contribution processes and tooling - Community management: Establish regular quality-focused working groups and feedback sessions #### OSPO - Policy & Governance: Develop security vulnerability disclosure policies aligned with industry CVE processes - License Compliance: Review testing frameworks and community contributions for license compatibility - Security Management: Establish coordinated vulnerability response procedures with TSC - Community Engagement: Create formal agreements for upstream test case sharing and collaboration #### Community - Maintainers: Support test case review processes and quality gate enforcement - Contributors: Develop onboarding materials for quality testing contributions - Sponsors: Fund testing infrastructure and automated validation systems - Ecosystem: Coordinate cross-vendor testing standards and shared methodologies #### FAIR - Technical independence: Ensure quality processes remain vendor-neutral and community-driven - Policies: Establish transparent criteria for image qualification and release readiness - Governance artifacts: Document quality gate processes and community contribution guidelines ## 🧪 Hypotheses - Community-contributed test cases will identify platform-specific issues earlier than centralized testing alone - Shared testing methodologies across organizations will reduce duplicate effort and improve coverage - Standardized quality gates will increase confidence in SONiC deployments across diverse environments - Early bug detection through enhanced community testing will significantly reduce production incident costs ### 📊 Methodology - Analyze test case coverage gaps across different hardware SKUs and network topologies - Survey community members on current testing practices and willingness to share methodologies - Implement pilot programs for upstream test case contribution and measure bug detection rates ### ❓ Questions - How can the community standardize test case formats and reporting mechanisms across different organizations? - What incentives would encourage broader community participation in quality assurance activities? - How should security vulnerability disclosure balance transparency with operational security requirements? ### 🔮 Future Research - Investigate automated testing integration possibilities with community CI/CD pipelines - Explore machine learning applications for predictive quality analysis in network operating systems - Study cross-platform compatibility testing strategies for diverse hardware ecosystems ### 💡 Implications - Practice: Network operators need structured processes for contributing test cases and receiving quality feedback - Policy: Security patch disclosure requires industry-standard CVE processes balancing transparency and security - Security: Community-driven testing may expose vulnerabilities requiring coordinated disclosure mechanisms - Education: Quality assurance knowledge sharing becomes critical for ecosystem-wide reliability improvements - Funding: Investment in shared testing infrastructure could benefit entire SONiC ecosystem ### 🧠 Learning Styles Assessment Reading level estimate: 12th Grade (technical presentation format) #### 👀 Visual learners - Present: Network architecture diagrams, process flow charts, tier-level illustrations - Improve: Add testing pipeline visualizations, quality gate flowcharts, community contribution workflows #### 🔊 Aural learners - Present: Video presentation with spoken explanations and Q&A session - Improve: Provide downloadable audio summary, podcast version of key concepts #### 🗣️ Verbal learners - Present: Technical terminology definitions, analogies to traditional networking concepts - Improve: Add glossary of SONiC-specific terms, comparison tables with traditional networking #### 🧩 Kinesthetic learners - Present: References to hands-on testing processes, practical deployment scenarios - Improve: Include testing laboratory setup guides, step-by-step quality validation checklists #### 🧠 Logical learners - Present: Structured process flows, staged quality gates, systematic approach descriptions - Improve: Add decision trees for quality gate progression, metrics-based quality criteria #### 🧑‍🤝‍🧑 Social learners - Present: Community collaboration emphasis, Q&A session, shared responsibility concepts - Improve: Include community working group structures, peer review processes, collaborative testing frameworks #### 🧘‍♀️ Solitary learners - Present: Individual organization testing considerations, self-assessment opportunities - Improve: Provide individual contribution assessment tools, private testing methodology evaluation guides #### 🌿 Naturalistic learners - Present: Ecosystem thinking about network infrastructure, organic community growth concepts - Improve: Include sustainability considerations for testing infrastructure, lifecycle analysis of quality processes ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Standard presentation structure with clear headings and sequential content - Improve: Add detailed alt text for network diagrams and process flow illustrations #### Keyboard navigation - Present: Standard video player controls with keyboard accessibility - Improve: Provide text-based navigation options for presentation content #### Contrast and readability - Present: High contrast slides with readable font sizes in presentation - Improve: Ensure transcript formatting meets accessibility contrast requirements #### Text resizing and reflow - Present: Video content maintains readability when resized - Improve: Optimize transcript layout for various screen sizes and zoom levels #### Media and captions - Present: Video content available, transcript provided through analysis - Improve: Add professional captions and downloadable transcript files #### Forms and interactions - Present: Q&A session allows for community interaction - Improve: Provide accessible feedback mechanisms for ongoing community engagement ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global Azure infrastructure context, international community acknowledgment - Improve: Include examples from diverse geographical deployments, multi-regional testing considerations #### Inclusive language - Present: Technical language appropriate for networking professionals - Improve: Define technical jargon more clearly, provide plain language summaries of key concepts #### Neurodiversity support - Present: Structured presentation format with clear sections and logical progression - Improve: Add content timing indicators, provide summary bullet points, include visual structure indicators #### Representation - Present: Focus on technical content rather than individual representation - Improve: Highlight diverse community contributors, showcase various organizational perspectives #### Psychological safety - Present: Collaborative tone emphasizing community support and shared responsibility - Improve: Explicitly encourage questions from all experience levels, provide multiple engagement channels ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Quality processes are essential → SONiC has grown complex → Therefore enhanced quality is critical (strong logical flow) - Inductive strength: Examples from Azure deployment success support broader applicability claims - Causal reasoning: Clear cause-effect relationships between testing stages and quality outcomes - Analogical strength: Hardware/software disaggregation analogies help explain architectural concepts #### Fallacies - Appeal to popularity: "Thank you to all community members" - while appreciative, doesn't validate technical approaches - Hasty generalization: Assumes Azure-specific quality needs apply universally to all SONiC deployments #### Tone of voice - Overall tone: Professional, collaborative, technically authoritative with community-focused messaging ### 🛡️ DARVO Analysis - Deny: No denial patterns observed - presentation acknowledges current limitations and areas for improvement - Attack: No attacking patterns - maintains collaborative stance toward community and stakeholders - Reverse victim/offender: No role reversal patterns detected - Emotional cues and rhetoric: Positive, collaborative language emphasizing shared responsibility and mutual benefit ## 🎯 Conclusion Microsoft's comprehensive approach to SONiC quality demonstrates the critical importance of community-driven quality assurance as open-source network infrastructure scales to cloud proportions. The multi-stage validation process from PR reviews through image qualification provides a robust framework, but success depends on - Document Tags: [[quality]] [[testing]] [[security]] - Summary: Comprehensive Approach on SONiC Quality - URL: https://www.youtube.com/watch?v=sDQQgV_8DSI good morning all uh this is Anil panala I'm the senior technical program manager at Microsoft my colleague Ying Shi actually is a principal software engineer manager unfortunately couldn't join us today I am I'm sure he's he's listening us today uh remotely and he's been he's been contributed in in this presentation as well so we we both uh was thinking what to share in this in this forum I mean I'm sure most of you knows about the Sonic and you've been very closely collaborating with us so and we just wanted to give a little Glimpse on the insights of how the son if qualification is done when any when there is a change request comes in and and and how we actually looks at it and why the Sony quality is important we wanted to give little uh insights to the to the community here and what kind of help or support we are looking towards from the community as well so uh in the quick agenda here so we'll do the quick introduction on Sonic I'm sure most of you we are in the Sonic Summit so we don't really have to explicitly mention but I'm sure we wanted to give you a little uh insights uh on the Sonic introduction as well and we wanted to give you why Sony quality is very important at the stage we are in we we did a lot of enhancements in the Sonic and why it is very important at this point and followed by the some of the Control process it's been following right now for any of the image validations and and what's working currently well and what support needed from the community so let's Dive In so uh I'm sure this is one of the classic slide I'm sure you might be knowing it's a Sonic software open source for networking in the cloud it's it's open source by Microsoft and its partners and then joined Linux Foundation there are many benefits of Sonic and some of the key benefits we wanted to highlight here is uh it's basically it disaggregates the hardware Network hardware and the software which is very very important at the the industry level we are at we don't have to lock in the hardware and the software together so that's the benefit of the of the Sonic and the container solution actually offers consistent you know operational maintenance as as we just heard from the Dell they they also were able to bring up a new solution like Suzy Q we can do a lot of enhancements uh on on top of Sonic and and even in the security level also we can have a lot of features and functionalities through applications we can keep adding them and the Beautiful benefit here is uh when we when we have a bigger scale organizations and we have a lot of physical switches and and either multi running in a multiple Network operating systems instead of that with Sonic we can actually replace with Azure one network operating system so that's it will be easy for the management and operations as well so what is Sonic right so again going to the basics of it uh Sonic is a Debian Linux based uh Network operating support system which support supported by many physical uh switching platforms and it also has all major Asics on the right side I'm sure this is one of the classic diagram you might have seen from from many time so all the physical layer actually we have a PSI which is like a switch abstraction interface which acts as a hypervisor for the whole solution and it actually helps in you know communicating the SDK communication back to the north north side of it so the Sonic is consists of lot of components uh and most of all all the components are containerized using Dockers which enable um unable to build uh whatever the calls you want or based on scenario which needs to be pulled out or the call we can do on the component level as well on the top uh since it's all containerized you can use any kind of containerized management tools like kubernetes or anything like that on top of it and keep adding more more applications for configuration and management tools as well and we just wanted to say thank you to all the community members so I think it's it's been growing pretty well I've seen a lot every year over year we have seen new new community members has been added and thank you for everyone for contributing into this and making together as a one of the best networks operating system in the world so uh it's it's just a glimpse of uh how Sonic is powering Azure Cloud basically if you see uh there are multiple layers in in the network we we have a tier zero which is like a top of rack we have a tier one which is actually Leaf we have tier two which is a spine and we have one and above on top of it uh the the blue one indicates where actually currently we already enable the Sonic so most of the t0 and tier one we already enabled the Sonic tier 2 we are actually currently in in progress we've been we've been working on that it's Sonic for chassis it's coming in in the tier 2 layer and it's going to go above that and even the van and everything we have plans for that as well on the management side we also have m0s which are enabled by Sonic already and we are working on the c0 Roses as well so this is a kind of a high level of how how the adaptation rate uh it's not just uh how the adaptation rate within the Azure itself how we are trying to do and and we've been going towards it all right so uh why Sonic is important right why Sonic quality is very important so uh we we came a way of Journey right so we did a lot of things uh the features has been increased a lot the functionalities has been increased a lot and the complexity of the whole Sonic itself uh we've been improving and there are a lot of new skus I've been entering into this Sonic supporting the Sonic and a lot of combinations which is t0 Stevens and t2s has been there and the adaptation rate has been increasing the footprint in all the things is increasing so it's actually we started with a small and we are in the very big stage at this point and everything what we are doing is uh quality it makes very very important and and also the shifting left the quality right so finding the early bugs which actually Gonna Save a lot of time and also the cost for the for the organizations as well so and and we are trying to see how the community can also help us to find the early bugs before we you know start getting into the into the production networks so we would like to share some of the process we follow currently so any uh anything in the community or anything we see right so there are a couple of ways we do it right one is the any of the feature requests uh we have it or the issue reporting so any feature requests will go through the high level design review and the test plan review and which concludes as a PR creation or any issue being remoted within the reporter within the community we do the issue triaging and we find the solution and we create as a as a as a pull request as a new change request into the into the system so and this is one of the key uh slide of this and what we do with respect to the process right so uh any of any any pull request anything any change request has been created so before it's merged with the code there is series of unit tests being done uh in the in the in the environment and that's very very important and every every change has to go through APR and and because we need to do the protect the image integrity and we we run we make sure all the unit test is actually passed and and before we merge the code and that's the first stage of it so this is a stage process so once we once we go through that cycle so it will go as an image build and then it goes to the image qualification so image qualification has more uh more advanced qualification more testing is in is done across in the in this stage where we have integration testing we do the data and the control plane testing and we also make sure that it's been tested across the multiple physical appliances uh before uh and we make sure that it's it's completely tested to 100 to 100 percent before we we call it as a ready to deployment and that's where this is the stages are very important and we will any any mechanism anything we feedback goes back any of the stages for example we find issues in the image qualification side or even even in the pilot or production whatever issues we observe we try to get back as a feature request are the bug fixes and we try to make the changes again with the same process so in this uh in this also we we wanted Community to help us any anything we find issues ahead of time we wanted your support to look into it and help us you know build it more faster Solutions on that all right so what what's currently working right this uh we we have full support from community and and it we have a lot of uh comprehensive quality Gatekeepers so as I mentioned we do unit testing integration testing control plan testing for building images we make sure before we merging the code itself we do a lot of checks and it's very important and every feature and change request has to go through a PR process and and the Control process in place before we merge it and we've been growing the number of test cases right so every year by year we've been improving and keep adding all the test cases and this is where we need help from you as well to increase uh and to support and help in building more test cases into that and for all different combinations and layers within the within the network and as I said Community has been has been helping us with the unit testing and repo management as well and these are the things which been going well for any of the qualifications so far ah what we actually need I mean and we we need we need support from you guys to share and any of the Upstream test cases you've been running across your organizations and we are okay and we we are we are open to listen some of the math test methodologies you've been using and we want to see them evaluate them and adapt them you know if there are any any new things to be you know innovated from your side uh any any any test case reporting tools you've been working on we wanted to hear from you and if you can share across that and we as I mentioned earlier we wanted you to help contribute in growing the the control plane and and all the test cases uh for the son if qualification and also not just the Sonic itself right so we also wanted your help or contribution towards the PSI test and the infrastructure uh on testing on that as well so yeah so thank you for listening and and we wanted you guys to help us in in all the ways you've been helping already but we we request you to look into this and see how how you can add or contribute contribute to the sort of qualification thank you any any questions foreign uh normally I think uh when we when we there is two different streams we do it so when we do the publicly available any images we have a generic images specifically which can be deployed into any of the skus you have and if if we work on any of the specific queues queues we also give a different images for that specific too um if if you don't have specific on that list I think you can come back to us or come back to the community and share that whatever the issues you've been observing with the specific images with the specific skews and we can do a triaging on that why this is not working and we can definitely look into it but the the cycle of the image processing we wanted to continue every six months and and we want to see if these issues can be identified ahead of the time or you can we can still maintain or support the older versions if you are still running that if you have issues in the newer versions uh but it's it's we are ready to listen them and as a community we can try to resolve them is foreign if you have any specific ones we can as Shin mentioned right you can bring it up in the in the triage meetings we can definitely take a look and yeah absolutely you know can you repeat the question sure sure um I think the question was a security patches right sign which is not part of the uh any of the upcoming releases I think he has a concern that uh what is the roadmap for that feature and all is it is it this question you're asking right I think Shin already shared what what we're planning for the 2022 202305 release uh we have a roadmap for every six months on that and and I'm sure if it's not there in the list and we'll definitely um so on a question of security release right so I think the security patch is not something that uh always go with the release right just vulnerability are getting uncovered at any random time so it may or may not go well with any release planning and they also uh in reality there's a lot of consideration that a a security patch issue have to follow the common industry like a cve process right so you you cannot just open up a security to everybody overnight because a lot of people are actually running on this uh in the infrastructure so uh this is something that in the in the Sonic TSC we have started the discussion of how do we want to manage the disclosure right there's a certainly Upstream disclosure of stuff like you've mentioned FR on news and maybe some CVS from Debian itself and maybe some CV in whatever SSL right so there's a staff upstream and the Sonic team is more a community is more on the receiving than of since how do we react to it and then there's certainly going to be things that it's actually part of course Sonic right maybe someone made a Sonic change that is having a security issue so I don't think the uh we have a valid kind of Ready Step yet the TSA is discovering uh discussing of how do we want to handle that collectively because we have a lot of operators that is probably need some time to clean up the fleet right in the infrastructure and then we'll have vendors who need to kind of prepare that for the operators so that they can load out right so yeah yeah so yeah [Music] you you will see some analysis on a specific FR since coming soon there's some there's actually some discussion in the TSC community on that front on the FR vulnerabilities on whether that's a real one that's impacting Sonic or not we will have those kind of analysis coming thank you that was a good question uh any other questions I think most of them automated and yes yes yes the final these are more of a platform you know specific on that kind of a thing yeah any questions all right thank you so much [Applause], Comprehensive Approach on SONiC Quality, YouTube, 01ka7meh2jr04j7y4ktjs0bmcd.