YouTube Video pH993eBUOdk  ## Metadata - Author: [[YouTube]] - Full Title: YouTube Video pH993eBUOdk - Category: #articles - Document Tags: [[devops]] [[monitoring]] [[security]] - Summary: YouTube Video pH993eBUOdk - URL: https://www.youtube.com/watch?v=pH993eBUOdk Good afternoon everyone. Thank you for coming and joining our session on automated governance using a unified pipeline platform. I am Deepangjan. I'm a director fullstack engineering at Fidelity Investments. I primarily look into the design, development and overall delivery of the compliance and security application in the asset management. And today I have with me my colleague Apurva. Hello. >> Hi. So I'm a Perva. Um I also work at Fidelity and I'm a senior cloud engineer. Uh I'm on a day-to-day basis I kind of work uh on fullstack side. So I in my in my past role I was working on DevOps. So that's where my some of this knowledge comes from. But in my previous role, I was kind of similarly doing very similar things to this. Uh, and then I slowly transition. So these days, I kind of work on more on a coding side. So backend, front end, and some of this DevOps stuff as well. So that's kind of my day-to-day role at Fidelity. And uh, yeah, I'm very excited to be here and thank you all for coming in and I know it's right after lunch, so we'll try so that you all don't fall asleep, but we'll do our best with that. >> Thank you, Apurva. So before we start I just want to call out something is like what we are going to present today is based on lot of proof of concepts some experimentation that we have done and for the next 30 minutes or so you are going to see some of those examples and all and tools and techniques but we are not endorsing any of those tools or techniques explicitly. Okay, with that being said, let's took a oh sorry. Okay, so let's take a look at what we have for the afternoon in the agenda. So I will start with setting up the context about why we are talking about governance in software delivery life cycle, why it is important, what are the challenges that we face and as engineering teams we always try to come up with some solutions when we see a problem. We will talk about the unified pipeline vision and then I will hand it over to Apurva who will take you over the solutioning part. He will talk a little bit about the architecture, the key capabilities that we have built and what is the way forward and we will have a Q&A at the end. Does that sound right? Okay, let's get started. So when I talk say governance in plain English, governance means control. Now when it comes to software delivery life cycle or software development life cycle, what does governance means to all of you? What does come to your mind? Anyone? Yes. Exactly. So for a software development life cycle whenever we are trying to put an application into production we all want to make sure that it meet each of those changes are auditable so that we don't have to spend sleepless nights when there is an issue. The next thing we want to make sure is each of those changes that has gone into production has met the security and the quality controls. It meets all the organizational as well as regulatory requirements. Now regulatory requirements is something which is more kind of you will see in finance or something like in a domain that where we work on. Okay. Now we want to govern or control all of this. It's very easy to say but it's very difficult to put guardrails around all of this. We see lot of challenges in that and we have encountered it especially in big organization. What are few of those challenges? The first one is fragmented governance or fragmented CI/CD techniques and pipelines. We have distributed teams across the globes and they build pipelines the way they want to. They have different controls and checks in place and it is not uniform. So that leads that slows us down. We don't have something which is uniform across the organization. The next thing that we see is okay engineering team is ready for a release. We go to the release engineering and say we are ready with the release. This is our release notes and they say no stop. You cannot do it. Why you have not followed you have you are failing this guardrail. Okay. Now manual approvals. So the moment manual approvals comes into picture what happens is the stand the those are not standardized always different people try to enforce them in different ways and that slows us down lack of visibility also we see because we don't have a unified dashboard where we can see like when we are putting an application in production what is going in and if it fails some of the control and the checks where it is failing. So net in net manual governance is not scalable at all that we always get developers get slowed down with that. So we started brainstorming like how we can solve this problem and that is where we came up with our unified pipeline vision. What is that vision? We should have reusable flexible pipeline templates which can be used across all engineering teams in the organization. Governance should be a part of code. Governance as code that's what we call it. It means that whatever the controls you want to put in those should be part of your pipeline templates. And finally we should build a unified dashboard where we can see like if anything is going wrong what is going wrong and what should be put in place. Now with this idea we went to the different engineering teams asking for their feedback and what we got and the some of the benefits of this unifications were consistency. We have a unified governance model across all the CI/CD pipelines in the organization speed because we are getting rid of the manual intervention which will speed us up. Transparency the unified dashboard will give us transparency and finally scalability. You can run thousands of your pipelines at the same time and they go through all the unified pipelines. With that being said, now I will hand it over to my colleague Apurva to take you through the next cycle of how we solutionized it. Apurva. Okay. So I'm a kind of walker on the stage. I'll try my best to stay I think this is the area but so thank you the pungeon for kind of giving on the theory and then the basically why we're here and why the solution came into place and I'm just going to look at the bullet points because it's very hard to see from here but I'm going to take a point at a time which is automated compliance checks so these are just some of the key capabilities usually there are way more than this but I just wanted to write few of them and just talk about few of them. Um I'm kind of I usually I'm a kind of different kind of speaker. I engage in the questions even while we're doing it. So if you have questions, ideas, feel free to raise a hand. We can even talk now. You're always going to have time at the end to start off. Um automated compliance checks. I think someone said rules um when when asked the questions and rules and guards. I think that was the answer. And so those are the things right that currently somehow we have to go through manually. It depends on your size on your organization the teams the engineers the leaders on how this looks from organization to organization but one of the key thing that we have seen is that with this solution what we're trying to avoid is manual approvals manual process and in all of those manual if something goes uh unnoticed we want to avoid all of that and so with this vision what we're trying to achieve and one of the uh key capability is having that compliance check integrated into the uh solution system that we're kind of talking about. And again, I'm going to refer to the system as different things. So, it could be you can name it however you want. You can call it a template catalog, just the functions, uh reusable library. They're all just talking about the same concept. Actually, we just have to go back. Actually, we're still going to be so um so that's kind of uh compliance. So, now seamless CI/CD integration. What I mean by that is with this solution regardless of what source code uh platform you're using uh we want this solution will be complied across all of them. So and also the CI/CD tool there might be different team focusing on different uh different team using different CI/CD tool different source code management tool and so one of the goals for this solution is that it works across all of them and that kind of plays role into scalability adoptability that I'm going to talk about in a in a bit and the enhanced efficiency and security. So with this solution um one of the biggest thing that we're going to achieve is efficiency. Um obviously the security is the key concern that I talked about. Uh we're solving that but I'm a developer by hand and so one of the thing I focus on is efficiency and how I can do less manual work or how I can provide more throughput outside. How can I bring value to the business um and also enjoy coding. So one of the thing that takes away that fun is manual uh manual governance and the fear of not having just in the back of my head I'm like what if I'm missing something uh what if if what if this is not something what I'm supposed to do. So this kind of get rid of all of that and so that with that it gives me one efficiency but also secure and reliable mindset and and the software itself and the final scalability and adoptability. So with large organizations and a lot of teams, it's very important that all of them can adapt to this. If we can't adapt to this, you're not going to be able to sell this or well sell this to your other teams to kind of consume it. So adaptability is very important and that also kind of lines with scalability. So if there's one if you build something that is not adoptable just for one team, there's no point of building this. You're just waiting. You're just you're just wasting kind of time and effort. uh but if you can if other teams can adopt and reuse uh even the best is uh scalability right if other people can contribute to it and make it better u that's going to give you a long-term benefit and so with this solution u those are kind of four key capabilities that we kind of offer but with that being said uh the platform components so here's where I kind of want to talk about it so I tend to give like broad view and then more detailed technical view as So to start off basically we're going to talk about I don't know how the background for DevOps in this room. So I'm just going to give two different examples. So we're going to start with the first so this vision and then the solution that we talked about. So what is part of this right? Um basically what are the component uh what's really what am I even talking about? So to give you first to start off like a explain to me in a 5-year-old way we have a we're going to call this a reusable library. That's going to be my moving forward definition and you can just call that um if you were to kind of move forward with this. But let's say you have three different stack. So we have one stack here that has a 100 books and each of those books are call it a modular function. Uh but for for this sake it's just a book and there is piled up in a stack and there's 100 books right next to it. There is another stack that also has a 100 book but it's tied. So what I mean by that is there is 50 books and a 50 books. So if I want to pick up that I can only take 50 books with me versus if I on the first one I can take any book I want. I can put in back any book I want to put back. And the third one is there's again 100 books but I can only pick all 100 of them. So just to summarize there's three stacks. one that has 100 books and I can only take 100. I have no other flexibility. And then there is the second stack which has 50 and 50 and I can take one of them or both of them or none of them. And then there is a third one that has 100 books. I can pick and choose. It's up to me. But then that's more responsibility but I do get full flexibility. And so that's kind of basic theory of or not basic theory but that's like fivey old example explained um explanation. And so now I'm going to talk more of a technical what that means into the next you'll see uh I was just again it's not like real professional uh way but you'll understand what I did there and it's easy to understand. Um so I'm going to go through one next slide. Oh before we go to the example and explain you how and what that means and what that looks like. One of the slide I have here is the module workflow and reason why I have slide for that is that is very big key component to it. I really think think having this uh in a modular uh module style is very important uh because again it I already talked about adoption and one of the other key thing we get with that adoption is consistency. So if I have some team or as an organization if I want to if I want to make sure that all of the teams are using the same exact thing and and as the pundant said earlier with teams doing having across regions and sites and different time zone it's very hard to press that consistency and so one of the thing with modularity is we can pick and choose and that is also consistent to the governance that we have decided as an organization and so that's why I really believe the mod this whole modular workflow is is kind of benefited and that's why I have a slide about that but that also being localized u what that means is again one of the thing is every team is different and from my experience what I've seen is teams want to do things differently and they have their needs it's there's nothing wrong with it but every application is different every business case is different and so with that we need to provide some kind of customization but while still allowing and enforcing that governance and so we want to give them the freedom and flexibility but do not want to lose that control. So wouldn't it be best if you have control and the governance on it but also while still give them just a bit of freedom so they can localize and customize it as per their as per their need and so with this modular workflow that's something what we achieve and that being said sorry there's there's two more points sorry I'm taking few points but so balancing governance with flexibility I think I just kind of talked about it um and this also allows us to scale with pick and choose. Um I'll show you that that 100 books I talked about that can be 200 books. Usually it's it's couple hundreds thousands of books and I talked about only three types and three examples of stacks right but in reality that is just example in reality that could be five stacks six stacks and those are different. What I mean by that again is you can have a stack where you combine the full flexible one and not flexible one and have something else. And so with each different stack you can depending on your business need you can customize that make it your own but just the goal is to start somewhere um and I'm going to talk about that in my last slide is to just talk to your developers uh just look at what's out there uh what is basically causing you issue for your governance and then build according to that. This is just a start point and so one more slide. Okay, so example that's where I wanted to go for all all of this time. So we're going to look at the example of all the things I kind of talked about and okay so we're going to look at here if I don't fall down. Okay. So to start off on this side which is full flexible and what that is something I was talking about and that is just an example but in your devops you want to do build you want to do test you want to verify you want to have anything this is literally endless but rather than you writing a code rather than developers writing the same script and pipelines over and over again what if you have something like that where you can just pick and choose. So if you want to build your project and test it and then verify, deploy, promote, uh whatever you want to do and I'm guessing if you're doing that, other teams are probably doing that as well. And if you look at it, most likely it a lot of teams are just rewriting that or what they're doing is they're copying the same code. And then what happens in that case is when some when this team X team changes something and if they had copied from that well now Y team doesn't know about it unless they check it. But with having modular functions like this that is kind of that case is gone and that's again to bringing back to the original idea of consistency and adoptability. So again I put in a lot of functions there and then there's example and those are meant to do whatever you want u that include different kinds of testing different kinds of building uh code scanning security gates your governance um and then audit checks etc and I'm promised you're well not promised but I'm pretty sure your your security team will be very happy with this because every pipeline that is run across your organization has to go through all of these And let's say you want to change a version from Maven or Java from 17 to 21 and if you want to enforce it across your entire org well and you have a modular function for build Java projects well you can just do that in the one place and if everyone is using that and if everyone has adapted to it well now you don't have to worry about that everyone is compliant and as a as a team I don't have to worry about if am I compliant with 17 or 19. So that's kind of where I come from on the full flexibility that also brings that's a lot of writing even with that that is a lot of function. So again now so just to summarize where we are we have a 100 books now and someone wrote 100 books which is good now whoever comes in next don't have to write entire book but they just had to pick and choose but what if we take a step forward and now we create basically stacks of book from the already written books. So first let's say we had 50page book now we are creating 500page book. So that's where the medium flexibility comes in. So you're basically essentially using the same modular functions but now you have created another library that is using that. So now you come in and first for this one was you have you can do build test verify etc. This is kind of predefined templates. So if you are just doing build and test well there is instead of using build function and test function you just do one build and test or in this case option one I guess uh if you want to do verify and or I guess deploy rather than putting in four functions you just say option two and then there is that and what if we go on a little bit more for some of the customerf facing applications if there needs to be extremely compliant they need to have governance and they need to be more strict and and there's reason for it. So uh again one biggest one being customerf facing but for all of that what if you already have we can call it template catalog however you want but a full stack everything ready to go you just pick and choose and then the fourth and fifth which is not here and which is what I was kind of talking about is it's up to you to uh to kind of move forward with fourth and fifth is you have full what can you do can you add in you have a full stack can you add in one from here one from the medium flexibility how do you want to do that and so That's where kind of all of this comes from. But this is just an example of visual workflow for what that might look like. But at the end, if you want to have an idea and if you want to chat about it, we'll more than happy to see how what ideas you have and how else can we do this. But again this is just a starting point a vision that that can help u to solve the challenges that we're we're talking about the governance as a code because by adding governance as a code to this this is just a pipeline and by adding all of this you are basically forcing all the teams to be governed and all of your softwares are basically tested compliant making sure that before the before the delivery happens it's all ready to go and So this is just a continuous slide from the last one what the user flow might would would look like. So if user A is using a full flexible workflow or full flexible library as I said they can just say okay go ahead and print that I am starting a build uh maybe notify X team that I am starting my build for Java and then connect connect clone build test etc. And so one user could choose that or one team can use that and then there's a second user or team that don't want to write that much. Uh this is already someone has contributed someone has created uh a library for you to use which is option three and one from the last slide. So you can just have two lines again I'm just saying here uh this is very simple and to see here but when you put it in the code's perspective you have hundreds of line behind this functions. And so now by doing this you're already eliminating this person who is using it all. You're already eliminating a lot of those reusable stuff and saving them a lot of time. With this you're saving more and more. And so that's kind of where all of this kind of vision comes up in and practical knowledge. So behind this just visualize your code. But that's kind of where this is coming from. But if you have feedback questions on that let me know. I'm I'm very happy and excited to talk about this. So that being said, let me give some of the lessons learned. But that was kind of the basic architecture uh in place for this for this vision. And so some of the lessons learned here is start small and as I said again I I'm keep repeating the same thing is every team is different, every developer is different, every application is different. So just start first start and then pivot as you go. Most likely I've worked into similar things before and so most likely things pivot. In my case 100% of the times things have pivoted. But I we started we start somewhere and then we pivot. If you don't start we're never going to get there. And so talk to the developers, talk to people who are governing it. So security team the he talked about release engineering team. So just talk to developers and the release team and everyone all your stakeholders and and then go from there. Um I again as a developer one of the thing I struggle with is I tend to not ask question and do everything. I think I know everything and I'm going to do everything and then we all know about the documentation reading. Same thing I don't read documentation. and I don't ask stuff and then I go for it and then I realize oh I should have talked to people and so that's a lesson learned from me and so I just wanted to share with you all is do start and talk to your stakeholders and you will learn and get to the solution much faster. The second is governance as a code is a cultural as much as technical. So this whole practice of governance as a code it's a cultural shift. Uh your leadership has to be on board your development team has to be on board. uh because one without the other one does not work. First obviously your leadership needs to be on board but if your developers are not on board that also does not work. If uh let's say you have something that says if you're not compliant with the latest update of this new latest Python library and in that case you can't release well your developer needs to agree that everyone is going to be okay with that and this is what they want at least majority of them not everyone might agree but as at least the majority of them and so it's a cultural shift for your organization um it obviously it's technical but apart from that everyone needs to be on board happy excited um well not happy but at least understanding for it and so with that development team and then as I taking the example of uh outdated vulnerable library maybe it works for one of them it doesn't work for other or or anything like that but talk to your developer they're the one who are pushing the code they're the one who are running the pipelines might as well talk to them and get some feedback and see what if that helps with anything and lastly metrics so all this you got to show some value to your leadership to your business and see if it if it's working and even if it's working you need to show them the metrics and with this p uh with this uh consistency and with this solution one of the things you're going to get is metrics how many pipelines have run how many pipelines are using this workflow what portion of those uh basically uh modular workflows are using XYZ softwares or or the library and So all of that being said, it's very important that you use and so metrics that now you're saving time with this change and you are also being compliant and more governed. So you build software faster, you build software safer, and you're also making you're also making everything compliant for developers. And so the last key takeaways I talked about before is governant does not have to be it doesn't have to slow you down to kind of get rid of the very first slide is manual effort manual review manual reviews that takes a lot of time. Not just that along with that you have you're you're putting a risk that something might go unnoticed and something might be missing. So the governance doesn't have to slow you down with this solution. You can again have something that's consistent, something that's adoptable, something that's more secure, and something that's governed. And so with this unified pipeline, we get velocity plus compliance. your software will be delivered faster, more secure and all of your your uh pipeline and software will have scans to it giving you more more more mindset more more mindset that is that can sleep at night I guess um as I was talking before some of the times I think what I if the things I'm pushing are they really supposed to be pushing so that's that's giving that peace of mind at night um and so automate policies enforce on scale infinitely. What that means by that is the governance as a code. Now you have your so policies change dynamically depending on your company and organizations. All of you know that policies change constant and it's dynamic and so with this um you have a centrally managed policies. So with this if there's a change you just change it at one place and then it's applied and applied and enforced throughout your all thousands of pipelines that you have and finally create culture of innovation. So I'm going to hand it over to the pundant but last thing I want to say is you might be wondering how is this creating a culture of innovation? Well, when you let developers do what they do best, which is writing code or spending more time on learning about coding and other things and not spending more time on writing pipelines or governance code or worry about the governance, they are going to lead to innovation. So it's a cultural shift of even the developer mindset and by giving them freedom and time to focus on what they do best or what they want to do best and not spending time and effort on manual things that could be avoided and so that's what I mean by creating culture of innovation and with that being said I'm going to give to the punan to kind of take it us to the end on the future of what that governance mean >> thank you Apurva so we spoke talk a lot about governance. So what's next? What we can build better? So as you heard from Apurva, the unified pipeline vision, it's great, but can we do better? Can we have self-governing pipelines? And what do we mean by self-governing pipelines? Now today you have a pipeline which is monitoring and giving saying that okay something is failing. Now can we have some pipelines which will like monitor that can adapt and that can respond to a compliance issue. Let's think about it like you are rolling out a production change and that change has a software update which doesn't meet the security standards. What will happen today? Your pipeline will fail. Now still some developer has to come in. He has to put in a fix. What we are so we are still being re reactive to the situation. What we can do is we can have pipelines which can send automated me teams messages give an call on call to the engineer and they can come and fix but that is still reactive right. What we are aiming for is being proactive. How is there a opportunity that we can build something which can roll it back that particular release? Can it do an automated upgrade upgrade of that particular security fix and then push it into production. So these are some of the areas that we are exploring and the future is limitless and very exciting. With that said, we are concluding our talk here and I want to open the floor for any Q&A. Thank you., YouTube Video pH993eBUOdk, YouTube, 01kac3ss4ar6trz4rrkp99pwsm.