devops vs devsecops

src: https://github.com/bridgecrewio/checkov

Abstract

There are 3 things that make DevSecOps different than Devops:

1. a secure software supply chain
2. improve culture and collaboration
3. automation and guardrails

I’ve never really been sure how DevSecOps differs from plain-old DevOps, but over the past year I think there’s finally something enough there to have a notion. To be concise, DevOps-think makes software delivery better by moving operations concerns closer to development with the help of a lot of automation and process change. There are three things that I think make DevSecOps different and additive to DevOps:

• A secure software supply chain – This is a fancy way of saying “we know all the components that went into building and deploying this software and trust those components.” It also includes the actual CI/CD pipeline that you trust and that’s resistant to third parties including malicious code, as we’ve seen happen in recent years.
• Improved culture and collaboration – Increasing collaboration and understanding between developers and security staff. As with many governance practices, with security, the governed (developers) and the governors (security staff) usually have an antagonistic relationship. Developers see security as unstoppable masters of “no,” and security people see developers as clueless coders. Well, that relationship isn’t helpful! As with DevOps, transforming “culture” to be more helpful is part of DevSecOps.
• Automation and guardrails – Automating security policy enforcement, and providing defaults and templates to make it as easy as possible for developers to write secure code and applications configuration from the start. Historically, verifying that developers are writing secure code has been a manual, error-prone process. Much of this can be automated now with good platforms.

Sure, we could dig deep into whether each of those is actually new and significant enough to deserve a new DevOps portmanteau infix. But who has time for that? Why not spend that time reading the below instead?!

DevSecOps integrates a secure software supply chain into DevOps

A simple supply chain

In recent years, the idea of a “secure software supply chain” has become a fundamental part of DevSecOps. A secure software supply chain starts with a “supply chain,” which is, mostly, just a renaming of “build pipeline.” The diagram above shows a simplistic example. A pipeline is filled with the activities that go into compiling, integrating, testing, and eventually releasing software. Let’s look at the three parts of a secure software supply chain.

Governing inputs and dependencies

First, you should think of a “software supply chain” as all the inputs that go into building and deploying an application. These inputs start with the code your developers write; the configuration used throughout the application lifecycle; the third-party frameworks, libraries, and other services your application relies on (e.g., to send text messages); and any other ingredients that go into making the application. Developers depend on open source and third-party services more than ever, meaning that your supply chain also needs to verify and trust that third-party code. 

Dependencies on third-party code quickly bring in the transitive dependency problem. For example, let’s say you’re using a PDF-generation library that uses a font-rendering library that depends on a logging framework. If that logging framework has a security problem, you now have that security problem regardless of how secure, verified, and trusted the PDF-generation and font-rendering libraries are. Indeed, this is exactly what happened recently with the Log4j security problem.

One of the major innovations in DevOps-think is treating the configuration needed to deploy and run the application as code as well. Since there are different groups involved in developing, testing, deploying, and even managing applications in production, it’s easy to lose track of this configuration throughout the application lifecycle. But, that’s no bueno in the DevSecOps mindset: that configuration is a major input to be tracked and secured.

Building

The second part of a software supply chain is the build process used to build and verify your applications, known as continuous integration. This is a decades-old, well-established practice and set of tools. In DevSecOps-think, this build phase might enforce policies and swap out different components for the software. For example, when a security issue comes out, an advanced secure software supply chain can rebuild your applications with patches for operating systems and frameworks, without having to trouble developers. This is exactly what Wells Fargo and others have been doing, which enables them to not only patch production quickly, but to rebuild production entirely several times a week to blow out any malware.

Secure delivery

Third, a software supply chain usually consists of another part: continuous delivery. This means automating releasing of the application builds to production. (There’s a distinction between continuous delivery and continuous deployment, but it’s mostly lexical hair-splitting that’s best deployed to clear out the room when you want to wrap up your cocktail party.)

The secure part of a secure supply chain is ensuring that the inputs—code, configuration, and third-party frameworks and services—are secure and follow your security policy. This largely means tracking and verifying that those inputs are what you think they are, and that dastardly people haven’t inserted malware into your applications. The diagram below illustrates some possible supply chain security problems.

Vectors of attack across the software supply chain

Obviously, a secure software supply chain seeks to prevent these and other security problems.

DevSecOps improves culture and collaboration

“Security is now everyone’s responsibility” is a commonly heard phrase in DevSecOps discussions. With the right tools and process in place, following that principle is possible, but getting people to change their behavior and how they work with each other is necessary as well. That is, part of applying DevSecOps is changing the culture of security in your organization. This should make you nervous, as changing “culture” is extremely difficult.

First of all, what even is “culture”? Thankfully, the school of DevOps has been whittling away at defining what culture is in an application-delivery context. My colleague Hannah Foxwell recently cataloged how a DevOps culture framework applies to security.

I won’t simply list the attributes here—just read Hannah’s write-up!—but instead want to point out what I see as a major mindset shift: thinking about security as a continuously evolving product. Instead of thinking of security as project-driven audits and reactive incident responses, DevSecOps staff think of developers as their customers and the security process as a product they’re creating and delivering for developer use. This is a product management approach to security. 

Applying product management thinking to security shifts security’s focus from enforcing compliance to making the right thing the easiest thing for developers to do. This is why the tools and automation components of DevSecOps are so important: those are the products. Because the practices of product management are so widely understood and proven over decades, you don’t need to reinvent the wheel when it comes to applying DevSecOps. As one example, check out a concept we call “platform as a product” that transforms traditional operations processes into a product-driven approach.

Once you take a product management approach to security, you’ll have a better understanding of why talk about DevOps culture tends to obsess over empathy, kindness, and, simply, “talking with people more.” Aside from, you know, being a good person, those are key tools for product management. A good product manager has a clear idea of what people using their products need, what those people are doing with the product, and how all that activity links to those people’s organizations’ goals. They can “get into the mind” of their customers. The goal of a DevSecOps product manager is to make it easier to create and deliver secure applications. And, instead of rolling your eyes at “those developers” who “don’t know anything about security,” you’re creating products to make developers awesome at security.

That product-oriented mindset about security is a helpful way to think about one of the more popular phrases in DevSecOps talk: “shift left.” At first glance, “shifting left” seems to mean giving developers more responsibility over writing secure code, ensuring compliance, and controls. I think that’s sort of true; part of DevSecOps is getting developers to write more secure code. But, expecting developers to handle the majority of security tasks is too much to put on an application developer’s shoulders. Instead, “shifting left” means moving security activities and policy enforcement closer to developers, treating them as “customers,” and building out the guardrails, platforms, and tools they need to build secure applications. Let’s go over those tools and platforms next.

DevSecOps automates verification and sets guardrails

DevSecOps enables automated verification

With all of the tracking that goes on in a secure software supply chain, you can automate much of the verification and governance that security teams require to check and then release software. This verification has been done manually for years—often, literally with email and spreadsheets. It is time consuming, error prone, and generally unpleasant. For organizations that want to start deploying their software weekly, if not daily, manual security checks are impossible to work with. Just imagine how long it takes to even schedule a meeting across different organizations like development and security!

Thankfully, the automation that comes with DevSecOps build pipelines helps reduce the toil of governance. It makes it easy, for example, to create a bill of materials (BoM) and sign container images. You can store all of that in a repository, even add it to version control, and security staff can create starter application designs and configurations that template out security best practices. 

Automating these activities is made easier because of the infrastructure standardization at the heart of Kubernetes. One of the main goals of Kubernetes is to define a common standard for the lifecycle of containers, networking configuration and policy enforcement, and runtime customizations. The application development model, favoring stateless components that run in containers, allows Kubernetes to enforce the desired state of applications. This desired state is the configuration of the application and infrastructure in production. It’s stored and can also be versioned. Simply knowing the current state of your system is a huge step up from the rat kings that most production systems end up looking like. 

The configuration enforcement that Kubernetes performs also improves your security capabilities. This means that when and if configuration drift occurs, Kubernetes can usually kill and then redeploy applications into their secure, trusted state. This doesn’t solve all your problems, of course, but it does make cleaning up production easier and deploying patches much faster. And if you regularly redeploy, or “repave,” all of production from scratch, you reduce the window of time malicious actors have to mess around. For example, following this pattern, Wells Fargo can rebuild production multiple times a week and can deploy numerous patches throughout the week.

DevSecOps empowers security through guardrails

The second capability a platform gives security staff is the ability to put guardrails in place that make it easier for developers to do the right thing. The notion of “guardrails” here means providing baseline templates for secure applications and including code- and image-scanning tools in your supply chain to enforce the inclusion of secure code and frameworks.

Another good example of a security feature you can bake into a platform is logging. In too many cases, each application team will come up with its own format and practices for logging. There may be clusters of teams that follow the same scheme, but in an organization with thousands of developers, security staff will encounter many different logging cultures. A centralized platform standardizes how logs are created, stored, and retrieved. This means that security staff don’t have to learn and adapt to each application’s logging scheme, spending time coming up with clever regular expressions to search for errors, or just figuring out how to get ahold of the logs in the first place.

Zero-trust networking is another major set of capabilities to build into your security-as-a-product platform for developers. Most application designs rely heavily on communicating with other components and services over a network. Traditionally, those connections were mostly secured because the network was considered secure. This isn’t really viable nowadays as developers are using more and more third-party services over a network. Thus, you want to make sure your developers are doing things like requiring TLS for connections.

That’s just one area that security can start standardizing. A DevSecOps approach to platforms can also standardize controls like: 

• Image signing 
• Data classification 
• Secrets management 
• Input/output validation 
• And more! 

There are many more details on this topic, and how you automate and put guardrails in place depends on how you build your internal platform. Check out this talk from David Zendzian for an example of using these kinds of controls on a Kubernetes-based platform. Also, check out Adib Saikali’s book, Securing Cloud Applications.

Transitioning from DevOps to DevSecOps

Now comes the tricky part, and where I’m going to let you down: putting this all into practice. It’d be simple to say that you just need to install the right tools, create and automate secure supply chains, and hire a security product manager. That’s not enough. But, there is no missing ingredient, or occult knowledge. The simple secret is to actually do it and keep doing it. Changing habits, however, is difficult. Let’s look at four steps to start.

Understand your goals

Hopefully, the above has given you an idea of the goals you should have beyond “be more secure.” However, as CISOs and other security wonks will tell you, it’s good to start with an understanding of the trade-offs between capabilities and security policy, your risk management strategy and plans, and the ongoing threats to security. Security is much more than just bugs and hackers. In the context of DevSecOps, this deeper understanding should revolve around your goals for the software you’re building and the business that software is running.

Audit your current DevOps practices

The first step to putting DevSecOps into practice is to evaluate what you’re already doing. Usually, if you ask your various teams—apps, security, and ops—if they’re doing a good job, following best practices, and otherwise being useful they’ll—surprise!—say yes. People in larger organizations are pretty good at delivering the outcomes you’ve asked for. They will be doing something. So, what you need to focus on in this audit is assessing whether your people are delivering the new outcomes you want, the goals of DevSecOps. 

Map your end-to-end process

This audit should start with looking at the complete, end-to-end lifecycle it takes to get software out the door. Take a hypothetical case of adding a new feature to an existing application and whiteboard out everything that needs to happen, from idea to coding to securing to deploying to running in production and finally to a person actually using the feature. Note the time each activity takes and the time it takes between each activity. That time between each activity is wait time and is usually caused by passing off responsibility from one group to another, for example, security compliance and audits. 

Implement change

Mapping out your end-to-end process helps get over the local optimization that exists in most organizations. Each team is typically optimized, but the handoffs and coordination across the end-to-end lifecycle are often, to use the technical term, not cool. Take this end-to-end stream and ask “how can we improve security tools and processes at each stage to speed up the release cycle and put in more security controls and policy?” and start there.

As you start, we of course have help to offer when it comes to the tools and automation part. First, check out our learning path on putting DevSecOps into practice. Next, check out VMware Tanzu Application Platform, which is all about making the right thing the easiest thing. You can start using the Tanzu supply chain choreographer (based on Cartographer) to create a secure software supply chain and the developer portal in Tanzu Application Platform (based on Backstage) to work on better collaboration between application developers and security staff. There are great, ready-to-use secure software supply chains with guardrails included. You can and should customize them and make them your own, of course. You can see a lot of that pulled together in this demo from Ryan Baker.

Additional DevSecOps resources

Now, I didn’t want to create an exhaustive overview and list of DevSecOps intricacies. Instead, I just wanted to lay out a simple, baseline understanding. There are many more details to learn once you dig in, of course. If you want to start putting DevSecOps into practice, I recommend the following:

1. “Creating a Culture of DevSecOps” – Hannah Foxwell goes in-depth on transforming your culture to be more DevSecOps aligned.
2. “DevSecOps: Integrating security with software development” – Integrating DevSecOps into the developer’s inner-loop and Kubernetes
3. “Secure Your Software Supply Chain with New VMware Tanzu Application Platform Capabilities” – Tiffany Jordan and Tazin Progga go deep on the secure software supply chain notion.
4. DevSecOps for Dummies – A straightforward guide to implementing DevSecOps practices into your organization

Good luck out there!