DevSecOps and containers: How a cloud-native model places new demands on organizational security
More often than not, container projects start out small―as experiments on developers' laptops. One developer shares with another, and after a few months of tinkering, a team brings the packaged-up microservice into the organization's production environment. After a quick security check, it will be ready to deploy.
That's when things derail. The security team finds vulnerabilities in the container library and halts the project. Security leaders send the app back for reprogramming and demand extensive documentation. Costs pile up, managers start pointing fingers, and the problem gets escalated to the C-suite.
App delivery snafus like this are nothing new. The development, operations, and security legs in an organization have traditionally fit together like a wobbly stool. But in recent years, these segments have started to pull together, bringing forward a cultural shift toward a new concept called DevSecOps. In DevSecOps, security teams and concepts are embedded in the DevOps process. This ensures that security is addressed early and often in the app delivery chain rather than getting bolted on at the end.
The strategy of "shifting security left" is particularly important for organizations developing container applications. Containers are continuing to grow in popularity and are becoming more integral to the app delivery process. They're harder to secure than traditional virtual machines (VMs). On the other hand, the way containers work indirectly provides an opportunity to facilitate better cooperation between development and security teams.
Containers can be harder to secure because their attack surfaces are larger than VMs'. In a typical configuration a hypervisor has dozens of VMs on it. If a security team finds a vulnerability on one VM, it can simply patch the running VM. Since container platforms use a shared operating system kernel, an attacker can exploit the whole system through a single point of failure.
So, either through an exploit vulnerability or through poorly programmed apps, an individual container can impact the availability of all containers running on top of that kernel. Whereas with virtual machines, each VM has a separate kernel, and an exploit (or poorly programmed app) is limited to cause damage only to that one instance of the VM.
Containers thus provide an environment that's conducive to DevSecOps collaboration. The fact that containers are immutable means they're replaced with new versions, rather than getting patched during runtime. This provides a kind of environmental parity for developers, operations, and security. It increases consistency and predictability of workloads, facilitating easier collaboration among these disparate stakeholders.
The culture shift has to start at the top. In the past, executives took a hands-off approach to technology, focusing more on ensuring that a product gets released on time and is popular with customers. They viewed security as a cost center―paying for insurance they hoped to not have to use.
With containers, it's important to take a security-first approach. Security needs to be addressed at every level of the product development lifecycle. If executives aren't buying into this approach, a container project is likely to fail. You need to have top-down support to be able to support the structure and initiatives that need to happen: training, finding the right people, and putting an effective security program into place. Executives have the ultimate responsibility to ensure that their technical teams are putting the correct security controls in place, such as code assurance tools built into the CI/CD pipeline or an approach to secrets management.
Executives need to commit to putting the proper training programs in place. Developers want to ship high-quality code, but they're often not trained in concepts like ensuring the importance of using verified container images in their platforms. Providing access to secure coding education internally can help developers understand the possibilities around secure coding as well as their roles in DevSecOps.
It's also important to identify internal champions as a way of promoting the importance of DevSecOps. Senior people on the development team can coach and mentor junior developers and at same time act as an interface to the executive team. Executives need to know the money they've invested in enabling a culture change toward DevSecOps is generating a return—in terms of, perhaps, achieving a lower defect rate or releasing higher quality software.
Shifting toward DevSecOps
How does a shift toward DevSecOps take place? An organization can start with three goals: adopt a secure by design approach, executing the concept of shifting left, and following a motto of being secure by default.
Secure by design
Work in cloud-native, or container-oriented, environments tends to be fluid. Developers seldom start with an up-front design. They change designs often as they stumble onto new ways to do things more efficiently. So it's hard for a security team to walk in and do a threat modeling environment late in the game.
It's best to build secure design principles into the development process from day one. Starting early, a specialist can mitigate risk factors in the design stage, achieve better visibility into the software supply chain, and make the right security recommendations. It's also an opportunity to involve the security team in the review process early.
In traditional waterfall development, testing and, by extension, security testing has always been positioned as a last-mile activity—something that gets done after the software has been created and just before it gets passed over to operations. If you bring in security the day before a product goes into production, security becomes a bottleneck in development. Plus, the security team ends up sitting on the bench in between release cycles.
By shifting left, you move the testing cycle earlier in the process. Software testers get involved in product development discussions. They can understand how a product is likely to be used so they can develop their testing schemes based around that. This helps them identify problems earlier on. Because DevOps is all about small iterative changes to a product, the concept of a release day is no longer valid. Instead, you work toward creating a minimum viable product and add functionality as you go along.
Secure by default
Since developers are not security experts, it makes a lot of sense to follow a secure-by-default mentality. This is where you provide developers and operations staff with a set of tooling to build secure applications. In a container environment, you have a concept of a registry: a directory where you can download container images. You start with a container image and then build upon that to add your own functionality.
Anyone can download a container image from the web, but these images can have vulnerabilities in them. Adopting a secure-by-default posture means rather than allowing developers to download container images from a public source, you maintain a curated registry inside your organization. The only container images a developer can download come from an accepted source. Each of the orchestration platforms offers that capability. By having a private registry, a security team can control the lifecycle, as well as the libraries and the security of the containers, and replace those images as vulnerabilities get fixed.
When you have a container orchestration platform, you can turn on controls that are useful—like secure registries, container vulnerability scanning, and sequence management. But very often, you don't have things like real-time monitoring of the containers or the ability to provide firewalling between containers or the whole plug-in to the CI/CD pipeline. There are some things orchestration platforms can help with, but there are others that need to be added on in addition to them.
Today, there is no single vendor or platform that provides true end-to-end security for container environments, so it's important to build a security architecture based upon industry best practices and best-of-breed security technologies. You need to understand the marketplace, the challenges, and the right security tools to plug those holes. It will end up being a combination of things the orchestration platform offers, things third-party tools offer, things open source offer, and changes in processes.
Keeping the culture strong
How do you ensure that the DevSecOps culture you've built is equipped to keep providing top-tier container security? Here are a few tips:
- Make sure you stay up to speed with new challenges as they come out.
- Make sure your container security program is integrated with the rest of your security management program. A lot of organizations have a CERT (computer emergency response team) that keeps its eyes open for new vulnerabilities that come out, making sure they're addressed properly, and it's important for this team to also be aware of the container toolset in use.
- Make sure you're keeping up to date with developments in the container security space.
- Make sure assessments from a compliance perspective extend out to the container environment as well.
It's also important to note that the container ecosystem of vendors is still young and evolving. Docker, the acknowledged container king, recently changed its strategy, announcing the sale of its enterprise business to Mirantis to focus on the open source component going forward. Kubernetes is emerging as the container orchestration platform of choice. Other start-ups will get acquired or merge or go out of business. Make sure you make appropriate technology choices, and make sure they're not going to be unusable. Don't pin your whole program on one technology.
Secure from the start
Container technologies deliver a lot of opportunity in terms of productivity and flexibility in app development. But unless you get buy-in and executive support, and unless you do the appropriate due diligence and risk awareness, you risk opening up a big security hole. That's something that can be addressed when you move toward DevSecOps: getting security involved in the container development lifecycle and making sure developers are aware of the need for secure coding development practices.
DevSecOps and containers: Lessons for leaders
- Containers can introduce big security holes. A DevSecOps approach to containerization ensures security is addressed early rather than bolted on at the end.
- Focus on three goals: Adopt a secure-by-design mentality, execute on the concept of "shifting left," and follow with a secure-by-default approach.
- Efforts to build security into the container development lifecycle require top-down support. If you don't have that, those efforts are likely to fail.
- Why containers will drive transformations in the 2020s
- The telecom network is modernizing with containers, cloud-native
- How containers and open source Kubernetes accelerate innovation
This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.