Seventy-five percent of running containers have high or critical vulnerabilities, according to our recent study. Worse yet, these flaws have patches available, so they could be (but haven’t been) fixed. Many industry veterans wouldn’t be surprised by statistics like these, but weren’t things supposed to be better in the cloud?
Things will be better, and for some people they already are. The most advanced and meticulous teams can reduce the number of running vulnerable containers to 5% or less. They accomplish this by shifting security testing to the left in their software delivery pipelines and building streamlined, easy remediation workflows for developers and operators alike. Creating good processes around shiny technology has always been the greatest of all security struggles, and it’s no different in the cloud.
Bringing Bad Habits to the Cloud
Cloud migration does not magically modernize workloads or the processes around them, and security is no exception. In fact, security is often the last thing we want to address because it tends to slow down everything else.
Let’s take the example of multifactor authentication (MFA). Most of us know, or at least have heard, that this is something we should implement, especially for accounts that are the most important to protect. But do you have MFA set up on all your bank accounts? Most of us probably don’t. We never seem to have the time, and the extra prompt asking you to confirm your identity every time becomes annoying
The cloud isn’t all that different because it’s operated by humans. Sysdig data shows that 48% of organizations don’t have MFA enabled on their most privileged account, the root user. Further, 27% of organizations use this account for administrative tasks, against the advice of cloud best practices and Center for Internet Security (CIS) benchmark guidelines.
Because identity and access management (IAM) is one of the most critical cloud security controls, we should strive to develop new, cloud-native processes around it. Cloud teams should create IAM roles scoped to specific tasks with no extra permissions, as well as train their users on how assumed roles work.
Oh, and please enable MFA!
The Security “Shift Left” Is Incomplete
Transforming a complex process flow takes time, and it usually happens in phases. The data shows that 48% of images are scanned for vulnerabilities for the first time either in the CI/CD pipeline or in the container registry — that is, before they are ever deployed into runtime.
On the one hand, this means that many companies are successfully “shifting security left.” On the other hand, that remaining 52% of images are scanned for the first time in runtime, delaying the discovery of potentially critical vulnerabilities. Part of the reason is that we still have a tendency to delay security assessment to save time. Another possible explanation is that a subset of workloads isn’t being included in the “shift left” mandate. Specifically, many third-party applications that do not contain any custom code created by the organization itself fall into this category. For example, Kubernetes components, Web servers, and load balancers may not get configured until long after software testing stages are completed.
In either case, the “shift left” is happening, but it’s incomplete. The most advanced teams are using their CI/CD pipelines to test all artifacts for security, including infrastructure components like the deployment manifests for hosts, clusters, containers, and more.
Risk Acceptance Can and Should Shift Left, Too
When we talk about accepting risk, we often think of it as the last resort: “There’s nothing else I can do at this point, so I document it and hope for the best.”
However, the sooner we discover our sources of risk, the better equipped we will be to create effective mitigations for them. We may still end up running 75% of our containers with high or critical vulnerabilities, but if we have actively considered the risk associated with those vulnerabilities and implemented controls to reduce it, our security posture may not be so bad.
As we worry more and more about software supply chain security, it’s worth asking how many runtime vulnerabilities are in a third-party component we didn’t even build — and that our software doesn’t even use. Identifying vulnerable dependencies earlier in the software delivery process can save a lot of time and significantly reduce risk exposure. You won’t fix every flaw, but many can be managed with additional security controls. At the very least, the flaws should be documented for visibility
Is There No Hope?
Billions of containers are running in the cloud today. That number will only increase, expanding the attack surface and potential risks. Indeed, a new environment with new tools can be an opportunity for innovation, or it can be the beginning of yet another mountain of technical debt. Cloud is our chance to start fresh and get it right this time. We may just have to slow down at first and invest in building those cloud-native processes to make sure we get it right.
