Guiding Teams to Deploying Safer Software Faster
I won't waste your time on the business case for the security and throughput benefits of DevOps. They are well known. That, however, does not mean that it is easy to integrate security into a DevOps pipeline. It requires preparation and planning, but even a small commitment to doing so will lead to significant improvements across the entire process. The key for executives to focus on is making security a natural part of the pipeline rather than introducing additional bottlenecks that might slow deployment.
There are three fundamental principles to DevSecOps from both an execution and GRC perspective:
- Treating everything as code
- Dealing with a serverless architecture
- Container orchestration
It is also essential to understand at which points automation can be maximized without compromising (and even improving) security.
I’ve found the following to be best practices for implementing risk-based approaches at all phases of DevOps.
Planning in DevOps
The planning stage for DevSecOps is similar to any security planning - threat modeling, policy development, and security assurance models. The difference in a DevSecOps model is that you set the stage for integrating security into the deployment pipeline in the planning stage. In this phase, your team should be determining tooling, test acceptance criteria, and providing guidance and education to developers and operations teams alike.
This phase will set the stage for the entire project. Ideally, your plan will inject enough security to be effective, but not so much that it can overwhelm teams with unnecessary false positives and obstruct release schedules. The safest way to do this is to spread security out across the entire pipeline, so each stage has a natural and incremental security checkpoint. At each checkpoint, you perform security checks that only apply to that stage. As deployment progresses, the security checks should evolve from simple automated pattern matches to more complex human-driven penetration tests.
Coding in DevOps
The coding stage provides many opportunities for both introducing and automating security.
First is the environment the developers use for writing code. As you shift security left on the DevOps pipeline, ensure that developers are working in secure environments. Virtual machines, Docker containers, VPNs, and user desktop machines can expand the attack surface and increase risk and need to be secured accordingly.
It can be daunting to determine the tooling that is provided to developers to identify common security mistakes. IDE add-ons, linters, and similar tools do not catch complex logic errors but are well-suited for spotting common vulnerability patterns. Most importantly, defining how developers handle secrets such as passwords, API keys, or private certificates is critical. The best approach is to use tools such as Azure Key Vault, AWS Secrets Manager, or HashiCorp. Remember to make such tools available throughout the entire development and deployment pipeline.
There are numerous benefits of security automation during the coding phase of a DevSecOps pipeline. Having a process that is repeatable with the same results each time ensures consistency. The implementation of policies and configurations becomes self-documenting and easy to audit. Plus, it adds the ability to scale up as necessary or roll back configurations when needed.
Throughout the coding process, developers regularly commit code to source control. This is another opportunity for security checks to be fully automated using pre-commit git hooks and static application security testing (SAST) tools. At this point, only include checks for the most severe or show-stopping security issues. During code commit, it is also an opportunity to automate scanning for hard-coded secrets or API keys that developers may have accidentally left in the code. It can be wise to run some custom unit tests at this point, but again, limit these tests to the most serious issues to avoid overwhelming developers at each commit.
Lastly, be sure to treat everything as if it is code. Usually, this refers to software-defined networking and other configuration data, but teams should also treat security as code. They should review hardened docker files, chef cookbooks, digital certificates, hardening scripts, firewall configurations, and user permissions in this context. Having teams approach these items as code makes them more visible, more accessible to audit, simple to reproduce, and easier to identify security issues.
Building in DevOps
As your integration platform runs continuous builds, you have the opportunity to perform additional security checks. At this phase, the focus should still be on the most severe issues. Still, it can be prudent to perform other tests such as checking for vulnerabilities in third-party components and dependencies or performing more in-depth risk-based checks.
Builds don't need to be broken unless serious or large numbers of vulnerabilities are encountered or publicly disclosed issues with third-party components are found. For all other security concerns, trigger alerts for the security team. This stage is also an ideal time to gather security metrics, digitally sign and store build-time artifacts.
Testing in DevOps
This phase is the last chance to identify security vulnerabilities before the code goes into production. Unlike waterfall or other development methodologies, DevOps allows for rapid and continuous deployments. Some organizations push dozens of builds for deployment every day. But fast deployment cycles can lead to security checks delaying deployments, and security teams can be overwhelmed with a never-ending stream of builds to test.
The DevSecOps pipeline approaches these issues by automating all testing up to this point and only breaking builds for the most serious issues. All other issues found in testing can be addressed through the normal feedback loops, such as bug reports and security team alerts.
That said, more in-depth testing shouldn't be neglected entirely. Teams should perform manual testing periodically and particularly when automated results are concerning.
At this point in the deployment, many obvious bugs should have been identified with earlier automated tests. Over time, existing code will become more secure, and vulnerabilities reported from other sources will already have been documented. Therefore, much of the testing in this phase will continuously verify and document policy compliance.
Nevertheless, there is always room for advanced testing, especially to address emerging threats, including:
- Static application security testing (SAST) tools can check for a broader range of vulnerabilities, and any unit tests should address any additional items identified and your threat modeling
- Dynamic application security testing (DAST) tools can test applications in a running state and cover the entire execution environment
- Interactive application security testing (IAST) tools can interact with the applications' user interface controls and API models to identify vulnerabilities
- Fuzzing tools can feed irregular characters and data lengths to identify common flaws with bounds checking and error handling
This testing phase should include application code and OS Configurations, databases and other back-end software, hardening scripts, other automation files, and certificates and certificate chains.
Deploying in DevOps
Once the application is approved for deployment, the security team will need triggers to perform any deployment-related security steps. These might include:
- Creating or updating any secrets necessary for deployment or operations
- Documenting what has been updated and by whom
- Producing any compliance documentation and final security scan output as artifacts
- Performing any final configuration or hardening
- Determining the authenticity of any third-party components
Operating in DevOps
Continuous feedback loops are a big part of what makes DevSecOps so effective. In addition to the traditional bug reports, error logs, and telemetry that provide feedback to developers, integrating security offers additional sources of feedback, including:
- Pen-testing reports
- Monitoring metrics
- User-reported issues
- Vulnerability reports and update notification
- Intrusion detection, antivirus, and SIEM alerts
There is a lot more that goes into creating a secure and repeatable DevSecOps process. However, it’s essential to ensure that the highest impact activities are implemented early and continually scrutinized and honed.
About Ed Adams, CEO
Ed Adams is a software quality and security expert with over 20 years of experience in the field. He served as a member of the Security Innovation Board of Directors since 2002 and as CEO since 2003. Ed has held senior management positions at Rational Software, Lionbridge, Ipswitch, and MathSoft. He was also an engineer for the US Army and Foster-Miller earlier in his career.
Ed is a Ponemon Institute Research Fellow, Privacy by Design Ambassador by the Information & Privacy Commissioner of Canada, Forbes Technology Council Member, and recipient of multiple SC Magazine’s Reboot Leadership Awards. He sits on the board of Cyversity, a non-profit committed to advancing minorities in the field of cyber security, and is a BoSTEM Advisory Committee member.
