The cost of fixing a defect in testing is a worn-out argument, so I won’t beat a dead horse here. Rather, I’ll provide some insight into some aspects of penetration testing that isn’t as commonly discussed. I want to talk about how testing fits into a mature security development lifecycle, and how altering your process can actually make penetration testing less traumatic for your schedule and your budget.
Development teams often wait to “Test” until the “Test phase”. This could be true whether you use agile or waterfall or something in between. What often gets lost in translation is that testing is an activity, not a phase, and should be a persistent activity throughout development. Perhaps a better word would be inspection. You inspect your design with a design review, you can inspect your code with a code review and you can inspect your binaries through penetration test methods. The goal of penetration testing is to find anything that was missed during design review or code review. Penetration testing should not be a single line of defense or the only point at which you look for problems.
One way to measure the maturity of your software development effort is to look at the severity of bugs found in testing. If you are finding really nasty security vulnerabilities (or performance or stability issues) then you should ask yourself what you can improve upstream in early parts of your process. When teams find nasty vulnerabilities during verification, that’s a great testament to their skill as penetration testers, but it is not a testament to your team’s ability to create a secure application. Instead, you should be thinking about penetration testing as a backstop that is used a last line of defense, to catch anything that was missed in the earlier stages. If your application is designed and built right, vulnerabilities should be minimized and only a few critical one’s should surface in penetration testing. And vulnerabilities, after all, are architecture, design or coding mistakes that ideally should be prevented or caught well before verification.
Below are some key best practices. To learn more, feel free to view my “Six Key Security Engineering Activities” webcast.
- Understand the impact that an effective secure SDLC will have on this problem:
- Conduct architecture reviews to find problems before coding
- Conduct code reviews on modules, not final code base – you can catch coding mistakes early and reduce vulnerabilities
- Conduct frequent, smaller-scale testing (regression testing) – ensures you are vulnerable for much shorter amount of time
- Use Threat Models to optimize and drive security test planning – ensures that you focus on the high-risk areas of your application
- Analyze your vulnerabilities to improve your coding techniques - most vulnerabilities result from the same coding mistake
- Leverage testing as a backstop activity to ensure your design and code was implemented correctly
- Unless accompanied by complimentary design effort, testing as a stand-alone activity reduces the effectiveness of every dollar and minute spent
