Joe is the Senior Director of Product Security at Highspot and has been in the security industry for almost two decades. Before transitioning to Product Security, he held leadership roles for security consulting and development teams.
During our Ed TALKS on Scaling AppSec, Joe provided his best tips on:
- Getting early/easy wins
- Methodically choosing a toolset
- Selectively cross-training teams on offensive & defensive techniques
Quick Early Wins
Joe stressed that having some quick wins early on is essential. He does this by meeting the engineering team where they are, e.g., showing them one or two select things that are and are not working, then solving one of their critical problems by working side-by-side with them so they see for themselves what an effective strategy can yield.
Being the proverbial bull in a China shop doesn't work – telling teams what they are doing is wrong or making assumptions about what they need without hearing from them will not win anyone over. You need to be sympathetic to how development teams are measured. If you can demonstrate how they can improve a process by even 10% and show how it's not going to take any more time, require they learn a new tool, or slow down the delivery of features, now you've got their attention. While disagreements will happen, it's important to avoid relationships turning adversarial. It isn't the security team "catching the dev team" doing something wrong; it's about ensuring objectives are positioned jointly and mapping out slow incremental improvements. Now, there are going to be certain things that you're just going to have to throw away, of course, and that's fine. But this is an easier conversation when positioned as "us not meeting our shared objectives" versus "what you are doing isn't working."
Watch Now: Ed TALKS: Scaling AppSec – Getting Tools to Perform
Modern application design and the continued adoption of DevOps expand the scope of automated security testing and push tools to the limit. At the same time, complex platforms like IoT and Blockchain require more specialized tools and skills. Hear how product and appsec pros plan to scale software securely in 2022.
Choosing the RIGHT tools
Joe said that five years ago, he would have insisted that all tools are all terrible and there's nothing a tool could do that a team of expert security engineers couldn't do better. Today, he feels that choosing the right tools is a significant component of developing a mature software security strategy. As a SaaS provider, Highspot writes a lot of code in-house; however, it also depends on open-source, external libraries and software. This leads to software supply chain challenges, heavy CSP & API use, licensing requirements, and other areas beyond their immediate control. This complex ecosystem makes the prioritization of security-focus a challenge. Like many security teams, they can't go to the product team and say, "Okay, I just scanned our entire code base, and here are our 5,000 out-of-date libraries. Let's go." You're not going to win any friends that way, nor are you likely to address genuine product or enterprise risk with that tactic.
Joe's take is that those who work in security are ultimately in a sales position. We are selling security internally to non-security stakeholders. Like any sales effort, to be effective, you first need to help your target understand the value of your proposition. In sales, you need to help solve problems for your clients. He suggests starting by highlighting one specific problem, importantly, coupled with a recommended solution that is sensitive to the context of your client (in this case, a dev leader tasked with getting a set of features out the door by a specific deadline.) Take your time to wade through that morass to figure out what will solve a particular problem in the short term (maybe something recurring or present in multiple teams) and start there. Once you've vetted some tools, how do you figure out what to do without getting yourself into proof-of-concept hell? This is where tool requirements and peer feedback are invaluable. Determine the tool's objectives, if it will work with your primary tech stacks, ease of use, and other factors relative to your team's goals. This will eliminate many tools from the start and narrow down to a targeted list.
During the panel, we were asked a question from a Security Innovation intern alumnus - a nifty bit of serendipity. He asked, "Any advice for deciding when to build tooling in-house versus licensing?" This also related directly to a previous Ed TALKS where Trupti Shiralkar from Datadog talked about source code analysis scanning testing (SAST) products she had bought. She measured how many vulnerabilities they found. It was 17%. The products found only 17% of known security vulnerabilities. As a result, she asked her team to write their own automation or parsers to save the company hundreds of thousands of dollars.
Having been the leader of a security engineering team that built and used dozens of internally developed tools for expert-led penetration testing, Joe's reaction was that you need to be highly cautious about taking that path. It might solve one problem short-term but can quickly become obsolete. For example, once that little parser or library is done, and then two years down the road, we're parsing XML, and we find some vulnerability in it. We go, "Oh, well, who owns that?" Well, that person left two years ago. There are all kinds of challenges like that to consider. I assured Joe this should never be a problem as we know how well developers document their code.
He suggested there are several questions to be asked when deciding to build vs. buy:
- How will vulnerabilities be discovered? If external, how will we be notified, and who owns the alerts? If internal, who owns the testing?
- How will they patch, and what are the SLAs and response commitments?
- Is it going to become abandoned?
Expanding the colors of security beyond red (attack) and blue (defend)
While the role (and formation) of red and blue teams are well understood, many are unfamiliar with the concepts of orange, purple, or green teams. April Wright initially came up with the "Color wheel of InfoSec" concept, which she presented at Black Hat, and which Joe finds particularly useful and refers to often. He has heard all the security vs. development team banter… "Security slows us down." "Builders just need to build." "We don't need/want to understand attacker techniques," and so on. Since Joe has been on both sides of this proverbial fence, I asked him about it.
His initial take was that he's a huge fan of these short hands for meaning and understanding in cybersecurity. One challenge he identified for red and blue teams is that it naturally adopts an adversarial relationship. The red team's attacking us, and the blue team's defending, and then the developers are just in the middle going, "Oh man, parents are fighting again." But April's key points in that talk are red breaks, blue defends, and yellow fixes. I like that. When we think of things adversarially, we develop what Joe calls "Marco Polo security," like that kid's game in a pool where someone yells "Marco." Everybody else says "Polo," and you must find an elusive "Polo" while blindfolded.
Read Part 3 of Our 3-Part Scaling AppSec Blog Series
In the same episode of Ed TALKS, Rajan Gupta, Chief Product Security Officer, Honeywell, also provided some of his best tips for scaling application security.
That's how a lot of security teams still operate. Essentially the dev team does their best and throws it over the wall, and security tells them it's broken. They try again, and security says, "It's still broken." This is something I dubbed the Application Security Conundrum and first wrote about back in 2005. It's an inefficient way to manage security, more akin to tennis than anything related to software. April goes on to talk about how the red and the blue teams work hand in hand to educate the yellow team. She developed that orange team to facilitate interaction and education to avoid organizations looking for this perfect unicorn kind of person that knows all about software development, defense, and attacks.
I ask all my panelists for their Final Take, and here is Joe's: we need to think about security collaboratively, from the software we build or buy, to the teams we build, to the education we provide. Understand and remind ourselves frequently that we want to develop great software that meets a customer need. Meeting customer need means delivering software that builds trust; this is the core of security.
About Ed Adams, CEO
Ed Adams is a software quality and security expert with over 20 years of experience in the field. He served as a member of the Security Innovation Board of Directors since 2002 and as CEO since 2003. Ed has held senior management positions at Rational Software, Lionbridge, Ipswitch, and MathSoft. He was also an engineer for the US Army and Foster-Miller earlier in his career.
Ed is a Ponemon Institute Research Fellow, Privacy by Design Ambassador by the Information & Privacy Commissioner of Canada, Forbes Technology Council Member, and recipient of multiple SC Magazine’s Reboot Leadership Awards. He sits on the board of Cyversity, a non-profit committed to advancing minorities in the field of cyber security, and is a BoSTEM Advisory Committee member.
