Understanding development teams

Implementing any cross-functional program requires empathy. Collaborations between security and development teams commonly break down due to perspectives and goals being so different. While both teams may want the same end-result — secure applications — the preferred workflows and measurements of success can be very different. Fortunately, empathy can drive alignment.

When rolling out a modern AppSec program, it’s crucial for those who interact and support the development teams to have a clear understanding of the problems, frictions, and frustrations the developers currently face. Only once they understand those, can they see how to integrate security into the development process. Let’s dig deeper into the areas that are worth assessing and being conscious of before starting new collaborations. Below are some questions you can ask yourself before starting a rollout.

"[Security] has got to be a partnership. A fundamental reason that a lot of our security fails occur is that we try and dictate to, rather than truly partnering with, the groups that we’re trying to help be successful. My role is to help our engineering teams at Mandiant produce good, secure, functional code quickly. We need to move fast as a business, partnering is how we produce things that solve the business needs and security needs."

Tim Crothers, SVP, Chief Security Officer at Mandiant

How empathetic and understanding are you to your development teams?

Understanding the way your development teams work and how they make decisions is critical. By understanding your team’s general approach and development maturity, you can set realistic expectations of what you want them to do, determine how and where they can actually change, and see how you can best support and empower them. Without this, you will not be able to understand their point of view, creating friction and decreasing the chances of success. Also, remember that when thinking about the following topics, the answers will likely differ from team to team within the same business unit or even department. Be sure not to make assumptions about development teams and consider how they’re likely to differ in their culture and approach.

What is their appetite for change?

Some development teams are attracted to the latest tech — sometimes too much — and are willing to try new tools, technologies, libraries, programming models, etc. just to stay modern or to see if it might be a better solution. Other teams only make these types of changes because of an existing pain point or because they are certain that the change will fix a problem that currently affects their application. And then there are other teams that will avoid change by default, operating under the philosophy of “if it’s not broken, don’t fix it.”

Recognizing the behavior and mindset of your development teams in terms of how much they are willing to change will help to determine how best to collaborate with them in finding a common goal or destination for your AppSec program. For example, if they have resisted testing via Git pull requests (PRs) in the past, don’t assume adding security testing in their PRs is going to be any different from previous attempts.

An important distinction to make when estimating the maturity and adaptability of your development teams is technical adoption vs. process adoption. It’s common for less mature or earlier stage companies to be more focused on technical adoption in comparison to process standardization. For example, startups need to release fast to be first to market, which pushes process and standards maturation lower down on their priority list. It will be quite difficult to get a company in that phase to implement any security practice that acts as a gate.

How well defined is the team or projects?

A major difference between the security view and the developer view is ownership of assets. From the perspective of the security team, there are a number of assets owned by the engineering teams, some of which are more critical than others. From the developer perspective, each team has a focus, a number of projects or services that they own, projects they contribute to that other teams own, and projects they contribute to that many teams use and rely on, but don't have a specific team owning them.

With this in mind, asking a development team to own the security of their code and projects will have varying success depending on the ownership of each individual project. The more crisply defined the area is, the more likely a development team will be to take on the responsibility for the security of that project.

Additionally, the age of the team can also have an impact on the way you roll out. Ideally, standards and processes can be added from the start for a new team. But a more likely alternative is to take advantage of natural disruptions. For example, if a team has a major change unrelated to your program (new manager, team overhaul, etc.), they’ll be more likely to adopt additional development standards as part of defining how they want to work.

What is your team’s current capacity?

Like most teams, developers aren’t waiting around idly for work to arrive. They’re prioritizing what is most important for them to work on next, knowing full well that they can’t get through everything on their ever-growing todo list. Simply adding more tasks to a developer or team’s already overflowing plate is not constructive or supportive. This is amplified when teams are under-resourced and doing their best to go sprint after sprint while also trying to keep their heads above water.

"We realize they have a lot of other responsibilities. They have to build features and products. They have to worry about performance reliability. We want to make participating in security as easy as possible."

Jason Chan, VP Security at Netflix

What’s the breadth of team abilities across the organization?

The reason to take the time to understand the dynamic of your development teams is because they’re all different. It’s extremely common to find a smaller number of teams on the leading edge that adopt newer technologies and processes first, with the rest of the teams in the organization following them over time. Majority adoption tends to happen slowly at first, but then ramps up when enough automations and best practices are in place to lower the barrier to adoption for other teams. Of course, requiring the bar to be significantly lower for broad adoption to occur means that many teams will take much longer to adopt — if at all. Before you can drive developer adoption, you’ll need to know how many of your teams are going to need a lowered bar, as well as which teams can pilot to create the automations and practices.

An example of the breadth of team abilities can be seen in a team’s tendencies to add integrations and to want to receive feedback. Typically, mature teams want feedback, and they want it as early as possible — in their IDE or in automations in their local build processes as well as in their Git repositories. Less mature teams may only want to automate in the CI process, meaning they’ll get feedback late, typically just before a production deployment. Trying to add both of these types of teams in the same group of rollouts is unlikely to be successful, and most likely going to overwhelm the less mature teams.

A common cause of disparity across adoption is the complexity and age of the projects. An old, complex service that is being maintained rather than actively developed is a good example of a project that would not be a good candidate, as you can expect greater resistance when changing processes. Similarly, if your organization expanded inorganically (through acquisitions for example), you will be inheriting different technologies, pipelines, and cultures, which decreases the consistency across the wider organization, and increases the chances that team assumptions will be inaccurate.

How do your dev teams integrate security practices into their pipeline today?

Having taken time to assess the development teams in your organization, it’s important to identify which security practices they currently follow and how they follow them, so you can determine a baseline to build from. Development teams typically start in a very hands-off way initially, perhaps periodically running tests that are for visibility purposes only, and integrating these into the pipeline where possible. There would likely be no blocking actions or gates anywhere to start with, allowing the development team to continue to release at the speed they’re accustomed to.

Then take note of their preferences in integrations. For example, are they scanning or testing in CI, or earlier in PRs? Are they using an out-of-the-box integration, or have they done any scripting and automation to make it work for their pipeline or project? Do they test in IDEs or is their approach more reactive? A team that leans towards integration will be more likely to adopt an integrated developer security solution.

"The key, from my perspective, is just understanding our engineering teams’ preferred practices. What are those patterns so that we can partner to put in guardrails, rather than controls? We want to support the outcomes that the teams want. We need to be able to ensure that the appropriate practices and processes that they’ve determined are correct. And typically, we’ll collaborate on those. If you really simplify it, the consistent thing is looking for gaps — gaps in our processes, gaps in our [collaboration]."

Tim Crothers, SVP, Chief Security Officer at Mandiant

How do they prioritize security during sprints?

Another good indicator of where the team is on their security journey is whether their security focus is more on future development or their backlog. Often, backlogs can be quite overwhelming with thousands of issues or vulnerabilities, versus a new feature that might only introduce a few. An important aspect of this is also understanding how the team triages issues to understand what is important (or necessary) to fix, as well as when and how to escalate. If a team has OKRs around their backlog, as well as processes in place to create forward momentum, they’ll be more likely to adopt tools that will help them get there faster.

How do they incorporate security testing and fixing into their sprints? Do they require a clean security test to deliver code, or do they add security tasks to their technical debt backlog and spend a sprint every few months fixing their issues? The latter is often common for teams that aren’t quite as mature, or even some lower-performing teams that can’t contain the work in sprints. In these instances, it’s equally common to see these types of dedicated sprints for scaling, performance, or reliability — all competing for time with security sprints.

Finally, does the team have any internal documentation of how they test, or SLAs they’re working to for their issues? These are all good questions that don’t just give you an understanding of where the team is today, but also tell you what the next step is to make it easier for the team to test and focus on the right things.