DevSecOps lessons learned during a pandemic

Implementing a DevSecOps strategy during a pandemic taught our team these real-world lessons

As we’ve seen over the past year and a half, the pandemic has accelerated digital transformation and forever changed workplace culture. Increased reliance on digital tools has elevated the value of DevSecOps, as enterprises of all sizes and across all industries realize the importance of automating and integrating security at every phase of the software development lifecycle – from initial design through integration, testing, deployment, and product delivery.

My engineering team was no exception to this shift – we had to quickly prepare to build a new Virtana SaaS platform and deliver several new modules, all while working remotely.

Here I’ll share some observations, pain points, and lessons learned to help others intelligently embrace DevSecOps best practices within their teams.

[ How can automation free up more staff time for innovation? Get the free eBook: Managing IT with Automation. ]

Two essential steps during our big shift

It is essential to recognize the differences between legacy systems and new enterprise software, so we started by equipping our team with expertise on SaaS product development, mindset, application, security, and delivery. About 25 percent of the team pivoted to an iterative, future-thinking model that involved two steps:

Step 1: Learning

We set a goal for team members to learn product development methodology
We built awareness of the need for enterprise-class security at all stages of work
As a team, team members participated in a two-week internal boot camp on SaaS development
The team went through this process together, researching, testing, and sometimes failing but always learning

Step 2: Building

The team built a product from scratch with requirements, acceptance criteria, and guidance on what to use, using a simple SaaS product as a learning exercise
The product was functional and deployed, but completely throwaway
The team added business logic and a fully automated deployment pipeline
The team provided complete end-to-end security for all workflows (note that the majority of the team had never done development in the cloud)
The team developed the build and deploy pipelines, security controls, and application over four weeks. At the end, they had a functional (but not perfect) product

DevSecOps can thrive even during a pandemic

DevSecOps is not just about technology or security; it requires the right mixture of process and people transformations to succeed. We went through this transformation in 2019 and were well-positioned to build on it through the pandemic in 2020.

[ Want a shareable primer on DevSecOps and its benefits? See What is DevSecOps? ]

At its core, DevOps is about making it easier to build, deliver, and maintain software, allowing teams to focus on business outcomes instead of the mechanics of implementing security protocols at every step. All aspects of product security are considered from the beginning, including who is allowed to build a service, what credentials are needed to deploy, automated testing for vulnerabilities, securing inter-service communication, architecture reviews, etc.

Once you understand security constraints and requirements, incorporating security into the DevSecOps process increases organizational agility. We were able to transition our products to SaaS, provide quick turnaround for updates, and support flexible deployment models. All of this was possible because the team had fully accepted and adopted a DevSecOps mindset.

Building a robust build/deploy/monitor pipeline has always been important, but the pandemic has shown that it is mandatory to business success. Before the pandemic, some teams would build features in isolation in a private branch shared by multiple developers. A few individuals were responsible for adding the appropriate test, security, and automation at the end of the cycle before the code was merged.

Once you understand security constraints and requirements, incorporating security into the DevSecOps process increases organizational agility.

That strategy changed during the pandemic: Each individual needed to become self-sufficient and could not be blocked or gated by other team members. As a result, all team members followed the DevSecOps best practices for all code development.

Building a culture of collaboration

In my many years as an engineer, team member, and product builder, I’ve always worked at an office (with the occasional work-from-home Friday). That changed in March 2020. With an all-remote workforce, collaboration was critical to the success of our team. Initially, we over-rotated on the number of meetings to share information, maintain a cultural fabric, and provide clarity on priorities and expectations. Over a few months, we found the right balance of synchronous and asynchronous ways to collaborate across our teams.

To find that balance, you must be cognizant of team members’ needs. People need to interact and share ideas to advance work, but they also need solitude to think. Be aware that meeting fatigue is real. Based on feedback from the team, we instituted “no-meeting Wednesdays.” We also encourage team members to block their calendars for lunch, walks, coding time – and simply time to think.

Ensure clarity around roles, responsibilities, and priorities. Confusion often results from under-communication, so be very clear on the vision and strategy for your products and the company.

Don’t assume that every message you share is understood. There is an adage in marketing: A person does not get a message until they hear it seven times. This is also true with teams; members need to hear and see the vision and strategy repeatedly to assure everyone is on board. Consider repeating your message in seven different ways, such as via email, verbally in meetings, in PowerPoint presentations, and reminders in communications channels. Remember, we all learn differently – some of us are visual learners, others understand information better by hearing it. Communicating seven times, in seven ways, will help ensure that your message is clear.

In the end, teams should feel they are solving problems for customers and moving the company’s mission forward.

Reinforce how the work being done every quarter aligns with that vision. Ask customers, salespeople, and product managers to share details about problems that need to be solved. If there is new information that changes the priority (or feasibility) of a particular feature, get to a stopping point and move to something else. In the end, teams should feel they are solving problems for customers and moving the company’s mission forward. No one wants to work on something that won’t be used or doesn’t provide value.

[ Also read: 9 DevOps and DevSecOps best practices for the hybrid work era. ]

As people, processes, and technology come together to deliver the outcome, we continuously adapt and change as needed. We use tools to increase visibility into decision-making, explaining not only what needs to be done but also why. Teams, especially remote teams, need to believe in a vision and understand the impact for customers in order to do their job effectively.

A successful remote culture requires planning. With distributed teams working on different areas of a product, it is imperative that all pieces come together to deliver the desired outcome for customers. This requires various team leaders to coordinate work in advance so that within a given iteration, or sprint, all the work across teams supports the same outcome.

This is the “people and process” aspect of transformation, also known as agile transformation (I prefer to call it an “iterative development model”). We are continuously improving on this aspect, and it has proved that with the right tools and processes, a remote team is just as capable as an on-site one.

The biggest takeaway

Change is hard. It’s not convenient. If you’re happy with your outcomes, you should not change your process, people, or technology. However, if you’re looking to transform your products, grow revenue, provide increased value for your customers, and accelerate velocity, that usually requires transformation.

DevSecOps and iterative development require change, which often clashes with the cultural norms within a company. Often this change is thought of as something the engineering or product teams need to do rather than as a company-wide initiative. This outlook is bound to fail.