Achieving Substantial Gains in IT Performance Across Government Through DevSecOps

Guest Blogger:  Chris Yates, Senior Solutions Architect, Red Hat

Eli Whitney, famed inventor of the cotton gin, demonstrated the value of Interchangeable parts in the United States in 1801 to the US Congress, President John Adams, and President Elect Thomas Jefferson. This was a critical demonstration of the impact and value such a feature could bring to the military. Whitney demonstrated the viability and value of interchangeable parts by stripping down several muskets, and then reassembling a functional musket from random parts from the disassembled muskets.

Today, we take for granted that parts are interchangeable, from the bolt carrier of a rifle to the alternator on a transport vehicle, we assume that one is as good as another. But, as with information systems developed today, muskets of that era were bespoke artisanal creations. The parts for any given musket were custom fitted to accommodate the variation in manufacturing for the other components comprising the whole. A gunsmith would be necessary to replace the hammer or pan of a musket, to return it to working condition, as it would need to be fitted and modified to complement the other components it interacted with. The same can be said for information systems today that often require a specialist or team of specialists to configure, deploy, modify, or repair in the instance of a failure.

To address the bespoke nature of information systems, and to gain the same types of benefits for information systems as interchangeable parts brought to manufacturing, the Department of Defense (DoD) is adopting DevSecOps. DevSecOps has seen accelerated growth in the public sector over the last two years, especially within DoD. In this first in a series of blog posts, we’ll investigate the impacts the adoption of DevSecOps is having on the federal government. DevSecOps enables the adoption of modern, advanced application architectures and behaviors. The fundamental principles of DevSecOps significantly influence the development, capabilities, architecture, and the path of innovation of new and existing technologies in the industry. Further, we'll explore how DevSecOps principles may change the way government relies on industry and how it may impact the resource available to government in the future.

First, what is DevSecOps? DevSecOps is a combination of processes, tools, and people, which combine with enterprise values across the disciplines of Development, Security, and Operations.  DevSecOps form a unique culture to enable more efficient delivery and management of secure software. The integration of security augments the DevOps practices seen in industry. It’s important to integrate security throughout the process in government adoption to effectively reap the benefits of iterative improvements. Traditional development processes more often incorporate security as a checkpoint that needs to be passed, but does not integrate security concerns throughout the process. DevSecOps elevates security into a first-class citizen, instead of a bolt-on checkpoint. The adoption of DevSecOps is extensive across the federal government including the General Services Administration, Air Force, Army, and Navy.

DevSecOps Processes

From a process standpoint, DevSecOps can be seen as an extension beyond the software development lifecycle practices found in agile development and Continuous Integration and Continuous Delivery (CI/CD) methodologies to improve the operational behaviors of the deployed system, including security. Taking the principles of continuous deployment and applying them towards operations management, and introducing configuration as code, operational tasks can be automated, and through this automation, increase resiliency. The golden apple of this process being push button automation; the ability to completely redeploy a component of infrastructure from bare metal to full operational capability by kicking off the appropriate automation playbook.

DevSecOps Tools

The list of tools utilized in DevSecOps are myriad, and it’s more important to have the right classes of tools, than to have precisely the same tools as another DevSecOps practicing organization might carry. DevSecOps is a combination of all three pillars of processes, tools, and people, and DevSecOps isn’t a single product that can be purchased. Unfortunately, there’s no such thing as a DevSecOps box we can install in a datacenter. The collection of tools are focused around source code version control, build automation, test automation, security validation, performance testing, configuration management, and extend into project management systems that permit prioritization, issue tracking, and team collaboration.

DevSecOps People

The people component of DevSecOps is often the most challenging in the DoD. Conway’s Law says that organizations tend to build products with a design that reflects the communication structure of the organization. Organizations organized into silos trend towards building applications built into silos. With the tight integration of roles required for effective DevSecOps adoption, many government agencies are seeing a need to flatten their organization structure and integrate IT professionals into cross functional teams aligned across a product, rather than maintaining role based internal organizations that communicate through ticketing systems. This restructuring and alignment of personnel helps drive results driven outcomes, by bringing everyone together, working towards the same goals - the successful release of their product - rather than being focused on an organizational functional role and ticket-ops.

DevSecOps Advantages

DevSecOps brings several advantages to the table for DoD agencies, including shorter time to value, faster iteration to field new capabilities, and moving risk left. The most valuable advantage is shortened time to value, whether that value be measured as innovation, reliability, or reduced rollout lead time. The Waterfall process, the most common traditional development process, however, focuses on extensive requirements documentation, development up front, which can set goals for the development of features, and capabilities for the first release of a product that do not all have significant impact or provide wide reaching value across the user base. DevSecOps is able to bring more value to the enterprise, more quickly, through its iterative feedback process.

DevSecOps Hurdles

One of the more significant hurdles in adoption of DevSecOps in the government is the concept of the Minimum Viable Product (MVP). However, the MVP is a key component in the value achieved by moving risk left. The Pareto Principle, or 80/20 rule, resonates even in product development. Often, 80% of users only use 20% of a product’s capabilities. If a product is developed through the traditional waterfall model, 80% of the users may be blocked from getting value out of the product while waiting for the development and testing of features they won’t use. The saying “perfect is the enemy of good” applies when trying to satisfy completing every desired feature into a waterfall release schedule. This not only complicates the integration and testing of all of the features, but delays the delivery of valuable features.

DevSecOps is positioned to significantly influence the traditional monolithic acquisition and development processes the federal government has canonized through the nearly 2,000 page long Federal Acquisition Regulation (FAR). FAR defines the processes that must be followed for the acquisition of products and services by executive agencies. Because of the guidance of these regulations, the rhythm of the FAR processes strongly influenced the adoption of waterfall development practices. However, industry has demonstrated that the waterfall model of software development has significant shortcomings when it comes to the development of innovative, novel solutions.

Problem Solving with DevSecOps

When solving a new problem, there are two types of unknowns. There are known unknowns, and unknown unknowns. Known unknowns are challenges we know upfront we will need to tackle. Unknown unknowns are those unforeseen challenges we will face as we break new ground in the production of a novel solution to an unsolved problem. When dealing with the production of innovative solutions, the unknown unknowns are the true stumbling blocks that cannot be discovered or predicted when following a Waterfall model until after the schedule is set. This is another area where moving risk left is accomplished by maintaining small, strategic, incremental changes through the DevSecOps feedback loop. If a particular feature is intractable or sufficiently complex, it can be reprioritized, abandoned, dissected, or reimagined much earlier in the development process, before it can impact and possibly prevent the delivery of other valuable capabilities.

Many people, when thinking of DevSecOps, imagine cloud-native web applications using microservice or serverless architectures, or container-based workloads. But DevSecOps is not a process only for the development of these more sophisticated, modern architectures. DevSecOps can be used to develop traditional monolithic applications, n-tier architectures, service-oriented architectures, and even embedded systems. DevSecOps, however, significantly improves the viability of producing these more sophisticated systems.

One of the key concepts being developed in the DoD is known as the “software factory.” Software factories are a realization of the concepts of DevSecOps, and are focused on the remediation of the bespoke, artisanal nature of IT systems fielded by the DoD today. Software factories correlate with another development of the industrial revolution, the assembly line. A software factory is dependent on the adoption of DevSecOps principles, but the implementation of DevSecOps does not imply the creation of a software factory, just as the success of the assembly line is dependent on interchangeable parts. Through the adoption of DevSecOps principles, the DoD is moving quickly towards leaner, more responsive, more efficient armed forces.

Future posts in this series will continue to explore these issues in greater detail.