DevOps helps government IT reinvent itself
The long and winding road of government IT history is paved with huge, failed projects. Among them: the original Affordable Care Act healthcare exchange website, which crashed and burned on its first day; the U.S. Citizenship and Immigration Service’s Transformation project, now $1 billion and four years behind; the Air Force’s Expeditionary Combat Support System, which wasted $1 billion; and California’s IT modernization projects for the Department of Motor Vehicles and for Medicaid, both junked after the local governments spent tens of millions of dollars on them.
But now the same DevOps methodologies that are proving useful in business are being adopted in government IT projects as well. The use is primarily at the federal level, but also for state and local government.
[ Keep up with infrastructure and IT ops with enterprise.nxt’s newsletter. Subscribe now ]
It’s because of the size of government IT projects that DevOps holds such promise. “Just 13 percent of respondents in a recent MeriTalk/Accenture survey of 152 U.S. federal IT managers believed they could ‘develop and deploy new systems as fast as the mission requires,’” writes analyst Michael Coté. “The impact of improving on that could be huge. For example, the U.S. federal government, by conservative estimates, spends $84 billion a year on IT. And yet, the Standish Group believes that 94 percent of government IT projects fail. These are huge numbers that, with even small improvements, can have massive impact.”
A crumbling federal government IT infrastructure
The U.S. federal government is turning to DevOps because, as with roads and bridges, its aged IT infrastructure is crumbling. Applications on which millions of people depend are written in languages like COBOL that run on old hardware such as ancient mainframes. It’s increasingly difficult to find staffers who can maintain the hardware and software as the “silver tsunami” of government retirees hits. Finally, as the infrastructure ages, it becomes more prone to failure.
Fixing those issues costs money. Some studies have found that as much as 85 percent of the government’s IT budget goes toward maintaining these legacy systems. That doesn’t leave a lot of money for developing new applications or for innovation in general, much less initiatives like smart cities.
Generally, efforts to start over and write new versions of the applications using hardware and software from this century have not worked out well. It’s simply too complex to write out the specifications for an entire new system that’s based on 30 or 40 years’ worth of jury-rigging the original system. Agencies that have tried to do so, on the whole, have crashed and burned spectacularly.
Where DevOps works
A number of federal agencies have been working with DevOps over the past couple of years and are starting to report successes. For example, NASA’s Jet Propulsion Laboratory (JPL) is migrating its old systems to new technology. The IT team started with an aging mission-critical system. It would have taken 130 people to physically visit JPL to manually input their passwords, and the team fixed it with a Docker container, according to Greg Otto in FedScoop.
Tom Soderstrom, chief technology officer of JPL, recalled, “One of our guys said, ‘I can do that. I’ll take the application, wrap it in Docker, and move it to GovCloud, and I’ll run it from there,’” Otto writes. “By the time the off-site people said, ‘It won’t work,’ I said, ‘We already did it; it works.’”
Using DevOps methodologies and tools also helped the Federal National Mortgage Association (commonly known as Fannie Mae), a U.S. government-sponsored enterprise that is a leading source of financing for mortgage lenders. After initiating DevOps in 2015, including a set of preapproved development tools, overall productivity increased by an average of 30 percent to 40 percent while costs dropped by 30 percent. Moreover, Fannie Mae’s overall quality grew by 32 percent, with some projects realizing a quality score as high as 70 percent, writes John Edwards in InformationWeek.
Why does quality improve? It’s because important components, such as security and testing, are involved right from the beginning, says Tom Suder, CEO of the Advanced Technology Academic Research Center. Development is also improved with the use of microservices, the process of dividing development into small, self-sustained pieces, Suder says, “instead of having a monolithic system with so many dependencies you can’t do anything with it.”
That means IT and development can keep a closer eye on feature creep. You don’t get to the end of the contract and then discover there's no money left but there’s plenty of flaws.
In addition, DevOps enables government agencies to more easily divide work among multiple vendors, Suder says. “They have ongoing competitions all the time and use whoever is doing the best work,” he says. “You have one vendor; if they get behind, you don’t give them any more work, and you allocate more work to the contractor who’s doing a better job or with more resources. That limits your risk.”
Think globally, DevOps locally
While much of the emphasis on government DevOps is at the federal level, it’s happening at the state and local level as well. For example, the state of Colorado is working to implement DevOps. Its IT staff has learned that the biggest component is the cultural change, according to Milo Knezevic, officially the state’s program manager but unofficially its DevOps guru. DevOps helps Colorado IT staffers create processes and tools to enable sharing and collaboration, increase efficiencies, and automate resource-intensive manual steps, he says.
Colorado isn’t alone. "Challenges and Trends in State and Local IT Operations: United States," a recent Ponemon Institute study that polled 220 decision-makers in state and local government IT, found that 38 percent of respondents intended to increase DevOps spending in the next year. In addition, 34 percent said they had already adopted DevOps, while 43 percent planned to adopt DevOps at some unspecified future time. Moreover, 43 percent feel that DevOps has made their IT functions simpler.
These governments aren’t just using DevOps for minor applications. Among Ponemon survey respondents, 77 percent plan to use DevOps on mission-critical systems, and 71 percent expect DevOps use on back-office operations such as human resources and enterprise resource planning.
A process shift
As in any industry, implementing DevOps methodologies in the government requires a new mindset. Thinking of a government IT project in agile terms (continuous iterations of small improvements) is different from thinking of it as a massive start-from-scratch implementation. Both agile and DevOps adopt a process that depends on incremental improvement, automation, and continuous feedback.
The traditional development process uses a waterfall methodology. “The big RFP [request for proposal] goes out, with the needs analysis up front. You develop the project for months and months—up to a year. You say, ‘Here’s what we have,’ and then the government people say, ‘This isn’t really what I wanted,’” Suder says. “A lot of times, government projects fail, and they fail because of a lack of communication.”
One barrier to implementing DevOps methodologies is the government’s procurement system, which traditionally is set up for awarding large amounts for a single project as opposed to the continuous development process called for in DevOps. “The traditional government approach to any system acquisition is the waterfall model,” writes Hasan Yasar, technical manager of Carnegie Mellon University’s Software Engineering Institute, in its DevOps blog. “Unfortunately, this model often starts with the development of rigid requirement specifications and sets the project on a path for failure on schedule, technical objectives, and overall project cost.”
“We may like this screen better—how do you put that in an RFP?” Suder asks. “You have to allocate buckets of hours, how far you’re going to go, and what you’re going to accomplish.”
In fact, according to the way some government procurement specifications are written, IT departments couldn’t test the software until it was completely developed, says Mark Schwartz, CIO of the U.S. Citizenship and Immigration Services, which is migrating to DevOps. Even making some changes to a web page—something he could do himself in a few minutes—would take a year, Schwartz was told.
Ultimately, what makes DevOps so powerful—in government as well as any other organization—is the cultural change it brings about, said Jennifer Hoover, DevOps program manager for the Transportation Security Administration, at a recent MeriTalk event. “DevOps is really a misnomer,” she said. “DevOps really should include all the business lines as well. Inherently, DevOps is about the people. It’s about the culture. Somewhere in the middle of that is DevOps. IT requires education and communication throughout all agencies to define that.”
Lessons for leaders
- DevOps isn’t just a technical issue; it’s a cultural one as well. Look at your change management procedures to ensure that people have time and space to make the change in thinking that DevOps requires.
- Similarly, make sure you look at back-office processes. Adopting a new process isn’t going to help if there’s no mechanism to pay for it.
- Take advantage of the microservices component of DevOps by making sure that key aspects such as testing and security are included from the beginning.
This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.