The state of patch management
You may have missed it because the events happened in extreme slow motion, but the software industry got something right: Software updates are no longer the disruptive horror show they once were.
A long time ago, even as they were recovering from Y2K, IT departments could be struck at any moment with the report of a drastic vulnerability in some key software, from Microsoft Office or Windows to Adobe Acrobat. As often as not, the news hit as the vendor released an update that represented a finger in the vulnerability dike, all knowing that another hole would open up soon. The implication was that IT should drop everything and apply the updates ASAP. For people with other work to do, the situation became impossible.
But today, patch management is not a major burden. This is partly due to maturity in the products we use and in the IT community, but it’s also because software companies have for the most part taken control of the situation from IT.
The best example of this is mobile devices. Every now and then, not even on a regular schedule, Apple and your mobile carrier (typically the update source for Android phones) release updates and new versions. You aren’t given many choices to make. IT may hold off on releasing the update to users through its mobile device management system, but release it you will. Arguably, the fact that customers are willing to accept such an arrangement with their mobile devices made it acceptable for desktop and server platforms to adopt it, too.
Where patch management is headed
I asked Troy Hunt, a prominent writer, trainer, Microsoft regional director, and MVP, for perspective. He agrees that this is the way things are going.
“In a perfect world, updates will occur frequently, automatically, and without disruption," says Hunt. "Barring a handful of super-geeks, this is pretty unanimously well-received proposition because 99.x percent of people simply don’t want to think about this stuff; they just want it to work. Your [mobile OS] example is perfect: Every now and then, it pops up a message, which you accept, and the update happens without further human interaction and with a very high success rate."
Hunt adds, "Windows desktop environments are always going to be different due to the fundamentally different architecture of the platform and wide range of hardware it runs on. And as much as I see my (very techie) audience complain about Windows updates, it’s only ever when the process isn’t seamless," such as with an unexpected reboot or driver conflict.
It's not surprising that consumers are happy not having to think about updates. But even enterprises accept the loss of control, and they probably will remain happy unless significant problems develop in update quality.
On the release of an update, the best practice for IT is to test it before deploying it. The only practical way to do a good job of this in a large organization is to call on volunteers all over the enterprise (even if you have to order them to volunteer) to accept the updates on release and report quickly on any problems. If you spread the volunteers out enough, you can get a representative sample of real-world configurations. (The semi-official term for this is “flighting,” but it may be more accurate and honest to call the volunteers "guinea pigs.") Much more common is just to wait a few days to see if early adopters in the world at large are reporting problems.
Service model taking over
You may have heard that the software industry is moving to a service model, and this phenomenon of mandatory updates is part of it. Microsoft already refers to Windows 10 as Windows as a service. One of the central features of the service model is that there is only one current version. If the software is locally installed, as with Windows or Office, you may be able to turn off updates altogether, but you can’t pick and choose among updates. Turning off updates from a service generally makes no sense. And if the services are web-based, such as Salesforce or G Suite, there are no updates for you to apply; the changes happen, and you may or may not notice them.
And yet, even though control of updates is disappearing, testing opportunities are improving. Some of your bold volunteer users should be signed up for the Windows Insider Program, which delivers pre-release builds of Windows. These give you more than a couple of days to test your own systems and configurations on the coming, inevitable Windows update.
If you are not already living this experience, you soon will be. The days of Microsoft supporting a product for 10 years are over. Windows 7 security updates will end on (Patch) Tuesday, Jan. 14, 2020. Even the regular Windows 10 builds have only a limited service period, as you can tell by the fact that since the release of Windows 10 (Version 1507) in July 2015, there were four updated versions in just over two years (to version 1709).
There is an exception worth mentioning: Windows 10 Enterprise LTSC (Long-Term Servicing Channel), which is designed for deployments on special-purpose devices with a need for stability and tight security controls, such as an MRI controller. This is a special, stripped-down edition of Windows 10 Enterprise, lacking the Edge browser and end-user candy such as the Music app. The normal semi-annual feature releases for other Windows 10 builds don’t happen with the LTSC. Instead, there is a release every two to three years; organizations can choose to apply these as in-place upgrades or even skip them for a whole 10-year lifecycle.
In another example of the service model taking over, the upcoming Microsoft Office 2019 will be delivered only as Click-to-Run executables, not as .MSI installers. Click-to-Run is based on Microsoft’s App-V technology for streaming the app from a server and running it in a sandbox, but it also keeps the version of the program the user is running current. Once again, IT’s options for controlling such updates will be, at best, limited.
Linux has its own issues
Steven J. Vaughan-Nichols, well-known Linux author and former Unix/Linux sysadmin, says that the situation with Linux administration is different. “Patch management is alive and well on Linux servers," he says. "Much of it is still done <groan> manually. Typically, you keep an eye on your Linux distribution's and none-distro open-source software security mailing lists. As the new security notifications come in, sysadmins review them and then patch as necessary.”
It's not totally manual, he says: “There are Linux patch management tools. They tend to be distro specific. For example, Spacewalk/Satellite for Red Hat Enterprise Linux (RHEL) and its close relative CentOS; SUSE Manager for SUSE; and Landscape for Canonical's Ubuntu.… Forward-looking Linux sysadmins are also quickly embracing DevOps tools such as Ansible, Chef, Puppet, and Salt.”
The sort of central vendor control we see from Microsoft, Apple, and Google is more difficult on Linux because a distribution is composed of numerous packages from different authors. It’s possible, but part of what Linux admins like about the OS is the control they get over it.
The real problem today
Microsoft regional director Hunt is also proprietor of Have I Been Pwned, a service for learning whether your logon credentials have been included in a data breach, which raises a worthy point: Exploits of the vulnerabilities addressed by security updates aren’t the only security problems. Indeed, in recent years, attacks based on social engineering and data theft have been of much greater importance than attacks directly through vulnerabilities. Just ask John Podesta.
This isn’t a reason to downplay the importance of security updates; quite the contrary. Vulnerabilities have been less effective on attack because security updates have been more effective on defense, and mostly because prompt delivery and installation of updates is more the norm than in the past.
Important attacks based on vulnerabilities still show up now and then, particularly when users have not applied updates, as was the case with the WannaCry ransomware attack that began on May 12, 2017. Governments, large corporations, and ordinary people all over the world were affected by it. And yet, Microsoft had released an update in March 2017 that closed the vulnerability on which WannaCry relied. Almost all affected systems were running Windows 7, two years after the release of Windows 10. Meaning that the attacks would have been minimized had these organizations had a policy in place to regularly and rapidly apply updates that closed vulnerabilities.
It’s reasonable to think that many of the organizations hit by WannaCry changed policy afterward to apply updates and OS upgrades more promptly, though experience shows us that this is often not the case. I hope we’ve all learned this lesson by now, especially if it was not at our own expense.
The state of patch management: Lessons for leaders
- Patches are not an option; they are a requirement for secure computing.
- Taking end users out of the patch management process will result in more secure environments.
- These aren't OS-specific issues; everyone is vulnerable.
This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.