Patching for SCADA and ICS Security: The Good, the Bad and the Ugly

In my last blog, I discussed the reasons why critical industrial infrastructure control systems are so vulnerable to attacks from security researchers and hackers, and explained why patching for such systems is not a workable solution.

But let’s now examine the good, the bad and the ugly details of patching as a means to secure SCADA and ICS systems. And to begin, let’s suppose patches could be installed without shutting down the process (for example, through the staged patching of redundant controllers)...

“You may run the risks, my friend...” Image Credit: pictureshowpundits.com

The Impact of Patching for SCADA and ICS Security

In a landmark study of the patches for post-release bugs in OS software, Yin et al showed that between 14.8% and 24.4% of all fixes are incorrect and directly impact the end user. And if that’s not bad enough, 43% of these faulty ‘fixes’ resulted in crashes, hangs, data corruption or additional security problems.

What’s more, patches don’t always solve the security issues they were designed to address. According to Kevin Hemsley, a member of the Industrial Control Systems Cyber Emergency Response Team (ICS-CERT), in 2011, ICS-CERT saw a 60% failure rate in patches fixing the reported vulnerability in control system products.

Even Good Security Patches Can Cause Issues

Most patches require the shutdown and restart of the manufacturing process. Some can also break or remove functionality previously relied on by the control system. For example, one of the vulnerabilities the Stuxnet worm exploited was a hardcoded password in Siemens’ WinCC SQL database.

At the time, Siemens were widely criticized for not quickly releasing a patch to remove the password. However, customers who took it upon themselves to manually change the password soon discovered that many critical control functions depended on this password to access accounts. In this case, the ‘cure’ was even worse than the disease.

Patching Often Requires the Presence of Experts

Another ugly truth about patching is that the process itself often requires staff with special skills to be present.

For example, the vulnerability exploited by the Slammer worm in January 2003 actually did have a patch (MS02-039) that was released in 2002. Unfortunately, this didn’t help an oil company with numerous production platforms in the Gulf of Mexico. The company started rolling out the patch in the summer of 2002, but issues with server restarts required Windows experts to be present during patching. Since very few of these experts were safety certified for platform access, most platforms were still not patched when Slammer hit six months later.

When There Are No Patches

Of course, you can only use patches to fix vulnerabilities if the vendor has created a patch. Unfortunately, this is the exception rather than the rule. At the SCADA Security Scientific Symposium (S4) in January 2012, Sean McBride noted that less than half of the 364 public vulnerabilities recorded at ICS-CERT had patches available at that time.

Some accuse the vendors of indifference or laziness, but there are many factors that prevent the quick release of a patch.

In 2010, a major ICS vendor told me that internal testing of a mission critical product had revealed security issues. Unfortunately, these vulnerabilities were part of an embedded OS supplied by a 3rd party. Now the OS supplier refused to address the vulnerabilities, and so the ICS vendor (and its customers) faced a situation where patching was not possible.

In a 2011 case involving another ICS vendor, vulnerable backdoors were found in a PLC by an independent security researcher, who publically exposed them. The vendor designed a patch to remove backdoors, but then learned that these backdoors were widely used by troubleshooting teams for customer support. To complicate matters, the company’s quality assurance (QA) process for product changes required four months to complete. This meant that even if customers were willing to sacrifice support for security, they were faced with a four month window of exposure while they waited for the proper testing of patches to be completed.

When it comes to patching for SCADA and ICS system security, the cure may well be worse than the disease itself. Image Credit: www.time.com

Many SCADA/ICS Users Choose Not to Patch

My last example highlights a core problem with a patch-based strategy for control system security. Many customers simply don’t want to run the risk of degrading service and increasing downtime. The vendor noted in the previous example privately told me that they have a 10% patch download rate for released patches.

My own experience with an ICS security product confirms the reality of low patch acceptance in the field.

In September 2010, we released Tofino Industrial Security System version 1.6. This upgrade addressed a number of security and performance issues and was offered to all registered users at no charge - if downloaded within 30 days. Initial acceptance was low, so we repeated the offer for an additional 30 days. After two months, only 30% of users had downloaded the free upgrade. And that doesn’t necessarily mean they all installed it!

Planned Patching is Good. Reactive Patching is Bad. Rushed Patching is Ugly

Let’s be clear - patching bugs is an important process for any control system. And patching for vulnerabilities is critical for good security. But the IT strategy of reactive, continuous patching on a monthly or weekly basis just won’t work for SCADA and ICS systems. Patching in a hurry is just plain dangerous.

SCADA/ICS vendors face multiple issues when trying to create “quick” patches - they have to consider both safety and QA requirements that often delay patch releases. In other cases, a reasonable and safe patch just isn’t possible.

SCADA/ICS customers face similar concerns. And quite frankly, who can blame them for not wanting to increase downtime or expose their critical controller or server systems to safety risks?

Patch support for legacy products is also an issue – many expect a control product to operate for 20 years, putting it well outside the typical IT support window. Finally, as noted in the Slammer worm example, patches can require significant staff resources to install safely.

So create a patching plan that works for your process environment. Make sure that it includes processes for proper tests and change management controls.

Just don’t expect patches to be a quick fix for your control system’s security problems. If you do, you may discover new problems that are worse than the bugs the patches cure.

Do you have stories to support or contradict the opinions expressed here? Let me know your thoughts.

In my next blog, I’ll share some secrets on how to successfully use patching in SCADA and control systems.

Related Content to Download

Published Paper - "Patching for Control Systems - A Broken Model?"

 

Download this paper presented at the Texas A&M Annual Instrumentation Symposium and learn about:

  • The challenges of patching for control systems
  • Vendor data on patching deployment rates on ICS products and what can be achieved in the future
  • Compensating control-based solutions for security vulnerabilities

This document is vendor neutral and is ideal for serious consideration of the topic.

Related Links

 

RSS Feed Subscribe to the "Practical SCADA Security" news feed

Author Eric Byres

Comments

6

Spot on comments.

Folks really need to have a test environment to be able to try out patches before inflicting them on production systems.

Some may complain that a test environment is expensive. Maybe so, but how expensive is downtime if you put a debilitating patch on your production system?

Some may say that you cannot totally replicate the production system for a test system. Certainly. But you can evaluate the parts of your system that do cause the most problems with patches, or the most grief if they malfunction due to a patch.

I agree - some companies cheap out on the test environment, but then wonder why they have so much down time. It is poor economy IMHO.

And for those unlucky engineers that have to live with the lack of a good test system, there is still an out. Pick a few sacrificial systems and try out the patches on those. Or test out the patches on redundant systems (one at a time of course). I will blog more about that next week.

Is it right to rail on patching so much? Sure, the process is about as much fun as visiting the dentist or dealing with a recall notice on your car.

Organizations consider many factors in deciding if, when, and what systems will be maintained. Getting good at this process is part of being a control engineer these days.

In the early days of ICS, if the vendor said this patch is mandatory, you grunted your teeth and rolled it out. No brainer - you knew the process was at risk.

Fortunately most of what once constituted nasty functional bugs have been worked out of the software. Sadly, times have changed and we have a new class of nasty.

To make matters worse policies driven by regulation can be seemingly overreaching and counter-intuitive. Aircraft unable to fly because of a broken tray table but ok to fly with a missing firmware update?

Oops that one is too much to tackle in a blog. Chalk it up to the fruitcake world we live in.

I might not have been clear on what I am railing against. It is not patching per se. It is patching as a silver bullet. We need to patch, but we need to get smart about it. I promise that next week I will discuss what I see as the smart way to patch in ICS/SCADA, versus the purely reactive way.

Never mind the vendors own patches, what about patches for the underlying OS and other core 3rd party components (e.g. SQLServer)? How many vendors test their applications against patches released for these products in a timely manner (if at all)?

Some interesting thoughts on Patching. I would like to understand, what are the specific major challenegs facing individual control systems such as SCADA, PLC and DCS systems and what are the current measures being adopted to address the security issues in these systems?

Add new comment