SCADA Security: Welcome to the Patching Treadmill

As regular readers of this blog know, after Stuxnet, security researchers and hackers on the prowl for new targets to exploit shifted their efforts to critical industrial infrastructure.

Unfortunately, the Supervisory Control and Data Acquisition (SCADA) and Industrial Control Systems (ICS) applications they are now focusing on are sitting ducks.

Up until recently SCADA and ICS systems have been designed with reliability and safety in mind; security has been a minor consideration. Products that have never faced security tests are now under attack from sophisticated vulnerability discovery tools, and major control system security flaws are being continuously exposed.

SCADA/ICS applications are easy targets for security researchers and hackers.
Image Credit: Active Rain Caption

In recent years, we have seen a staggering growth in government security alerts for these systems, and have witnessed some of the most sophisticated cyber-attacks on record.

The US government’s ICS-Computer Emergency Response Team (ICS-CERT) tracks and publishes Security Advisories for known security vulnerabilities found in industrial products. In the entire decade prior to the discovery of Stuxnet (July 2010), ICS-CERT published 5 security advisories involving 3 vendors.

Compare this to 2011, when there were:

  • 215 publicly disclosed vulnerabilities
  • 104 security advisories
  • 39 vendors involved in those security advisories

By late 2012, the total publically disclosed vulnerabilities topped 569.

And remember that these vulnerabilities are typically disclosed to the world prior to ICS vendors having patches available.

Furthermore, 40% of disclosed vulnerabilities include working attack code. This means that individuals can download exploit tools and run them against a target with little understanding of control systems or the consequences of their actions. And download and attack they do - ICS-CERT reported over 20,000 reports of unauthorized internet access to control systems in the last half of 2012.

Welcome to the Industrial Security Patching Treadmill!

Since security researchers, hackers, and issues have essentially migrated from the IT environment, it’s not surprising that we look to that world for a solution. There, security vulnerabilities are addressed by applying software patches - a constant cycle involving multiple patches over the life of a product. In fact, the typical IT computer needs patching (with a full reboot) at least once per week.

You don’t need to be a SCADA/ICS expert to realize that shutdowns of that frequency just aren’t feasible for critical infrastructure control systems.

On and on and on it goes... Image Credit: Learning to Fly

So How Many Patches Does a Control System Need?

One might argue that a control system requires fewer patches than an IT system…or that the software footprint is smaller, the code quality better… If so, then maybe the patching cycle could be synchronized with annual maintenance shutdowns. In this scenario, patching could be a workable solution to address software vulnerabilities.

To determine if this was an option, in 2008 I participated in the analysis of a U.S. refinery process control network (PCN). There were 85 computers on the refinery PCN, and a similar number of industrial controllers. Although we could only gather reliable data for 78 of the computers, we determined that they were running 272 distinct processes/applications.

A search of the National Vulnerability Database (NVD) found that 48 of these processes had one or more serious security vulnerabilities. Across the refinery PCN, there were 5,455 publically known vulnerabilities, an average of 70 per machine. An aggressive Windows OS patch program reduced this number by almost 50%, but that still left 2,284 published vulnerabilities remaining. Why? Because the applications involved did not have a means of automated patching.

Latent PLC Vulnerabilities Are Not All Disclosed at Once

And let’s not forget the latent vulnerabilities lurking in the control system. Academic research tells us that most commercial software contains 3 - 10 defects for every thousand lines of code (KLOC), and that 1% to 5% of these result in vulnerabilities. That works out to between 0.03 and 0.5 vulnerabilities per KLOC.

So what does that mean in real life?

Take Windows XP, for example…it contains about 40 million lines of code (40,000 KLOC). As of October 2012, about 1106 moderate or severe vulnerabilities have been listed in the NVD for Windows XP - that’s a Vulnerability/KLOC ratio of about 0.0276.

Whatever your experience using Windows XP, seems it’s on the low end of vulnerabilities and pretty good from a security point of view!

Looking back at the history of Windows XP vulnerabilities, we have to assume that SCADA/ICS vulnerabilities won’t all be disclosed at one time. We’ll likely see a relatively small number of disclosures in the first few years, as researchers begin to investigate the products in the industrial space. Then, after SCADA/ICS products have been exposed to widespread security scrutiny, a virtual avalanche of vulnerabilities may occur, resulting in the need to install control system patches on a weekly basis.

 Prepare for an avalanche of vulnerabilities once SCADA/ICS products have been exposed to security scrutiny. Image Credit: The Alaska Avalanche Information Center

We Can’t Ignore SCADA/ICS Firmware

It’s also obvious that the firmware in PLC and DCS controllers will also have vulnerabilities and will require patching. Controllers typically contain between 1,000 KLOC and 5,000 KLOC of firmware. Based on the analysis used above, this means that each is likely to contain between 30 and 150 vulnerabilities. If the vulnerability disclosure curves are similar to those we’ve seen in the IT sector, we can expect a low number of patches in the immediate future, followed by an epidemic in a few years.

The above analysis clearly indicates that the frequency of patching needed to address future SCADA/ICS vulnerabilities in both controllers and computers is likely to exceed the tolerance of most SCADA/ICS operators for system shutdowns.

Tune in to next week’s blog and learn about the impact of patches, what happens when there are no patches, and why many SCADA/ICS customers simply don’t want to patch...

Do you think that patching is a workable solution for securing SCADA/ICS control systems? Do you have any patching success or horror stories to share? Let me know your thoughts.

Related Content to Download

Presentation - "Patching for Control Systems - A Broken Model?"

 

Download this presentation and learn about:

  • The challenges of patching for control systems
  • Vendor data on patching deployment rates on ICS products and what can be achieved in the    future
  • Compensating control-based solutions for security vulnerabilities

This document is vendor neutral and is ideal for serious consideration of the topic.

Related Links

RSS Feed Subscribe to the "Practical SCADA Security" news feed

Author Eric Byres

Comments

8

While not an answer to everything (e.g. firmware patches), the use of application whitelisting on PC based SCADA, and PLC engineering tools can extend the periodicity between patches, allowing them to be scheduled with plant outages.

Absolutely - application whitelisting is a very promising compensating control for vulnerabilities on the server or desktop. I will discuss a few others over the next few weeks, including "network whitelisting" for non-PC control assets like PLCs.

Regards
Eric

Hello,

I would like to ask from where comes statistic on vulnerabilities in this article?
--------
By late 2012, the total publically disclosed vulnerabilities topped 569.
--------

Regards,

Maryna

The statistic comes from Sean McBride of Critical Intelligence Inc. Sean presented an excellent paper last year at S4 2012 called "Documenting the “Lost Decade:” An Analysis of Publicly-Disclosed ICS-Specific Vulnerabilities since 2001”, and shared his 2013 update with us in February.

Regards
Eric

Thank you for your answer. However I am greatly confused now (I hope I am getting something wrong!). In the "SCADA safety in numbers" report form Positive Technologies, the numbers are TOTALLY different and are much smaller (like less than 200 vulnerabilities in total by the end of 2012). However, their report was admitted as being accurate by the community (I am speaking about comments in mailing list scadasec@news.infracritical.com). I quickly looked up the differences and realized, that e.g. Honeywell vulnerabilities are not included in the report from Positive Technologies. Seems like finding a reliable source of information is not a trivial task (what is not surprising). But I am surprised that their report differs so much from McBride's.

Just to be clear, Sean McBride informed us that there are 569 SCADA/ICS published vulnerabilities as a cumulative total since 2001. Only 248 new vulnerabilities were actually announced in 2012. So that might be part of the difference.

And for yet another number, see page 12 http://ics-cert.us-cert.gov/pdf/Year_in_Review_FY2012_Final.pdf . In this case I think the number is lower than Sean's because one ICS-CERT ticket can include several vulnerabilities.

Thank you very much. I have my clarity now.

Another comment on patching. Some industries, like pharmaceutical cannot patch their systems without re-certifying the entire plant, what is a very lengthy and costly procedure (up till 1 year and millions of whatever currency). So, what is their solution? In this case I support the general attitude of Tofino, that Digital Bond is inadequate in their demands and expectations.

Add new comment