It’s been almost four months since electric suppliers serving large parts of western Ukraine – called “oblenergos” — were hacked December 23. While such an occurrence in that part of the world might be discounted due to tensions between Russia and Ukraine, cyber security managers for utilities in the U.S. and elsewhere now recognize that this was the first confirmed attack on a power grid anywhere in the world.
About 225,000 customers reportedly were without power for up to six hours. The hack is important to understand mainly because the hackers made it very difficult for utilities to recover by “burning bridges” and forcing networks into “manual mode.” Those handicaps complicated administrators’ efforts to rescue their systems using backup servers.
What Happened in the Ukraine Could Happen in Anywhere in the World
“It could be translated into a potential outage here,” said Mark Engels, Dominion Virginia Power’s Director of Enterprise Technology, Security and Compliance, told a rapt Virginia Chamber of Commerce energy conference earlier this month in Richmond.
“One thing is for certain – by understanding what happened there we can help prevent a similar occurrence here at home,” wrote Ted Gutierrez, the Product Manager at the SANS Institute who was among the technical experts who analyzed and reported on the Ukraine attack.
The SANS Institute is a private company in Fredericksburg, Virginia that trains security professionals to guard against and respond to cyberattacks.
Numerous Industry and Government Groups Are Sharing Lessons They Think They’re Learning
Institute employees joined with other members of the Electricity Information Sharing and Analysis Center in assessing lessons from the Ukraine incident. Together they issued a report consolidating the open source information to correct media reports, “clarify important details” surrounding the attack and recommend ways to guard against similar attacks. It is one of several on-going collaborations to grasp and share lessons still being learned.
Despite the every-growing assessments and organizations involved utilities are still very much exposed to similar and even more potent attacks.
“We’re definitely better than we were five years ago. But we’re not even close to being where we should be,” said Robert M. Lee, Founder and CEO of San Antonio-based Dragos Security, who has participated in many of the assessments both via written reports, private industry meetings and government-led seminars that media have been invited to cover.
“The most significant problem is we don’t fully understand the scope of the threat we face,” Lee added. “It’s not nearly as difficult to cause real impact as many people believe.”
Where utilities can strengthen their defenses most effectively and efficiently, Lee asserts, is building in-house competencies, monitoring their virtual and physical networks 24/7 and identify all the devices, equipment and other assets that could be compromised.
Lee cited a Texas natural gas pipeline company which he would not identify for how it is discovering devices and assets “they didn’t know they had.” He urged every energy utility to be take a similar, comprehensive, inventory.
Five Themes from one of the Industry Assessments
The overriding conclusions in a report Lee helped write are captured by five “themes”:
Theme 1: The operation relied upon intrusions that appear to have come from a broader access campaign conducted in the spring of 2015 aimed at the obleenergo’s Industrial Control Cystems, or ICSs.
Theme 2: There were actually separate attacks on three oblenergos which were timed for a desired impact matched with a “thoroughness of the adversary sequence of events in achieving their goals.”
Theme 3: Early media reports reflected a misunderstanding of linkages to “Blackberry 3”, “BlackEnergy” and “KillDisk” malware. KillDisk erases selected files on targeted systems and corrupts the master boot record, rendering systems inoperable. That said, the report cautions against focusing excessively on specific malware because it “places defenders into a mindset in which they are simply waiting for guidance on the specific attack components so they can eliminate them.”
Theme 4: The concept for the attack designed it to perform across multiple Supervisory Control And Data Acquisition (SCADA) and Distribution Management Systems (DMS) and then target common susceptible elements. Examples of those elements include storage overwrites for workstations and servers using Windows operating systems.
“The attackers likely developed a significant amount of unobservable adversarial testing prior to introduce the attack into the environment,” the report states. A separate report by the Industrial Control Systems Cyber Emergency Response Team also found the hackers “acquired legitimate credentials prior to the cyber-attack to facilitate remote access.”
Theme 5: The sharing of lessons learned is important to identifying and responding to future attacks. Among other industry groups and government agencies assessing the Ukraine attack include:
- Department of Homeland Security
- North American Electric Reliability Council and its Cybersecurity Risk Information Sharing Program
- Electric Sub-sector Coordinating Council
- National Cybersecurity and Communications Integration Center
- S. Computer Emergency Readiness Team
The timing for the report, as well as, these and industry-confidential lessons learned from the attack comes on the eve of U.S. utilities facing a July 1 deadline for complying with the North American Electric Reliability Council’s (NERC’s) latest “Critical Infrastructure Protection” standards (CIPs) update.
In a modeling of how much damage a cyberattack on electric power systems in the Northeast U.S. alone might cause, Lloyds of London estimated the economic impact to be at least $243 billion to $1 trillion. The blackout in that exercise estimated power restoration taking up to several weeks in some locations.
The unique nature of utility operations and regional power grids operating across many states makes the U.S. especially vulnerable. Here are five different facets of that exposure, drawn from a recent white paper by Cisco and Intel, “Utility Security: Exceeding Mandates to Mitigate Risk”:
- Very few industries control such a widely distributed infrastructure that connects so directly to consumers.
- With the advent of smarter grids, connected homes and the “Internet of Things” (IoT) technology, utilities now manage networks far larger than their IT departments ever had to. Efficiently managing, analyzing and protecting the large volume of data that these vast networks generate is still proving to be a considerable leap for many utilities.
- The vast scope of IoT technology often exceeds the skill-set of the majority of utility workers and they don’t necessarily grasp the myriad security implications of everything they do.
- The interconnected nature of high-voltage power transmission networks presents cybersecurity risks all its own.
- Utilities increasingly are reliant upon third parties to maintain the operational health of their equipment. This typically requires internet-based access to that equipment and thereby introduces a commensurately larger exposure than most other industries.
July 1 is Deadline for U.S. Utilities to Comply with New ‘Critical Infrastructure Protection’ Standards
Hardly a month goes by without new expertise being shared. And that’s only what’s becoming available publicly and through the media. How the mushrooming volume of information is being parsed is becoming a huge undertaking with many authorities claiming unique expertise.
Last Thursday, the Edison Electric Institute weighed in with a seminar adding its own pages to the industry defense strategy playbook. It relived challenges identified during an industry “war games” planning exercise in November 2016 which simulated cyber and physical attacks. Among responses not touched on above included:
- Utilities are weighing rules for blacking out parts of the grid deliberately to facilitate recovery.
- Stockpiling replacement high-voltage transformers which are in short supply and take several months and possibly more than a year to acquire.