A few weeks ago, another Cybersecurity attack hit and automation control systems integrator Huffman Engineering helped rescue one of their customers.

Blog post by Keith Mandachit, Senior Engineer

Ransomware Message“We’ve been hit and we are down!”

This is not what you want to hear from one of your industrial control system (ICS) clients.  This is definitely not what you want to see on the screen of your computer, especially the ones on your control systems.  It’s an automation engineer’s worst nightmare.  Or is it?

A few weeks ago, the latest cybersecurity ransomware attack, dubbed “NotPetya” or “SortaPetya,” affected many companies across the world including some in the U.S. and one of our customers, a global leader in their industry with many facilities across the world.  As their trusted system integrator partner for one of the facilities, Huffman Engineering was called upon to help them recover from this mess.  Upon arriving at the scene, we found that every Windows-based computer, virtual or local, was encrypted.  Domain controllers, human machine interface (HMI) servers and clients, historians and engineering workstations…all rendered useless.

Where to start?  Paying the ransom and allowing the hackers to decrypt the compromised systems was not an option.  The only choice was to rebuild each system either from scratch, or hopefully with a backup.  We were about to find out how good –  or not good –  the customer’s backup plan was.  Most of the HMI interfaces on the shop floor were easily recovered using a whole hard drive image.  Some of the virtual servers were recovered with a backup.  Not all of the backups were current; one was found to be two years old.  Before systems were placed back on the shop floor network (SFN), each system was updated with the latest Windows patches and antivirus updates.  Within two days, the Huffman Engineering team had the critical systems back up and running.  Within a week, even with the 4th of July holiday in the middle, the automation control systems were recovered and functioning.  Other facilities within the company had not yet come even close to recovering and resuming production.  One of the keys to the automation systems’ quick recovery was the strong relationship built between Huffman Engineering and our relatively new customer.   This partnership allowed Huffman Engineering to gain knowledge of the customer’s industrial control systems and use it in their time of need.  The Huffman Engineering team understood the many control system applications that were in place and how they were being used which made the learning curve a lot smaller.

Now what?  What was learned during this crisis?  Huffman Engineering recommends the following security precautions to help prevent not just cybersecurity attacks, but to mitigate a variety of potential negative security events to your mission-critical systems.

  1. Backups!  Ensure systems are backed up on a regular basis.  Image all hard drives, backup virtual machines, and store configurations and programs on a non-Windows based file server.  No matter how secure you think your defense is, prevention is never 100% so backups are key to a quick and painless recovery.
  2. Keep Windows-based systems up to date with the latest patches.  In the industrial control systems world, some applications don’t play nicely with every update, so it may not be possible to apply them all.  This particular attack was patched by Microsoft update MS17-010 which was released on March 14, 2017.  Disabling and/or removing the Server Message Block version 1 (SMBv1) networking protocol accomplishes the same action as the patch. If you have an app or hardware device that requires SMBv1, it’s probably time to ditch it.  Too busy or don’t know how?  Consider a service agreement with Huffman Engineering to help you keep your data backed up and secure.
  3. Consider installing antivirus software. If it’s already installed, make sure it’s up to date.  Only 16 out of 61 endpoint antivirus software packages were able to detect this particular ransomware according to VirusTotal, so consider a non-signature based antivirus package such as Cylance.
  4. Consider keeping your control system computers off the internet.  Sometimes that isn’t practical, especially with the emerging IOT/IIOT protocol.  If the control systems are connected to the internet, protect the SFN with a firewall.  Don’t allow the use of control system computers for personal ‘internet surfing’ or checking email.  Need remote access?  Use VPN for secure remote access.
  5. If possible, do not reuse passwords.  It’s very common for all machines on the network to have the same local administrator password.  How safe is that?  Once a shared password is compromised (especially if it has administrator rights) an attack has the ability to propagate broadly.  Although it may be inconvenient, consider using different local administrator passwords.
  6. Implement least-privilege administrative models.  In other words, don’t put every domain user in the domain administrator group.  Users should log on with a user account that has the absolute minimum permissions necessary to complete the current task and nothing more.
  7. Develop a relationship with a control systems integrator.  Another set of eyes that knows your automation systems inside and out is an asset.  While there are many integrators out there, finding a certified CSIA (Control System Integrators Association) integrator to partner with can be a great advantage.  The CSIA certification provides an extra level of assurance that your control systems integrator is operating a well-managed and stable business.  Integrators with CSIA certification have been audited for their processes and adhere to best practices that reduce risk and contribute to successful partnerships with their clients.

Cyberattacks on industrial control systems are a nightmare, no question.  Imagine going through the recovery process on your own; it’s probably your worst nightmare.  In this particular case, the expertise of CSIA-certified Huffman Engineering and the trusted partnership with their customer led to a quick and successful recovery of their mission-critical industrial control systems…and a better night’s sleep for all.