Legacy Integrations : Hacking a Defunct SecondWind SCADA
TLDR: Automating GUI interactions for a defunct SCADA HMI allowed us to make the wind farm dynamically dispatch-able. This allowed us to make money on an older site by renegotiating the PPA without upgrading investing in a new SCADA system.
Where it began…
There was a company that sold a SCADA (Supervisory Control and Data Acquisition) application called SecondWind specifically for the control and operation of wind turbines at a wind farm. By 2010 the system was severely outdated but owners of the SCADA application where locked into it because of its proprietary communication to the control systems on the wind turbine. A snippet from an outdated press release can be seen below:
“Our ADMS Wind SCADA and Wind Turbine Monitoring Systems are excellent technologies with a significant purpose in today’s growing renewable energy market,” said Second Wind CEO Larry Letteney. (see article here)
They where bought out by another company, CG-automation, that stopped supporting the application in any meaningful way. The application ran on Windows 2003 and polled a series of wind turbine controllers sequentially to draw the state of the turbine on a GUI (Graphical User Interface) that had text-boxes, and buttons to send back control signals in order to manage individual turbine states and the sites over-all power output.
How It Worked
When a wind farm is bought or built there is usually a contract to secure a certain amount of power from that wind farm over the course of some time. It is called a Power Purchase Agreement (PPA) . The PPA’s are common but there are merchant style wind farms who are not setup with a contract (I wont be discussing those much here). The PPA for this wind farm included the method to dispatch the site through a phone call. (Yes, a phone call). The operators would receive a phone call from the Independent Service Operator (ISO), a regulatory body that manages the approved output of energy produced on the grid, and then they would remote into the HMI (Human Machine Interface) and enter the set-point requested by the ISO. The ability to follow the set-point within a certain amount of time gave some leeway to this outdated process. For the most part it worked.
Challenges
The site came up for PPA renewal. The ISO now had an API (application programmable interface) and was providing incentives to operators who would set up new PPA’s without the need for phone calls. So the question was asked to the SCADA controls team : Can we dispatch this SecondWind site dynamically through the use of an API, without spending any money upgrading the site or its out-dated SCADA application?
The initial reaction of my manager was, “No! We wont support that.” . But, then he spoke to his new employee who came over from IT and asked is there something we can do? I immediately said, “Yes”. All I needed was an agent that could automate the manual entry of an integer into the HMI’s set-point input-box. I would then simulate the proper button clicks to approve the change. The agent would receive the set-points through an API client that would route the signal from an internet facing business network down into the control network at the site. (Easy, peasey) Here where some things to consider :
- Older Version of Microsoft Windows
- Multiple versions of the SecondWind HMI
- Locating windows GUI form using Spy++
- Running along-side a Human Dispatcher (Raise Conditions)
- End-to-End testing between a Business Network and Control Network
- Operating (ICS) Industrial Control Systems with a hacky solution : Safety , Safety, Safety!
Creating an API Client
The ISO provided an API endpoint. I was hoping it was REST but it turned out to be a complicated SOAP endpoint. After hours of XML I built a contract in .NET to to handle the calls and set the appropriate polling rate to be every 15 minutes. (This is typical for most power markets). The response contained the timestamp and set-point. A handshake process was required to validate the set-point was received. I would reply with the set-point feedback with another prebuilt soap POST. The communication of this API client would require a secure certificate that was issued by the ISO. Every API submission had to be first validated by sending the security information as well. The way these API’s are set up can vary. Luckily we had example clients from the ISO and it was pretty easy to test this part of the project.
The API client was hosted in our datacenter at the time. I added a few other features outside of just being a client :
- Only one of its kind can be running on the machine at any given time , this prevented double set-points to be sent to the site.
- Rolling log files to validate any set-point sent and to track if there was any errors.
- Email notifications on any errors.
- Watchdog service to monitor the API client if it every had issues as a precaution, also it would send email notification if the API did not reply every 15 minutes with a set-point.
- Distribution of redundant services across multiple servers with a RabbitMQ topic to validate that at least one service is running.
Routing Into A Control Network
Most Wind Farm operators will have a tiered network architecture. A business layer, control layer and device layer. They are seperated by firewalls and other security measures. Getting my signal from a datacenter server on the business layer all the way to the control network was not too bad.
Most sites already have these systems in place to send signals to a Park PLC (Programmable Logic Controller). The Park PLC (in this case a Modicon M1) can do several tasks but one of the basic tasks is to send signals directly to the wind turbines to reset faults or apply some type of site control on-top of what the SCADA is meant to do. These PLC’s use standard protocols to receive values (Modbus, OPC, DNP3). The ISO’s set-point was sent to the PLC using Modbus over ethernet. The API client server was whitelisted by the network team to send a signal down into the Park PLC.
By sending the signal through the Modbus protocol I could reduce the attack vectors for a cyber criminal since its relatively difficult to “root” a PLC with Modbus traffic. Modbus is an open protocol and relatively easy to build apps against. (check out a great dotnet library for it).
The set-point would be embedded in a Modbus protocol tcp packet using 2 Modbus ‘holding’ registers, 16-bits each, to send a 32-bit set-point value. Other Modbus settings need to be considered such as bit-order and endianness. (more on Modbus)
The PLC is then programmed using Schneider Electric’s Unity Pro IDE. Something that I didn’t realize at first is that all of these PLC brands will have specific programmers. They usually are designed with specific chipsets that require a licensed utility to re-program them. (great for safety and vendor-lock-in)
The Unity program allows the user to connect to a PLC on the network and load new programming logic to the PLC. Once loaded the PLC will reboot and run its program managing all the IO and logic. For this project all I had to do was use the PLC to route my signal back to the SecondWind host computer on the control network. The PLC has two network cards (incoming from business layer) and a control layer network card. The Modbus/TCP register is read on every scan of the PLC logic (in milliseconds) and is then moved over to another Modbus/TCP connection on a control network IP address. Similar communication can be done for Modbus (serial) through a function block. Or create a custom TCP pack and send it over ethernet using functions blocks.
Integrating with SecondWind HMI
As mentioned before the set-point is captured with an API and routed into a PLC then relayed over Modbus to the host computer running the SecondWind HMI. At this point the agent I built running on the HMI is a hosting a Modbus server. The Modbus server is part of the agent that is accepting Modbus data on a pre-defined register. It captures the values on multiple holding registers and converts it into an int32 value. (Remember the feedback signal?) The agent also returns values to the PLC as a feedback set-point value.
The agent is built to take the set-point and then log the value it is reading as it changes. On a change of the set-point on the Modbus register it runs a routine to insert the value into the HMI’s SecondWind GUI. The goal here is to run through a series of commands that will emulate a user on the computer that clicks through the form and inserts the set-point. (This is also known as Robotic Process Automation, RPA .)
Lets identify the clicks …
The SecondWind application could only run on Windows 2003, this presented some challenges. I had to upgrade the local .NET 1.1 framework to build an agent that could support the libraries I wanted to use. After much research around compatibility issues I built the courage to continue with it. I then used a Windows Interop and Windows Handler libraries to be able to grab access to the hWnd (handler) of the SecondWind form control.
- Detect the process that is running “SecondWind” . This allowed me to make sure the app is running and identify the Process ID (PID).
- Using the PID grab on to the windows form handle (hWnd).
- Detect the name of the windows form , this gave me the title of the application. I used this to determine the version of the form I was dealing with. ie. “SWI Supervisory Computer 2.1.0” , or “SWI Supervisory Computer 2.1.2”
- This is when I needed to use an old Microsoft utility called “Spy++”. It allowed me to detect the forms and its sub controls . By walking through the windows handler (hWnd) , I could pick out which object in the array to create another handle for and then finally access the property of the control object like an input-box.
The code I wrote is still proprietary so unfortunately I cant give away to much of the details but the idea is pretty simple. With the access to the form properties it is relatively easy to create a sequence of writing to to the input-box and clicking on the form buttons using the Windows Messaging library for the form controls. These methods have gotten a lot easier with the introduction of .NET 5.
Additional features:
- Install the agent as a windows service
- Slow down the form interactions
- Create “writing and reading” sequence steps to validate what has been sent to the form .
- Make sure that only one target application is running
- Create different sequences for different versions of the target application ( form arrays/properties could be different and require another walkthrough with spy++).
- Build a redundant service to “watch” the first service and make sure its running.
- Implement a rolling-log file for error handling using Serilog. (rolling so that it doesn’t fill up the hard drive over time)
- Modbus feedback and heartbeat signal (incremental value) to validate the PLC is still accessible and communicating (polled ever 5 seconds).
Summary
By applying some windows form automation and tying it to an API connection we where able to renegotiate a power purchase agreement and make a legacy SCADA application for a wind farm dynamically dispatch-able. Considerations around redundancy and reliability where taken into account. As similar systems around the country begin to age owner/operators need to stay address what methods will be put in place to remain competitive in the renewables market.
