Many grid operators are using specialized security sensors to monitor their central OT networks. The sensors can automatically learn what normal network communication looks like, and then alert on deviations from normal to detect security incidents. ENCS evaluated such sensors in the member project on security monitoring in 2017. Since then, these sensors have become an important tool for most SOC and CSIRT teams working on OT systems.
Vendors of these sensors are also offering industrial PCs or embedded sensors that could be placed in field locations. The prices of such sensors have dropped considerably since 2017. So, it is becoming possible to deploy sensors in the internal networks of high voltage substations. ENCS has therefore started a project to evaluate the sensors for monitoring inside substations.
A market survey was conducted based on questionnaires to determine their capabilities. Unfortunately, it turns out the sensors are hard to evaluate on paper. Based on the questionnaires, they can all detect all vulnerabilities and all incidents. We found no real differences between sensors in the market survey.
This raises the question of how good the sensors are in practice. Can they detect all the vulnerabilities and incidents that vendors claim they do? And can they present these in a meaningful way to analysts?
To answer these questions, ENCS tested six sensors. All of them were tested in our lab where they were trained on 24 hours of substation traffic collected from members. Traffic related with test cases was injected in other 24 hours traffic to see if the sensors detect it. Two of them could also tested at our members substations focusing more on their usability.
The lab tests were performed in two rounds. The results of the first round did not live up to the expectations and called to question the use cases, methodology and preparedness of the sensors for the tests. Therefore, a second round was agreed with the vendors to jointly address these issues. One vendor could not join for the second round due to more urgent commitments. Another one joined only in the second round.
This whitepaper overviews the results of the second round.