WP-049-2021 Rhebo Industrial Protector test results

This report presents the test results for the Rhebo Industiral Protector security sensors for a laboratory tests on substations traffic.

Our members have shown an increasing interest in deploying OT security monitoring solutions, which provide specialized intrusion detection capabilities for OT systems. The solutions see real-time network communications and monitor for deviations from known baselines or matches to attack signatures.

We first assessed these solutions in 2017, when they were mostly new and unknown. They were also quite expensive to deploy sensors in each high-voltage substation, which would require installing tens to hundreds of sensors. Therefore, most operators deployed them in their central systems, such as their SCADA system. Because of this, our assessment focused on the detection capabilities for protocols used between substations and central systems, such as IEC 104.

Lately, the cost of OT security monitoring solutions has been decreasing, making them more suitable for deployments in substations. Due to this, we revisited these solutions in 2020, this time focusing on the detection capabilities for protocols used between IEDs and gateways, such as IEC 61850.

We divided the new assessment in two stages:

  1. A market survey based on questionnaires, which allowed us to assess new features in previously assessed solutions, solutions that we did not assess before and how they all support the IEC 61850 protocol. It turned out that all solutions are quite balanced on paper.
  2. A test of the solutions in a virtualized environment at ENCS’ laboratory and in our members’ substations. The results allowed us to validate the claims made by vendors in the first stage and to understand how the features work in practice.

WP-048-2021 Omicron StationGuard test results

This report presents the test results for the Omicron StationGuard security sensors for a laboratory tests on substations traffic.

Our members have shown an increasing interest in deploying OT security monitoring solutions, which provide specialized intrusion detection capabilities for OT systems. The solutions see real-time network communications and monitor for deviations from known baselines or matches to attack signatures.

We first assessed these solutions in 2017, when they were mostly new and unknown. They were also quite expensive to deploy sensors in each high-voltage substation, which would require installing tens to hundreds of sensors. Therefore, most operators deployed them in their central systems, such as their SCADA system. Because of this, our assessment focused on the detection capabilities for protocols used between substations and central systems, such as IEC 104.

Lately, the cost of OT security monitoring solutions has been decreasing, making them more suitable for deployments in substations. Due to this, we revisited these solutions in 2020, this time focusing on the detection capabilities for protocols used between IEDs and gateways, such as IEC 61850.

We divided the new assessment in two stages:

  1. A market survey based on questionnaires, which allowed us to assess new features in previously assessed solutions, solutions that we did not assess before and how they all support the IEC 61850 protocol. It turned out that all solutions are quite balanced on paper.
  2. A test of the solutions in a virtualized environment at ENCS’ laboratory and in our members’ substations. The results allowed us to validate the claims made by vendors in the first stage and to understand how the features work in practice.

WP-047-2021 Nozomi Guardian test results

This report presents the test results for the Nozomi Guardian security sensors for a laboratory tests on substations traffic.

Our members have shown an increasing interest in deploying OT security monitoring solutions, which provide specialized intrusion detection capabilities for OT systems. The solutions see real-time network communications and monitor for deviations from known baselines or matches to attack signatures.

We first assessed these solutions in 2017, when they were mostly new and unknown. They were also quite expensive to deploy sensors in each high-voltage substation, which would require installing tens to hundreds of sensors. Therefore, most operators deployed them in their central systems, such as their SCADA system. Because of this, our assessment focused on the detection capabilities for protocols used between substations and central systems, such as IEC 104.

Lately, the cost of OT security monitoring solutions has been decreasing, making them more suitable for deployments in substations. Due to this, we revisited these solutions in 2020, this time focusing on the detection capabilities for protocols used between IEDs and gateways, such as IEC 61850.

We divided the new assessment in two stages:

  1. A market survey based on questionnaires, which allowed us to assess new features in previously assessed solutions, solutions that we did not assess before and how they all support the IEC 61850 protocol. It turned out that all solutions are quite balanced on paper.
  2. A test of the solutions in a virtualized environment at ENCS’ laboratory and in our members’ substations. The results allowed us to validate the claims made by vendors in the first stage and to understand how the features work in practice.

WP-046-2021 Forescout SilentDefense test results

This report presents the test results for the Forescout SilentDefense security sensors for a laboratory tests on substations traffic and two tests in members' substations.

Our members have shown an increasing interest in deploying OT security monitoring solutions, which provide specialized intrusion detection capabilities for OT systems. The solutions see real-time network communications and monitor for deviations from known baselines or matches to attack signatures.

We first assessed these solutions in 2017, when they were mostly new and unknown. They were also quite expensive to deploy sensors in each high-voltage substation, which would require installing tens to hundreds of sensors. Therefore, most operators deployed them in their central systems, such as their SCADA system. Because of this, our assessment focused on the detection capabilities for protocols used between substations and central systems, such as IEC 104.

Lately, the cost of OT security monitoring solutions has been decreasing, making them more suitable for deployments in substations. Due to this, we revisited these solutions in 2020, this time focusing on the detection capabilities for protocols used between IEDs and gateways, such as IEC 61850.

We divided the new assessment in two stages:

  1. A market survey based on questionnaires, which allowed us to assess new features in previously assessed solutions, solutions that we did not assess before and how they all support the IEC 61850 protocol. It turned out that all solutions are quite balanced on paper.
  2. A test of the solutions in a virtualized environment at ENCS’ laboratory and in our members’ substations. The results allowed us to validate the claims made by vendors in the first stage and to understand how the features work in practice.

WP-045-2021 Cisco Cyber Vision test results

This report presents the test results for the Cisco Cyber Vision security sensors for a laboratory tests on substations traffic.

Our members have shown an increasing interest in deploying OT security monitoring solutions, which provide specialized intrusion detection capabilities for OT systems. The solutions see real-time network communications and monitor for deviations from known baselines or matches to attack signatures.

We first assessed these solutions in 2017, when they were mostly new and unknown. They were also quite expensive to deploy sensors in each high-voltage substation, which would require installing tens to hundreds of sensors. Therefore, most operators deployed them in their central systems, such as their SCADA system. Because of this, our assessment focused on the detection capabilities for protocols used between substations and central systems, such as IEC 104.

Lately, the cost of OT security monitoring solutions has been decreasing, making them more suitable for deployments in substations. Due to this, we revisited these solutions in 2020, this time focusing on the detection capabilities for protocols used between IEDs and gateways, such as IEC 61850.

We divided the new assessment in two stages:

  1. A market survey based on questionnaires, which allowed us to assess new features in previously assessed solutions, solutions that we did not assess before and how they all support the IEC 61850 protocol. It turned out that all solutions are quite balanced on paper.
  2. A test of the solutions in a virtualized environment at ENCS’ laboratory and in our members’ substations. The results allowed us to validate the claims made by vendors in the first stage and to understand how the features work in practice.

 


WP-033-2020: Test results for OT security sensors monitoring inside substations

Many grid operators are using specialized security sensors to monitor their central OT networks. The sensors can automatically learn what normal network communication looks like, and then alert on deviations from normal to detect security incidents. ENCS evaluated such sensors in the 2017 member project on security monitoring . Since then, these sensors have become an important tool for most SOC and CSIRT teams working on OT systems.

Vendors of these sensors are also offering industrial PCs or embedded sensors that could be placed in field locations. The prices of such sensors have dropped considerably since 2017. So, it is becoming possible to deploy sensors in the internal networks of high voltage substations. ENCS has therefore started a project to evaluate the sensors for monitoring inside substations.

A market survey was conducted based on questionnaires to determine their capabilities. Unfortunately, it turns out the sensors are hard to evaluate on paper. Based on the questionnaires, they can all detect all vulnerabilities and all incidents. We found no real differences between sensors in the market survey.

This raises the question of how good the sensors are in practice. Can they detect all the vulnerabilities and incidents that vendors claim they do? And can they present these in a meaningful way to analysts?

To answer this questions, ENCS tested five sensors in our lab. The sensors were trained on substation traffic collected from members. Different test cases were injected in the traffic to see if the sensors detect them.


Darktrace – First Impressions

First impressions of the Darktrace sensor from a StedIn pilot.


New Sensors to Monitor OT Security

Capabilities and limitations of network-based sensors for OT


OT Security Monitoring Architecture

Reference functional architecture for a security monitoring system for Operational Technology (OT)


OT Security Monitoring Market Survey

Broad market survey of SIEM systems, intrusion detection systems, and vulnerability scanners