A traffic storm in the substation network is almost always caused by a loop: a patch cable, a device with two ports bridged, or a port between switches without RSTP. Other causes are misconfigured redundancy, GOOSE and sampled values multicast that switches forward to every port, and a faulty device transmitting uncontrolled. IEDs saturate, GOOSE messages are delayed and the HMI loses communication.
Symptom
The HMI loses communication with several bays at once, the switches show their activity LEDs lit continuously, some IEDs stop responding or restart their ports, and lost-GOOSE alarms arrive. It usually happens after an intervention: a technician plugged in a laptop, a device was added or a switch was replaced.
An Ethernet network does not tolerate loops. A broadcast frame that finds a loop circulates endlessly, is duplicated at every switch and saturates the network in seconds. Redundancy protocols such as RSTP block one link to prevent it; PRP and HSR duplicate frames over independent paths and discard the copy. When one of those mechanisms does not cover a link, or when the substation multicast traffic is not filtered, the network fills up. To go further: ALBEDO xGenius Ethernet and IEC 61850 tester.
Possible causes, from the most to the least likely
| Cause | Why it happens | What gives it away |
|---|---|---|
| Loop through a cable or a new device | Patch cable between two ports, laptop with two bridged interfaces, IED with two ports connected to the same switch | The storm began when something was connected; it disappears when it is disconnected |
| Port between switches without RSTP | RSTP disabled on a trunk link, or a mix of incompatible ring protocols | Continuous topology changes in the logs; loop not blocked |
| PRP or HSR redundancy wrongly connected | LAN A and LAN B joined, or a non-HSR device inserted in the ring | Excessive duplicates, frames circulating or an open ring |
| Unfiltered GOOSE or SV multicast | Switches forward layer 2 multicast to all ports if there are no VLANs or filters | IEDs with high load, no loop; multicast traffic on ports that do not need it |
| Faulty device transmitting uncontrolled | Defective network card or firmware bug | A single port with very high traffic from one source |
What to check first
- Recent changes: devices or cables connected, switches replaced, configurations loaded. Most storms begin at that point.
- Per-port counters of the switches: broadcast traffic, multicast and errors. The port with the spiking traffic points to the segment.
- RSTP logs: topology changes, root port and blocked ports. A port between switches must run RSTP; ports toward end devices are configured as edge ports with BPDU protection.
- Redundancy topology: that PRP LAN A and B do not touch each other and that the HSR ring contains only HSR devices or RedBoxes.
- VLANs and multicast filtering of GOOSE and sampled values. IGMP snooping does not filter them because they are not IP traffic. See VLAN and IEEE 802.1Q priority and the substation Ethernet switch.
- Did the network saturate when a device or a cable was connected?
- Yes: Physical loop: disconnect that element and check whether the port ran RSTP
- No: Is the excess traffic broadcast?
- Yes: Is there a port between switches without RSTP, or joined PRP networks?
- Yes: Undetected loop: correct the redundancy configuration
- No: Device transmitting uncontrolled: locate it with the port counters
- No: Unfiltered GOOSE or SV multicast: check VLANs and multicast filters
- Yes: Is there a port between switches without RSTP, or joined PRP networks?
How it is confirmed
- Traffic capture on a mirror port: dominant frame type, source address and whether the same frames repeat (loop) or come from a single sender (faulty device).
- Orderly disconnection of links in a window coordinated with operations, starting with recent changes, while watching the counters.
- Controlled loop test: connect a cable between two edge ports and verify that the switch blocks it.
- Measurement of the GOOSE and SV load per port, compared with the capacity of the connected IEDs.
How to keep it from happening again
- Enable RSTP on all ports between switches and configure ports toward end devices as edge ports with BPDU protection.
- Configure storm control on the switches: a broadcast and multicast limit per port bounds the damage of a loop.
- Separate the traffic with VLANs and static multicast filters, so each IED receives only the GOOSE and SV it needs.
- Disable unused switch ports and control who connects devices to the network.
- Test the network under load at commissioning, including GOOSE timing during a reconvergence. See GOOSE not arriving or arriving late.
- Document the physical and logical topology in the substation communications network.
What not to do
- Do not restart all the switches at once. If the loop is still connected, the storm returns; and you lose the counters that locate it.
- Do not disable RSTP to “simplify” a link between switches.
- Do not connect laptops with two interfaces to the substation network without checking that they do not bridge.
- Do not join PRP LAN A and B to “give redundancy” to a simple device: use a RedBox.
What to look for in a capture
Substation traffic has a recognizable footprint. In a mirror-port capture, these are the signals worth identifying:
| Traffic | How to recognize it | What seeing it in excess indicates |
|---|---|---|
| GOOSE | Ethertype 0x88B8; multicast addresses that by convention start with 01-0C-CD-01 | Bursts after an event are normal; a constant, very high flow indicates a publisher that does not settle |
| Sampled values | Ethertype 0x88BA; multicast addresses 01-0C-CD-04 | Presence on ports that do not need them: missing filtering |
| Broadcast | Destination address with all bits set to one | Loop or faulty device |
| The same frames repeated | Same source address and same content at very short intervals | Loop: frames going around |
The address prefixes are a recommendation of IEC 61850-8-1, not an obligation, and some projects use others. Always compare with the project SCD.
Why legitimate GOOSE can also trip a storm control
When a GOOSE signal changes, the publisher repeats the frame quickly and spaces out the repetitions until it returns to the steady-state rate. This is correct design: it shortens delivery time in case of loss. But it means that, after an event with many signals (a busbar trip, for example), the multicast burst is the highest on the network. A multicast storm control with a low threshold can cut exactly that traffic and cause lost GOOSE at the worst moment. The threshold is calculated from the maximum expected burst and not from the steady-state traffic.
What a topology change does
Every topology change in RSTP makes the switches flush their address tables. While they learn them again, unicast frames to unknown destinations are flooded out of all ports, which produces a brief traffic rise. If the change repeats (a link that comes and goes, a cable with a poor contact), the network alternates between flooding and calm, and the IEDs see intermittent losses without a real loop. That is why the topology change log is as useful as the broadcast counters: a port with constant changes explains symptoms similar to those of a storm.
Devices with two ports
Many IEDs have an internal switch with two ports for daisy-chaining in a ring. If both ports are connected to the same switch without the ring or the redundancy being configured for it, a loop forms inside the device itself. It is a frequent installation error on new equipment and is prevented by verifying, for each IED, the network mode declared in its engineering (independent ports, RSTP, PRP or HSR) before connecting it.
Illustrative example
After an intervention, the SCADA loses several bays at once and the switch LEDs never stop blinking. The counters show an edge port with broadcast traffic far higher than the rest. That port was connected to a service laptop whose adapter bridged two interfaces. When it was unplugged, the network normalized in seconds. The per-port counter pointed to the source without restarting anything.
When to call us
If your substation network saturates and it is not clear where the loop is or what traffic is filling it, our team reviews the switch configuration, the redundancy and the GOOSE and SV filtering, and tells you what to correct. We configure IEDs, switches and gateways remotely, worldwide, with remote relay and communications configuration, with a backup before and after each change. On-site network configuration is done with our IEC 61850 engineering, in Colombia, or elsewhere on request, and we design the network from communications engineering. Contact us.
Related guides
- RSTP and Ethernet redundancy
- PRP and HSR (IEC 62439-3)
- Substation Ethernet switch
- VLAN and IEEE 802.1Q priority
- GOOSE not arriving or arriving late
- Substation communications network
- Lost sampled values on the process bus
- ALBEDO xGenius: Ethernet, PTP and C37.94 Testing
References
- IEEE 802.1Q, Bridges and Bridged Networks (includes RSTP and VLAN).
- IEC 62439-3, Industrial communication networks – High availability automation networks – Part 3: PRP and HSR.
- IEC 61850-90-4, Network engineering guidelines.
- IEC 61850-3, General requirements (environmental requirements for substation network equipment).
Frequently asked questions
What is a broadcast storm in an Ethernet network?
It is the saturation of the network by broadcast frames that multiply endlessly, normally because a loop makes every frame re-enter the switch and be forwarded again to all ports. Within seconds it consumes the bandwidth and the processing capacity of the connected devices.
Can a traffic storm affect the protections?
Yes, those that depend on the network. Relays keep protecting through their hardwired inputs, but trips, blocking and permissive signals sent by GOOSE can be delayed or lost, and some IEDs with a saturated processor restart their communications. That is why GOOSE-based functions are tested with the network loaded.
How do you locate the loop in a substation network?
With the per-port broadcast traffic counters of the switches and their RSTP logs: the port where traffic spikes or where the topology keeps changing points to the segment. Disconnecting links in an orderly way, starting with recent changes, confirms the spot.