Substation Communications Network: Levels and Protocols

Short answer

The communications network of a substation organizes the devices into levels (process, bay, station and control center) and defines which network joins each level and which protocol runs on it. In an IEC 61850 substation it carries GOOSE trips, sampled values, commands and reports, so it is part of the protection system: it is designed with substation-grade switches, zero-recovery-time redundancy (PRP or HSR), traffic separation and cybersecurity.

What the communications network of a substation is

The communications network of a substation joins the relays, the bay controllers, the merging units, the HMI, the RTU or gateway and the clocks. In an IEC 61850 digital substation the trips and currents travel over it, so a network failure is a protection failure.

It helps to distinguish two planes. The architecture is the logical plane: which devices talk to which, over which network and with which protocol. The physical network is the hardware that supports it: switches, fibers, RedBoxes and clocks. The architecture is decided first and the network is sized afterwards.

Levels and buses

IEC 61850-5 describes the automation functions in three levels within the substation (station, bay and process) and the logical interfaces between them. Its second edition (2013) widened the scope to power system automation, with interfaces to other substations and to control centers. In practice four levels are used:

Process: primary equipment, merging units and breaker IEDs.
Bay: protective relays and bay controllers.
Station: HMI, RTU or gateway, engineering server, recorders and clock.
Control center and other substations: outside the fence, over the wide area network (WAN).

Between process and bay is the process bus; between bay and station, the station bus. A concrete implementation with yard units and point-to-point fibre is GE’s HardFiber architecture, described in Process Bus over Fiber.

What traffic each level carries

Segment	What travels	Protocol
Process bus	Currents and voltages from merging units to relays; trips to breaker IEDs and positions back; high-precision time	Sampled values (SV), GOOSE and PTP (IEC/IEEE 61850-9-3), all layer 2 multicast with no IP address
Station bus	Reports, select-before-operate commands, fault files and settings between IEDs and their clients; interlocks, breaker failure initiation and blocks between bays; time	MMS, GOOSE and SNTP (IEC 61850-8-1 profile) or PTP
To the control center	States, measurements, alarms and commands	IEC 60870-5-104 or DNP3 from the gateway; IEC TR 61850-90-2 for end-to-end IEC 61850
Between substations	Teleprotection, line differential, GOOSE between substations and synchrophasors to a concentrator	Dedicated channels or multiplexers with IEEE C37.94 interface; IEC TR 61850-90-1 for GOOSE between substations

Timing and bandwidth requirements

GOOSE trip: IEC 61850-5 asks for a 3 ms transfer time for fast trip messages.
Event time stamping: class T1 (±1 ms).
Sampled values: ±1 µs (class T5), so that samples from different merging units are comparable.
Synchrophasors: well below 26 µs, which at 60 Hz would by themselves consume 1 % total vector error; in practice the same ±1 µs reference is used.
Bandwidth: the process bus dominates. Each SV stream takes about 5 Mbps with the usual profile of 80 samples per cycle, while MMS and GOOSE on the station bus are small bursts that grow with events.

To check that a trip meets its time, a latency budget is built: the delay of each link, of each switch and the number of hops between sender and receiver, as proposed by the technical report IEC TR 61850-90-4. That report (edition 2.0, 2020) is the network engineering guide inside the substation: it compares topologies, redundancy, GOOSE and SV traffic and synchronization, and analyzes both station and process buses on separate and combined physical networks. It leaves the WAN to IEC TR 61850-90-12 and cybersecurity to IEC 62351 and IEC 62443.

The network equipment

Substation switches, which meet IEC 61850-3 and often IEEE 1613: requirements for construction, environment, electromagnetic compatibility and safety. See the substation Ethernet switch.
Dual-port IEDs and RedBox, which connects single-port devices to a redundant network.
Optical fiber between the yard and the control house, and copper inside the panels.
Firewall at the edge toward the control center and remote access.
Clocks and switches with PTP support; see time synchronization in substations.
Power supply from the DC system, with two sources when the network is redundant.

Redundancy and traffic separation

For trips and sampled values, zero-recovery-time redundancy defined in IEC 62439-3 is used. With PRP each device connects to two independent networks, LAN A and LAN B, and sends each frame on both; with HSR the devices form a ring and each frame travels in both directions. In both cases the receiver keeps the first copy. RSTP also gives redundancy, but by reconfiguring the network, and frames are lost during that time. The detail is in PRP and HSR.

GOOSE and SV carry an IEEE 802.1Q tag with VLAN and priority, so that switches deliver them only to the ports that need them and forward them ahead of MMS or engineering traffic. Without VLANs and multicast filtering, SV streams can saturate the station bus; see VLAN and IEEE 802.1Q priority. The allocation of IP addresses, VLANs and priorities for the whole project is documented in an IP and VLAN addressing plan.

The most common topologies are these. The simple star is economical and has no redundancy, acceptable in medium voltage and industry when the philosophy allows it. The PRP double star is common in high and extra-high voltage. The HSR ring is useful in the process bus and medium-voltage switchgear. Combinations of these are also used, joined by RedBox. One non-negotiable criterion: the paths of main 1 and main 2 protection must be independent. Topologies are compared in detail in substation network topologies, and what changes between medium, high and extra-high voltage in substation communications by voltage level. To decide between PRP and HSR by network size and total cost, see PRP vs HSR.

Design criteria

Start from the functions: which protection depends on which message and what happens if that message is missing.
Separate traffic by VLAN and priority (IEEE 802.1Q) and filter multicast.
Choose redundancy by level and keep the paths of main 1 and main 2 protection independent.
Design the timing: clock, PTP profile and compatible switches along the whole path.
Define the IP address plan and the security edge toward the WAN.
Document the network together with the SCD. Edition 2 of IEC TR 61850-90-4 includes an example SCL configuration with the network topology.

Conventional versus digital

In a conventional substation, AIS or GIS, the network only carries supervision and control: trips and measurements stay on copper. With an IEC 61850 station bus, interlocks and some trips move to GOOSE. With a process bus, the whole protection depends on the network and on timing, and network design becomes part of the protection study.

Cybersecurity

IEC 62351 defines security mechanisms for power system protocols; its part 6 deals with IEC 61850, including authentication of GOOSE and SV with message authentication codes so as not to affect timing. IEC 62443 gives the framework for industrial automation systems: zones, controlled conduits and component requirements. In practice: segment, close unused ports, use individual accounts with roles, centralize event logging and allow remote access only through one controlled point.

Supervision and testing

The network is supervised like any protection device. IEC 61850-7-4 includes logical nodes for that: LCCH supervises the physical channels and LGOS and LSVS the reception of each GOOSE and SV subscription. Their alarms must reach the control center.

At commissioning, five points are checked. Redundancy works: cutting LAN A, LAN B or the ring loses no GOOSE or SV. Each IED receives only the streams it should. GOOSE trips meet their time. PTP reaches the most distant IEDs with the required accuracy. The security configuration and its backups are in order. IEC TR 61850-90-4 devotes a chapter to these tests (VLAN, priority, multicast, redundancy and synchronization), which its second edition expanded, and the end-to-end tests close the cycle with teleprotection and the control center.

Common faults and what to check first

LAN A and LAN B crossed in a PRP network, or an IED connected twice to the same network.
VLAN not allowed on a port after replacing a switch.
SV streams reaching the station bus and saturating ports: check VLAN and multicast filtering with a capture on the affected port.
Data that does not reach the control center although the IED has it: gateway mapping or MMS reports not enabled.
Events with inconsistent time stamps between bays: two time sources or a switch without PTP in the path.
Main 1 and main 2 protection on the same physical path, or loops created with a test cable.
Dirty or attenuated fiber, with intermittent errors, and switch configurations without backup.
Outdated SCD compared with the real network, after changes made in the field.

Traffic capture on a mirror port, switch logs and supervision nodes are the first tools, always against an up-to-date SCD.

Network design and diagnosis with Tiza Group

Are you losing GOOSE or sampled values, or does your redundant network not behave as it should? At Tiza Group we review the configuration of the IEDs and their communications with our remote relay configuration, or on site with our IEC 61850 engineering in Colombia, or elsewhere on request, in medium, high and extra-high voltage, in substations and industry. Write to us and we will propose the scope.

References

IEC 61850-5: communication requirements for functions and device models.
IEC TR 61850-90-4: network engineering guidelines for power system automation.
IEC TR 61850-90-1 and 90-2: communication between substations and with control centers.
IEC TR 61850-90-12: wide area network engineering guidelines.
IEC 61850-3: general requirements for communication equipment in substations.
IEC 61850-6: substation configuration language (SCL).
IEC 61850-7-4: supervision logical nodes LCCH, LGOS and LSVS.
IEC 61850-8-1 and IEC 61850-9-2: GOOSE, MMS and sampled values.
IEC 62439-3: PRP and HSR redundancy protocols.
IEC/IEEE 61850-9-3: PTP profile for power system automation.
IEC 62351 (series), in particular part 6, and IEC 62443 (series): cybersecurity.
IEC 60870-5-104 and IEEE 1815 (DNP3): telecontrol.
IEEE 1613 and IEEE 802.1Q.

Frequently asked questions

What is the difference between the station bus and the process bus?

The station bus joins the relays and bay controllers with the HMI, the gateway and the servers, and carries MMS reports, commands and GOOSE between bays. The process bus joins the merging units and breaker IEDs with the relays, and carries sampled values, GOOSE trips and PTP synchronization.

Which protocol goes to the control center if the substation is IEC 61850?

Usually a gateway translates the IEC 61850 data to IEC 60870-5-104 or DNP3, the protocols that control centers typically support. IEC TR 61850-90-2 describes how to do it with IEC 61850 end to end, but its use depends on the operator.

Why does a relay stop receiving a GOOSE after a switch is replaced?

The usual causes are a VLAN not allowed on the port, multicast filtering badly configured, the port connected to the wrong LAN in a PRP network, or a priority that is not respected. Compare the configuration of the new switch with the network design and the SCD.

Can the station bus and the process bus share the same physical network?

Yes. IEC TR 61850-90-4 analyzes both separate and combined networks. If they are combined, VLANs and multicast filtering must keep sampled values from reaching the station bus ports, because their throughput can saturate those links.