Tiza Group

Home / Guides / Protection, control and communications

Logs and SIEM in Substations: What to Send

Which substation events to send to a central log or SIEM, from which devices, with what retention, and how to avoid alert overload.

By Shadai García, CEO and protection engineer at Tiza Group.

21LAN ALAN Bmanagement networknetwork managerto SCADAnetwork managerall links upSNMPv3 trap: SW LAN A, port 3 linkDownsyslog: port 3 downIED 21: receiving on LAN B onlySCADAno network alarmsalarm: LAN A degradedseparate management ports: LAN A and B never joinSNMPv3: authenticated and encryptedGridSwitchSNMPv3 trap/syslogManagerGatewaySCADA© tizagrp.com21LAN ALAN Bmanagement networknetwork managerto SCADAnetwork managerall links upSNMPv3 trap: SW LAN A, port 3 linkDownsyslog: port 3 downIED 21: receiving on LAN B onlySCADAno network alarmsalarm: LAN A degradedseparate management ports: LAN A and B never joinSNMPv3: authenticated and encryptedGridSwitchSNMPv3 trap/syslogManagerGatewaySCADA© tizagrp.com
Network monitoring: SNMP and syslog. Location in the system and operating principle. Illustrative: real values come from the nameplate and the manufacturer's manual.
Short answer

A SIEM receives the logs of many devices, joins them, correlates them and warns of anything suspicious. In a substation, the useful things to send are logins and their failures, configuration changes, communications status and security alarms from relays, switches and firewalls, with a synchronized clock and defined retention. NERC CIP-007 gives a concrete reference: log those events, alert on logging failures, keep 90 days and review a summary at least every 15 days.

What problem it solves

A substation has dozens of devices that record what happens to them: relays, controllers, switches, firewalls, clocks, the HMI and the SCADA server. Each keeps its events in its own memory, which is short and gets overwritten. If the device is switched off, restarted or tampered with, the log can be lost just when it is needed. A central log keeps a copy off the device, and a SIEM (security information and event management) adds capabilities to use it.

In a SIEM these functions are usually distinguished: aggregation of data from several sources, correlation to turn data into information, alerts directed to a responsible person, dashboards and support for compliance with reports. It is the central tool of a security operations center (SOC), in-house or contracted.

Where it fits: network supervision with SNMP and syslog answers how the device is; the SIEM answers whether someone has done something they should not have. Both are fed by the same sources.

What to send

Not everything a device records is useful, and sending everything saturates. The criterion is the event type, not the volume:

SourceUseful security eventsWhy
Relays and controllersLogins and failures, change of settings or group, account creation and change, restart, firmware updateUnauthorized changes and brute force
Switches and routersConfiguration change, logins, port up or down, topology changeDevice connected where it should not be, or a broken link
FirewallsBlocked connections, rule changes, administrative accessScans and unplanned zone crossings
SCADA server and HMILogins, commands with user, configuration changes, actions on alarmsWho operated what
Remote access gatewaySessions opened and closed, user, destination, durationTraceability of engineering and vendor access
Clocks and time serversLoss of reference, source changeWithout reliable time, events cannot be ordered

NERC CIP-007 is a useful reference even if the installation is not subject to it. For the systems it covers, in the edition we consulted, it asks to log at a minimum successful logins, failed access and login attempts and detected malicious code; to generate alerts for malicious code and for failure of the logging itself; to retain logs for at least 90 consecutive calendar days; and to review a summary or sample at intervals no longer than 15 calendar days to find undetected incidents. That last requirement is the one most often forgotten: storing without reviewing is not monitoring.

How the data arrives

Syslog (IETF RFC 5424) is the most common format for switches, firewalls and many IEDs to send their messages. Some practical points:

  • Transport matters. Over UDP a message can be lost without notice; over TCP, and better still TLS, delivery is confirmed and the content protected. Older devices send only over UDP.
  • Relays sometimes do not speak syslog. Some expose their events through the substation protocol or a vendor tool; in that case, the gateway, the SCADA or a collector acts as the intermediary.
  • A local collector per substation caches if the link to the control center fails, and forwards when it is restored.
  • The logging network is a management network, not the process network. Syslog traffic should not share a VLAN with GOOSE or sampled values; see OT network segmentation.

Time and event order

A SIEM correlates events from different devices by their time stamp. If a relay has a different time from the firewall, the sequence of an incident comes out disordered and correlation fails. All devices that send must be synchronized to the same source; see time synchronization in substations. Also define the time zone of the logs, preferably UTC at the collector. The rules for service laptops, third-party equipment and USB drives connected to relays and switches are in Service Laptops and USB in Substations.

Avoiding alert overload

An untuned SIEM produces hundreds of alerts that nobody reads. To avoid this:

  1. Start with a few useful rules: repeated failed logins, configuration change outside a maintenance window, a new account, an unknown device on the network, failure of the logging itself.
  2. Distinguish maintenance windows from normal operation, so as not to alert on authorized work.
  3. Assign an owner and a response time to each alert. An alert without an owner is not an alert.
  4. Review and retire rules that never help.
  5. Connect alerts to response: who is called and what is done? The response plan is covered in OT cybersecurity gap assessment.

An intrusion detection system adds events the SIEM can receive; see intrusion detection in substation networks.

What to take care of in design

  • Logs are sensitive: they contain user names, addresses and network behavior. Protect them accordingly.
  • The SIEM must not become an entry path: the flow is from the substation outward, and the collector sits in a controlled zone.
  • Capacity: calculate storage from the events per device and the chosen retention.
  • Periodic test: generate a test event and check that it arrives, with the right time and to whoever should see the alert.

What could be verified

The logging requirements cited come from the CIP-007 text as reproduced by a 2017 vendor compliance matrix; the current version is confirmed at nerc.com. The description of a SIEM’s functions was checked against industrial cybersecurity training material, used only as a topic map. Syslog support, transport and event content depend on each model and firmware; they are verified in each device’s manual.

How we help

At Tiza Group we configure the forwarding of events from relays, controllers, switches and firewalls to a central log, with a synchronized clock and defined retention, within our substation and plant cybersecurity service. Write to us. How to put the plan together, with roles, escalation and annual tests, is covered in OT Incident Response Plan for a Substation.

References

  • IETF RFC 5424, The Syslog Protocol.
  • NERC, CIP-007, Systems Security Management, requirement R4 (the current version is confirmed at nerc.com).

Frequently asked questions

Is a SIEM the same as a syslog server?

No. A syslog server receives and stores messages. A SIEM also joins data from many sources, correlates them to find patterns, generates alerts and dashboards, and helps prepare compliance reports. Many installations start with a well-managed syslog and move up to a SIEM when they have someone to operate it.

Should all the relay events go to the SIEM?

No. Protection and process events, such as trips and pickups, belong to the operations event log and the SCADA. The SIEM gets the security events: logins, failed access, changes of settings or accounts, restarts, device connection and alarms from network equipment.

How long should logs be kept?

Your policy sets it and, if it applies, the standard. NERC CIP-007, in the edition we consulted, asks to retain event logs for at least 90 consecutive calendar days, where technically feasible, and to review a summary or sample at intervals no longer than 15 calendar days. Many organizations keep more for investigation reasons.

WhatsApp