Logging Support

Introduction

The system produces a wealth of information in log files. Most of the log messages generated in the system are that of general events from the various system components. For some of the more important events that are logged, see Security Related Events.

The following methods for logging are available:

  • Logging of default log files: By default all logs are written to an internal RAM disk. Log files written to the internal RAM disk are not persistent across reboots.

  • Logging of default log files to USB: When enabled it will log the same things that are logged to the internal RAM disk, but it will be done to a connected external USB drive.

  • Logging to console port: It is possible to direct logging messages to the console port. Messages of severity level DEBUG or higher are shown on the console port.

  • Logging to custom logging sink: For more advanced logging, including logging to remote machines, the system supports creation of user defined logging sinks. Each individual sink describes a logging target that can be either a remote logging server, a file on external media (USB or SD Card), or a local file. In addition, for each sink the operator defines filters for what log messages to be written based on Facility (subsystem) and Severity (level). The filter provides a greater control in regards to the messages that should be logged.

    • All sinks can be configured to use a certain pre-defined log format, defined in RFC 3164 and RFC 5424. For remote syslog servers a third format is supported for compatibility reasons, called BSD. The two RFC formats support timestamp and must be explicitly enabled by the operator.

For example an use-case see:

Overview

The system produces and presents a lot of information in the form of log messages that are stored in log files. Any message that is generated by the system will be associated with a so called Facility and a Severity, in accordance with the syslog standard specified in RFC 5424.

Facility

The Facility is supposed to signify what specific area any particular message belongs to. All facilities and how the system attempts to conform to them can be seen in the table below.

# Facility Description
0 kern Kernel log messages
1 user User-level messages
2 mail Unused
3 daemon System daemons
4 auth Security/Authorization messages
5 syslog Unused
6 lpr Unused
7 news Unused
8 uucp Unused
9 cron Unused
10 authpriv Unused
11 ftp Unused
12 ntp NTP
13 security Log audit, for audit trail
14 console Log alert
16 local0 Alarm sub-system
17 local1 Unused
18 local2 PPP
19 local3 Unused
20 local4 OpenVPN, IPsec
21 local5 Reserved, OEM customer specific
22 local6 Unused
23 local7 Unused

Table 1: All of the standard syslog facilities and their uses in the system.

Note

The facilities security and console are displayed as LOGAUDIT and LOGALERTin Wireshark, which is the de facto tool to debug all network traffic.

Severity

The Severity is supposed to represent the level of criticality for each individual message. All severities and how the system attempts to conform to them can be seen in the table below.

# Severity Description
0 emerg System level service only
1 alert System level service only
2 crit System level service only
3 err Severe error, daemon/service may restart
4 warning Significant problem, e.g. no connection to Radius server
5 notice General log message, e.g. login successful
6 info Informational, less important
7 debug Developer/low-level debug messages

Table 2: All of the standard syslog severities and their general usage in the system.

Simple Example

As an example, a normal configuration setup for remote logging is to log all messages that fall within a specific severity and above. In the following example below, the device is configured to send all log messages of severity notice and above to a remote log server located at foobar.example.com:

example:/#> configure
example:/config/#> logging
example:/config/logging/#> sink
example:/config/logging/sink-1/#> target udp address foobar.example.com
example:/config/logging/sink-1/#> priority *.notice
example:/config/logging/sink-1/#> end
example:/config/logging/#> show sink
IDX  TARGET                          PRIORITY                           FORMAT
1    udp address foobar.example.com  *.notice                          rfc3164
example:/config/logging/#> leave
example:/#>

Default Log Files

The system have a number of default logging files that are generated. In order to see all the available log files on the system use the command:

example:/#> dir log
/log/                 MOD. DATE/LINK    SIZE  MD5                             
auth.log              2019-11-28 17:35   264
messages              2019-11-28 18:02  7435
[...]

The messages log file is the main log file in the system. Use the show log command to show it, without any additional information it displays the messages log:

example:/#> show log
[...]

To show a specific log file:

example:/#> show log://auth.log
[...]

Remember the short-keys to the show command when listing long files:

  • R: show the rest of the file
  • q: Quit viewer

Also, the follow command is very useful when monitoring log files. It continuously lists new lines as they are appended to a file. Use Ctrl-C to stop and return to the command prompt.

The system produces a lot of different log messages, but there are a few that are considered a bit more important in terms of security. As such they are listen below:

Event Facility Severity Example
Login successful auth notice Authentication successful for user ‘admin’ from 10.0.0.1
Login failed auth warning Authentication failed for user ‘admin’ from 10.0.0.1
Audit trail security notice User ‘admin’ changed the configuration
System startup console notice \\/ Westermo WeOS v5.7, entering runlevel 3
System shutdown console notice \\/ Westermo WeOS v5.7, entering runlevel 6
System update console notice \\/ Westermo WeOS v5.7, entering runlevel 9
Enter Maintenance mode user notice Entering maintenance mode for upgrade to WeOS-5.7.x.pkg …
Exit Maintenance mode user notice Exiting maintenance mode for upgrade
Service start console notice Starting mdnsd:1, PID: 1203
Service stop console notice Stopping mdnsd:1, PID: 1203, sending SIGTERM…
Service restarting console notice Restarting mdnsd:1, PID: 1203, sending SIGHUP…
Service Died console warning Service mdnsd:1 died, restarting (1/10)
Service FAIL console warning Service mdnsd:1 keeps crashing, not restarting
Link up console notice Link up on eth1
Link down console notice Link down on eth1
DHCP Unknown host ID console warning DHCPDISCOVER(11:22:33:44:55:66) vlan1 no address available
NTP reply not synced ntp info reply from 10.0.0.1: not synced (alarm), next query 64s
NTP synced ntp info clock is now synced
NTP peer valid ntp info peer 10.0.0.1 now valid
NTP update FAIL ntp err no reply received in time, skipping initial time setting
TTDP stack ready daemon notice Bypass relays in normal position (upon startup; IEC 61375)
ECSC comm. established daemon notice ECSP_CTRL traffic heard from ECSC (upon startup; IEC 61375)

Table 3: Security related events that are logged in the system.

Example sink configuration to match the listed security related events listed above, sent to two remote log servers located at 192.168.1.10 and 192.168.1.12:

example:/#> configure
example:/config/#> logging
example:/config/logging/#> sink
example:/config/logging/sink-1/#> target udp address 192.168.1.10
example:/config/logging/sink-1/#> priority auth;security;console;ntp
example:/config/logging/sink-1/#> end

example:/config/logging/#> sink target udp address 192.168.1.12 priority auth;
security;console;ntp
example:/config/logging/sink-2/#> end
example:/config/logging/#> show
Console syslog : Disabled
USB/SD logging : Disabled
Secure mode    : Enabled
______________________________________________________________________________
Sinks
IDX  TARGET                          PRIORITY                           FORMAT
1    udp address 192.168.1.10        auth.*;security.*;console.*;n...  rfc3164
2    udp address 192.168.1.12        auth.*;security.*;console.*;n...  rfc3164

Configuration

The logging in the system can be configured from the top-level configuration context in the CLI:

example:/#> configure
example:/config/#> logging
example:/config/logging/#>
[no] console

Use to enable or disable logging directly on the console.

Default: Disabled

no
Disable logging to the console.
[no] external-logging [SIZE [COUNT]]

Enable or disable logging of default log files external media, e.g. USB/SD-Card.

When enabled the same log files that are logged to the local RAM disk will also be logged to the specified external media.

A folder called /log/ will be created on the external media if it does not exist, this is where the log files will be created.

All log files are automatically compressed from generation .1

Example:

Enable external logging and Keep 10 log files, each 10MiB (uncompressed):

example:/config/logging/#> external-logging 10M 10

Default: Enabled

Note

Sinks can be configured to log to a file on external media, regardless if this setting is enabled or not. This setting only handles logging of the systems default log files to the specified external media.

no
Disable logging of default log files to usb.
SIZE
Max size of each log file, in uncompressed form. Takes an optional k/M size modifier.
COUNT
Number of log files to keep, in total.
[no] external-logging-media [MEDIA]

Specify the target external media (USB/SD-Card) for the logging.

The selectable medias are based on the configured medias in the boot configuration context. Any media name can be configured, but a warning will be displayed if it is not also configured in the boot configuration.

Tip

Refer to this page for help on how to correctly set up access to external media devices.

no
Reset the option to default, the default external media.
MEDIA

Dynamically created using the media command in the boot configuration context. Use the do show media command in the current context to list all existing media definitions. All usable media definitions will be listed under Active. In addition, two default media definitions exist if no changes have been made to medias in the boot context:

  • internal: The representation of the built-in flash. This cannot be changed or removed.

  • external: The default media definition for any connected external media device (e.g. USB, SD-Card). It is configured to always match the first partition on said external media device. This definition is not static or locked, it can be freely adjusted by the user, even removed.

[no] secure-mode

Enable or disable secure mode. This setting determines whether the unit reject or accept any logging messages from remote machines. By default secure mode is enabled, meaning the unit operates in a secure way and do not accept any messages from remote machines.

Default: Enabled

no
Disable secure mode.
[no] template NAME priority [EXPR [;EXPR]]

Manage syslog sink priority templates. These templates can be applied to a sinks priority configuration for a quick filtering setup.

The system also provide a number of default template, that cannot be removed or changed.

Default: N/A

no
Use no template to remove all template, except for default templates (default templates are not removable). Use no template NAME to remove a specific template (e.g. no template my-template).
NAME
A free form string.
EXPR
facility.[!|=]severity
facility
[auth | console | daemon | kern | ntp | security | user | local0 | local1 | local2 | local3 | local4 | local5 | local6 | local7]
severity
[debug | info | notice | warning | err | crit | alert | emerg]
[no] sink [INDEX]

Manage syslog sinks, for custom logging to file and remote.

This command is used to manage and create syslog sinks. Each sink defines a logging target and filter on what messages to send towards the target destination based on log message Facility and Severity.

Logging sinks can be configured to log to remote machines, local files, or files on a connected USB drive.

Note

Enters a sub-configuration context for the sink.

no
Use no sink to remove all configured logging sinks, and use no sink INDEX to remove a specific sink definition (e.g. no sink 3).
INDEX

Numerical id of the sink, between 1-8.

When creating a new sink the INDEX argument is optional, when it is left out the next free index is automatically selected for you.

Sink configuration

Each sink also have its own configuration context:

example:/config/logging/#> sink
example:/config/logging/sink-1/#>
[no] target [udp [address [IP|FQDN] | dhcp] | file [MEDIA] PATH]

Specify the target destination of the sink. The target can specify the following types of destinations for the sink:

  • File on local RAM disk: Log to a local file specified by a file path, relative to the systems log file folder.

  • File on an external media device (USB or SD-Card): Log to a file on a connected external media, specified by a file path relative to the target MEDIA.

  • Remote machine: Specify a remote machine to log to by providing an IP address or and FQDN.

  • Remote machine (Dynamic): Specify that the sink should log to a remote machine specified by an IP address received from a DHCP lease providing a log server (DHCP Option 7).

Note

The target cannot be the same for multiple different sink instances.

Note

Folders specified in the PATH when creating a file target, must exist before, they will not be created by the sink, only the file will.

Examples:

example:/config/logging/sink-1/#> target udp address 10.0.0.1


example:/config/logging/sink-1/#> target udp dhcp


example:/config/logging/sink-1/#> file internal my-file.log


example:/config/logging/sink-1/#> file my-media my-file.log

no
A target is required for every sink. Therefore, it is not possible to remove, it can only be changed by configuring another target.
IP
IP address in standard quad-dotted notation, e.g. 192.168.1.1.
FQDN
A fully qualified domain name.
PATH
String definition of the file path and name.
MEDIA

Dynamically created using the media command in the boot configuration context. Use the do show media command in the current context to list all existing media definitions. All usable media definitions will be listed under Active. In addition, two default media definitions exist if no changes have been made to medias in the boot context:

  • internal: The representation of the built-in flash. This cannot be changed or removed.

  • external: The default media definition for any connected external media device (e.g. USB, SD-Card). It is configured to always match the first partition on said external media device. This definition is not static or locked, it can be freely adjusted by the user, even removed.

[no] priority <EXPR [;EXPR] | template NAME>

Configure a sink priority expression, defines what type of messages are sent to the sink target.

The priority per sink consist of up to 16 different expressions. Each expression consists of a facility, a modifier, and a severity. These expressions are combined to define the syslog priority, which in turn dictates what type of messages are sent to the sink target.

Note

Another thing to keep in mind, ordering of the expression have an effect on how the filtering is applied. Meaning that expressions are processed in the order that they appear, i.e. the order they are added.

As an example an expression *.* is added specifying that all messages are to be logged, of any facility and severity. If additional rules, say auth.!* and authpriv.!*, are added this means that we will now exclude all messages with the auth and authpriv facility based on preceding expression rules. If we would have added the exclude rules first, in this example, it would not do anything, since no base to exclude them from existed.

Filter Expression Modifiers

Mod Usage Description
. facility.severity Include messages of facility with severity equal or greater
.! facility.!severity Exclude messages of facility with severity equal or greater
.= facility.=severity Include messages equal to this facility and severity
.!= facility.!=severity Exclude messages equal to this facility and severity

Wildcard Matching On Any Facility or Severity

Usage Description
*.severity Include all messages of a severity
facility.* Include all messages of a facility
*.* Include all messages of any facility or severity
EXPR
facility.[!|=]severity
facility

[kern | user | mail | daemon | auth | syslog | lpr | news | uucp |cron | authpriv | ftp | ntp | security | console | local0 .. local7 | *]

Note

The use of '*' act as a wildcard, indicating that all facilities

will match.

severity

[debug | info | notice | warning | err | crit | alert | emerg| *]

Note

The use of '*' act as a wildcard, indicating that all severities will match.

[no] format bsd|rfc3164|rfc5424

Set the formatting to be used when sending the messages.

BSD

Legacy formatting for remote syslog servers;

<PRI> message

RFC3164

Standardized formatting;

<PRI>Nov 18 09:36:42 hostname proc[123] message

RFC5224

Modern syslog formatting;

<PRI>1 2019-11-18T09:36:42.000321+01:00 hostname proc 123 - - message

Example

This is a simple example how to configure a sink:

example:/#> configure
example:/config/#> logging
example:/config/logging/#> sink
example:/config/logging/sink-1/#> target udp address 10.0.0.1
example:/config/logging/sink-1/#> format rfc3164
example:/config/logging/sink-1/#> priority console.notice;security.warning
example:/config/logging/sink-1/#> show
Target : udp address 10.0.0.1
Format : rfc3164
______________________________________________________________________________
Priority
Expr  Facility     Modifier     Severity                                      
1     console      .            notice
2     security     .            warning
example:/config/logging/sink-1/#> leave
example:/#>

The same setup can also be configured with a one-liner:

example:/#> configure
example:/config/#> logging
example:/config/logging/#> sink target udp address 10.0.0.1 format rfc3164 pri
ority console.notice;security.warning
example:/config/logging/sink-1/#> show
Target : udp address 10.0.0.1
Format : rfc3164
______________________________________________________________________________
Priority
Expr  Facility     Modifier     Severity                                      
1     console      .            notice
2     security     .            warning
example:/config/logging/sink-1/#> leave
example:/#>

This sink now sends any message of facility console and severity notice or higher, and facility security with severity warning or higher, to a syslog server at 10.0.0.1, UDP port 514. All messages are also formatted according to RFC 3164, see above for details.