WeOS
5.20.1

# Logging Support

## Introduction

The system produces a wealth of information in log files. Most of the log messages generated in the system are that of general events from the various system components. For some of the more important events that are logged, see Security Related Events.

The following methods for logging are available:

- Logging of default log files: By default all logs are written to an internal RAM disk. Log files written to the internal RAM disk are not persistent across reboots.

- Logging of default log files to USB: When enabled it will log the same things that are logged to the internal RAM disk, but it will be done to a connected external USB drive.

- Logging to console port: It is possible to direct logging messages to the console port. Messages of severity level DEBUG or higher are shown on the console port.

- Logging to custom logging sink: For more advanced logging, including logging to remote machines, the system supports creation of user defined logging sinks. Each individual sink describes a logging target that can be either a remote logging server, a file on external media (USB or SD Card), or a local file. In addition, for each sink the operator defines filters for what log messages to be written based on Facility (subsystem) and Severity (level). The filter provides a greater control in regards to the messages that should be logged.

  - All sinks can be configured to use a certain pre-defined log format, defined in RFC 3164 and RFC 5424. For remote syslog servers a third format is supported for compatibility reasons, called BSD. The two RFC formats support timestamp and must be explicitly enabled by the operator.

For example an use-case see:

- Logging to remote machines

## Overview

The system produces and presents a lot of information in the form of log messages that are stored in log files. Any message that is generated by the system will be associated with a so called Facility and a Severity, in accordance with the syslog standard specified in RFC 5424.

##### Facility

The Facility is supposed to signify what specific area any particular message belongs to. All facilities and how the system attempts to conform to them can be seen in the table below.

# | Facility | Description
--- | --- | ---
0 | kern | Kernel log messages
1 | user | User-level messages
2 | mail | Unused
3 | daemon | System daemons
4 | auth | Security/Authorization messages
5 | syslog | Unused
6 | lpr | Unused
7 | news | Unused
8 | uucp | Unused
9 | cron | Unused
10 | authpriv | Unused
11 | ftp | Unused
12 | ntp | NTP
13 | security | Log audit, for audit trail
14 | console | Log alert
16 | local0 | Alarm sub-system
17 | local1 | Unused
18 | local2 | PPP
19 | local3 | Unused
20 | local4 | OpenVPN, IPsec
21 | local5 | Reserved, OEM customer specific
22 | local6 | Unused
23 | local7 | Unused

Table 1: All of the standard syslog facilities and their uses in the system.

Note

The facilities `security` and `console` are displayed as `LOGAUDIT` and `LOGALERT`in Wireshark, which is the de facto tool to debug all network traffic.

##### Severity

The Severity is supposed to represent the level of criticality for each individual message. All severities and how the system attempts to conform to them can be seen in the table below.

# | Severity | Description
--- | --- | ---
0 | emerg | System level service only
1 | alert | System level service only
2 | crit | System level service only
3 | err | Severe error, daemon/service may restart
4 | warning | Significant problem, e.g. no connection to Radius server
5 | notice | General log message, e.g. login successful
6 | info | Informational, less important
7 | debug | Developer/low-level debug messages

Table 2: All of the standard syslog severities and their general usage in the system.

##### Simple Example

As an example, a normal configuration setup for remote logging is to log all messages that fall within a specific severity and above. In the following example below, the device is configured to send all log messages of severity notice and above to a remote log server located at foobar.example.com:

    example:/#> configure
    example:/config/#> logging
    example:/config/logging/#> sink
    example:/config/logging/sink-1/#> target udp address foobar.example.com
    example:/config/logging/sink-1/#> priority *.notice
    example:/config/logging/sink-1/#> end
    example:/config/logging/#> show sink
    IDX  TARGET                          PRIORITY                           FORMAT
    1    udp address foobar.example.com  *.notice                          rfc3164
    example:/config/logging/#> leave
    example:/#>

### Default Log Files

The system have a number of default logging files that are generated. In order to see all the available log files on the system use the command:

    example:/#> dir log
    /log/                 MOD. DATE/LINK    SIZE  MD5
    auth.log              2019-11-28 17:35   264
    messages              2019-11-28 18:02  7435
    [...]

The messages log file is the main log file in the system. Use the `show log` command to show it, without any additional information it displays the messages log:

    example:/#> show log
    [...]

To show a specific log file:

    example:/#> show log://auth.log
    [...]

Remember the short-keys to the `show` command when listing long files:

- `R`: show the rest of the file
- `q`: Quit viewer

Also, the `follow` command is very useful when monitoring log files. It continuously lists new lines as they are appended to a file. Use `Ctrl-C` to stop and return to the command prompt.

### Security Related Events

The system produces a lot of different log messages, but there are a few that are considered a bit more important in terms of security. As such they are listen below:

Event | Facility | Severity | Example
--- | --- | --- | ---
Login successful | auth | notice | Authentication successful for user ‘admin’ from 10.0.0.1
Login failed | auth | warning | Authentication failed for user ‘admin’ from 10.0.0.1
Audit trail | security | notice | User ‘admin’ changed the configuration
System startup | console | notice | \\/ Westermo WeOS v5.7, entering runlevel 3
System shutdown | console | notice | \\/ Westermo WeOS v5.7, entering runlevel 6
System update | console | notice | \\/ Westermo WeOS v5.7, entering runlevel 9
Enter Maintenance mode | user | notice | Entering maintenance mode for upgrade to WeOS-5.7.x.pkg …
Exit Maintenance mode | user | notice | Exiting maintenance mode for upgrade
Service start | console | notice | Starting mdnsd:1, PID: 1203
Service stop | console | notice | Stopping mdnsd:1, PID: 1203, sending SIGTERM…
Service restarting | console | notice | Restarting mdnsd:1, PID: 1203, sending SIGHUP…
Service Died | console | warning | Service mdnsd:1 died, restarting (1/10)
Service FAIL | console | warning | Service mdnsd:1 keeps crashing, not restarting
Link up | console | notice | Link up on eth1
Link down | console | notice | Link down on eth1
DHCP Unknown host ID | console | warning | DHCPDISCOVER(11:22:33:44:55:66) vlan1 no address available
NTP reply not synced | ntp | info | reply from 10.0.0.1: not synced (alarm), next query 64s
NTP synced | ntp | info | clock is now synced
NTP peer valid | ntp | info | peer 10.0.0.1 now valid
NTP update FAIL | ntp | err | no reply received in time, skipping initial time setting
TTDP stack ready | daemon | notice | Bypass relays in normal position (upon startup; IEC 61375)
ECSC comm. established | daemon | notice | ECSP_CTRL traffic heard from ECSC (upon startup; IEC 61375)

Table 3: Security related events that are logged in the system.

Example `sink` configuration to match the listed security related events listed above, sent to two remote log servers located at 192.168.1.10 and 192.168.1.12:

    example:/#> configure
    example:/config/#> logging
    example:/config/logging/#> sink
    example:/config/logging/sink-1/#> target udp address 192.168.1.10
    example:/config/logging/sink-1/#> priority auth;security;console;ntp
    example:/config/logging/sink-1/#> end

    example:/config/logging/#> sink target udp address 192.168.1.12 priority auth;
    security;console;ntp
    example:/config/logging/sink-2/#> end
    example:/config/logging/#> show
    Console syslog : Disabled
    USB/SD logging : Disabled
    Secure mode    : Enabled
    ______________________________________________________________________________
    Sinks
    IDX  TARGET                          PRIORITY                           FORMAT
    1    udp address 192.168.1.10        auth.*;security.*;console.*;n...  rfc3164
    2    udp address 192.168.1.12        auth.*;security.*;console.*;n...  rfc3164

## Configuration

The logging in the system can be configured from the top-level configuration context in the CLI:

    example:/#> configure
    example:/config/#> logging
    example:/config/logging/#>

`[no] console`

  Use to enable or disable logging directly on the console.

  Default: Disabled

  `no`
    Disable logging to the console.
`[no] external-logging [SIZE [COUNT]]`

  Enable or disable logging of default log files external media, e.g. USB/SD-Card.

  When enabled the same log files that are logged to the local RAM disk will also be logged to the specified external media.

  A folder called /log/ will be created on the external media if it does not exist, this is where the log files will be created.

  All log files are automatically compressed from generation .1

  Example:

  Enable external logging and Keep 10 log files, each 10MiB (uncompressed):

      example:/config/logging/#> external-logging 10M 10

  Default: Enabled

  Note

  Sinks can be configured to log to a file on external media, regardless if this setting is enabled or not. This setting only handles logging of the systems default log files to the specified external media.

  `no`
    Disable logging of default log files to usb.
  `SIZE`
    Max size of each log file, in uncompressed form. Takes an optional k/M size modifier.
  `COUNT`
    Number of log files to keep, in total.
`[no] external-logging-media [MEDIA]`

  Specify the target external media (USB/SD-Card) for the logging.

  The selectable medias are based on the configured medias in the boot configuration context. Any media name can be configured, but a warning will be displayed if it is not also configured in the boot configuration.

  Tip

  Refer to this page for help on how to correctly set up access to external media devices.

  `no`
    Reset the option to default, the default `external` media.
  `MEDIA`

    Dynamically created using the `media` command in the boot configuration context. Use the `do show media` command in the current context to list all existing media definitions. All usable media definitions will be listed under Active. In addition, two default media definitions exist if no changes have been made to medias in the boot context:

    - `internal:` The representation of the built-in flash. This cannot be changed or removed.

    - `external:` The default media definition for any connected external media device (e.g. USB, SD-Card). It is configured to always match the first partition on said external media device. This definition is not static or locked, it can be freely adjusted by the user, even removed.

`[no] secure-mode`

  Enable or disable secure mode. This setting determines whether the unit reject or accept any logging messages from remote machines. By default secure mode is enabled, meaning the unit operates in a secure way and do not accept any messages from remote machines.

  Default: Enabled

  `no`
    Disable secure mode.
`[no] template NAME priority [EXPR [;EXPR]]`

  Manage syslog sink priority templates. These templates can be applied to a sinks priority configuration for a quick filtering setup.

  The system also provide a number of default template, that cannot be removed or changed.

  Default: N/A

  `no`
    Use `no template` to remove all template, except for default templates (default templates are not removable). Use `no template NAME` to remove a specific template (e.g. `no template my-template`).
  `NAME`
    A free form string.
  `EXPR`
    `facility.[!|=]severity`
  `facility`
    `[auth | console | daemon | kern | ntp | security | user | local0 | local1 | local2 | local3 | local4 | local5 | local6 | local7]`
  `severity`
    `[debug | info | notice | warning | err | crit | alert | emerg]`
`[no] sink [INDEX]`

  Manage syslog sinks, for custom logging to file and remote.

  This command is used to manage and create syslog sinks. Each sink defines a logging target and filter on what messages to send towards the target destination based on log message Facility and Severity.

  Logging sinks can be configured to log to remote machines, local files, or files on a connected USB drive.

  Note

  Enters a sub-configuration context for the sink.

  `no`
    Use `no sink` to remove all configured logging sinks, and use `no sink INDEX` to remove a specific sink definition (e.g. `no sink 3`).
  `INDEX`

    Numerical id of the sink, between `1-8`.

    When creating a new sink the `INDEX` argument is optional, when it is left out the next free index is automatically selected for you.

### Sink configuration

Each sink also have its own configuration context:

    example:/config/logging/#> sink
    example:/config/logging/sink-1/#>

`[no] target [udp [address [IP|FQDN] | dhcp] | file [MEDIA] PATH]`

  Specify the target destination of the sink. The target can specify the following types of destinations for the sink:

  - File on local RAM disk: Log to a local file specified by a file path, relative to the systems log file folder.

  - File on an external media device (USB or SD-Card): Log to a file on a connected external media, specified by a file path relative to the target `MEDIA`.

  - Remote machine: Specify a remote machine to log to by providing an IP address or and FQDN.

  - Remote machine (Dynamic): Specify that the sink should log to a remote machine specified by an IP address received from a DHCP lease providing a log server (DHCP Option 7).

  Note

  The target cannot be the same for multiple different sink instances.

  Note

  Folders specified in the PATH when creating a file target, must exist before, they will not be created by the sink, only the file will.

  Examples:

      example:/config/logging/sink-1/#> target udp address 10.0.0.1

      example:/config/logging/sink-1/#> target udp dhcp

      example:/config/logging/sink-1/#> file internal my-file.log

      example:/config/logging/sink-1/#> file my-media my-file.log

  `no`
    A target is required for every sink. Therefore, it is not possible to remove, it can only be changed by configuring another target.
  `IP`
    IP address in standard quad-dotted notation, e.g. 192.168.1.1.
  `FQDN`
    A fully qualified domain name.
  `PATH`
    String definition of the file path and name.
  `MEDIA`

    Dynamically created using the `media` command in the boot configuration context. Use the `do show media` command in the current context to list all existing media definitions. All usable media definitions will be listed under Active. In addition, two default media definitions exist if no changes have been made to medias in the boot context:

    - `internal:` The representation of the built-in flash. This cannot be changed or removed.

    - `external:` The default media definition for any connected external media device (e.g. USB, SD-Card). It is configured to always match the first partition on said external media device. This definition is not static or locked, it can be freely adjusted by the user, even removed.

`[no] priority <EXPR [;EXPR] | template NAME>`

  Configure a sink priority expression, defines what type of messages are sent to the sink target.

  The priority per sink consist of up to 16 different expressions. Each expression consists of a facility, a modifier, and a severity. These expressions are combined to define the syslog priority, which in turn dictates what type of messages are sent to the sink target.

  Note

  Another thing to keep in mind, ordering of the expression have an effect on how the filtering is applied. Meaning that expressions are processed in the order that they appear, i.e. the order they are added.

  As an example an expression `*.*` is added specifying that all messages are to be logged, of any facility and severity. If additional rules, say `auth.!*` and `authpriv.!*`, are added this means that we will now exclude all messages with the auth and authpriv facility based on preceding expression rules. If we would have added the exclude rules first, in this example, it would not do anything, since no base to exclude them from existed.

  Filter Expression Modifiers

  Mod | Usage | Description
  --- | --- | ---
  `.` | `facility.severity` | Include messages of facility with severity equal or greater
  `.!` | `facility.!severity` | Exclude messages of facility with severity equal or greater
  `.=` | `facility.=severity` | Include messages equal to this facility and severity
  `.!=` | `facility.!=severity` | Exclude messages equal to this facility and severity

  Wildcard Matching On Any Facility or Severity

  Usage | Description
  --- | ---
  `*.severity` | Include all messages of a severity
  `facility.*` | Include all messages of a facility
  `*.*` | Include all messages of any facility or severity

  `EXPR`
    `facility.[!|=]severity`
  `facility`

    `[kern | user | mail | daemon | auth | syslog | lpr | news | uucp |cron | authpriv | ftp | ntp | security | console | local0 .. local7 | *]`

    Note

    The use of `'*'` act as a wildcard, indicating that all facilities

    will match.

  `severity`

    `[debug | info | notice | warning | err | crit | alert | emerg| *]`

    Note

    The use of `'*'` act as a wildcard, indicating that all severities will match.

`[no] format bsd|rfc3164|rfc5424`

  Set the formatting to be used when sending the messages.

  `BSD`

    Legacy formatting for remote syslog servers;

    `<PRI> message`

  `RFC3164`

    Standardized formatting;

    `<PRI>Nov 18 09:36:42 hostname proc[123] message`

  `RFC5224`

    Modern syslog formatting;

    `<PRI>1 2019-11-18T09:36:42.000321+01:00 hostname proc 123 - - message`

## Example

This is a simple example how to configure a sink:

    example:/#> configure
    example:/config/#> logging
    example:/config/logging/#> sink
    example:/config/logging/sink-1/#> target udp address 10.0.0.1
    example:/config/logging/sink-1/#> format rfc3164
    example:/config/logging/sink-1/#> priority console.notice;security.warning
    example:/config/logging/sink-1/#> show
    Target : udp address 10.0.0.1
    Format : rfc3164
    ______________________________________________________________________________
    Priority
    Expr  Facility     Modifier     Severity
    1     console      .            notice
    2     security     .            warning
    example:/config/logging/sink-1/#> leave
    example:/#>

The same setup can also be configured with a one-liner:

    example:/#> configure
    example:/config/#> logging
    example:/config/logging/#> sink target udp address 10.0.0.1 format rfc3164 pri
    ority console.notice;security.warning
    example:/config/logging/sink-1/#> show
    Target : udp address 10.0.0.1
    Format : rfc3164
    ______________________________________________________________________________
    Priority
    Expr  Facility     Modifier     Severity
    1     console      .            notice
    2     security     .            warning
    example:/config/logging/sink-1/#> leave
    example:/#>

This sink now sends any message of facility `console` and severity `notice` or higher, and facility `security` with severity `warning` or higher, to a syslog server at 10.0.0.1, UDP port 514. All messages are also formatted according to RFC 3164, see above for details.

« Previous Next »