WeOS
5.29.0

# Watchdog

## Introduction

The watchdog is a functionality that can help the system recover from various possible malfunctions that could cause the system to enter a state where it is not able to operate correctly. Anything from a hardware fault or program errors may cause the watchdog to trigger.

The watchdog is essentially just a timer that will cause the system to reset if it runs out. In order to prevent the watchdog from triggering, the system will periodically “kick” the watchdog in order to inform it that the system is still operating within established parameters.

## Overview

WeOS allows users to perform adjustments to the watchdog timeout along with the kick interval. The watchdog timeout is not allowed to be set to a value equal or smaller than the kick interval.

In addition, the system also allows for disabling the watchdog. However, this is not something that is recommended. It is suggested that if adjustments to the watchdog is deemed necessary for any reason, tweak the timers instead.

### Reset Information

Further, the watchdog in the system can also provide some useful information to the user. One of the more prominent of these is information on the last reset of the system. The information will be provided with a reset reason and cause. The following are the possible reset reasons and causes that the watchdog is able to provide:

Reset Cause | Reset Reason
--- | ---
0 | None
1 | System OK
2 | Failed subscription
3 | Failed kick
4 | Failed unsubscription
5 | Failed to meet deadline
6 | Forced reset
8 | Descriptor leak
9 | Memory leak
10 | CPU overload

### Monitors

The watchdog also provides three different types of monitors, that are enabled by default as well. These monitors are configured to log information at provided intervals. The following are the three monitors supported by the watchdog:

- loadavg: Monitoring the average load on the system.

- meminfo: Monitoring the memory in order to detect memory leaks.

- filenr: Monitoring file descriptors in order to detect file descriptor leaks.

Currently only these three default monitors are configurable for the watchdog.

## Configuration

### Watchdog Settings

The watchdog can be configured from the top-level configuration context in the CLI.

    example:/#> configure
    example:/config/#> watchdog
    example:/config/watchdog/#>

#### Syntax

`[no] enable`

  Enable or disable watchdog daemon.

  Default: enabled.

  no
    Disable the watchdog.
`[no] interval [SEC]`

  Set watchdog kick interval, seconds.

  The time between kicks from watchdogd to the kernel wdt driver. Recommended to set to 1/3 of the kernel wdt timeout.

  Min: 1 sec, Max: 128 sec, Default: 20 sec.

  no
    Reset to the default value: 20.
  SEC
    Time in seconds within the allowed range: 1-128.
`[no] timeout [SEC]`

  Set kernel wdt timeout, seconds.

  The timeout before the kernel watchdog timer elapses. This can be an external hardware watchdog, depending on the device. Recommended to set to 10 seconds or greater.

  Min: 1 sec, Max: 128 sec, Default: 60 sec.

  no
    Reset to the default value: 60.
  SEC
    Time in seconds within the allowed range: 1-128.
`[show] monitor [loadavg | meminfo | filenr]`

  Handle watchdog monitors.

  Note

  Enters a sub-configuration context.

  show
    Show the configured monitors.
  `loadavg`
    Monitor for average CPU load.
  `meminfo`
    Monitor for memory usage.
  `filenr`
    Monitor for file descriptor count.

### Monitor Settings

The monitors can be configured from the watchdog configuration context in the CLI. When entering the monitor context a existing monitor name must be provided.

    example:/#> configure
    example:/config/#> watchdog
    example:/config/watchdog/#> monitor NAME
    example:/config/watchdog/monitor-NAME/#>

#### Syntax

`[no] enable`

  Enable or disable the monitor.

  Default: Enabled.

  no
    Disable the monitor.
`[no] interval [SEC]`

  Set the monitor interval.

  Default: 3600 seconds.

  no
    Reset to the default value..
  SEC
    Time value in seconds.
`[no] description [STRING]`

  Free form description of this watchdog monitor.

  no
    Remove any defined description.
  STRING
    Free from string.
`[no] logmark`

  Enable or disable logging to at the configured `interval`.

  Default: Enabled.

  no
    Disable logging.
`[no] warning [PERCENTAGE]`

  Set the percentage when to trigger a warning.

  If this threshold is exceeded, a warning will be logged at the next `interval` invocation.

  no
    Disable the warning.
  PERCENTAGE

    Percentage value in the range `1` - `100`.

    Note

    For a `loadavg` monitor, the maximum percentage value is dependent on the number of CPU cores in the product. As an example, two cores would produce a max value of 200%.

## Status

The current status of the watchdog can be observed from the top level in the CLI:

    example:/#> show watchdog
    Status              : Enabled
    Timeout (sec)       : 60
    Kick interval       : 20
    Watchdog ID         : 0
    Boot counter        : 821
    Reset counter       : 1
    Reset date          : 2019-06-18T09:05:12Z
    Reset reason        : 6 - Forced reset
    PID                 : 1
    Label               : init

The boot counter will be incremented on all boots, both Power on Reset (cold boot) and after a watchdog reset. The reset counter is only incremented on watchdog reset. And set to 0 on a Power on Reset. A reboot from cli will also increment the reset counter since it uses the watchdog for power cycling. The reset reason will differ though. A reboot will be “6 - Forced reset” and real watchdog reset will show as “7 - Unknown failure”. See Reset Information for reset reason for watchdogd.

« Previous Next »