Health Monitor HowTo

About

This document provides basic Health Monitor use-cases, and how the feature is configured.

For an overview of all available configuration settings for Health Monitor, refer to the port configuration guide:

• Port Configuration Guide

Health Monitor utilises port error states when the health of a link is considered to have deteriorated. When the criteria configured with Health Monitor are met, it sets the port to Err-Disabled. For more information regarding port error states, refer to the error state section in the port configuration guide:

• Port Error States

Introduction

When individual components degrade over time, small and localized faults may escalate into widespread network disruptions. These kind of faults may not only have a large impact on the whole network, but they can also be very difficult to isolate, and time-consuming to troubleshoot. Health Monitor provides a centralized capability to continuously monitor the health of physical network connections.

Health Monitor offer the ability to track link state changes (UP/DOWN) of a port. If the number of state changes exceed a given threshold during a given time-window, it places the port in Err-Disabled. If a timeout is configured, it will automatically clear the error. If no timeout is given, the error has to be manually cleared.

One example use case for this feature is to configure it on a port that is part of a ring. A faulty SFP or a faulty cable between two devices could potentially disrupt the whole ring. A blocked port may start forwarding if the link between two devices goes down in order to recover. If the link then goes up again, the ring protocol may start forwarding on the link that was down and put the other link in blocking mode. A fault could trigger this kind of reconvergence over and over again, and cause the network to never reach a stable state.

Examples

In this section we show different example use-cases of how one could configure Health Monitor.

Health Monitor with manual recovery

In this example, we configure Health Monitor state-changes with a threshold of 10 and 60 seconds as time-window.

example:/#> configure
example:/config/#> port eth2
example:/config/port-eth2/#> health-monitor
example:/config/port-eth2/health-monitor/#> state-changes threshold 10 window 60
example:/config/port-eth2/health-monitor/#> leave
example:/#>

If Health Monitor observes 10 or more state-changes on port eth2 within 60 seconds, the port will be placed in Err-Disabled. Since we gave no timeout parameter, the error will have to be manually cleared by either port clear-error eth2 or by administratively forcing the port up with port up eth2.

Health Monitor with automatic recovery

Here we configure Health Monitor state-changes with a threshold of 5 and 120 seconds as time-window with a 20 minute timeout.

example:/#> configure
example:/config/#> port eth2
example:/config/port-eth2/#> health-monitor
example:/config/port-eth2/health-monitor/#> state-changes threshold 5 window 120 timeout 20
example:/config/port-eth2/health-monitor/#> leave
example:/#>

If there are 5 or more state-changes on port eth2 within 120 seconds, the port will be placed in Err-Disabled. As we set a 20 minute timeout, the error will be cleared automatically after 20 minutes. It is still possible to manually clear the error state if needed.

Protecting multiple ports

It is possible to configure Health Monitor for multiple ports at once. Here we configure Health Monitor state-changes for the port range eth0..eth7.

example:/#> configure
example:/config/#> port eth0..eth7
example:/config/port-eth0..eth7/#> health-monitor
example:/config/port-eth0..eth7/health-monitor/#> state-changes threshold 5 window 120 timeout 20
example:/config/port-eth0..eth7/health-monitor/#> leave
example:/#>

Verifying if Health Monitor has triggered an error state

To the determine if Health Monitor have placed the port in Err-Disabled or not, we first use show port command in the admin exec context. Any link reporting Err-Disabled indicates that a service has disabled the port.

example:/#> show port
PORT             LINK     TYPE       SPEED  STATE           OWNER/VLAN        
eth1             Down      N/A  ----------  No-Link         vlan U:1
eth2             Down      N/A  ----------  No-Link         vlan U:1
eth3             Down   1000-T  ----------  Err-Disabled    vlan U:1
eth4             Up     1000-T  1000M-Full  Forwarding      vlan U:1
...
example:/#>

To determine the exact cause for the port being rendered Err-Disabled, we can show a detailed view by using show port PORT. This view also display the current configuration for the port.

example:/#> show port eth3
eth3                                                                          
Link           : Down
Oper. status   : Disabled
Admin status   : Enabled
Description    : eth3
Type           : 1000-T
State          : Err-Disabled
...
State Changes  : threshold 5 window 120 timeout 20
Shielded mode  : Unshielded
PHY power mode : Low power (CLASS B)
Bridge group   : Disabled

Error State: Err-Disabled (effective)
STATE             SERVICE          REASON                                     
Err-Disabled      Health Monitor   Excessive link state changes detected

Here we can see that Health Monitor was the service that place the port in Err-Disabled, and that the reason was "Excessive link state changes detected".

Disabling Health Monitor

To disable Health Monitor state-changes you can either do no health-monitor in the port config context or no state-changes in the health-monitor context:

example:/#> configure
example:/config/#> port eth2
example:/config/port-eth2/#> no health-monitor
example:/config/port-eth2/#> leave
example:/#>

example:/#> configure
example:/config/#> port eth2
example:/config/port-eth2/#> health-monitor
example:/config/port-eth2/health-monitor/#> no state-changes
example:/config/port-eth2/health-monitor/#> leave
example:/#>