Logging to Remote Machines HowTo

About

The goal of this document is to provide a comprehensive guide on how to configure client devices to send logging messages to a remote logging server. This process, known as remote logging, is a crucial aspect of maintaining and monitoring the health of distributed systems.

In this guide, we will walk you through the steps required to set up remote logging from a device. We will cover different protocols that can be used for transmitting log data, specifically UDP, TCP, and TLS. Each of these protocols has its own advantages and trade-offs in terms of speed, reliability, and security, and we will discuss these in the context of remote logging.

By the end of this guide, you will have a clear understanding of how to configure remote logging on a WeOS device, and how to choose the most appropriate protocol for your specific needs and environment.

Introduction

The system generates a wealth of valuable data in the form of log messages. In a network comprising numerous devices, centralizing these log messages on a remote logging server can be beneficial. This consolidation of messages onto a single receiver simplifies the monitoring and analysis of the system’s behavior in a distributed setup.

Moreover, directing log messages to a remote server enables the storage of a larger volume of log messages over an extended period. This is particularly useful considering the internal log storage on the device is limited, both in terms of capacity and persistence, as log messages do not survive reboots.

For this use case, we consider the topology shown in Figure 1 below. There is a dedicated syslog server and a switch S1 that is configured to send log messages to the syslog server. In this simplified view we focus on only one switch, but the same principles apply to multiple devices in a network.

             .---------.
             | Syslog  |
             | Server  |
             |         |
             '----+----'
                  |.99
                  |
                  |
------------------+------------------ 192.168.1.0/24
                  |
                  |
                  |.10
             .----+----.
             |         |
             |   S1    |
             |         |
             '---------'

Figure 1: Example network topology with a syslog server and a switch S1 configured to send log messages to the syslog server.

For the use case presented in this guide, how to configure each of the available transport protocols (UDP, TCP, and TLS) for remote logging will be presented. For your specific use case, you should choose the most appropriate protocol based on your requirements and constraints. Some considerations for choosing different transport types for remote logging include:

Security Requirements: If security is not a primary concern for the logging data, or if it is handled through other means, UDP or TCP can be used. These protocols transmit messages in clear text. However, if security is a concern, TLS should be used. TLS encrypts the messages before they are sent over the network, providing an additional layer of security.
Server Capabilities: The log server’s capabilities may dictate the transport type. If the server does not support TLS, TCP or UDP may be the only options. It’s important to verify the server’s compatibility with the chosen transport protocol.
Network Reliability: If the network is unreliable, TCP or TLS (which is built on top of TCP) should be used. These protocols provide mechanisms to ensure all log messages are delivered. UDP, on the other hand, does not guarantee delivery and may be more suitable for reliable networks or scenarios where occasional message loss is acceptable.
Performance Needs: UDP generally has lower overhead than TCP or TLS, making it faster and more efficient in terms of network resources. If performance is a critical factor and security or guaranteed delivery is not a concern, UDP may be the preferred choice.
Regulatory Compliance: Certain industries or applications may have specific regulatory requirements that dictate the use of secure transport protocols. In such cases, TLS would likely be the required choice.

In conclusion, the choice of transport protocol for remote logging depends on a variety of factors, including security requirements, server capabilities, network reliability, performance needs, and regulatory compliance. Understanding these factors will help you make an informed decision that best suits your specific use case.

In the next section, we will delve into the practical aspect of this guide. We will walk you through the step-by-step process of configuring the switch S1 for remote logging using each of the transport protocols discussed in this section.

Configuration

In this section, we will cover the configuration steps required to set up remote logging on the switch S1. It is quite straightforward, we need to set up a logging source, destination and then lastly the sink, that will connect the source and destination components.

Note

For this configuration example, we assume that we already have the underlying interfaces with associated IP addresses configured on the switch.

In this example we do not use any logging filters. This means that all log messages generated by the switch will be sent to the remote syslog server.

Step 1: Source Configuration

We want the source of the log messages to be from the switch S1 itself.. So we configure a source that we will call local:

Note

By default, from the factory default configuration, a logging source named local is already present on the switch. But for the sake of this example, we will create the source just to illustrate the configuration process.

S1:/#> configure
S1:/config/#> logging
S1:/config/logging/#> source local
S1:/config/logging/source-local/#> userspace
S1:/config/logging/source-local/#> kernel
S1:/config/logging/source-local/#> end
S1:/config/logging/#>

We specified that our source local should include both userspace and kernel messages in the logs generated by the source. The next step is to configure the destination for these log messages.

Step 2: Destination Configuration

We will configure a destination, that we will call remote, to send the log messages to the syslog server located at IP address 192.168.1.99:

S1:/config/logging/#> destination remote
S1:/config/logging/destination-remote/#> type remote
S1:/config/logging/destination-remote/#> host address 192.168.1.99

Tip

It is also possible to specify a hostname instead of an IP address for the destination host. If we assume that the syslog server has a hostname syslog-server, we can specify it as follows:

S1:/config/logging/destination-remote/#> host name syslog-server

Note

By default the IP-Protocol to use for the remote logging is IPv4. If you want to specify IPv6 as the IP-Protocol, you can do so by setting the protocol:

S1:/config/logging/destination-remote/#> ip-protocol ipv6

Then we can specify the IPv6 address of the syslog server:

S1:/config/logging/destination-remote/#> host address 2001:db8::1

Depending on which transport protocol to use for the remote logging, the configuration for each protocol is as follows:

UDP: For UDP we simply specify the transport type as UDP:

S1:/config/logging/destination-remote/#> transport udp

TCP: For TCP we specify the transport type as TCP:

S1:/config/logging/destination-remote/#> transport tcp

TLS: For TLS we first specify the transport type as TLS:
```
S1:/config/logging/destination-remote/#> transport tls
```
Now, depending on a few factors, we may need to specify a certificate and/or a ca-certificate for the TLS connection.

Note

We assume for the sake of this example that the appropriate certificates are already present on the switch. Refer to the Certificate Management documentation for more information on how to manage certificates on the switch.

The following is how to handle the certificates for the TLS connection:
- Certificate: If the syslog server requires a client certificate for authentication, we can specify the certificate to use:
```
S1:/config/logging/destination-remote/#> certificate my-example-cert
```
  If the server does not require a client certificate, this step can be skipped. However, for optimal security, it is recommended to use a client certificate if the server supports it.
- CA Certificate: Depending on how the the peer-verify option has been configured, we may need to specify a CA certificate to verify the server’s certificate. If the peer-verify option is set to require a trusted CA certificate:
```
S1:/config/logging/destination-remote/#> peer-verify trusted
```
  A certificate must be provided by the server during the TLS handshake and this certificate must be verifiable by the CA certificate that is specified on the client side:
```
S1:/config/logging/destination-remote/#> ca-certificate my-example-ca-cert
```
  If the peer-verify option is set to untrusted, the CA certificate is not required. In this case, it is not necessary to verify the server’s certificate against the CA certificate. However, for optimal security, it is recommended to use a CA certificate to validate the server’s certificate.

Note

By default, the port used for remote logging will be set to the default for the selected transport protocol. If you want to specify a different port, you can do so by using the port command.

When we have configured the selected transport protocol, we can leave the destination configuration context.

S1:/config/logging/destination-remote/#> end
S1:/config/logging/#>

The final step is to configure the sink that will connect the source and destination components.

Step 3: Sink Configuration

Now we will configure our sink, that we will call remote to connect the source and destination components:

S1:/config/logging/#> sink remote
S1:/config/logging/sink-remote/#> source local
S1:/config/logging/sink-remote/#> destination remote
S1:/config/logging/sink-remote/#> end
S1:/config/logging/#>

The configuration overview should now look something like this:

S1:/config/logging/#> show
Console syslog   : Disabled
External logging : Disabled
Secure mode      : Enabled

╒ Sources ═══════════════════════════════════════════════════════════════════╕
│NAME                                                                        │
│Type: local      USERSPACE  KERNEL                                          │
│local            Yes        Yes                                             │
└────────────────────────────────────────────────────────────────────────────┘

╒ Destinations ══════════════════════════════════════════════════════════════╕
│NAME                                                                        │
│Type: remote     TRANSPORT  TO-HOST       PORT   FORMAT                     │
│remote           udp        192.168.1.99  514    rfc5424                    │
└────────────────────────────────────────────────────────────────────────────┘

╒ Filters ═══════════════════════════════════════════════════════════════════╕
│NAME             INVERT        TYPE  FILTER                                 │
├────────────────────────────────────────────────────────────────────────────┤
│No filter(s) currently configured...                                        │
└────────────────────────────────────────────────────────────────────────────┘

╒ Sinks ═════════════════════════════════════════════════════════════════════╕
│NAME             SOURCES       DESTINATIONS   OP FILTERS                    │
│remote           local         remote                                       │
└────────────────────────────────────────────────────────────────────────────┘

S1:/config/logging/#>

This should be enough to get the remote logging up and running on the switch, so we exit the configuration context, to apply the configuration:

S1:/config/logging/#> leave
S1:/#>

Verification

At this point any log message generated by the switch S1 should be sent to the syslog server located at IP address 192.168.1.99. To verify that the configuration is working as expected, you can check the syslog server for any incoming log messages from 192.168.1.10, or we can generate a test log message on the switch and check if it is sent to the syslog server:

S1:/#> shell
Domain shell started.
admin@example:~ # logger -t proc -I 123 -p user.notice "This is a test log message"
admin@example:~ #

When checking for the log message on the server side, it should look something like this:

May 22 13:34:04 192.168.1.10 proc[123]: This is a test log message

The logging status on the device can also be checked to see if the log messages are being processed correctly:

S1:/#> show logging destination
╒ Destinations ════════════════════════════════════════════════════════════════════════════════╕
│TYPE    NAME         DESCRIPTION           PROCESSED  DROPPED  WRITTEN  QUEUED  MEMORY_USAGE  │
│file    ntp          /var/log/ntp.log      2          0        2        0       0             │
│file    syslog       /var/log/syslog       125        0        125      0       0             │
│file    auth         /var/log/auth.log     3          0        3        0       0             │
│file    alarm        /var/log/alarm        1          0        1        0       0             │
│file    messages     /var/log/messages     88         0        88       0       0             │
│file    kern         /var/log/kern.log     37         0        37       0       0             │
│remote  sink-remote  udp,192.168.1.99:514  106        0        106      0       0             │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
S1:/#>

As per the output above, the sink-remote destination, the number of PROCESSED and WRITTEN log messages should be the same, indicating that the log messages are being sent to the remote syslog server correctly.

If for instance the server would not be reachable, the number of QUEUED log messages would increase, indicating that the log messages are being queued up for transmission to the server when it becomes reachable again.

Note

If the syslog server is not reachable, when the switch’s configuration is applied, the switch will not be able to send the log messages to the server. When the server becomes reachable it may take a short while before the switch attempts to connect to the server again.

Of course, it is always a good idea to simply check if the syslog server is reachable from the switch:

S1:/#> ping 192.168.1.99
Press Ctrl-C to abort PING 192.168.1.99 (192.168.1.99) 56(84) bytes of data.
64 bytes from 192.168.1.99: icmp_seq=1 ttl=64 time=0.852 ms
64 bytes from 192.168.1.99: icmp_seq=2 ttl=64 time=0.766 ms
^C
--- 192.168.1.99 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.766/0.809/0.852/0.043 ms
S1:/#>

If it is not reachable, you should check the network configuration on the switch, server, and the network in between.

Conclusion

In this HowTo, we have gone through the necessary steps to configure remote logging, in order to transmit all log messages generated by the device to a remote syslog server.

This is, of course, a simplified example, but it should give you a good starting point for setting up remote logging on your device. The configuration can be further customized to suit your specific requirements. For a comprehensive list of configuration options, refer to the Logging Configuration documentation.