Logging Relay HowTo

About

This document presents an example use-case for a logging relay setup. A logging relay acts as an intermediary, receiving log messages from hosts and forwarding them to another remote syslog server. This is particularly useful for hosts that can only send User Datagram Protocol (UDP) messages, which are not encrypted.

In our use-case, the logging relay receives these UDP messages and forwards them to a syslog server using Transport Layer Security (TLS). TLS provides encryption, ensuring that the log messages are secure during transit. This is an effective method for encrypting log messages when dealing with end devices that lack the capability to send encrypted messages themselves.

Furthermore, a logging relay can be a solution for transmitting messages over an untrusted network. By encrypting the messages, it mitigates the risk of sensitive log information being intercepted or tampered with.

Introduction

It is not uncommon for network devices to generate log messages that need to be stored and analyzed. In some cases, these devices may not have the capability to send encrypted log messages. This is where a logging relay comes in. A logging relay acts as an intermediary, receiving log messages from hosts and forwarding them to the designated syslog server.

Consider a scenario similar to the one depicted in Figure 1 below.

                           .---------.
                           | Syslog- |
                           | Server  |
                           |         |
                           '----+----'
                                |.99
                                |
                             .--.-.
                            ( (    )__
                           (_,  \ ) ,_)  Untrusted Network
                             '-'--`--'   172.16.1.0/24
                                |
                                |vlan2: .1
                           .----+----.
                           | Syslog- |
                           | Relay   |
                           |         |
                           '----+----'
                                |vlan1: .1
                                |
                                |
               ---+-------------+-------------+--- 192.168.1.0/24
                  |             |             |
                  |.10          |.20          |.30
               .--+--.       .--+--.       .--+--.
               | H1  |       | H2  |       | H3  |
               '-----'       '-----'       '-----'

Figure 1: Example network topology with a syslog relay. The syslog relay receives log messages from hosts H1, H2, and H3, and forwards them to the syslog server.

In this example use-case we will focus on the configuration of a WeOS device to act as a the Syslog-Relay in the example topology. This entails configuring the device with a remote logging source, that will be combined with a remote logging destination.

The remote source will be configured to receive any UDP log message that reaches the device on the vlan1 interface. The remote destination will be configured to send log messages to the Syslog-Server located at 172.16.1.99, using TLS as the transport protocol.

Note

We assume that all the end devices are correctly configured to send their log messages to the Syslog-Relay at 192.168.1.1 using UDP as the transport protocol.

In the next section we will focus on the configuration of the Syslog-Relay device.

Configuration

We will focus on the configuration of the Syslog-Relay device. We take each step in turn, configuring the remote source, remote destination, and finally combining them in a sink configuration.

Prerequisites

We assume that the device is already configured with the necessary network settings, i.e. the IP addresses and the VLANs are correctly configured. For the example presented in the topology, we assume that the device has the following interface configuration:

vlan1: Configured with an IP address in the 192.168.1.0/24 subnet.
vlan2: Configured with an IP address in the 172.16.1.0/24 subnet.
hostname: Is set to Syslog-Relay.

Remote Source

We start with the configuration of the remote source, that we will call remote:

Syslog-Relay:/#> configure
Syslog-Relay:/config/#> logging
Syslog-Relay:/config/logging/#> source remote
Syslog-Relay:/config/logging/source-remote/#> type remote
Syslog-Relay:/config/logging/source-remote/#> transport udp
Syslog-Relay:/config/logging/source-remote/#> listen vlan1
Syslog-Relay:/config/logging/source-remote/#> keep-hostname
Syslog-Relay:/config/logging/source-remote/#> end
Syslog-Relay:/config/logging/#>

This source configuration will listen for any UDP log messages that reach the device on the vlan1 interface. We also enable the keep-hostname option to ensure that the hostname of the source is preserved in the log messages. Otherwise, when the messages are relayed to the syslog server, the hostname of the Syslog-Relay would be used instead. This option is useful for distinguishing between different sources when analyzing the log messages.

When we want to have a remote source, we must also ensure that we disable the secure-mode option. While this is enabled, the device will not accept any remote log messages. Disable it in the top level logging configuration context:

Syslog-Relay:/config/logging/#> no secure-mode

This configuration is enough to receive the log messages from the devices on the 192.168.1.0/24 subnet. The next step is to configure the destination.

Remote Destination

Next, in order to relay the log messages to the Syslog-Server we need to configure a remote destination. We will call this destination relay:

Syslog-Relay:/config/logging/#> destination relay
Syslog-Relay:/config/logging/destination-relay/#> type remote
Syslog-Relay:/config/logging/destination-relay/#> transport tls
Syslog-Relay:/config/logging/destination-relay/#> host name Syslog-Server
Syslog-Relay:/config/logging/destination-relay/#> peer-verify trusted
Syslog-Relay:/config/logging/destination-relay/#> certificate my-cert
Syslog-Relay:/config/logging/destination-relay/#> ca-certificate my-ca-cert
Syslog-Relay:/config/logging/destination-relay/#> end
Syslog-Relay:/config/logging/#>

In this configuration we specify that we will reach the remote destination at the hostname Syslog-Server using TLS as the transport protocol. In addition, we specify a certificate to use and verify the client against the server.

We have also specified a CA certificate to use for verifying the server certificate, which will be necessary because the peer-verify trusted option has been set. This option requires that the server must also provide a certificate to the client, which will be validated against the specified CA certificate.

Note

Any necessary certificates must be imported into the device before configuring the remote destination. This includes the CA certificate, the client certificate. However, for the client the need for a certificate depends on how the syslog server is configured, if it requires a certificate for the client then it must be imported into the device.

The client can also be configured the verify the server certificate, if the peer-verify trusted has been set. If this is the case the server also needs to provide a certificate to the client, that will be validated against the specified CA certificate.

We assume that we have already imported the necessary certificates into the device. Refer to the Certificate Management page for more information on how to import certificates into the device.

Note

Depending on how the certificates are generated, it may be necessary to specify the destination with its hostname. This is because the certificate may be generated with the hostname as the common name (CN). If this is the case, the hostname must be specified in the destination configuration.

In this example we have simply added the hostname as a static host entry in the device:

Syslog-Relay:/#> configure
Syslog-Relay:/config/#> dns
Syslog-Relay:/config/dns/#> host Syslog-Server 172.16.1.99
Syslog-Relay:/config/dns/#> leave
Applying configuration.
Configuration activated.  Remember "copy run start" to save to flash (NVRAM).
Syslog-Relay:/#>

This configuration should be enough to relay the log messages to the Syslog-Server.

Sink Configuration

The final step is to combine the source and destination in a sink configuration. We will call this sink to-relay:

Syslog-Relay:/config/logging/#> sink to-relay
Syslog-Relay:/config/logging/sink-to-relay/#> source remote
Syslog-Relay:/config/logging/sink-to-relay/#> destination relay
Syslog-Relay:/config/logging/sink-to-relay/#> end
Syslog-Relay:/config/logging/#>

The final configuration should look like this, when we check the overview from the top level logging configuration context:

Syslog-Relay:/config/logging/#> show
Console syslog   : Disabled
External logging : Disabled
Secure mode      : Disabled

╒ Sources ═══════════════════════════════════════════════════════════════════╕
│NAME                                                                        │
│Type: local      USERSPACE  KERNEL                                          │
│local            Yes        Yes                                             │
│Type: remote     TRANSPORT  LISTEN               PORT   FORMAT              │
│remote           udp        vlan1 (192.168.1.1)  514    rfc5424             │
└────────────────────────────────────────────────────────────────────────────┘

╒ Destinations ══════════════════════════════════════════════════════════════╕
│NAME                                                                        │
│Type: remote     TRANSPORT  TO-HOST        PORT   FORMAT                    │
│relay            tls        Syslog-Server  6514   rfc5424                   │
└────────────────────────────────────────────────────────────────────────────┘

╒ Filters ═══════════════════════════════════════════════════════════════════╕
│NAME             INVERT        TYPE  FILTER                                 │
├────────────────────────────────────────────────────────────────────────────┤
│No filter(s) currently configured...                                        │
└────────────────────────────────────────────────────────────────────────────┘

╒ Sinks ═════════════════════════════════════════════════════════════════════╕
│NAME             SOURCES       DESTINATIONS   OP FILTERS                    │
│to-relay         remote        relay                                        │
└────────────────────────────────────────────────────────────────────────────┘

Syslog-Relay:/config/logging/#> leave

Now when we leave the configuration context, the configuration will be applied, and the device should start relaying the UDP log messages to the Syslog-Server.

Syslog-Relay:/config/logging/#> leave
Applying configuration.
Configuration activated.  Remember "copy run start" to save to flash (NVRAM).
Syslog-Relay:/#>

Verification

If everything has been configured correctly, we should be able to check the status of the logging source, with it looking something like this:

Syslog-Relay:/#> show logging source
╒ Sources ═════════════════════════════════════════════════════════════════════╕
│TYPE        NAME             DESCRIPTION       PROCESSED                      │
│remote      sink-remote      192.168.1.1       15                             │
│unix-dgram  local-userspace  /dev/log          358                            │
│file        local-kernel     /dev/kmsg         1                              │
└──────────────────────────────────────────────────────────────────────────────┘
Syslog-Relay:/#>

We should see that we have a remote source active called sink-remote. If messages are being received, the PROCESSED counter should be increasing.

Next we can check the status of the logging destination:

Syslog-Relay:/#> show logging destination
╒ Remote Destinations ═══════════════════════════════════════════════════════════════════╕
│NAME        REMOTE_PROTOCOL_IP_PORT  PROCESSED  DROPPED  WRITTEN  QUEUED  MEMORY_USAGE  │
│sink-relay  tls,Syslog-Server:6514   15         0        15       0       0             │
└────────────────────────────────────────────────────────────────────────────────────────┘
Syslog-Relay:/#>

Using tcpdump on the Syslog-Relay we can verify that the log messages are being received on the vlan1 interface:

Syslog-Relay:/#> tcpdump iface vlan1 expr "proto 17 and port 514"
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vlan1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:50:34.158829 IP 192.168.1.20.35038 > 192.168.1.1.syslog: SYSLOG user.notice, length: 76
09:50:37.414279 IP 192.168.1.30.45581 > 192.168.1.1.syslog: SYSLOG user.notice, length: 76
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
Syslog-Relay:/#>

If we see log messages being received on the vlan1 interface, then the we can check that the messages are being relayed to the Syslog-Server by using tcpdump on the interface towards the Syslog-Server, interface vlan2:

Syslog-Relay:/#> tcpdump iface vlan2 expr "proto 6 and port 6514"
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vlan2, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:54:26.278686 IP 172.16.1.1.41217 > server1.6514: Flags [P.], seq 1392486109:1392486134, ack 36688746, win 4005, length 25
09:54:26.279468 IP server1.6514 > 172.16.1.1.41217: Flags [.], ack 25, win 6560, length 0
09:54:26.279493 IP 172.16.1.1.41217 > server1.6514: Flags [P.], seq 25:123, ack 1, win 4005, length 98
09:54:26.280229 IP server1.6514 > 172.16.1.1.41217: Flags [.], ack 123, win 6560, length 0
09:54:28.618872 IP 172.16.1.1.41217 > server1.6514: Flags [P.], seq 123:148, ack 1, win 4005, length 25
09:54:28.620004 IP server1.6514 > 172.16.1.1.41217: Flags [.], ack 148, win 6560, length 0
09:54:28.620042 IP 172.16.1.1.41217 > server1.6514: Flags [P.], seq 148:246, ack 1, win 4005, length 98
09:54:28.620751 IP server1.6514 > 172.16.1.1.41217: Flags [.], ack 246, win 6560, length 0
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel
Syslog-Relay:/#>

If everything is working correctly, we should see something similar to the above.

Troubleshooting

The following is an example of some troubleshooting steps that can be taken if the configuration is not working as expected:

On the Syslog-Relay, create a local file destination and add it to the same sink that we created to send messages to the Syslog-Server. This will allow us to verify that the messages are at least reaching the Syslog-Relay correctly, and are being processed by the sink.

Syslog-Relay:/#> configure
Syslog-Relay:/config/#> logging
Syslog-Relay:/config/logging/#> destination local-file
Syslog-Relay:/config/logging/destination-local-file/#> type file
Syslog-Relay:/config/logging/destination-local-file/#> media internal
Syslog-Relay:/config/logging/destination-local-file/#> file-name syslog-relay-debug.log
Syslog-Relay:/config/logging/destination-local-file/#> end
Syslog-Relay:/config/logging/#> sink to-relay
Syslog-Relay:/config/logging/sink-to-relay/#> destination local-file
Syslog-Relay:/config/logging/sink-to-relay/#> end
Syslog-Relay:/config/logging/#> leave
Applying configuration.
Configuration activated.  Remember “copy run start” to save to flash (NVRAM).

Now any UDP log messages that are received by the Syslog-Relay should be written to the file syslog-relay-debug.log. We can check this file to see if the messages are being received correctly:

Syslog-Relay:/#> show log://syslog-relay-debug.log
May 27 10:05:49 H2 admin: test 1
May 27 10:05:51 H3 admin: test 2
________________________________________________________________________
log://syslog-relay-debug.log                    Mon May 27 10:05:51 2024

Check the local log files on the Syslog-Relay to see if there are any errors or warnings that may indicate a problem with the configuration, especially regarding the TLS connection.
Perform a similar check on the Syslog-Server to ensure that it is correctly configured to receive log messages over TLS. Verify if there are any errors or warnings in the log files that may indicate a problem with the TLS connection.
Verify that the correct certificates have been imported into the device and that the correct certificates are being used in the configuration.
Ensure that the correct port numbers are being used for the transport protocols (UDP and TLS), if either any of the clients or the server are using a different non-standard port number.

Ensure that the configured host can be reached, using for example the ping command.

Syslog-Relay:/#> ping Syslog-Server
Press Ctrl-C to abort PING Syslog-Server (172.16.1.99) 56(84) bytes of data.
64 bytes from Syslog-Server (172.16.1.99): icmp_seq=1 ttl=64 time=1.22 ms
64 bytes from Syslog-Server (172.16.1.99): icmp_seq=2 ttl=64 time=2.56 ms
^C
— Syslog-Server ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.223/1.891/2.560/0.668 ms
client1:/#>

If the host cannot be reached, then the device will not be able to relay the log messages to the syslog server.

If a hostname is used to specify the destination, ensure that the hostname can be resolved correctly.

Conclusion

In this document we have presented an example use-case for a syslog relay setup. We have configured a WeOS device to act as a syslog relay, receiving log messages from hosts using UDP and forwarding them to a syslog server using TLS.

This setup is particularly useful for hosts that lack the capability to send encrypted log messages. By using a syslog relay, we can ensure that the log messages are secure during transit, even when transmitted over an untrusted network.