Notes from the field: Load balancing a UDP Syslog workload with NetScaler.

Introduction

The following sections talk about a use case for load balancing something simple, UDP. When first looking at this problem, it was not obvious to me quite what the issue was. As I simply thought that setting up a load-balancing virtual server for UDP would be trivial and would do the job.

The problem was that it did not behave as the customer wanted. Also, finding a simple fix was not obvious!

Problem statement: What is Syslog and why is it important? This Syslog problem was raised by the head of IT security, he considered Syslog to be very important for him as it provided a way to track what activity his devices were seeing on the network. He stated that in his security role he considered Syslog as mission-critical.

As part of the deployment, the customer used NetScaler to load balance the Syslog source devices onto the array of Syslog servers, which he could then query. The network looked like this diagram below. It should be noted that the diagram looks to show a two-arm deployment. It is only shown as it is to keep it simple. The appliance was deployed as a one-arm deployment.

Device1, 2,3 and 4 on the left-hand side are all Syslog source systems, they all send their entries to a VIP defined on the NetScaler which then load balances the traffic onto two or more Syslog servers. The customer raised a question about how NetScaler actually load balances UDP-based Syslog traffic.

• His NetScaler setup was resulting in uneven load balancing on the Syslog servers in some cases. This was unacceptable to the customer.
• He had also seen packet fragmentation in other configurations which then resulted in Syslog traffic being damaged, so losing live data. Another unacceptable situation. This packet fragmentation happened when the MTU was exceeded.

We need to make it clear that we are performing UDP load balancing and not SYSLOG load balancing. We could be load-balancing SYSLOG, DNS, or NTP traffic, which all make use of UDP traffic.

Naturally, this point about UDP is significant, as the traffic is ‘lightweight’ and lacks a lot of the options for traffic control that are present with TCP.

Impact: As stated, Syslog is important in this use case for this client. Having a good way to evenly load balance the traffic will ensure that Syslog does not overload any one Syslog server

What have they tried?Using session less load-balancing.

This first option is to configure the NetScaler to use Session Less load balancing. This re-evaluates the load balancing decision individually for each UDP packet, which will ensure that each request gets load balanced based on the selected load balancing method, regardless of the Source IP/Source Port/Destination Port tuple.

Result: It doesn’t work as expected, as fragmented packets resulted in lost data. There are also a number of issues with session less, one of which required that the appliance is deployed in two-arm mode. This alone was a significant piece of work if undertaken by the client due to the number of appliances that the customer had.

Using persistence

Option two was to look to use session persistence to ‘stick’ a network device to a particular Syslog server. The problem with this approach was that it resulted in this.