Let's Improve the Little Load Balancer with XDP

Previously…

#

In the previous episode, we saw how to distribute the load of a ping across h2 and h3. However, the ping command detects that the IP comes from h2 or h3 and not from the load balancer:

PING 10.0.0.10 (10.0.0.10) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=0.088 ms (DIFFERENT ADDRESS!)
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.098 ms (DIFFERENT ADDRESS!)
64 bytes from 10.0.0.3: icmp_seq=3 ttl=64 time=0.082 ms (DIFFERENT ADDRESS!)
64 bytes from 10.0.0.2: icmp_seq=4 ttl=64 time=0.088 ms (DIFFERENT ADDRESS!)
64 bytes from 10.0.0.2: icmp_seq=5 ttl=64 time=0.081 ms (DIFFERENT ADDRESS!)

We will now see why this is annoying and how we can solve the problem.

But what’s the problem?

#

Taking a step back

#

In the network we’ve built so far in this guide:

Everything is flat, everyone can communicate with everyone else.

In general, clients, a load balancer, and backends are in different networks. So the solution in the previous episode only works if the backends can route packets to the client, which is not necessarily the case for security reasons.

Furthermore, this “works” for ICMP. But for another protocol such as TCP, it would not work: During the connection phase (3-way handshake), the client would detect that the SYN sent to the load balancer does not have the same IP as the server’s SYN-ACK packet, and the connection would not proceed any further.

The current solution is therefore not ideal. How does a “real” load balancer work? The return path would need to be symmetrical to the forward path to avoid this problem.

Currently, the forward path (echo request) is: client → load balancer → backend

And as the return path (echo reply): backend → client

The return path should therefore be: backend → load balancer → client

This way, the client will believe that it is pinging the load balancer and not one of the two backends.

So what do we do?

#

One possible solution would be to install an XDP program on veth3 to redirect to lb. But this would require modifying each client’s interface, and we don’t necessarily have access to the client (imagine you want to create a load balancer for a cloud).

The solution is rather to make the backends believe that the source IP is not that of the client (10.0.0.1) but that of lb. This is called SNAT (Source Network Address Translation).

The first idea would be to continue the XDP program from the previous episode by adding SNAT.

But that doesn’t work: if we rewrite the source IP at the same time as the destination, the packet that just arrived on veth6 would appear to come from… veth6 itself.

To get around this problem, we need to create another XDP program attached to the next interface: veth7. We can then modify the source IP without creating any inconsistencies.

But that’s not all! You have to manage the return. Otherwise, the packet would stop at lb. So, on the return, you have to modify the packet that arrived in order to reach the client.

So here’s what we’re going to do today:

• write an XDP SNAT program on veth7
• handle the return path

Let’s start by creating the program that will change the source IP.

Let’s create the load balancer on the SNAT side

#

Creating the XDP Hello World

#

Let’s generate the Aya program:

cargo generate --name lb-xdp-snat \
               -d program_type=xdp \
               -d default_iface=veth7 \
               https://github.com/aya-rs/aya-template
cd lb-xdp-snat

In anticipation, we will use additional crates:

• network-types for Rust structures of level 1, 2, and 3 headers
• blog-xdp, the crate from this blog, so we don’t have to copy/paste previously written helper functions

So let’s modify the lb-xdp-snat-ebpf/Cargo.toml file and add the following to the dependencies section:

network-types = "0.1.0"
blog-xdp = { git = "https://github.com/littlejo/blog-xdp" }

Let’s build the “hello world” program and test it quickly:

cargo run

We run ping -c1 10.0.0.10, which returns received a packet on the cargo run side.

Keeping only pings

#

Now, we’re going to retrieve only the ICMP packets.

We’ll repeat everything we did previously to achieve this. We’ll modify the lb-xdp-snat-ebpf/src/main.rs file:

use blog_xdp::helper::{iph_csum, log_icmp, filter_icmp};

fn try_lb_xdp_snat(ctx: XdpContext) -> Result<u32, u32> {
    let (ipv4hdr, icmphdr) = match filter_icmp(&ctx) {
        Some(x) => x,
        None => return Ok(xdp_action::XDP_PASS),
    };
    log_icmp(&ctx, ipv4hdr, icmphdr);
    Ok(xdp_action::XDP_PASS)
}

The message src=10.0.0.10 dst=10.0.0.1 (0) is displayed only when an ICMP packet is sent. This is where we will develop the second part of the load balancer.

Let’s check that the eBPF program is running:

cargo run

On another terminal, let’s ping the future load balancer:

ping -c 2 10.0.0.10

On the cargo run terminal, the following is displayed:

[INFO  blog_xdp::helper] src=10.0.0.10 dst=10.0.0.1 (0)
[INFO  blog_xdp::helper] src=10.0.0.10 dst=10.0.0.1 (0)

Now that we have filtered to only include ICMP packets, let’s see how we can modify the address.

Changing the source IP

#

This is where it gets interesting.

All we need to do is modify one IP address with that of the load balancer to tell the backends (h2 and h3) to send requests back to the load balancer.

First, let’s see what happens when both eBPF programs are running at the same time.

Let’s install the XDP program we just created:

cargo run 

In another terminal, let’s install the XDP DNAT program we created in the previous episode:

ip netns exec lb cargo run 

ping -c 2 10.0.0.10

Here’s what we’ve done:

Let’s look at the logs for the SNAT program:

[INFO  blog_xdp::helper] src=10.0.0.10 dst=10.0.0.1 (5)
[INFO  blog_xdp::helper] src=10.0.0.1 dst=10.0.0.2 (8)
[INFO  blog_xdp::helper] src=10.0.0.10 dst=10.0.0.1 (5)
[INFO  blog_xdp::helper] src=10.0.0.1 dst=10.0.0.2 (8)

• Number 5 corresponds to a redirect request: we don’t care about that.
• Number 8 corresponds to an echo request.

We will handle only echo requests and echo replies, so we’ll add the following filter:

let type_ = unsafe { (*icmphdr).type_ };
match type_ {
    8 => {},
    0 => {},
    _ => return Ok(xdp_action::XDP_PASS),
}

We will now create the SNAT: we will change the source IP to make the backend believe that the original packet comes from the load balancer and not from the client:

unsafe {
        (*ipv4hdr).src_addr = [10, 0, 0, 10];
        let chksum = iph_csum(ipv4hdr);
        (*ipv4hdr).set_checksum(chksum);
}

Here’s what we did at the packet level:

The backend sends the packet back to the load balancer.

The main code is then:

fn try_lb_xdp_snat(ctx: XdpContext) -> Result<u32, u32> {
    let (ipv4hdr, icmphdr) = match filter_icmp(&ctx) {
        Some(x) => x,
        None => return Ok(xdp_action::XDP_PASS),
    };
    let type_ = unsafe { (*icmphdr).type_ };

    match type_ {
        8 => {},
        0 => {},
        _ => return Ok(xdp_action::XDP_PASS),
    }

    unsafe {
        (*ipv4hdr).src_addr = [10, 0, 0, 10];
        let chksum = iph_csum(ipv4hdr);
        (*ipv4hdr).set_checksum(chksum);
    }

    log_icmp(&ctx, ipv4hdr, icmphdr);
    Ok(xdp_action::XDP_PASS)
}

Let’s run again the two XDP programs and see what happens. It no longer pings, but we have the following log on the SNAT program:

[INFO  blog_xdp::helper] src=10.0.0.10 dst=10.0.0.2 (8)
[INFO  blog_xdp::helper] src=10.0.0.10 dst=10.0.0.3 (0)
[INFO  blog_xdp::helper] src=10.0.0.10 dst=10.0.0.2 (8)
[INFO  blog_xdp::helper] src=10.0.0.10 dst=10.0.0.2 (0)

The load balancer receives the return packet!

Furthermore, for the return path, we have already made the ping initiator believe that the source IP will be the load balancer and not the backend.

All that remains is to modify the program so that it returns to the original client’s IP address.

Let’s handle the return path

#

We’re not far from our goal! Now we just need this part to work:

We have two options:

• modify the XDP program on veth6 so that the destination IP is that of the client
• modify the XDP program on veth7 so that the destination IP is that of the client

Both are valid. Since we have already modified the destination address on veth6, the logical thing to do is to do the same on veth6.

So, for the rest of this article, we will only modify the program from the previous episode, lb_xdp(). The SNAT program code is now complete.

Without connection tracking

#

So if the packet is an echo reply, we need to modify the destination address. However, the eBPF program does not currently know this address. We will have to help it. Since we are testing on the host, we will first simplify things by hard-coding the host address (10.0.0.1):

let type_ = unsafe { (*icmphdr).type_ };

if type_ == 0 {
    unsafe {
        (*ipv4hdr).dst_addr = [10, 0, 0, 1];
        let chksum = iph_csum(ipv4hdr);
        (*ipv4hdr).set_checksum(chksum);
    }
}

Let’s test it: it pings fine!

At the veth6 interface:

[INFO  lb_xdp] src=10.0.0.1 dst=10.0.0.3 (8)
[INFO  lb_xdp] src=10.0.0.3 dst=10.0.0.1 (0)

At the veth7 interface:

[INFO  blog_xdp::helper] src=10.0.0.10 dst=10.0.0.3 (8)
[INFO  blog_xdp::helper] src=10.0.0.10 dst=10.0.0.1 (0)

Let’s improve the lb_dnat() function:

#[inline(always)]
fn lb_dnat(ipv4hdr: *mut Ipv4Hdr, icmphdr: *const IcmpHdr) {
    let dst_addr_mod = match unsafe {(*icmphdr).type_} {
        0 => [10, 0, 0, 1],
        _ => {
            let backend_index = unsafe { bpf_get_prandom_u32() % 2 + 2 };
            [10, 0, 0, backend_index as u8]
        }
    };

    unsafe {
        (*ipv4hdr).dst_addr = dst_addr_mod;
        let chksum = iph_csum(ipv4hdr);
        (*ipv4hdr).set_checksum(chksum);
    }
}

And the main function then becomes:

fn try_lb_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let (ipv4hdr, icmphdr) = match filter_icmp(&ctx) {
        Some(x) => x,
        None => return Ok(xdp_action::XDP_PASS),
    };
    lb_dnat(ipv4hdr, icmphdr);
    log_icmp(&ctx, ipv4hdr, icmphdr);

    Ok(xdp_action::XDP_PASS)
}

With connection tracking

#

Hidden headers

#

This works if we ping from IP 10.0.0.1, but if we ping from another namespace, it doesn’t work because we hard-coded it. How can we avoid hard-coding this part of the code? The eBPF program needs to remember which source IP was used so that it can send it back.

The idea is to create an eBPF map that records the source IP that entered veth6. This allows it to be sent back to the correct IP during the echo reply.

However, there is a problem: if several clients ping at the same time, we need to be able to identify who to send the ping back to. To do this, we need to find a piece of data that is common to all pings.

Let’s recall what the ICMP header contains:

• Type: on arrival it is always 8 (echo request) and on exit always 0 (echo reply). This won’t help us.
• Code: this is a subtype that specifies the type; it will always be equal to 0 for type 8 and 0. This won’t help us.
• Checksum: the checksum will change between the incoming packet and the outgoing packet. This won’t help us.

So there is nothing in the header that can identify anything in common.

But if you read RFC 792, there are other headers that vary depending on the type.

So we can see that there are some common elements:

• Identifier: an identifier created by the client
• Sequence Number: the sequence number that increments with each ping (you know, the icmp_seq=?).

One approach would therefore be to create an eBPF map: {[id, seq] → source ip}

How can these headers be retrieved in Rust?

#

If you look at the network-types documentation:

We can see that there is data included in the icmp header. Does it contain Identifier and Sequence Number? Yes:

The data field contains type-specific data such as echo identifiers/sequence numbers, redirect gateway addresses, or pointers to errors in received packets.

There are even methods to retrieve them easily:

So we have the echo_id() and echo_sequence() methods. To retrieve them, we can write this:

let id_: u16 = unsafe { (*icmphdr).echo_id().map_err(|_| 0u32)? };
let seq_: u16 = unsafe { (*icmphdr).echo_sequence().map_err(|_| 0u32)? };

To create the eBPF map:

use aya_ebpf::maps::LruHashMap;
use aya_ebpf::macros::map;

#[map]
pub static SEQ_ID: LruHashMap<[u16; 2], [u8; 4]> = LruHashMap::with_max_entries(16, 0);

When the packet arrives (echo request), we will insert the IP into the map:

let src_addr = unsafe { (*ipv4hdr).src_addr };
let _ = SEQ_ID.insert(&[seq_, id_], &src_addr, 0);

When the packet returns (echo reply), we will retrieve the IP from the map to know where to send it:

let src_ip = unsafe {*SEQ_ID.get(&[seq_, id_]).ok_or(0u32)?}

After adapting it for the lb_dnat function, we end up with:

use aya_ebpf::maps::LruHashMap;
use aya_ebpf::macros::map;
#[map]
pub static SEQ_ID: LruHashMap<[u16; 2], [u8; 4]> = LruHashMap::with_max_entries(16, 0);

#[inline(always)]
fn lb_dnat(ipv4hdr: *mut Ipv4Hdr, icmphdr: *const IcmpHdr) -> Result<u32,u32> {
    let id_: u16 = unsafe { (*icmphdr).echo_id().map_err(|_| 0u32)? }; //ADD
    let seq_: u16 = unsafe { (*icmphdr).echo_sequence().map_err(|_| 0u32)? }; //ADD

    let dst_addr_mod = match unsafe {(*icmphdr).type_} {
        0 => unsafe {*SEQ_ID.get(&[seq_, id_]).ok_or(0u32)?}, //INSTEAD OF 10.0.0.1
        _ => {
            let src_addr = unsafe { (*ipv4hdr).src_addr };
            let _ = SEQ_ID.insert(&[seq_, id_], &src_addr, 0); //ADD
            let backend_index = unsafe { bpf_get_prandom_u32() % 2 + 2 };
            [10, 0, 0, backend_index as u8]
        }
    };

    unsafe {
        (*ipv4hdr).dst_addr = dst_addr_mod;
        let chksum = iph_csum(ipv4hdr);
        (*ipv4hdr).set_checksum(chksum);
    }
    Ok(0)
}

And the main code is slightly modified:

fn try_lb_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let (ipv4hdr, icmphdr) = match filter_icmp(&ctx) {
        Some(x) => x,
        None => return Ok(xdp_action::XDP_PASS),
    };
    lb_dnat(ipv4hdr, icmphdr)?; //Just a ?
    log_icmp(&ctx, ipv4hdr, icmphdr);

    Ok(xdp_action::XDP_PASS)
}

One last test

#

You can test with parallel pings, for example:

hping3 --icmp-ipid 1 10.0.0.10 --count 5 & \
       ip netns exec h3 hping3 --icmp-ipid 3 10.0.0.10 --count 5 & \
       ip netns exec h2 hping3 --icmp-ipid 2 10.0.0.10 --count 5

What did we do, anyway?

#

To summarize what we set up in these two parts:

• When receiving an ICMP Echo Request (ping from the client):
  • we rewrote the destination address with an IP from one of the backends → DNAT
  • we stored the client’s IP in an eBPF map in anticipation of the return path → CONNTRACK
  • We rewrote the source address with the IP of the load balancer so that the return path goes through the load balancer → SNAT
• Upon receiving an ICMP Echo Reply (response from the backend):
  • We retrieved the client’s IP from the eBPF map → CONNTRACK
  • We rewrote the destination address with the client’s IP → DNAT
  • We rewrote the source address with the load balancer’s IP so that the client does not detect that the IP has changed → SNAT

This episode is now complete!

We have seen the basics of creating an XDP load balancer: DNAT, SNAT, CONNTRACK, and checksum.

If you want to go further in creating a load balancer, you can:

• implement a UDP and TCP load balancer
• implement a load balancer on a virtual machine to get closer to a production environment

I hope you enjoyed this series of articles!

You can find all the small programs created for these articles: