Let's Create a Little Load Balancer with XDP

We saw what an XDP program is in part one: it can be used to load balance packets.

In this part, I will show you how to create an XDP program that load balance ICMP packets. We’ll do that using the Rust Aya framework.

Follow the guide!

I assume you are already in an environment for developing with Aya. If not, you can use the Killercoda lab:

What Shall We Do Now?

#

ICMP Load Balancer

#

We are going to create an ICMP (Internet Control Message Protocol) load balancer. So when we ping the load balancer, we are actually pinging another server called the backend.

Let’s briefly review how the ICMP protocol works. The client sends an echo request ICMP packet to a server. The server responds with another ICMP packet of the echo reply type.

To learn more about ICMP, you can visit this page.

Why not TCP or UDP?

#

It’s true that an ICMP load balancer may not seem particularly useful. However, I think it’s the best place to start because it’s the easiest to create. Furthermore, most of the concepts we’ll be discussing in this article will still be useful for these load balancers.

As a guide, here is the order of difficulty in my opinion:

1. ICMP load balancer
2. UDP load balancer
3. TCP load balancer

Let’s naively create the load balancer

#

Setting up the dev environment

#

As we saw in the first part, I suggest installing namespaces:

git clone https://github.com/littlejo/eunomia.dev
cd eunomia.dev/docs/tutorials/42-xdp-loadbalancer/
./setup.sh

Let’s recall what this generated:

We will create an XDP load balancer at the veth6 interface level that will distribute the load between h2 and h3. We will change the destination address in XDP from 10.0.0.10 to 10.0.0.2 or 10.0.0.3:

This is called DNAT (Destination Network Address Translation).

To start with, we will redirect network packets only to h2 (10.0.0.2), which would look like this:

Creating the XDP Hello World

#

Let’s generate the Aya program:

cargo generate --name lb-xdp \
               -d program_type=xdp \
               -d default_iface=veth6 \
               https://github.com/aya-rs/aya-template
cd lb-xdp/

In anticipation, we will use additional crates:

• network-types for Rust structures of level 1, 2, and 3 headers;
• blog-xdp, the crate from this blog, so we don’t have to copy/paste previously written helper functions.

So let’s modify the lb-xdp-ebpf/Cargo.toml file and add the following to the dependencies section:

network-types = "0.1.0"
blog-xdp = { git = "https://github.com/littlejo/blog-xdp" }

Let’s build the hello world program:

cargo build

Let’s test it quickly:

ip netns exec lb cargo run

We run ping -c1 10.0.0.10, which returns received a packet on the cargo run side.

Keeping only pings

#

Now, we’re going to modify the Aya program on the kernel side: lb-xdp-ebpf/src/main.rs. We’re going to retrieve only the ICMP packets.

We have already written similar code in a previous episode, so here are the results of the changes:

use network_types::{
    eth::{EthHdr, EtherType},
    ip::{Ipv4Hdr, IpProto},
    icmp::IcmpHdr,
};
use blog_xdp::helper::ptr_at;

#[inline(always)]
fn filter_icmp(ctx: &XdpContext) -> Option<(*const Ipv4Hdr, *const IcmpHdr)> {
    let ethhdr: *const EthHdr = ptr_at(ctx, 0).ok()?;

    match unsafe { (*ethhdr).ether_type() } {
            Ok(EtherType::Ipv4) => {}
            _ => return None,
        }

    let ipv4hdr: *const Ipv4Hdr = ptr_at(ctx, EthHdr::LEN).ok()? ;
    let icmphdr: *const IcmpHdr = ptr_at(&ctx, EthHdr::LEN + Ipv4Hdr::LEN).ok()? ;

    match unsafe { (*ipv4hdr).proto } {
         IpProto::Icmp => {}
         _ => return None,
    }

    Some((ipv4hdr, icmphdr))
}

fn try_lb_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let (ipv4hdr, icmphdr) = match filter_icmp(&ctx) {
        Some(x) => x,
        None => return Ok(xdp_action::XDP_PASS),
    };
    let dst_addr = unsafe { (*ipv4hdr).dst_addr };
    let src_addr = unsafe { (*ipv4hdr).src_addr };

    info!(&ctx,
        "src={:i}, dst={:i}", src_addr, dst_addr);
    Ok(xdp_action::XDP_PASS)
}

The message of type: src=10.0.0.1, dst=10.0.0.10 is displayed only when an ICMP packet is sent. This is where we will develop the load balancer.

Let’s check:

ip netns exec lb cargo run

On another terminal, let’s ping the future load balancer:

ping -c 2 10.0.0.10

On the cargo run terminal, the following is then displayed:

[INFO  lb_xdp] src=10.0.0.1, dst=10.0.0.10
[INFO  lb_xdp] src=10.0.0.1, dst=10.0.0.10

Feel free to test other protocols to verify that nothing is displayed.

Improving logs

#

To better understand how this load balancer will work, it is useful to display a little more detail about the received packet.

Let’s look at the contents of an ICMP header:

• Type: the type of ICMP request (8 corresponds to echo request and 0 corresponds to echo reply)
• Code: the code specifies the type (not useful for echo request and echo reply)
• Checksum: for packet integrity

To retrieve these elements, we can use the network-types crate.

So to get the ICMP packet type, you can do it this way:

let type_ = unsafe { (*icmphdr).type_ };

Let’s create a function dedicated to logs:

#[inline(always)]
fn log_icmp(ctx: &XdpContext, ipv4hdr: *const Ipv4Hdr, icmphdr: *const IcmpHdr) {
    let type_ = unsafe { (*icmphdr).type_ };
    let dst_addr = unsafe { (*ipv4hdr).dst_addr };
    let src_addr = unsafe { (*ipv4hdr).src_addr };

    info!(ctx,
        "src={:i} dst={:i} ({})", src_addr, dst_addr, type_);
}

The main code becomes:

fn try_lb_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let (ipv4hdr, icmphdr) = match filter_icmp(&ctx) {
        Some(x) => x,
        None => return Ok(xdp_action::XDP_PASS),
    };
    log_icmp(&ctx, ipv4hdr, icmphdr); //MODIFY
    Ok(xdp_action::XDP_PASS)
}

Let’s start ip netns exec lb cargo run again and then ping:

[INFO  lb_xdp] src=10.0.0.1 dst=10.0.0.10 (8)

We can now verify that the ICMP packet is indeed an echo request (8).

Changing the destination IP

#

This is where it gets interesting. Remember that the goal is to achieve the following first:

We therefore need to replace the destination address with 10.0.0.2:

let backend_ip = [10, 0, 0, 2];
unsafe {
    (*ipv4hdr).dst_addr = backend_ip;
}

This is not possible because the variable ipv4hdr is not mutable. We therefore need to replace const with mut:

let ipv4hdr: *const Ipv4Hdr = ptr_at(&ctx, EthHdr::LEN)? ;

The function ptr_at returns a Result<*const T, u16>. We could recreate an almost identical function but with Result<*mut T, u16> as the return value. But there is a more appropriate way by taking the original function and casting the type:

#[inline(always)]
fn ptr_at_mut<T>(ctx: &XdpContext, offset: usize) -> Result<*mut T, u16> {
    let ptr = ptr_at::<T>(ctx, offset)?;
    Ok(ptr as *mut T)
}

We then end up with the following main code:

#[inline(always)]
fn filter_icmp(ctx: &XdpContext) -> Option<(*mut Ipv4Hdr, *const IcmpHdr)> { //MODIFY
    let ethhdr: *const EthHdr = ptr_at(ctx, 0).ok()?;

    match unsafe { (*ethhdr).ether_type() } {
            Ok(EtherType::Ipv4) => {}
            _ => return None,
        }

    let ipv4hdr: *mut Ipv4Hdr = ptr_at_mut(ctx, EthHdr::LEN).ok()? ; //MODIFY
    let icmphdr: *const IcmpHdr = ptr_at(&ctx, EthHdr::LEN + Ipv4Hdr::LEN).ok()? ;

    match unsafe { (*ipv4hdr).proto } {
         IpProto::Icmp => {}
         _ => return None,
    }

    Some((ipv4hdr, icmphdr))
}

fn try_lb_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let (ipv4hdr, icmphdr) = match filter_icmp(&ctx) {
        Some(x) => x,
        None => return Ok(xdp_action::XDP_PASS),
    };
    let backend_ip = [10, 0, 0, 2]; //ADD
    unsafe { //ADD
        (*ipv4hdr).dst_addr = backend_ip; //ADD
    } //ADD

    log_icmp(&ctx, ipv4hdr, icmphdr);
    Ok(xdp_action::XDP_PASS)
}

Let’s test the compilation and installation of the eBPF program:

ip netns exec lb cargo run

No problem.

Let’s test the ping:

ping -c 2 10.0.0.10

In cargo run, we can clearly see the packets with the new destination:

[INFO  lb_xdp] src=10.0.0.1 dst=10.0.0.2 (8)
[INFO  lb_xdp] src=10.0.0.1 dst=10.0.0.2 (8)

BUT it no longer pings:

PING 10.0.0.10 (10.0.0.10) 56(84) bytes of data.

--- 10.0.0.10 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1006ms

We seem both close to the goal and far from it. Let’s see where the problem might be coming from.

Let’s fix the bug

#