Let's Build a Tiny Firewall with XDP

We saw what an XDP program is in part one: it can be a way to filter a network stream.

In this part, I will take you on a brief tour of the different headers and show you how filtering can be done. We will see how to redevelop xdp-filter with the Rust Aya framework.

Follow the guide!

I assume you are already in an environment for developing with Aya. If not, you can use the Killercoda lab:

Let’s build an XDP hello world program

#

Let’s recreate the development environment

#

As we saw in the previous section, I suggest installing namespaces:

git clone https://github.com/littlejo/eunomia.dev
cd eunomia.dev/docs/tutorials/42-xdp-loadbalancer/
./setup.sh

Let’s recall what we created:

Let’s create the “hello world” program

#

As we did with xdp-filter, we will install the XDP program on veth0.

Let’s generate the Aya program:

cargo generate --name browser-xdp \
               -d program_type=xdp \
               -d default_iface=veth0 \
               https://github.com/aya-rs/aya-template

Let’s build and install the “hello world” program:

cd browser-xdp/
cargo run

On another terminal, let’s test the program:

ip netns exec lb ping -c 2 10.0.0.1

On the cargo run terminal, we can clearly see two packets:

[INFO  browser_xdp] received a packet
[INFO  browser_xdp] received a packet

So every time the interface receives a packet, the XDP program is launched.

Now we’ll see how to stop a packet and what type of packet we’re dealing with.

The basics of creating an XDP program

#

XDP actions

#

Let’s first look at the code generated by cargo.

The most important file is this one: browser-xdp-ebpf/src/main.rs, i.e., the kernel-side code.

Overall, the code looks like the code generated for other types of eBPF programs such as Tracepoints:

#[xdp]
pub fn browser_xdp(ctx: XdpContext) -> u32 {
    match try_browser_xdp(ctx) {
        Ok(ret) => ret,
        Err(_) => xdp_action::XDP_ABORTED,
    }
}

fn try_browser_xdp(ctx: XdpContext) -> Result<u32, u32> {
    info!(&ctx, "received a packet");
    Ok(xdp_action::XDP_PASS)
}

However, there is one small difference: xdp_action::*.

This determines what the packet does once it has finished traversing the eBPF program.

There are five different actions:

• XDP_ABORTED: Stops the packet with an error
• XDP_DROP: Stops the packet silently
• XDP_PASS: Lets the packet continue through the network stack.
• XDP_REDIRECT: Redirects the packet to another interface or to an AF_XDP socket attached to an interface.
• XDP_TX: Redirects the packet to the same interface

In this article, we will only use XDP_PASS and XDP_DROP.

The XDP context and its methods

#

As with all other eBPF programs, to go beyond hello world, you need to understand how to work with the context.

Let’s take a look at the documentation:

Unlike previous types of programs where only one method was used, there are four different methods for XDP:

• data() returns the memory address of the start of the network packet.
• data_end() returns the memory address of the end of the network packet.
• metadata() returns the memory address of the start of the XDP metadata linked to the Linux kernel or driver.
• metadata_end() returns the memory address of the end of the metadata. In the same way as data_end(),

In this article, we will only use the data() and data_end() methods to retrieve the different headers.

How to retrieve the headers?

#

Let’s recall the following diagram:

• data() is located at the beginning of the ethHdr header
• data_end() is located at the end of the payload

How do we retrieve the Ethernet header (ethHdr)? It is located at the very beginning of the packet.

let ethhdr = ctx.data();

To retrieve the next header, simply add the length of the Ethernet header. In Rust pseudo-code, this would look like:

let ipv4hdr = ctx.data() + sizeof(ethhdr);

Etc.

Let’s focus on the Ethernet header. How can we retrieve its elements? ctx.data() is of type usize. This is a natural integer that represents a memory address. We therefore need to retrieve its content. How can we do this?

In the same way that *const u8 represents a pointer to a character string, we must do the same for the Ethernet header.

We must therefore convert it into a structure that represents the Ethernet frame header. There are several ways to do this, such as looking at the C code and converting it to Rust, which is a good exercise but a little tedious. Fortunately, since Rust is now well established in the eBPF ecosystem, there is already a Rust crate that has done the work for us: network-types.

To retrieve this crate, we will modify the browser-xdp-ebpf/Cargo.toml file and add the following to the dependencies section:

network-types = "0.1.0"

We will then be able to include it in the main code (browser-xdp-ebpf/src/main.rs):

use network_types::eth::EthHdr;

[...]

let ethhdr = ctx.data() as *const EthHdr;

Now that we have retrieved the Ethernet header structure, we can retrieve the various elements the header consists of:

• DST MAC: the destination MAC address
• SRC MAC: the source MAC address
• ETHERTYPE: determines the upper-layer protocol (such as IPv4, IPv6, or Arp, to name the most well-known).

How can this be done with Aya? Let’s review the documentation:

So to retrieve the destination address, simply code:

let dst_addr = unsafe { (*ethhdr).dst_addr};

info!(&ctx, "dst_addr {:mac} ", dst_addr);

We have to use unsafe because we are dereferencing a raw pointer.

Now let’s try it. Here is the modified main code:

use network_types::eth::EthHdr;

fn try_browser_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let ethhdr = ctx.data() as *const EthHdr;

    let dst_addr = unsafe { (*ethhdr).dst_addr};

    info!(&ctx, "dst_addr {:mac} ", dst_addr);

    info!(&ctx, "received a packet");
    Ok(xdp_action::XDP_PASS)
}

Let’s build the program:

cargo build

The compilation works.

Let’s install it in the kernel:

cargo run

We then get the following error:

Error: the BPF_PROG_LOAD syscall failed. Verifier output: 0: R1=ctx() R10=fp0
; unsafe { (*self.ctx).data as usize } @ xdp.rs:16
0: (61) r1 = *(u32 *)(r1 +0)          ; R1_w=pkt(r=0)
; let dst_addr = unsafe { (*ethhdr).dst_addr}; @ main.rs:20
1: (71) r0 = *(u8 *)(r1 +1)
invalid access to packet, off=1 size=1, R1(id=0,off=1,r=0)
R1 offset is outside of the packet
verification time 65 usec
stack depth 0
processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0


Caused by:
    Permission denied (os error 13)

The verifier doesn’t really like this part of the code:

let ethhdr = ctx.data() as *const EthHdr;

The important part of the error is:

invalid access to packet,
R1 offset is outside of the packet

Not very clear. Let’s try to explain it better. When we try let ethhdr = ctx.data() as *const EthHdr;, there is memory access. The verifier requires us to prove that this access is always within the packet (between data() and data_end()).

We therefore need to perform a check to reassure it before this access:

if ctx.data() + EthHdr::LEN > ctx.data_end() {
    return Ok(xdp_action::XDP_PASS);
}

This can be translated as: if the access is not at the packet level, the XDP program does not process the packet: it lets it continue on its way in the Linux kernel network stack.

Otherwise, we can convert it to EthHdr:

let ethhdr = ctx.data() as *const EthHdr;

Here is the complete code for the main function:

fn try_browser_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let start = ctx.data() ;
    let end = ctx.data_end() ;

    if start + EthHdr::LEN > end { //Check for the verifier
        return Ok(xdp_action::XDP_PASS);
    }

    let ethhdr = start as *const EthHdr;

    let dst_addr = unsafe { (*ethhdr).dst_addr};

    info!(&ctx, "dst_addr {:mac} ", dst_addr);
    info!(&ctx, "received a packet");

    Ok(xdp_action::XDP_PASS)
}

Let’s check that it works:

cargo run

Let’s run some pings. We successfully retrieve the MAC address:

[INFO  browser_xdp] dst_addr de:ad:be:ef:00:01
[INFO  browser_xdp] received a packet
[INFO  browser_xdp] dst_addr de:ad:be:ef:00:01
[INFO  browser_xdp] received a packet

Now that we have successfully retrieved the ethHdr header, how do we retrieve the other headers? Simply add the lengths of the preceding headers.

Obviously, we will use very similar methods for all headers. It would be wise to create a function. Here it is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

use core::mem::size_of;

#[inline(always)]
fn ptr_at<T>(ctx: &XdpContext, offset: usize) -> Result<*const T, u16> {
    let start = ctx.data();
    let end = ctx.data_end();
    let len = size_of::<T>();

    if start + offset + len > end {
        return Err(1);
    }

    Ok((start + offset) as *const T)
}

A few notes on creating this function:

• (L3) We inline the function for performance reasons
• (L4) <T>: allows us to define a generic function that doesn’t depend on a specific structure
• the offset lets us to navigate from header to header (offset = 0 for ethHdr)
• (L7) We had to change how the structure length is calculated, using size_of, to make it work properly.

Then the main code becomes:

fn try_browser_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let ethhdr: *const EthHdr = ptr_at(&ctx, 0)?;

    let dst_addr = unsafe { (*ethhdr).dst_addr};

    info!(&ctx, "dst_addr {:mac} ", dst_addr);

    info!(&ctx, "received a packet");
    Ok(xdp_action::XDP_PASS)
}

Now that we have created the function that allows us to traverse the different headers, let’s see how we can block or allow packets based on network headers.

Guided tour of the different headers

#

Level 1: Network access

#

What we can filter

#

We already saw this for the first level: we can retrieve the source MAC address, the destination MAC address, and the EtherType. This allows us to filter based on the MAC address

So we’ll use the function we created earlier, ptr_at():

let ethhdr: *const EthHdr = ptr_at(&ctx, 0)?;

To retrieve the MAC addresses and display them:

let dst_addr = unsafe { (*ethhdr).dst_addr };
let src_addr = unsafe { (*ethhdr).src_addr };

info!(&ctx, "src: {:mac} => dst: {:mac}", src_addr, dst_addr);

Let’s test it:

cargo run

And ping. Let’s look at the result:

[INFO  browser_xdp] src: de:ad:be:ef:00:10 => dst: de:ad:be:ef:00:01
[INFO  browser_xdp] src: de:ad:be:ef:00:10 => dst: de:ad:be:ef:00:01

To write cleaner code, we can create a function:

#[inline(always)]
fn display_ethhdr(ctx: &XdpContext, ethhdr: *const EthHdr) {
    let dst_addr = unsafe { (*ethhdr).dst_addr };
    let src_addr = unsafe { (*ethhdr).src_addr };
    let ethertype = unsafe { (*ethhdr).ether_type };
    info!(ctx, "EthHdr: src: {:mac} => dst: {:mac} ({})", src_addr, dst_addr, ethertype);
}

Let’s filter

#

So, to filter a MAC address, we can do the following:

let mac: [u8; 6] = [0xde, 0xad, 0xbe, 0xef, 0x00, 0x10]; //de:ad:be:ef:00:10

if mac == src_addr {
    info!(&ctx, "drop");
    return Ok(xdp_action::XDP_DROP);
}

Let’s test it:

cargo run

Let’s ping from lb: it no longer pings. Let’s look at the result in Aya:

[INFO  browser_xdp] drop
[INFO  browser_xdp] drop

The program correctly identifies the packet and stops it.

Let’s ping from the h2 namespace:

ip netns exec h2 ping -c 2 10.0.0.1

The ping goes through:

[INFO  browser_xdp] received ethhdr packet src: de:ad:be:ef:00:02 => dst: de:ad:be:ef:00:01
[INFO  browser_xdp] received ethhdr packet src: de:ad:be:ef:00:02 => dst: de:ad:be:ef:00:01

To make it more like xdp-filter, we can create this function that allows us to distinguish whether we want to block the source or the destination:

enum Mode {
    Src,
    Dst,
}

#[inline(always)]
fn drop_mac(ethhdr: *const EthHdr, mac: [u8; 6], mode: Mode) -> bool {
    let addr = match mode {
        Mode::Src => unsafe {(*ethhdr).src_addr},
        Mode::Dst => unsafe {(*ethhdr).dst_addr},
    };

    mac == addr
}

In the main code, this gives us:

fn try_browser_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let ethhdr: *const EthHdr = ptr_at(&ctx, 0)?;
    let mac: [u8; 6] = [0xde, 0xad, 0xbe, 0xef, 0x00, 0x10]; //de:ad:be:ef:00:10
    display_ethhdr(&ctx, ethhdr);
    if drop_mac(ethhdr, mac, Mode::Src) {
       return Ok(xdp_action::XDP_DROP);
    }

    Ok(xdp_action::XDP_DROP);
}

Level up

#

The Ethertype determines the protocol of the packet in the upper layer.

To retrieve it, we can use the following code:

let ethertype = unsafe {(*ethhdr).ether_type} ;

But there is a cleaner way with the ether_type() method, which checks whether the protocol number actually exists:

Instead of memorizing protocol numbers, the network-types crate simplifies things with the EtherType type:

This allows us to filter in the following way:

match unsafe { (*ethhdr).ether_type() } {
        Ok(EtherType::Ipv4) => {}
        _ => return Ok(xdp_action::XDP_PASS),
}

Flows other than IPv4 (e.g., IPv6, Arp) pass through directly and are no longer analyzed.

Level 2: Internet

#

What we can filter

#

So let’s assume that we only want to analyze IPv4 traffic.

As with Ethernet, let’s look at what data we can retrieve with the IPv4 header:

For clarity, the header is shown on several lines. There is significantly more information to retrieve. The ones we will use for this article are:

• PROTOCOL: the upper-level protocol (TCP or UDP, for example)
• SOURCE ADDRESS: the source IP address
• DESTINATION ADDRESS: the destination IP address

In Rust, we can retrieve these headers:

To retrieve the IPv4 headers, simply shift the pointer to the length of the Ethernet header structure using the function we created:

let ipv4hdr: *const Ipv4Hdr = ptr_at(&ctx, EthHdr::LEN)? ;

In the same way as for the lower layer, we can create a function:

#[inline(always)]
fn display_iphdr(ctx: &XdpContext, iphdr: *const Ipv4Hdr) {
    let dst_addr = unsafe { (*iphdr).dst_addr };
    let src_addr = unsafe { (*iphdr).src_addr };
    let proto = unsafe { (*iphdr).proto };
    info!(ctx, "Ipv4Hdr: src: {:i} => dst: {:i} ({})", src_addr, dst_addr, proto as u8);
}

Let’s test the following code:

use network_types::{eth::{EthHdr,EtherType},
                    ip::Ipv4Hdr,
};

fn try_browser_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let ethhdr: *const EthHdr = ptr_at(&ctx, 0)?;
    match unsafe { (*ethhdr).ether_type() } {
        Ok(EtherType::Ipv4) => {}
        _ => return Ok(xdp_action::XDP_PASS),
    }
    let ipv4hdr: *const Ipv4Hdr = ptr_at(&ctx, EthHdr::LEN)? ;
    let mac: [u8; 6] = [0xde, 0xad, 0xbe, 0xef, 0x00, 0x10]; //de:ad:be:ef:00:10
    //display_ethhdr(&ctx, ethhdr);
    display_iphdr(&ctx, ipv4hdr);
    Ok(xdp_action::XDP_PASS)
}

Let’s ping again:

ip netns exec lb ping -c 2 10.0.0.1

On the cargo run side, this gives:

[INFO  browser_xdp] Ipv4Hdr: src: 10.0.0.10 => dst: 10.0.0.1 (1)
[INFO  browser_xdp] Ipv4Hdr: src: 10.0.0.10 => dst: 10.0.0.1 (1)

In the same way as with the Mac address, to prevent the lb IP, we can use this function:

#[inline(always)]
fn drop_ip(iphdr: *const Ipv4Hdr, ip: [u8; 4], mode: Mode) -> bool {
    let addr = match mode {
        Mode::Src => unsafe {(*iphdr).src_addr},
        Mode::Dst => unsafe {(*iphdr).dst_addr},
    };

    ip == addr
}

The main code then gives:

fn try_browser_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let ethhdr: *const EthHdr = ptr_at(&ctx, 0)?;
    match unsafe { (*ethhdr).ether_type() } {
        Ok(EtherType::Ipv4) => {}
        _ => return Ok(xdp_action::XDP_PASS),
    }
    let ipv4hdr: *const Ipv4Hdr = ptr_at(&ctx, EthHdr::LEN)? ;
    display_iphdr(&ctx, ipv4hdr);
    if drop_ip(ipv4hdr, [10, 0, 0, 10], Mode::Src) {
       return Ok(xdp_action::XDP_DROP);
    }
    Ok(xdp_action::XDP_PASS)
}

The ping no longer works:

ip netns exec lb ping -c 2 10.0.0.1

[INFO  browser_xdp] Ipv4Hdr: src: 10.0.0.10 => dst: 10.0.0.1 (1)
[INFO  browser_xdp] Ipv4Hdr: src: 10.0.0.10 => dst: 10.0.0.1 (1)

The program correctly identifies the packet and stops it.

Level up

#

To go to the next level, you need to know which protocol you want to look at. To do this, use the proto option. It is of type IPProto, which is an enumeration:

match unsafe { (*ipv4hdr).proto } {
         IpProto::Udp => {}
         _ => return Ok(xdp_action::XDP_PASS),
}

It is easy to filter if you want to block certain protocols.

Level 3: Transport

#

What we can filter

#

For UDP, we can retrieve the header in this way:

let udphdr: *const UdpHdr = ptr_at(&ctx, EthHdr::LEN + Ipv4Hdr::LEN)? ;

Now let’s look at the UDP segment header:

The most interesting part to filter are the ports.

In Rust:

Ports are not integers but arrays of 2 bytes. This is not practical. Fortunately, there are the following methods:

We can filter at the UDP port level:

#[inline(always)]
fn drop_udp_port(udphdr: *const UdpHdr, port: u16, mode: Mode) -> bool {
    let packet_port = match mode {
        Mode::Src => unsafe {(*udphdr).src_port()},
        Mode::Dst => unsafe {(*udphdr).dst_port()},
    };

    port == packet_port
}

Let’s test the following code:

use network_types::{eth::{EthHdr,EtherType},
                    ip::{Ipv4Hdr,IpProto},
                    udp::UdpHdr,
};

fn try_browser_xdp(ctx: XdpContext) -> Result<u32, u32> {
    let ethhdr: *const EthHdr = ptr_at(&ctx, 0)?;

    match unsafe { (*ethhdr).ether_type() } {
        Ok(EtherType::Ipv4) => {}
        _ => return Ok(xdp_action::XDP_PASS),
    }
    let ipv4hdr: *const Ipv4Hdr = ptr_at(&ctx, EthHdr::LEN)? ;

    match unsafe { (*ipv4hdr).proto } {
         IpProto::Udp => {}
         _ => return Ok(xdp_action::XDP_PASS),
    }

    let udphdr: *const UdpHdr = ptr_at(&ctx, EthHdr::LEN + Ipv4Hdr::LEN)? ;
    display_udphdr(&ctx, udphdr);
    if drop_udp_port(udphdr, 8888, Mode::Dst){
        return Ok(xdp_action::XDP_DROP);
    }
    Ok(xdp_action::XDP_PASS)
}

Let’s install the XDP program:

cargo run

Let’s create a UDP server with netcat:

nc -u 0.0.0.0 -l 8000

Now let’s create a UDP client:

ip netns exec lb netcat -u 10.0.0.1 8000
coucou

This command opens a UDP client and sends the message coucou.

On the UDP server side, we can clearly see:

coucou

But above all, we can clearly see on the cargo run side:

[INFO  browser_xdp] UdpHdr: src: 44024 => dst: 8000

Now let’s test with a UDP server on port 8888:

nc -u 0.0.0.0 -l 8888

Now let’s create a UDP client:

ip netns exec lb netcat -u 10.0.0.1 8888
coucou

Nothing is displayed on the server, which is normal because the XDP program has blocked it:

[INFO  browser_xdp] UdpHdr: src: 39540 => dst: 8888

Level 4: Application

#

The crate network-types does not help retrieve application protocol headers. By knowing the port from the previous level, you can easily filter the application. If you filter port 53, you will filter DNS, for example. But if you want to filter a specific request, that’s another matter.

It is possible to go higher. But application headers can be:

• too large for the stack (limited to 512 bytes), requiring the use of eBPF maps;
• encrypted, as with HTTPS or SSH, making analysis impossible with XDP.

This episode is now over! We have seen the basics of creating a firewall like xdp-filter.

However, we haven’t seen how to make it a little smarter, for example by avoiding DDoS attacks.

That’s good timing! In the next episode, we will create an XDP program that will mitigate denial-of-service attacks.