Let’s Probe a Library with an eBPF uProbe

We saw what a uProbe-type program was in part one: a way to probe a library.

We will verify this with an Aya program that will retrieve the various arguments of the execve() function from Libc.

I assume you are already in an environment for developing with Aya and that you have installed bpftrace. If not, you can use the Killercoda lab:

What are we really going to do?

#

Libc

#

Unlike in the previous section, where we attached our eBPF program to a program, here we are attaching it to a shared library.

Libc is the standard C library. So every time a C (or C++) program is executed, an eBPF program could potentially be launched.

The execve function

#

execve is a system call (a syscall) from the Linux kernel. But it is also the name of a function in libc that calls this same syscall (a wrapper). Thus, each time the execve() function of libc is called, our uProbe-type eBPF program will be launched.

The arguments of the execve function

#

We also need to retrieve the various arguments of the execve() function.

To find its arguments, we could obviously look in the libc source code. But there is an easier way:

man execve

The part that interests us is the following:

int execve(const char *pathname, char *const _Nullable argv[],
                  char *const _Nullable envp[]);

We can see that the function has three arguments:

1. pathname: the name of the command with the full path (e.g., /bin/bash). It is of type const char * (equivalent in Rust to *const u8).
2. argv: an array of command arguments. It is of type char *const _Nullable[] (equivalent in Rust to *const *const u8)
   • argv[0]: the name of the command
   • argv[1]: the first option
   • etc.
3. envp: an array of command environment variables. It is of type char *const _Nullable[] (equivalent in Rust to *const *const u8).

How to trigger the eBPF program?

#

Let’s take a simple example. If you run a command in a terminal, for example ls, what will happen?

• Thanks to the PATH environment variable, the shell (for example bash) will find the right path to locate the ls binary:

/usr/bin/ls

• To execute the binary, the shell will then call the execve() function from libc:

execve("/usr/bin/ls", ['ls'], ["PATH=/bin:/usr/bin", ...])

• The eBPF program will finally be triggered.

Here is a brief summary of all this:

There are obviously other programs besides shells that call the execve() function in libc, such as systemd for starting various programs on a Linux system.

So we are going to create a program very similar to the one we created with the sys_enter_execve tracepoint in the introductory articles on eBPF, but this one will be attached at the user level to the execve() function of libc.

Let’s generate an uProbe-type Aya program

#

So we already have the answers to both questions:

🤷   Target to attach the (u|uret)probe? (e.g libc):
🤷   Function name to attach the (u|uret)probe? (e.g getaddrinfo):

Let’s see how to create an eBPF hello world program for this attachment point.

Let’s test it with bpftrace

#

First, let’s check that it works with the bpftrace command line:

sudo bpftrace -e \
  'uprobe:libc:execve { printf("Hello execve\n"); }'

• uprobe: the type of eBPF program
• libc: the name of the library
• execve: the function to debug
• { printf("Hello execve\n"); }: the bpftrace code

Each time we run a command on another terminal, we see Hello execve.

Now let’s do it with Aya.

Generating and compiling the Aya program

#

The following cargo generate command generates the eBPF program:

cargo generate --name test-uprobe-2 \
               -d program_type=uprobe \
               -d uprobe_target=libc \
               -d uprobe_fn_name=execve \
               https://github.com/aya-rs/aya-template

Now let’s compile it and install it in the Linux kernel:

cd test-uprobe-2/
RUST_LOG=info cargo run

Testing the program

#

On another terminal, launch any program:

ls

On the cargo run terminal, you will see:

[INFO  test_uprobe] function execve called by libc

In the previous section, we left off at this point regarding uProbes. Let’s look at how to retrieve the various arguments of the execve() function. Let’s start with the first one: the name of the binary.

Let’s retrieve the name of the binary

#

Let’s test with bpftrace

#

Before modifying the Aya code, let’s see how to do it with bpftrace. It’s a bit more complicated than a simple hello world.

To retrieve the first argument, we use arg0:

sudo bpftrace -e \
  'uprobe:libc:execve { printf("%d\n", arg0); }'

We retrieve the address where the first argument is located. How do we “convert” it to a string? Just use the str() function:

sudo bpftrace -e \
  'uprobe:libc:execve { printf("%s\n", str(arg0)); }'

Now that we have the draft with bpftrace, let’s see how to implement it with Aya.

Let’s modify the Aya code

#

We need to modify the following function in the test-uprobe-2-ebpf/src/main.rs file:

fn try_test_uprobe_2(ctx: ProbeContext) -> Result<u32, u32> {
    info!(&ctx, "function execve called by libc");
    Ok(0)
}

We therefore need to manipulate the ctx variable. Here is the documentation:

There is only one method that interests us:

The first element is the name of the binary that is executed.

So we need to add something like this:

let arg0: *const u8  = ctx.arg(0).ok_or(1u32)?;

To “convert” this pointer into a character string, we will do it in a similar way to how we did with Tracepoints in the introductory article on creating eBPF programs with Aya.

This gives us the following code:

fn try_test_uprobe_2(ctx: ProbeContext) -> Result<u32, i64> {
    let arg0: *const u8  = ctx.arg(0).ok_or(1u32)?;
    let mut buf = [0u8; 128];
    let filename = unsafe {
        let filename_bytes = bpf_probe_read_user_str_bytes(arg0, &mut buf)?;
        from_utf8_unchecked(filename_bytes)
    };
    info!(&ctx, "function execve called by libc {}", filename);
    Ok(0)
}

At the time, it wasn’t very clear in my mind.

Let’s explain this code in detail:

• The helper function bpf_probe_read_user_str_bytes() allows you to read the memory address from user space and retrieve its contents with a slice of bytes. You need a buffer for this.
• from_utf8_unchecked() converts a slice of bytes into an &str (without unchecked, otherwise the eBPF verifier will not accept it).

Finally, here is a small diagram explaining how to retrieve a character string from user space:

Now let’s test the change.

#

Let’s check that the code still works:

RUST_LOG=info cargo run

On another terminal, let’s run any command:

ls

On the cargo run terminal, you will see:

[INFO  test_uprobe_2] function execve called by libc /usr/bin/ls

We left off at this point in the introductory articles on eBPF with Aya. But we could have gone further by retrieving the command options and its environment variables. Let’s see how to do that.

Let’s retrieve the command options

#

Let’s test with bpftrace

#

Before doing this with Aya, we’ll look at how to do it with bpftrace. To retrieve the second argument, we must use arg1. Since arg1 is a pointer to a pointer, we cannot use the str() function directly. We must dereference arg1 to obtain a single pointer. To do this, simply use *.

This gives us:

bpftrace -e \
  'uprobe:libc:execve { printf("%s\n", str(*arg1)); }'

We then retrieve the first element of the array, which is the name of the command. We therefore need to move through the array if we want to retrieve the different options. Each element is 8 bytes in size (only valid in 64-bit).

To go to the second element of the array, i.e., the first option, simply move 8 bytes (by adding 8):

bpftrace -e \
  'uprobe:libc:execve { printf("%s\n", str(*(arg1+8))); }'

You will see that the difficulty will be much the same in Rust.

Let’s modify the Aya code

#

With Aya, to retrieve the second argument, you need to add this piece of code:

let argv: *const *const u8 = ctx.arg(1).ok_or(1u32)?;

How do we retrieve the nth option of the command? We need to use the add function to shift its pointer to the correct memory address:

For example, to retrieve the first option, we will shift by 1:

let argv1 = argv.add(1);

However, argv1 is still of type *const *const u8. You now need to dereference it to obtain *const u8.

There is a ready-made function for this:

So we have:

let argv1_deref: *const u8 = bpf_probe_read_user(argv1)?;

Now that argv1_deref is of type *const u8, we need to convert it to &str. This gives us code similar to that used to retrieve the name of the binary. It would probably be useful to create a function for a “serious” project.

Here is the complete code for retrieving the first option:

let argv: *const *const u8  = ctx.arg(1).ok_or(0u32)?; //arg1
let mut buf = [0u8; 16];
let argname = unsafe {
    let argv1 = argv.add(1); //arg1+8
    let argv1_deref: *const u8 = bpf_probe_read_user(argv1)?; //*(arg1+8)
    let argname_bytes = bpf_probe_read_user_str_bytes(argv1_deref, &mut buf)?;
    from_utf8_unchecked(argname_bytes) //str(*(arg1+8))
};
info!(&ctx, "function execve called by libc {}", argname); //printf("%s\n", str(*(arg1+8)));

Now let’s test the change.

#

Let’s check that the code still works:

RUST_LOG=info cargo run

On another terminal, let’s run a command with an option:

ls -lrt

On the cargo run terminal, you will see:

[INFO  test_uprobe_2] function execve called by libc /usr/bin/ls
[INFO  test_uprobe_2] function execve called by libc -lrt

This is the behavior we wanted.

What happens if we run a command without an option?

man

On the cargo run terminal, you will only see:

[INFO  test_uprobe_2] function execve called by libc /usr/bin/man

What happened?

This part of the code was not displayed:

info!(&ctx, "function execve called by libc {}", argname);

Since the command has no arguments, this part of the code threw an error:

let argname = unsafe {
    let argv1 = argv.add(1); //arg1+8
    let argv1_deref: *const u8 = bpf_probe_read_user(argv1)?; //*(arg1+8)
    let argname_bytes = bpf_probe_read_user_str_bytes(argv1_deref, &mut buf)?;
    from_utf8_unchecked(argname_bytes) //str(*(arg1+8))
};

And so the program crashed and never processed the last info.

This episode is now over! We have seen how to retrieve the arguments of a function in a C program, particularly for character strings and character string arrays, and how to display them.

In the next episode, we will see how to profile a program function.