Let’s Observe a Simple Function in Your Program with uProbes

We saw what a uProbe-type program was in part one: a way to probe the functions of your programs.

In this part, we will first create a very simple program in Go and make it interact with two eBPF programs:

• The first is a uProbe type
• The second is a uRetProbe type

I assume you are already in an environment for developing with Aya and that you have installed the go compiler and bpftrace. If not, you can use the Killercoda lab:

Let’s create a program to test the uProbes

#

A very simple code

#

For a change from Rust, we’re going to create a small program in Go:

// hello.go
package main

import "fmt"

func hello() int {
    fmt.Println("Hello, world!")
    return 3
}

func main() {
    ret := hello()
    fmt.Println("Returned:", ret)
}

How do we activate uProbes?

#

The purpose of this article is to activate:

• an eBPF program of type uProbe each time we enter the hello() function:

• an eBPF program of type uRetProbe each time we exit the hello() function:

Now that we have seen the ins and outs of the article, let’s move on to compiling the program.

Let’s compile the program

#

To compile it, just type:

go build -gcflags="all=-N -l" -o hello hello.go

You may notice that we added an option -gcflags="all=-N -l". Normally, you don’t need to use this option to compile a Go program:

go build -o hello hello.go

This also works. So why use this option?

• -gcflags means go compiler flags; these are options passed to the Go compiler.
• all means that the options apply to all compiled packages.

The options specified to the Go compiler are:

• -N: by default, the compiler modifies (mangles) the function name. This option disables this modification.

• -l: by default, the compiler inlines functions, i.e., it directly integrates the content of the function into the calling code, thus allowing programs to perform better. This option allows you to disable this inline.

These options will allow us to keep the functions and make them readable, which will make it easier for us to find the attachment point for our eBPF program.

How to find the attachment point?

#

Now that we have created and compiled the small program, we need to figure out how to trigger the uProbe or uRetProbe type eBPF program. When using cargo generate for the aya repo, we need to answer two questions: where is the binary name and which function is being observed. Let’s take a closer look.

Name of the binary

#

🤷   Target to attach the (u|uret)probe? (e.g libc):

To ensure the portability of the eBPF program, you must answer with the absolute path of the binary. For example, I created it here: /home/cloud_user/hello.

Function name

#

Once you have answered the first question, there is a second question:

🤷   Function name to attach the (u|uret)probe? (e.g getaddrinfo):

What should you answer? You might be tempted to answer hello since you set the -N option, which disables name decoration during compilation.

But it’s a little more complicated than that. The Go compiler still slightly modifies the function name during the compilation stage.

As we saw in the previous episode, to find all the functions in a binary, just use bpftrace:

bpftrace -l 'uprobe:/home/cloud_user/hello:*'

• The -l option lists all available probes
• uprobe: the type of eBPF program
• /home/cloud_user/hello: the location of the binary
• *: the wildcard (0 or more characters)

The display format is then:

uprobe:/home/cloud_user/hello:[function1]
uprobe:/home/cloud_user/hello:[function2]
uprobe:/home/cloud_user/hello:[function3]
uprobe:/home/cloud_user/hello:[function4]
etc

That’s unfortunate: the function name is the same as my file name… We can’t do | grep hello. How can we get around this? With awk, cut, or worse, a regex? Let’s keep it simple:

bpftrace -l 'uprobe:/home/cloud_user/hello:*hello*'

And it will respond with:

uprobe:/home/cloud_user/hello:main.hello

So the real name of the function is main.hello

Now that we have the answers to both questions, we can create our eBPF programs.

Let’s start with the uProbe-type program.

Let’s create an eBPF uProbe program

#

Let’s test with bpftrace

#

Before diving headfirst into an Aya program, let’s check that it works properly with bpftrace, the program that creates an eBPF program from a command line. That’s good timing: we already found the beginning in the previous section:

uprobe:/home/cloud_user/hello:main.hello

All that’s left to do is add a hello world:

sudo bpftrace -e \
'uprobe:/home/cloud_user/hello:main.hello { printf("Hello go\n"); }'

In another terminal, let’s launch the hello program (without root privileges):

./hello

On the terminal where bpftrace is running, you will see Hello go each time you launch the hello program.

Let’s check that we can create a similar program with Aya.

Generating the Aya program

#

Let’s define the different responses for the attachment point in variables. In my case, it will be:

target=/home/cloud_user/hello
fn_name=main.hello

Now let’s run the command to generate the Aya program:

cargo generate --name test-uprobe \
-d program_type=uprobe \
-d uprobe_target=$target \
-d uprobe_fn_name=$fn_name \
https://github.com/aya-rs/aya-template

Without the options we specified, cargo generate would run in interactive mode: we would have to answer questions.

You will get the following output:

🔧   program_type: "uprobe" (value from CLI)
🔧   uprobe_target: "/home/cloud_user/hello" (value from CLI)
🔧   uprobe_fn_name: "main.hello" (value from CLI)
🔧   Destination: /home/cloud_user/test-uprobe ...
🔧   project-name: test-uprobe ...
🔧   Generating template ...
[ 1/23]   Done: .gitignore
[ 2/23]   Done: Cargo.toml
[ 3/23]   Done: LICENSE-APACHE
[ 4/23]   Done: LICENSE-GPL2
[ 5/23]   Done: LICENSE-MIT
[ 6/23]   Done: README.md
[ 7/23]   Ignored: pre-script.rhai
[ 8/23]   Done: rustfmt.toml
[ 9/23]   Done: test-uprobe/Cargo.toml
[10/23]   Done: test-uprobe/build.rs
[11/23]   Done: test-uprobe/src/main.rs
[12/23]   Done: test-uprobe/src
[13/23]   Done: test-uprobe
[14/23]   Done: test-uprobe-common/Cargo.toml
[15/23]   Done: test-uprobe-common/src/lib.rs
[16/23]   Done: test-uprobe-common/src
[17/23]   Done: test-uprobe-common
[18/23]   Done: test-uprobe-ebpf/Cargo.toml
[19/23]   Done: test-uprobe-ebpf/build.rs
[20/23]   Done: test-uprobe-ebpf/src/lib.rs
[21/23]   Done: test-uprobe-ebpf/src/main.rs
[22/23]   Done: test-uprobe-ebpf/src
[23/23]   Done: test-uprobe-ebpf
🔧   Initializing a fresh Git repository
✨   Done! New project created /home/cloud_user/test-uprobe

Compilation and installation in the kernel

#

Now that we have generated the program, we need to compile it and install it in the Linux kernel:

cd test-uprobe/
RUST_LOG=info cargo run

This will take a little time the first time:

Updating crates.io index
     Locking 103 packages to latest compatible versions
      Adding which v6.0.3 (available: v8.0.0)
  Downloaded anstyle v1.0.11
  Downloaded cfg-if v1.0.1
  Downloaded anyhow v1.0.98
  Downloaded either v1.15.0
  Downloaded cargo_metadata v0.19.2
  Downloaded version_check v0.9.5
  Downloaded which v6.0.3
  Downloaded socket2 v0.6.0
  Downloaded mio v1.0.4
[...]
warning: test-uprobe@0.1.0:     Finished `release` profile [optimized] target(s) in 19.59s
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 50.16s
     Running `/root/build-cache/debug/test-uprobe`
Waiting for Ctrl-C...

Let’s test it now

#

Leave the Aya program running and, on another terminal, launch the program you want to examine. In my case:

./hello

On the terminal where you launched the cargo run command, you should see the following output each time you launch the program:

[INFO  test_uprobe] function main.hello called by /home/cloud_user/hello

You can see that the difficulty is the same as with bpftrace, but the compilation time is much longer.

Let’s create an eBPF uRetProbe program

#

We saw an example of a uProbe-type eBPF program where we didn’t have to modify the generated code. Now we’re going to make things a little more complex with a uRetProbe-type eBPF program and retrieve the return value.

Let’s test with bpftrace

#

The bpftrace code is then slightly modified:

sudo bpftrace -e \
'uretprobe:/home/cloud_user/hello:main.hello { printf("retval=%d\n", retval); }'

• uprobe is replaced by uretprobe
• The return variable has been added: retval

Run the hello program in another terminal and you will see in the bpftrace terminal:

retval=3

Let’s see how to produce the equivalent program in Aya.

Generating the Aya program

#

The code is generated in a similar way to uProbe:

target=/home/cloud_user/hello
fn_name=main.hello
cargo generate --name test-uretprobe \
               -d program_type=uretprobe \
               -d uprobe_target=$target \
               -d uprobe_fn_name=$fn_name \
               https://github.com/aya-rs/aya-template

Compilation and installation

#

In the same way, we will compile and install the eBPF program:

cd test-uretprobe/
RUST_LOG=info cargo run

At the end of each hello() function, the eBPF program displays:

[INFO  test_uretprobe] function main.hello called by /home/cloud_user/hello

So far, we have done pretty much the same thing as for uProbe. Now let’s see how to retrieve the return value 3.

Modifying the code

#

We need to modify the file test-uretprobe-ebpf/src/main.rs: the kernel space code. Specifically, this piece of code:

fn try_test_uretprobe(ctx: RetProbeContext) -> Result<u32, u32> {
    info!(&ctx, "function main.hello called by /home/cloud_user/hello");
    Ok(0)
}

We need to see how to use the RetProbeContext structure to display the return code.

Let’s look at the documentation:

So we need to use the ret() method. We’ll add something like this:

let retval: u32 = ctx.ret().ok_or(1u32)?;

We also need to modify the Aya macro info! to display this value.

This gives us the following result:

fn try_test_uretprobe(ctx: RetProbeContext) -> Result<u32, u32> {
    let retval: u32 = ctx.ret().ok_or(1u32)?;
    info!(&ctx, "retval={}", retval);
    Ok(0)
}

Let’s test it now.

#

Apply the changes:

RUST_LOG=info cargo run

Run the hello command in another terminal:

./hello

And on the Aya program side, we get the following display:

[INFO  test_uretprobe] retval=3

This is consistent with what we found with bpftrace.

This episode is now complete! We’ve seen the basics of making an uProbe and uRetProbe eBPF program react when launching a Go program.

However, we haven’t yet explored how to retrieve the arguments of a function.

That’s perfect timing! In the next episode, we’ll probe a well-known library by retrieving function arguments and processing them!