struct inode ctime fields changed again with Kernel >= 6.11

[agalauner-r7]

Description

Running tracee on a kernel with version >= 6.11 fails with the below error.

Reason for this is that the ctime fields in the inode struct changed again: torvalds/linux@3aa63a5

[...]
; if (bpf_core_field_exists(inode->__i_ctime)) { // Version >= 6.6 @ filesystem.h:61
3941: (15) if r1 == 0x0 goto pc+2     ; R1_w=0
3944: <invalid CO-RE relocation>
failed to resolve CO-RE relocation <byte_off> [620] struct inode___older_v66.i_ctime (0:0 @ offset 0)
processed 3273 insns (limit 1000000) max_states_per_insn 0 total_states 226 peak_states 226 mark_read 173
-- END PROG LOAD LOG --
{"level":"warn","ts":1732264697.5150177,"msg":"libbpf: prog 'tracepoint__sched__sched_process_exec': failed to load: -22"}
{"level":"warn","ts":1732264697.518446,"msg":"libbpf: failed to load object ''"}
{"level":"fatal","ts":1732264697.5203826,"msg":"Tracee runner failed","error":"cmd.Runner.Run: error initializing Tracee: ebpf.(*Tracee).Init: ebpf.(*Tracee).initBPF: failed to load BPF object: invalid argument"}

Output of tracee version:

Tracee version: v0.22.4

Output of uname -a:

Linux 500576e7ccce 6.11.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 01 Nov 2024 03:30:41 +0000 x86_64 GNU/Linux

Additional details

I would fix it myself and submit a PR, but there are multiple ways to do it and I am not sure what's the preferred way.

Right now you check using the CO-RE framework if a new field for kernels between 6.6 and 6.10 is present. If so, read that, otherwise cast the struct into an older mock version and use CO-RE to read the old field.

This doesn't work if we have three different versions now.

There is a way to check for the current linux kernel version by declaring an external variable:

extern int LINUX_KERNEL_VERSION __kconfig;

if (LINUX_KERNEL_VERSION >= KERNEL_VERSION(6, 11, 0)) {
    /* we are on v6.11+ */
}

So, naively, I ended up with code like this:

statfunc u64 get_ctime_nanosec_from_inode(struct inode *inode)
{
    struct timespec64 ts;
    if (LINUX_KERNEL_VERSION >= KERNEL_VERSION(6, 11, 0)) {
        ts.tv_sec = BPF_CORE_READ(inode, i_ctime_sec);
        ts.tv_nsec = BPF_CORE_READ(inode, i_ctime_nsec);
    } else if (LINUX_KERNEL_VERSION >= KERNEL_VERSION(6, 6, 0)) {
        ts = BPF_CORE_READ(inode, __i_ctime);
    } else {
        struct inode___older_v66 *old_inode = (void *) inode;
        ts = BPF_CORE_READ(old_inode, i_ctime);
    }

    return get_time_nanosec_timespec(&ts);
}

This doesn't work though, because vmlinux.h doesn't contain the new definition of struct inode.

Now there are two tways to solve this:

1. Introduce a struct inode___newer_v611 which contains the two new fields and use it like the else case:

statfunc u64 get_ctime_nanosec_from_inode(struct inode *inode)
{
    struct timespec64 ts;

    if (LINUX_KERNEL_VERSION >= KERNEL_VERSION(6, 11, 0)) {
        struct inode___newer_v611 *new_inode = (void *) inode;
        ts.tv_sec = BPF_CORE_READ(new_inode, i_ctime_sec);
        ts.tv_nsec = BPF_CORE_READ(new_inode, i_ctime_nsec);
    } else if (LINUX_KERNEL_VERSION >= KERNEL_VERSION(6, 6, 0)) {
        ts = BPF_CORE_READ(inode, __i_ctime);
    } else {
        struct inode___older_v66 *old_inode = (void *) inode;
        ts = BPF_CORE_READ(old_inode, i_ctime);
    }
}

2. Regenerate vmlinux.h so we have a current struct inode and introduce a struct inode___older_v611 which contains the old fields and use that in the else if branch like you do now in the else branch:

statfunc u64 get_ctime_nanosec_from_inode(struct inode *inode)
{
    struct timespec64 ts;

    if (LINUX_KERNEL_VERSION >= KERNEL_VERSION(6, 11, 0)) {
        ts.tv_sec = BPF_CORE_READ(inode, i_ctime_sec);
        ts.tv_nsec = BPF_CORE_READ(inode, i_ctime_nsec);
    } else if (LINUX_KERNEL_VERSION >= KERNEL_VERSION(6, 6, 0)) {
        ts = BPF_CORE_READ(inode, __i_ctime);
        struct inode___older_v611 *old_inode = (void *) inode;
        ts = BPF_CORE_READ(old_inode, __i_ctime);
    } else {
        struct inode___older_v66 *old_inode = (void *) inode;
        ts = BPF_CORE_READ(old_inode, i_ctime);
    }
}

So what's preferred? And if it's solution number 2, how do I regenerate vmlinux.h? On my local machine using bpftool? Do you have another process for that?

I personally would prefer the second case, because it feels better to work with more "current" code and have the special cases present for older kernels.

[rscampos]

Hello @agalauner-r7,

Thank you for using Tracee and for providing such a detailed issue report.

I was able to reproduce the issue in my environment as well and will take a closer look at it.

; if (bpf_core_field_exists(inode->__i_ctime)) { // Version >= 6.6 @ filesystem.h:61
3941: (15) if r1 == 0x0 goto pc+2     ; R1_w=0
3944: <invalid CO-RE relocation>
failed to resolve CO-RE relocation <byte_off> [620] struct inode___older_v66.i_ctime (0:0 @ offset 0)
processed 3273 insns (limit 1000000) max_states_per_insn 0 total_states 226 peak_states 226 mark_read 173
-- END PROG LOAD LOG --
{"level":"warn","ts":1732305951.2560215,"msg":"libbpf: prog 'tracepoint__sched__sched_process_exec': failed to load: -22"}
{"level":"warn","ts":1732305951.257808,"msg":"libbpf: failed to load object ''"}
{"level":"fatal","ts":1732305951.2585351,"msg":"Tracee runner failed","error":"cmd.Runner.Run: error initializing Tracee: ebpf.(*Tracee).Init: ebpf.(*Tracee).initBPF: failed to load BPF object: invalid argument"}

Kernel version:

Linux ip-172-31-3-75 6.11.0-061100-generic #202409151536 SMP PREEMPT_DYNAMIC Sun Sep 15 16:01:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[NDStrahilevitz]

I think we use the field check method as a proxy for version (although version is actually a proxy for field existence) exactly because we don't support global kconfig access (although maybe for no reason anymore, not sure).

[rscampos]

Hey @agalauner-r7, I'm sorry for the delay.

I liked both approaches you proposed and agree with @NDStrahilevitz that we can use field check as proxy for version.

After trying different approaches and do some research, I ended up reading this from using macro KERNEL_VERSION:

Version numbers do not always reflect the actual features available in the kernel. 
Some distributions backport features to older kernels without reflecting this in the 
version number. If possible, it is always better to probe for the availability of a feature 
directly instead of inferring it from the kernel version.

From my understanding, think its better continue using the probe feature directly. I arrived at this solution, which has some similarities to your approach, except for the kernel version detection. WDYT?
Check out the PR: #4457