Add support to use kernel uprobe multi bpf link #2810

olsajiri · 2023-10-22T12:03:46Z

Adding support to use uprobe_multi kernel for multiple uprobes.

It's useful for attaching bigger amount of uprobes, because it's faster and
saves file descriptors, because it does not create perf event for each uprobe.

Checklist

Language changes are updated in man/adoc/bpftrace.adoc and if needed in docs/reference_guide.md
User-visible and non-trivial changes updated in CHANGELOG.md
The new behaviour is covered by tests

danobi · 2023-11-02T17:29:33Z

src/bpffeature.cpp

+extern "C" __attribute__((noinline)) __attribute__((unused)) void uprobe_multi_test_func(
+    void)
+{
+}


I'm afraid this won't work. Packagers typically strip symbols. We've had a lot of issues in the past with BEGIN/END probes being implemented with uprobes inside bpftrace binary. Which is why that moved to raw tracepoint prog type.

Is there another way to feature probe? Perhaps by passing an invalid symbol name and checking for EINVAL (rather than ENOSYS or something).

right, actually we can do that, it's better.. I actually did that in libbpf, will change

danobi · 2023-11-02T17:31:23Z

man/adoc/bpftrace.adoc

+*--disable-feature* _feature,feature,..._::
+  Disable detected features, valid values are::
+
+  *uprobe_multi* to disable uprobe_multi link +
+  *kprobe_multi* to disable kprobe_multi link


How about --no-feature instead? It's shorter and is more similar to gcc. I plan on adding --feature at some point for the AOT stuff (soon I promise...)

sure, will change

danobi · 2023-11-02T17:34:01Z

src/attached_probe.cpp

+int bcc_load_cb(uint64_t v_addr, uint64_t mem_sz, uint64_t file_offset, void *p)
+{
+  std::vector<struct addr_offset> *addrs =
+      static_cast<std::vector<struct addr_offset> *>(p);
+
+  for (std::vector<struct addr_offset>::iterator it = addrs->begin();
+       it != addrs->end();
+       ++it)
+  {
+    struct addr_offset &a = *it;
+
+    if (a.addr >= v_addr && a.addr < (v_addr + mem_sz))
+    {
+      a.offset = a.addr - v_addr + file_offset;
+    }
+  }


Looks familiar to some of the code in bcc. bcc doesn't export any of this functionality?

I can't see any.. it's basically what we do for single uprobe but adapted to do it for multiple symbols,
but I guess we could have just one function to do that for both cases then.. I'll check on that

actually I don't think the unifying will make the code easier, I'd rather keep it as it is

The uprobe_multi link support got in recently so we need to add detection of the uprobe_multi libbpf interface. Setting HAVE_LIBBPF_UPROBE_MULTI macro when it's present. Signed-off-by: Jiri Olsa <[email protected]>

viktormalik

Overall LGTM, thanks!

I just have a couple of nits, mainly suggestions to use C++ features which make the code more safe and legible.

src/bpffeature.cpp

man/adoc/bpftrace.adoc

src/attached_probe.cpp

src/bpftrace.cpp

src/attached_probe.cpp

viktormalik

One more thing just came across my mind: let's update the libbpf submodule to a version that contains uprobe multi support.

viktormalik

Nice, thanks!

danobi · 2023-11-09T15:58:13Z

src/bpffeature.cpp

+                         ARRAY_SIZE(insns),
+                         &load_opts);
+
+  if (progfd >= 0)


Might be good to handle progfd < 0 but seems unlikely BPF_PROG_LOAD will ever return EBADF and confuse the logic here

you are right, I set err as 0 which should make it ok,
I wonder why there's no compiler warning

Adding feature check for uprobe_multi, the has_uprobe_multi function will return true if the uprobe_multi support is detected. Also changing the bpf_prog_load cast for kprobe_multi detection code. Signed-off-by: Jiri Olsa <[email protected]>

Adding support to attach multiple uprobes with uprobe_multi link. There's no change in the user interface, just when we detect wildcarded uprobe probe we attach it with multi_uprobe link, like: # bpftrace -vv -e 'uprobe:/lib64/libc.so.6:printf_* {}' INFO: node count: 3 Attaching 1 probe... Program ID: 200 The verifier log: processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0 Attaching uprobe:/lib64/libc.so.6:printf_* Attaching to 3 functions /lib64/libc.so.6:printf_positional:0x59fd0 /lib64/libc.so.6:printf_size:0x5aad0 /lib64/libc.so.6:printf_size_info:0x5c790 Signed-off-by: Jiri Olsa <[email protected]>

Adding --no-feature option to disable selected detected features. At the moment it's possible to disable kprobe_multi and uprobe_multi links support with: # bpftrace --info ... kprobe_multi: yes uprobe_multi: yes # bpftrace --no-feature=kprobe_multi,uprobe_multi --info ... kprobe_multi: no uprobe_multi: no # bpftrace --no-feature=uprobe_multi,krava --info ERROR: USAGE: --no-feature can only have values 'kprobe_multi,uprobe_multi'. Signed-off-by: Jiri Olsa <[email protected]>

Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the uprobe_multi branch 8 times, most recently from 2b7aa2d to 4eca22f Compare October 26, 2023 12:10

olsajiri marked this pull request as ready for review October 26, 2023 12:45

olsajiri requested review from ajor, viktormalik, danobi and fbs as code owners October 26, 2023 12:45

danobi reviewed Nov 2, 2023

View reviewed changes

Detect libbpf uprobe_multi api is present

778b868

The uprobe_multi link support got in recently so we need to add detection of the uprobe_multi libbpf interface. Setting HAVE_LIBBPF_UPROBE_MULTI macro when it's present. Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the uprobe_multi branch 2 times, most recently from 00c9fbb to f3500eb Compare November 6, 2023 13:45

viktormalik reviewed Nov 7, 2023

View reviewed changes

olsajiri force-pushed the uprobe_multi branch 2 times, most recently from 119ed66 to 5d8bbbe Compare November 7, 2023 13:15

viktormalik reviewed Nov 7, 2023

View reviewed changes

viktormalik approved these changes Nov 8, 2023

View reviewed changes

danobi approved these changes Nov 9, 2023

View reviewed changes

olsajiri added 6 commits November 10, 2023 08:33

Add feature check for uprobe_multi

94365ba

Adding feature check for uprobe_multi, the has_uprobe_multi function will return true if the uprobe_multi support is detected. Also changing the bpf_prog_load cast for kprobe_multi detection code. Signed-off-by: Jiri Olsa <[email protected]>

Add runtime tests for uprobe_multi link

9136d13

Signed-off-by: Jiri Olsa <[email protected]>

Update CHANGELOG

d6d22d8

Signed-off-by: Jiri Olsa <[email protected]>

Update libbpf submodule

74efacd

Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the uprobe_multi branch from ae9b0a6 to 74efacd Compare November 10, 2023 08:38

viktormalik merged commit ba92ae2 into bpftrace:master Nov 10, 2023
21 checks passed

acj mentioned this pull request Jan 24, 2024

Build failing on 32-bit ARM #2953

Closed

ajwerner mentioned this pull request Jul 16, 2024

support uprobe multi bpf link aya-rs/aya#992

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to use kernel uprobe multi bpf link #2810

Add support to use kernel uprobe multi bpf link #2810

olsajiri commented Oct 22, 2023 •

edited

Loading

danobi Nov 2, 2023

olsajiri Nov 5, 2023

danobi Nov 2, 2023

olsajiri Nov 5, 2023

danobi Nov 2, 2023

olsajiri Nov 5, 2023

olsajiri Nov 6, 2023

viktormalik left a comment

viktormalik left a comment

viktormalik left a comment

danobi Nov 9, 2023

olsajiri Nov 10, 2023

Add support to use kernel uprobe multi bpf link #2810

Add support to use kernel uprobe multi bpf link #2810

Conversation

olsajiri commented Oct 22, 2023 • edited Loading

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viktormalik left a comment

Choose a reason for hiding this comment

viktormalik left a comment

Choose a reason for hiding this comment

viktormalik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olsajiri commented Oct 22, 2023 •

edited

Loading