Last month I spoke at Facebook's Performance @Scale event about Linux BPF Superpowers. These are coming to Linux in the 4.x series, and I've been using them in new open source performance tools.
Video is on Facebook (30 mins):
Linux 4.x Performance: Using BPF Superpowers (Brendan Gregg) Posted by At Scale on Friday, February 26, 2016
Slides are on slideshare (PDF):
We've stopped calling it eBPF (extended Berkeley Packet Filter), and are now just calling it BPF, although we need a better backronym: Bytecode Probe Framework? Naming things is hard.
BPF is the in-kernel bytecode machine that can be used for tracing, virtual networks, and more. Alexei Starovoitov is the lead developer (he's now at Facebook), and there are developers from several companies contributing, including myself at Netflix, Daniel Borkmann at Cisco, and Brenden Blanco at PLUMgrid.
As an example of BPF, I opened with off-CPU analysis, and why BPF was making new things possible. I summarized some other examples as well, including gethostlatency, which instruments DNS lookups system wide without needing to restart anything:
# ./gethostlatency TIME PID COMM LATms HOST 06:10:24 28011 wget 90.00 www.iovisor.org 06:10:28 28127 wget 0.00 www.iovisor.org 06:10:41 28404 wget 9.00 www.netflix.com 06:10:48 28544 curl 35.00 www.netflix.com.au 06:11:10 29054 curl 31.00 www.plumgrid.com 06:11:16 29195 curl 3.00 www.facebook.com 06:11:25 29404 curl 72.00 foo
gethostlatency, and the other tools I demonstrated, are in bcc tools, which is a Python front end for BPF. For this talk I created a diagram of all the bcc tracing tools so far:
So many of my favourites (from other tracing languages) now have equivalents in bcc, which is pretty exciting. Tools like execsnoop, opensnoop, ext4slower, tcpretrans, tcpconnect, and runqlat.
These bcc tools are still in development and require at least Linux 4.1, which many people aren't running yet. You can think of them as a preview of things to come. But they are coming sooner rather than later: Ubuntu 16.04 (for example) will have a 4 series kernel, and isn't far away.
Please watch my talk video above, and check out the other talk videos which were pretty interesting as well (although that link plays the low-res versions in Chrome; high-res versions, like I linked to above, do exist).
Thanks to Facebook for having me – it was a great event.
Links from the talk
iovisor bcc:
- https://github.com/iovisor/bcc
- http://www.brendangregg.com/blog/2015-09-22/bcc-linux-4.3-tracing.html
- http://blogs.microsoft.co.il/sasha/2016/02/14/two-new-ebpf-tools-memleak-and-argdist/
BPF Off-CPU, Wakeup, Off-Wake & Chain Graphs:
- http://www.brendangregg.com/blog/2016-01-20/ebpf-offcpu-flame-graph.html
- http://www.brendangregg.com/blog/2016-02-01/linux-wakeup-offwake-profiling.html
- http://www.brendangregg.com/blog/2016-02-05/ebpf-chaingraph-prototype.html
Linux Performance:
Linux perf_events:
Flame Graphs:
- http://techblog.netflix.com/2015/07/java-in-flames.html
- http://www.brendangregg.com/flamegraphs.html
Netflix Tech Blog on Vector:
Click here for Disqus comments (ad supported).