I was made aware that eBPF was available now in RHEL via the below tweet from Brendan Gregg. As our main distro is RHEL I was intrigued to see what the release looked like and what was available. For those who don’t know the bcc-tools package contains the eBPF bcc toolset which allows new insights into tracing and performance analysis on linux.
As this is part of the tech preview in the 7.6 Beta we added this release into our satellite packaging system and ran a quick yum upgrade on the host. Once upgraded the install was simply
yum install bcc-tools
I have omitted some details of the install however we can see that the size is 84Mb.
Total download size: 23 M Installed size: 84 M Is this ok [y/d/N]: y Downloading packages: (1/5): bcc-0.6.0-3.el7.x86_64.rpm | 478 kB 00:00:00 (2/5): bcc-tools-0.6.0-3.el7.x86_64.rpm | 377 kB 00:00:00 (3/5): python-bcc-0.6.0-3.el7.x86_64.rpm | 53 kB 00:00:00 (4/5): python-netaddr-0.7.5-9.el7.noarch.rpm | 983 kB 00:00:00 (5/5): llvm-private-6.0.1-2.el7.x86_64.rpm | 21 MB 00:00:00 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 38 MB/s | 23 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : llvm-private-6.0.1-2.el7.x86_64 1/5 Installing : python-netaddr-0.7.5-9.el7.noarch 2/5 Installing : python-bcc-0.6.0-3.el7.x86_64 3/5 Installing : bcc-tools-0.6.0-3.el7.x86_64 4/5 Installing : bcc-0.6.0-3.el7.x86_64 5/5 Verifying : bcc-tools-0.6.0-3.el7.x86_64 1/5 Verifying : python-netaddr-0.7.5-9.el7.noarch 2/5 Verifying : bcc-0.6.0-3.el7.x86_64 3/5 Verifying : llvm-private-6.0.1-2.el7.x86_64 4/5 Verifying : python-bcc-0.6.0-3.el7.x86_64 5/5 Installed: bcc-tools.x86_64 0:0.6.0-3.el7 Dependency Installed: bcc.x86_64 0:0.6.0-3.el7 llvm-private.x86_64 0:6.0.1-2.el7 python-bcc.x86_64 0:0.6.0-3.el7 python-netaddr.noarch 0:0.7.5-9.el7
We can see we have a few packages with bcc in the name installed bcc-0.6.0.3.el7.x86_64.rpm,bcc-tools.x86_64 0:0.6.0-3.el7 & python-bcc.x86_64 0:0.6.0-3.el7
Looking into these we see the following contains the main python libraries
root# rpm -ql python-bcc-0.6.0-3.el7.x86_64 /usr/lib/python2.7/site-packages/bcc /usr/lib/python2.7/site-packages/bcc-0.6.0-py2.7.egg-info /usr/lib/python2.7/site-packages/bcc/__init__.py /usr/lib/python2.7/site-packages/bcc/__init__.pyc /usr/lib/python2.7/site-packages/bcc/__init__.pyo /usr/lib/python2.7/site-packages/bcc/libbcc.py /usr/lib/python2.7/site-packages/bcc/libbcc.pyc /usr/lib/python2.7/site-packages/bcc/libbcc.pyo /usr/lib/python2.7/site-packages/bcc/perf.py /usr/lib/python2.7/site-packages/bcc/perf.pyc /usr/lib/python2.7/site-packages/bcc/perf.pyo /usr/lib/python2.7/site-packages/bcc/table.py /usr/lib/python2.7/site-packages/bcc/table.pyc /usr/lib/python2.7/site-packages/bcc/table.pyo /usr/lib/python2.7/site-packages/bcc/tcp.py /usr/lib/python2.7/site-packages/bcc/tcp.pyc /usr/lib/python2.7/site-packages/bcc/tcp.pyo /usr/lib/python2.7/site-packages/bcc/usdt.py /usr/lib/python2.7/site-packages/bcc/usdt.pyc /usr/lib/python2.7/site-packages/bcc/usdt.pyo /usr/lib/python2.7/site-packages/bcc/utils.py /usr/lib/python2.7/site-packages/bcc/utils.pyc /usr/lib/python2.7/site-packages/bcc/utils.pyo
Here we have the main bcc libraries.
root# rpm -ql bcc-0.6.0-3.el7.x86_64 /usr/lib64/libbcc.so.0 /usr/lib64/libbcc.so.0.6.0 /usr/lib64/libbpf.so.0 /usr/lib64/libbpf.so.0.6.0 /usr/share/doc/bcc-0.6.0 /usr/share/doc/bcc-0.6.0/README.md /usr/share/licenses/bcc-0.6.0 /usr/share/licenses/bcc-0.6.0/COPYRIGHT.txt /usr/share/licenses/bcc-0.6.0/LICENSE.txt
And for bcc-tools we have the main toolset and also documentation containing examples and man pages.
root# rpm -ql bcc-tools-0.6.0-3.el7.x86_64 |egrep -v "doc|man" /usr/share/bcc /usr/share/bcc/introspection /usr/share/bcc/introspection/bps /usr/share/bcc/tools /usr/share/bcc/tools/argdist /usr/share/bcc/tools/bashreadline /usr/share/bcc/tools/biolatency /usr/share/bcc/tools/biosnoop /usr/share/bcc/tools/biotop /usr/share/bcc/tools/bitesize /usr/share/bcc/tools/bpflist /usr/share/bcc/tools/btrfsdist /usr/share/bcc/tools/btrfsslower /usr/share/bcc/tools/cachestat /usr/share/bcc/tools/cachetop /usr/share/bcc/tools/capable /usr/share/bcc/tools/cobjnew /usr/share/bcc/tools/cpudist /usr/share/bcc/tools/cpuunclaimed /usr/share/bcc/tools/dbslower /usr/share/bcc/tools/dbstat /usr/share/bcc/tools/dcsnoop /usr/share/bcc/tools/dcstat /usr/share/bcc/tools/deadlock_detector /usr/share/bcc/tools/deadlock_detector.c /usr/share/bcc/tools/execsnoop /usr/share/bcc/tools/ext4dist /usr/share/bcc/tools/ext4slower /usr/share/bcc/tools/filelife /usr/share/bcc/tools/fileslower /usr/share/bcc/tools/filetop /usr/share/bcc/tools/funccount /usr/share/bcc/tools/funclatency /usr/share/bcc/tools/funcslower /usr/share/bcc/tools/gethostlatency /usr/share/bcc/tools/hardirqs /usr/share/bcc/tools/inject /usr/share/bcc/tools/javacalls /usr/share/bcc/tools/javaflow /usr/share/bcc/tools/javagc /usr/share/bcc/tools/javaobjnew /usr/share/bcc/tools/javastat /usr/share/bcc/tools/javathreads /usr/share/bcc/tools/killsnoop /usr/share/bcc/tools/lib /usr/share/bcc/tools/lib/ucalls /usr/share/bcc/tools/lib/uflow /usr/share/bcc/tools/lib/ugc /usr/share/bcc/tools/lib/uobjnew /usr/share/bcc/tools/lib/ustat /usr/share/bcc/tools/lib/uthreads /usr/share/bcc/tools/mdflush /usr/share/bcc/tools/memleak /usr/share/bcc/tools/mountsnoop /usr/share/bcc/tools/mysqld_qslower /usr/share/bcc/tools/nfsdist /usr/share/bcc/tools/nfsslower /usr/share/bcc/tools/nodegc /usr/share/bcc/tools/nodestat /usr/share/bcc/tools/offcputime /usr/share/bcc/tools/offwaketime /usr/share/bcc/tools/oomkill /usr/share/bcc/tools/opensnoop /usr/share/bcc/tools/phpcalls /usr/share/bcc/tools/phpflow /usr/share/bcc/tools/phpstat /usr/share/bcc/tools/pidpersec /usr/share/bcc/tools/profile /usr/share/bcc/tools/pythoncalls /usr/share/bcc/tools/pythonflow /usr/share/bcc/tools/pythongc /usr/share/bcc/tools/pythonstat /usr/share/bcc/tools/reset-trace /usr/share/bcc/tools/rubycalls /usr/share/bcc/tools/rubyflow /usr/share/bcc/tools/rubygc /usr/share/bcc/tools/rubyobjnew /usr/share/bcc/tools/rubystat /usr/share/bcc/tools/runqlat /usr/share/bcc/tools/runqlen /usr/share/bcc/tools/runqslower /usr/share/bcc/tools/slabratetop /usr/share/bcc/tools/softirqs /usr/share/bcc/tools/solisten /usr/share/bcc/tools/sslsniff /usr/share/bcc/tools/stackcount /usr/share/bcc/tools/statsnoop /usr/share/bcc/tools/syncsnoop /usr/share/bcc/tools/syscount /usr/share/bcc/tools/tcpaccept /usr/share/bcc/tools/tcpconnect /usr/share/bcc/tools/tcpconnlat /usr/share/bcc/tools/tcpdrop /usr/share/bcc/tools/tcplife /usr/share/bcc/tools/tcpretrans /usr/share/bcc/tools/tcpstates /usr/share/bcc/tools/tcpsubnet /usr/share/bcc/tools/tcptop /usr/share/bcc/tools/tcptracer /usr/share/bcc/tools/tplist /usr/share/bcc/tools/trace /usr/share/bcc/tools/ttysnoop /usr/share/bcc/tools/vfscount /usr/share/bcc/tools/vfsstat /usr/share/bcc/tools/wakeuptime /usr/share/bcc/tools/xfsdist /usr/share/bcc/tools/xfsslower
Looking at the list it looks like the whole toolset minus some examples,llcstat,zfstools & lua support however I was wondering as the kernel was showing as 3.10.0-933.el7.x86_64 would all the tools work as expected as we had previous tools in old which targeted older 3.x kernels but would say tracepoints work etc.
Tracepoints
So taking a quick look at tracepoints with the urandomread examples, note these don’t ship in the RHEL rpm’s so I had to create them and they work!! Given that tracepoint support landed in the 4.7 kernel it looks like RedHat have done a great job in backporting this support.
root# chmod 755 urandomread root# ./urandomread TIME(s) COMM PID GOTBITS 36967.532695000 sshd 29524 112 36967.532710000 sshd 29524 384 36967.537117000 sshd 29525 384 36970.864592000 python 9245 128 36972.782119000 sshd 29524 128 36972.790661000 systemd 1 128 36972.824703000 sshd 29536 384
And again lets test urandomread-explicit from the examples
root# ./urandomread-explicit TIME(s) COMM PID GOTBITS 37171.040068000 python 9245 128 37173.119018000 dd 31629 8192 37173.119028000 dd 31629 8192 37173.119035000 dd 31629 8192 37173.119043000 dd 31629 8192 37173.119050000 dd 31629 8192
Uprobes
Looking at uprobes we have tplist which list the probes we can attach too, working through the examples for say tplist this looks good.
root# ./tplist -l pthread
/lib64/libpthread.so.0 libpthread:lll_futex_wake
/lib64/libpthread.so.0 libpthread:pthread_start
/lib64/libpthread.so.0 libpthread:pthread_create
/lib64/libpthread.so.0 libpthread:pthread_join
/lib64/libpthread.so.0 libpthread:pthread_join_ret
/lib64/libpthread.so.0 libpthread:mutex_init
/lib64/libpthread.so.0 libpthread:mutex_destroy
/lib64/libpthread.so.0 libpthread:mutex_acquired
/lib64/libpthread.so.0 libpthread:mutex_entry
/lib64/libpthread.so.0 libpthread:mutex_timedlock_entry
/lib64/libpthread.so.0 libpthread:mutex_timedlock_acquired
/lib64/libpthread.so.0 libpthread:mutex_release
/lib64/libpthread.so.0 libpthread:rwlock_destroy
/lib64/libpthread.so.0 libpthread:rdlock_entry
/lib64/libpthread.so.0 libpthread:wrlock_entry
/lib64/libpthread.so.0 libpthread:cond_init
/lib64/libpthread.so.0 libpthread:cond_destroy
/lib64/libpthread.so.0 libpthread:cond_wait
/lib64/libpthread.so.0 libpthread:cond_timedwait
/lib64/libpthread.so.0 libpthread:cond_signal
/lib64/libpthread.so.0 libpthread:cond_broadcast
/lib64/libpthread.so.0 libpthread:lll_lock_wait_private
/lib64/libpthread.so.0 libpthread:lll_lock_wait
root@# ./tplist -v block:block_rq_complete
block:block_rq_complete
dev_t dev;
sector_t sector;
unsigned int nr_sector;
int errors;
char rwbs[8];
__data_loc char[] cmd;
I haven’t used uprobes in anger yet however looking at the examples for uflow & ucalls these too look ok.
root# ./uflow -C '<stdin>' -l python 17323 Tracing method calls in python process 17323... Ctrl-C to quit. CPU PID TID TIME(us) METHOD 13 17323 17323 2.757 -> <stdin>.<module> 13 17323 17323 2.757 <- <stdin>.<module> root@# ./uflow -l python 17323 Tracing method calls in python process 17323... Ctrl-C to quit. CPU PID TID TIME(us) METHOD 13 17323 17323 3.660 -> /usr/lib64/python2.7/encodings/utf_8.py.decode 13 17323 17323 3.660 <- /usr/lib64/python2.7/encodings/utf_8.py.decode 13 17323 17323 3.660 -> <stdin>.<module> 13 17323 17323 3.663 <- <stdin>.<module> root# ./ucalls 17323 -S Attached 719 kernel probes for syscall tracing. Tracing calls in process 17323 (language: python)... Ctrl-C to quit. ^C METHOD # CALLS /usr/lib64/python2.7/site.py.__repr__ 1 . 1 . 1 /usr/lib64/python2.7/encodings/utf_8.py.decode 2 sys_rt_sigaction 8 sys_ioctl 28 sys_read 46 sys_select 46 sys_write 54 sys_rt_sigprocmask 200 Detaching kernel probes, please wait...
Main tools
A quick run through some of the main tools from the reference guide.
root# ./execsnoop
PCOMM PID PPID RET ARGS
ps 575 29537 0 /usr/bin/ps -ef
ps 584 29537 0 /usr/bin/ps -ef
df 586 29537 0 /usr/bin/df -k
root# ./opensnoop
PID COMM FD ERR PATH
1175 opensnoop -1 2 /usr/lib64/python2.7/encodings/ascii.so
1175 opensnoop -1 2 /usr/lib64/python2.7/encodings/asciimodule.so
1175 opensnoop 35 0 /usr/lib64/python2.7/encodings/ascii.py
1175 opensnoop 36 0 /usr/lib64/python2.7/encodings/ascii.pyc
15804 Client 18 0 /sbin
942 multipathd 50 0 /sys/devices/pci0000:00/0000:00:02.0/0000:05:00.0/host1/rport-1:0-0/target1:0:0/1:0:0:7/state
942 multipathd 50 0 /sys/devices/pci0000:00/0000:00:02.0/0000:05:00.0/host1/rport-1:0-0/target1:0:0/1:0:0:7/state
root# ./ext4slower 1
Tracing ext4 operations slower than 1 ms
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
22:37:54 dd 2106 W 8192 103720 1.86 testfile
22:37:56 dd 2106 W 8192 603160 1.82 testfile
22:37:56 dd 2106 W 8192 603176 3.77 testfile
22:37:56 dd 2106 W 8192 603208 3.36 testfile
22:37:56 dd 2106 W 8192 603320 3.31 testfile
root# ./biolatency
Tracing block device I/O... Hit Ctrl-C to end.
^C
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 112 |****************************************|
32 -> 63 : 4 |* |
root# ./biosnoop
TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms)
0.000000000 java 18389 sda W 107010624 4096 0.04
0.124609000 jbd2/dm-11-8 3831 sda W 19679984 135168 0.07
0.124677000 jbd2/dm-11-8 3831 sda W 19680248 4096 0.03
1.100362000 jbd2/dm-16-8 3807 sda W 97551064 45056 0.05
1.100420000 jbd2/dm-16-8 3807 sda W 97551152 4096 0.03
3.724337000 kworker/u49:0 21993 sda W 160844128 4096 0.04
3.724348000 kworker/u49:0 21993 sda W 160844136 4096 0.04
3.724352000 kworker/u49:0 21993 sda W 160994008 4096 0.04
root# ./cachestat
TOTAL MISSES HITS DIRTIES BUFFERS_MB CACHED_MB
413966 0 413966 20 389 1187
root# ./tcpconnect
PID COMM IP SADDR DADDR DPORT
4680 ssh 4 127.0.0.1 127.0.0.1 22
root# ./runqlat
Tracing run queue latency... Hit Ctrl-C to end.
^C
usecs : count distribution
0 -> 1 : 237 |***************************** |
2 -> 3 : 325 |****************************************|
4 -> 7 : 100 |************ |
8 -> 15 : 14 |* |
root# ./profile
Sampling at 49 Hertz of all threads by user + kernel stack... Hit Ctrl-C to end.
^C
cpuidle_enter_state
cpuidle_idle_call
arch_cpu_idle
cpu_startup_entry
start_secondary
start_cpu
- swapper/5 (0)
1
[unknown]
cpuidle_enter_state
cpuidle_idle_call
arch_cpu_idle
cpu_startup_entry
start_secondary
start_cpu
So I hope this helps cover off some of the examples in the beta release. Obliviously it doesn’t cover all tools in the package however it looks good to me so far. I will definitely be using these more now moving forward and hopefully in production if and when it hits the mainline 7.6 kernel.
Allan McAleavy's Tech Blog 
Hi Allan, thanks for nice overview and testing! urandomread.py actually ships with RHEL-7.6 Beta, it’s part of bcc-doc rather then bcc-tools:
# rpm -ql bcc-doc | grep urandomread.py$
/usr/share/doc/bcc/examples/tracing/urandomread.py
LikeLike
Fantastic I will update the post to reflect this later today. Thanks
LikeLike