Taking eBPF (bcc-tools) for a quick spin in RHEL 7.6 (beta)

I was made aware that eBPF was available now in RHEL via the below tweet from Brendan Gregg. As our main distro is RHEL I was intrigued to see what the release looked like and what was available. For those who don’t know the bcc-tools package contains the eBPF bcc toolset which allows new insights into tracing and performance analysis on linux.

As this is part of the tech preview in the 7.6 Beta we added this release into our satellite packaging system and ran a quick yum upgrade on the host. Once upgraded the install was simply

yum install bcc-tools

I have omitted some details of the install however we can see that the size is 84Mb.

Total download size: 23 M
Installed size: 84 M
Is this ok [y/d/N]: y
Downloading packages:
(1/5): bcc-0.6.0-3.el7.x86_64.rpm                                                                                                                                                                                     | 478 kB  00:00:00
(2/5): bcc-tools-0.6.0-3.el7.x86_64.rpm                                                                                                                                                                               | 377 kB  00:00:00
(3/5): python-bcc-0.6.0-3.el7.x86_64.rpm                                                                                                                                                                              |  53 kB  00:00:00
(4/5): python-netaddr-0.7.5-9.el7.noarch.rpm                                                                                                                                                                          | 983 kB  00:00:00
(5/5): llvm-private-6.0.1-2.el7.x86_64.rpm                                                                                                                                                                            |  21 MB  00:00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                         38 MB/s |  23 MB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : llvm-private-6.0.1-2.el7.x86_64                                                                                                                                                                                           1/5
  Installing : python-netaddr-0.7.5-9.el7.noarch                                                                                                                                                                                         2/5
  Installing : python-bcc-0.6.0-3.el7.x86_64                                                                                                                                                                                             3/5
  Installing : bcc-tools-0.6.0-3.el7.x86_64                                                                                                                                                                                              4/5
  Installing : bcc-0.6.0-3.el7.x86_64                                                                                                                                                                                                    5/5
  Verifying  : bcc-tools-0.6.0-3.el7.x86_64                                                                                                                                                                                              1/5
  Verifying  : python-netaddr-0.7.5-9.el7.noarch                                                                                                                                                                                         2/5
  Verifying  : bcc-0.6.0-3.el7.x86_64                                                                                                                                                                                                    3/5
  Verifying  : llvm-private-6.0.1-2.el7.x86_64                                                                                                                                                                                           4/5
  Verifying  : python-bcc-0.6.0-3.el7.x86_64                                                                                                                                                                                             5/5

Installed:
  bcc-tools.x86_64 0:0.6.0-3.el7

Dependency Installed:
  bcc.x86_64 0:0.6.0-3.el7                            llvm-private.x86_64 0:6.0.1-2.el7                            python-bcc.x86_64 0:0.6.0-3.el7                            python-netaddr.noarch 0:0.7.5-9.el7

We can see we have a few packages with bcc in the name installed bcc-0.6.0.3.el7.x86_64.rpm,bcc-tools.x86_64 0:0.6.0-3.el7 & python-bcc.x86_64 0:0.6.0-3.el7

Looking into these we see the following contains the main python libraries

root# rpm -ql python-bcc-0.6.0-3.el7.x86_64
/usr/lib/python2.7/site-packages/bcc
/usr/lib/python2.7/site-packages/bcc-0.6.0-py2.7.egg-info
/usr/lib/python2.7/site-packages/bcc/__init__.py
/usr/lib/python2.7/site-packages/bcc/__init__.pyc
/usr/lib/python2.7/site-packages/bcc/__init__.pyo
/usr/lib/python2.7/site-packages/bcc/libbcc.py
/usr/lib/python2.7/site-packages/bcc/libbcc.pyc
/usr/lib/python2.7/site-packages/bcc/libbcc.pyo
/usr/lib/python2.7/site-packages/bcc/perf.py
/usr/lib/python2.7/site-packages/bcc/perf.pyc
/usr/lib/python2.7/site-packages/bcc/perf.pyo
/usr/lib/python2.7/site-packages/bcc/table.py
/usr/lib/python2.7/site-packages/bcc/table.pyc
/usr/lib/python2.7/site-packages/bcc/table.pyo
/usr/lib/python2.7/site-packages/bcc/tcp.py
/usr/lib/python2.7/site-packages/bcc/tcp.pyc
/usr/lib/python2.7/site-packages/bcc/tcp.pyo
/usr/lib/python2.7/site-packages/bcc/usdt.py
/usr/lib/python2.7/site-packages/bcc/usdt.pyc
/usr/lib/python2.7/site-packages/bcc/usdt.pyo
/usr/lib/python2.7/site-packages/bcc/utils.py
/usr/lib/python2.7/site-packages/bcc/utils.pyc
/usr/lib/python2.7/site-packages/bcc/utils.pyo

Here we have the main bcc libraries.

root# rpm -ql bcc-0.6.0-3.el7.x86_64
/usr/lib64/libbcc.so.0
/usr/lib64/libbcc.so.0.6.0
/usr/lib64/libbpf.so.0
/usr/lib64/libbpf.so.0.6.0
/usr/share/doc/bcc-0.6.0
/usr/share/doc/bcc-0.6.0/README.md
/usr/share/licenses/bcc-0.6.0
/usr/share/licenses/bcc-0.6.0/COPYRIGHT.txt
/usr/share/licenses/bcc-0.6.0/LICENSE.txt

And for bcc-tools we have the main toolset and also documentation containing examples and man pages.

root# rpm -ql bcc-tools-0.6.0-3.el7.x86_64 |egrep -v "doc|man"
/usr/share/bcc
/usr/share/bcc/introspection
/usr/share/bcc/introspection/bps
/usr/share/bcc/tools
/usr/share/bcc/tools/argdist
/usr/share/bcc/tools/bashreadline
/usr/share/bcc/tools/biolatency
/usr/share/bcc/tools/biosnoop
/usr/share/bcc/tools/biotop
/usr/share/bcc/tools/bitesize
/usr/share/bcc/tools/bpflist
/usr/share/bcc/tools/btrfsdist
/usr/share/bcc/tools/btrfsslower
/usr/share/bcc/tools/cachestat
/usr/share/bcc/tools/cachetop
/usr/share/bcc/tools/capable
/usr/share/bcc/tools/cobjnew
/usr/share/bcc/tools/cpudist
/usr/share/bcc/tools/cpuunclaimed
/usr/share/bcc/tools/dbslower
/usr/share/bcc/tools/dbstat
/usr/share/bcc/tools/dcsnoop
/usr/share/bcc/tools/dcstat
/usr/share/bcc/tools/deadlock_detector
/usr/share/bcc/tools/deadlock_detector.c
/usr/share/bcc/tools/execsnoop
/usr/share/bcc/tools/ext4dist
/usr/share/bcc/tools/ext4slower
/usr/share/bcc/tools/filelife
/usr/share/bcc/tools/fileslower
/usr/share/bcc/tools/filetop
/usr/share/bcc/tools/funccount
/usr/share/bcc/tools/funclatency
/usr/share/bcc/tools/funcslower
/usr/share/bcc/tools/gethostlatency
/usr/share/bcc/tools/hardirqs
/usr/share/bcc/tools/inject
/usr/share/bcc/tools/javacalls
/usr/share/bcc/tools/javaflow
/usr/share/bcc/tools/javagc
/usr/share/bcc/tools/javaobjnew
/usr/share/bcc/tools/javastat
/usr/share/bcc/tools/javathreads
/usr/share/bcc/tools/killsnoop
/usr/share/bcc/tools/lib
/usr/share/bcc/tools/lib/ucalls
/usr/share/bcc/tools/lib/uflow
/usr/share/bcc/tools/lib/ugc
/usr/share/bcc/tools/lib/uobjnew
/usr/share/bcc/tools/lib/ustat
/usr/share/bcc/tools/lib/uthreads
/usr/share/bcc/tools/mdflush
/usr/share/bcc/tools/memleak
/usr/share/bcc/tools/mountsnoop
/usr/share/bcc/tools/mysqld_qslower
/usr/share/bcc/tools/nfsdist
/usr/share/bcc/tools/nfsslower
/usr/share/bcc/tools/nodegc
/usr/share/bcc/tools/nodestat
/usr/share/bcc/tools/offcputime
/usr/share/bcc/tools/offwaketime
/usr/share/bcc/tools/oomkill
/usr/share/bcc/tools/opensnoop
/usr/share/bcc/tools/phpcalls
/usr/share/bcc/tools/phpflow
/usr/share/bcc/tools/phpstat
/usr/share/bcc/tools/pidpersec
/usr/share/bcc/tools/profile
/usr/share/bcc/tools/pythoncalls
/usr/share/bcc/tools/pythonflow
/usr/share/bcc/tools/pythongc
/usr/share/bcc/tools/pythonstat
/usr/share/bcc/tools/reset-trace
/usr/share/bcc/tools/rubycalls
/usr/share/bcc/tools/rubyflow
/usr/share/bcc/tools/rubygc
/usr/share/bcc/tools/rubyobjnew
/usr/share/bcc/tools/rubystat
/usr/share/bcc/tools/runqlat
/usr/share/bcc/tools/runqlen
/usr/share/bcc/tools/runqslower
/usr/share/bcc/tools/slabratetop
/usr/share/bcc/tools/softirqs
/usr/share/bcc/tools/solisten
/usr/share/bcc/tools/sslsniff
/usr/share/bcc/tools/stackcount
/usr/share/bcc/tools/statsnoop
/usr/share/bcc/tools/syncsnoop
/usr/share/bcc/tools/syscount
/usr/share/bcc/tools/tcpaccept
/usr/share/bcc/tools/tcpconnect
/usr/share/bcc/tools/tcpconnlat
/usr/share/bcc/tools/tcpdrop
/usr/share/bcc/tools/tcplife
/usr/share/bcc/tools/tcpretrans
/usr/share/bcc/tools/tcpstates
/usr/share/bcc/tools/tcpsubnet
/usr/share/bcc/tools/tcptop
/usr/share/bcc/tools/tcptracer
/usr/share/bcc/tools/tplist
/usr/share/bcc/tools/trace
/usr/share/bcc/tools/ttysnoop
/usr/share/bcc/tools/vfscount
/usr/share/bcc/tools/vfsstat
/usr/share/bcc/tools/wakeuptime
/usr/share/bcc/tools/xfsdist
/usr/share/bcc/tools/xfsslower

Looking at the list it looks like the whole toolset minus some examples,llcstat,zfstools & lua support however I was wondering as the kernel was showing as 3.10.0-933.el7.x86_64 would all the tools work as expected as we had previous tools in old which targeted older 3.x kernels but would say tracepoints work etc.

Tracepoints

So taking a quick look at tracepoints with the urandomread examples, note these don’t ship in the RHEL rpm’s so I had to create them and they work!! Given that tracepoint support landed in the 4.7 kernel it looks like RedHat have done a great job in backporting this support.

root# chmod 755 urandomread
root# ./urandomread
TIME(s)            COMM             PID    GOTBITS
36967.532695000    sshd             29524  112
36967.532710000    sshd             29524  384
36967.537117000    sshd             29525  384
36970.864592000    python           9245   128
36972.782119000    sshd             29524  128
36972.790661000    systemd          1      128
36972.824703000    sshd             29536  384

And again lets test urandomread-explicit from the examples

root# ./urandomread-explicit
TIME(s)            COMM             PID    GOTBITS
37171.040068000    python           9245   128
37173.119018000    dd               31629  8192
37173.119028000    dd               31629  8192
37173.119035000    dd               31629  8192
37173.119043000    dd               31629  8192
37173.119050000    dd               31629  8192

Uprobes

Looking at uprobes we have tplist which list the probes we can attach too, working through the examples for say tplist this looks good.

root# ./tplist -l pthread
/lib64/libpthread.so.0 libpthread:lll_futex_wake
/lib64/libpthread.so.0 libpthread:pthread_start
/lib64/libpthread.so.0 libpthread:pthread_create
/lib64/libpthread.so.0 libpthread:pthread_join
/lib64/libpthread.so.0 libpthread:pthread_join_ret
/lib64/libpthread.so.0 libpthread:mutex_init
/lib64/libpthread.so.0 libpthread:mutex_destroy
/lib64/libpthread.so.0 libpthread:mutex_acquired
/lib64/libpthread.so.0 libpthread:mutex_entry
/lib64/libpthread.so.0 libpthread:mutex_timedlock_entry
/lib64/libpthread.so.0 libpthread:mutex_timedlock_acquired
/lib64/libpthread.so.0 libpthread:mutex_release
/lib64/libpthread.so.0 libpthread:rwlock_destroy
/lib64/libpthread.so.0 libpthread:rdlock_entry
/lib64/libpthread.so.0 libpthread:wrlock_entry
/lib64/libpthread.so.0 libpthread:cond_init
/lib64/libpthread.so.0 libpthread:cond_destroy
/lib64/libpthread.so.0 libpthread:cond_wait
/lib64/libpthread.so.0 libpthread:cond_timedwait
/lib64/libpthread.so.0 libpthread:cond_signal
/lib64/libpthread.so.0 libpthread:cond_broadcast
/lib64/libpthread.so.0 libpthread:lll_lock_wait_private
/lib64/libpthread.so.0 libpthread:lll_lock_wait

root@# ./tplist -v block:block_rq_complete
block:block_rq_complete
    dev_t dev;
    sector_t sector;
    unsigned int nr_sector;
    int errors;
    char rwbs[8];
    __data_loc char[] cmd;

I haven’t used uprobes in anger yet however looking at the examples for uflow & ucalls these too look ok.

root# ./uflow -C '<stdin>' -l python 17323
Tracing method calls in python process 17323... Ctrl-C to quit.
CPU PID    TID    TIME(us) METHOD
13  17323  17323  2.757    -> <stdin>.<module>
13  17323  17323  2.757    <- <stdin>.<module>


root@# ./uflow -l python 17323
Tracing method calls in python process 17323... Ctrl-C to quit.

CPU PID    TID    TIME(us) METHOD
13  17323  17323  3.660    -> /usr/lib64/python2.7/encodings/utf_8.py.decode
13  17323  17323  3.660    <- /usr/lib64/python2.7/encodings/utf_8.py.decode 
13  17323  17323  3.660    -> <stdin>.<module>
13  17323  17323  3.663    <- <stdin>.<module>


root# ./ucalls 17323 -S
Attached 719 kernel probes for syscall tracing.
Tracing calls in process 17323 (language: python)... Ctrl-C to quit.
^C
METHOD                                              # CALLS
/usr/lib64/python2.7/site.py.__repr__                     1
.                                          1
.                                          1
/usr/lib64/python2.7/encodings/utf_8.py.decode            2
sys_rt_sigaction                                          8
sys_ioctl                                                28
sys_read                                                 46
sys_select                                               46
sys_write                                                54
sys_rt_sigprocmask                                      200
Detaching kernel probes, please wait...

 

Main tools

A quick run through some of the main tools from the reference guide.

root# ./execsnoop
PCOMM            PID    PPID   RET ARGS
ps               575    29537    0 /usr/bin/ps -ef
ps               584    29537    0 /usr/bin/ps -ef
df               586    29537    0 /usr/bin/df -k

root# ./opensnoop
PID    COMM               FD ERR PATH
1175   opensnoop          -1   2 /usr/lib64/python2.7/encodings/ascii.so
1175   opensnoop          -1   2 /usr/lib64/python2.7/encodings/asciimodule.so
1175   opensnoop          35   0 /usr/lib64/python2.7/encodings/ascii.py
1175   opensnoop          36   0 /usr/lib64/python2.7/encodings/ascii.pyc
15804  Client             18   0 /sbin
942    multipathd         50   0 /sys/devices/pci0000:00/0000:00:02.0/0000:05:00.0/host1/rport-1:0-0/target1:0:0/1:0:0:7/state
942    multipathd         50   0 /sys/devices/pci0000:00/0000:00:02.0/0000:05:00.0/host1/rport-1:0-0/target1:0:0/1:0:0:7/state

root# ./ext4slower 1
Tracing ext4 operations slower than 1 ms
TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
22:37:54 dd             2106   W 8192    103720      1.86 testfile
22:37:56 dd             2106   W 8192    603160      1.82 testfile
22:37:56 dd             2106   W 8192    603176      3.77 testfile
22:37:56 dd             2106   W 8192    603208      3.36 testfile
22:37:56 dd             2106   W 8192    603320      3.31 testfile

root# ./biolatency
Tracing block device I/O... Hit Ctrl-C to end.
^C
     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 112      |****************************************|
        32 -> 63         : 4        |*                                       |

root# ./biosnoop
TIME(s)        COMM           PID    DISK    T  SECTOR    BYTES   LAT(ms)
0.000000000    java           18389  sda     W  107010624 4096       0.04
0.124609000    jbd2/dm-11-8   3831   sda     W  19679984  135168     0.07
0.124677000    jbd2/dm-11-8   3831   sda     W  19680248  4096       0.03
1.100362000    jbd2/dm-16-8   3807   sda     W  97551064  45056      0.05
1.100420000    jbd2/dm-16-8   3807   sda     W  97551152  4096       0.03
3.724337000    kworker/u49:0  21993  sda     W  160844128 4096       0.04
3.724348000    kworker/u49:0  21993  sda     W  160844136 4096       0.04
3.724352000    kworker/u49:0  21993  sda     W  160994008 4096       0.04

root# ./cachestat
   TOTAL   MISSES     HITS  DIRTIES   BUFFERS_MB  CACHED_MB
  413966        0   413966       20          389       1187


root# ./tcpconnect
PID    COMM         IP SADDR            DADDR            DPORT
4680   ssh          4  127.0.0.1        127.0.0.1        22

root# ./runqlat
Tracing run queue latency... Hit Ctrl-C to end.
^C
     usecs               : count     distribution
         0 -> 1          : 237      |*****************************           |
         2 -> 3          : 325      |****************************************|
         4 -> 7          : 100      |************                            |
         8 -> 15         : 14       |*                                       |

root# ./profile
Sampling at 49 Hertz of all threads by user + kernel stack... Hit Ctrl-C to end.
^C
    cpuidle_enter_state
    cpuidle_idle_call
    arch_cpu_idle
    cpu_startup_entry
    start_secondary
    start_cpu
    -                swapper/5 (0)
        1

    [unknown]
    cpuidle_enter_state
    cpuidle_idle_call
    arch_cpu_idle
    cpu_startup_entry
    start_secondary
    start_cpu

So I hope this helps cover off some of the examples in the beta release. Obliviously it doesn’t cover all tools in the package however it looks good to me so far. I will definitely be using these more now moving forward and hopefully in production if and when it hits the mainline 7.6 kernel.

2 thoughts on “Taking eBPF (bcc-tools) for a quick spin in RHEL 7.6 (beta)

  1. Hi Allan, thanks for nice overview and testing! urandomread.py actually ships with RHEL-7.6 Beta, it’s part of bcc-doc rather then bcc-tools:

    # rpm -ql bcc-doc | grep urandomread.py$
    /usr/share/doc/bcc/examples/tracing/urandomread.py

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s