A linux type sar -n DEV for ESXi 6

I have been looking a lot at ESX performance and today I was looking at network stats. I can get network data in two second samples from esxtop however parsing of the data is cumbersome.

I could use powershell however in this method I need to do a delta from two sets of data where this again pushes the sample time out to many seconds. What I would like is to run a tool similar to sar -n DEV 1 below we can see an ouput from a standard RedHat host.

21:29:41        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
21:29:42        bond0   8148.98   2310.20  11525.37   1933.02      0.00      0.00     28.57
21:29:42         eth0   8130.61   2310.20  11524.02   1933.02      0.00      0.00     14.29
21:29:42         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
21:29:42         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00
21:29:42         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00
21:29:42         eth4     18.37      0.00      1.36      0.00      0.00      0.00     14.29
21:29:42         eth5      0.00      0.00      0.00      0.00      0.00      0.00      0.00
21:29:42         eth6      0.00      0.00      0.00      0.00      0.00      0.00      0.00
21:29:42         eth7      0.00      0.00      0.00      0.00      0.00      0.00      0.00
21:29:42           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00

As a starting point what I have is as follows from the ESX command prompt.

TSTAMP       IFACE    rxpck/s  txpck/s  rxMB/s   txMB/s   rxsize   txsize   rxeps    txeps
10:14:46 AM  vmnic0   13       1        0.00     0.00     112      399      0        0
10:14:46 AM  vmnic4   9        0        0.00     0.00     86       0        0        0
10:14:46 AM  vmnic2   5        0        0.00     0.00     135      0        0        0
10:14:46 AM  vmnic6   4        0        0.00     0.00     94       0        0        0
10:14:46 AM  vmnic1   3        0        0.00     0.00     163      0        0        0
10:14:46 AM  vmnic5   2        0        0.00     0.00     94       0        0        0

With some data being copied between two VMs on the same node over two different vlans.

TSTAMP       IFACE    rxpck/s  txpck/s  rxMB/s   txMB/s   rxsize   txsize   rxeps    txeps
08:52:23 PM  vmnic0   11       5        0        0        78       233      0        0
08:52:23 PM  vmnic4   8        0        0        0        85       0        0        0
08:52:23 PM  vmnic2   23801    2572     34       0        1449     70       0        0
08:52:23 PM  vmnic6   2577     1784     0        33       70       18516    0        0
08:52:23 PM  vmnic1   2        0        0        0        94       0        0        0
08:52:23 PM  vmnic5   2        0        0        0        94       0        0        0
08:52:24 PM  vmnic0   11       6        0        0        78       214      0        0
08:52:24 PM  vmnic4   8        0        0        0        85       0        0        0
08:52:24 PM  vmnic2   44308    4182     64       0        1456     71       0        0
08:52:24 PM  vmnic6   4189     3005     0        61       71       20583    0        0
08:52:24 PM  vmnic1   2        0        0        0        94       0        0        0
08:52:24 PM  vmnic5   2        0        0        0        94       0        0        0
08:52:25 PM  vmnic0   12       6        0        0        79       209      0        0
08:52:25 PM  vmnic4   8        0        0        0        85       0        0        0
08:52:25 PM  vmnic2   46642    3814     68       0        1465     71       0        0
08:52:25 PM  vmnic6   3820     2810     0        65       71       23286    0        0
08:52:25 PM  vmnic1   2        0        0        0        94       0        0        0
08:52:25 PM  vmnic5   2        0        0        0        94       0        0        0
08:52:26 PM  vmnic0   12       15       0        0        92       227      0        0
08:52:26 PM  vmnic4   9        0        0        0        103      0        0        0
08:52:26 PM  vmnic2   42063    4161     61       0        1450     70       0        0
08:52:26 PM  vmnic6   4170     2994     0        58       71       19515    0        0
08:52:26 PM  vmnic1   2        0        0        0        94       0        0        0
08:52:26 PM  vmnic5   2        0        0        0        94       0        0        0
08:52:27 PM  vmnic0   14       9        0        0        126      239      0        0
08:52:27 PM  vmnic4   8        0        0        0        85       0        0        0
08:52:27 PM  vmnic2   47247    3996     69       0        1460     74       0        0
08:52:27 PM  vmnic6   4003     3028     0        66       74       21831    0        0
08:52:27 PM  vmnic1   3        0        0        0        94       0        0        0
08:52:27 PM  vmnic5   3        0        0        0        94       0        0        0
08:52:28 PM  vmnic0   10       61       0        0        80       238      0        0
08:52:28 PM  vmnic4   8        0        0        0        85       0        0        0
08:52:28 PM  vmnic2   39979    3471     58       0        1452     71       0        0
08:52:28 PM  vmnic6   3477     2784     0        55       71       19979    0        0
08:52:28 PM  vmnic1   2        0        0        0        94       0        0        0
08:52:28 PM  vmnic5   2        0        0        0        94       0        0        0

EDIT– I have updated the script to be floating point as this matches more what sar would show.

Or a test where we migrate a VM to this server.

TSTAMP       IFACE    rxpck/s  txpck/s  rxMB/s   txMB/s   rxsize   txsize   rxeps    txeps
10:15:58 AM  vmnic0   13       34       0.00     0.01     155      268      0        0
10:15:58 AM  vmnic4   8        0        0.00     0.00     85       0        0        0
10:15:58 AM  vmnic2   5        0        0.00     0.00     87       0        0        0
10:15:58 AM  vmnic6   5        0        0.00     0.00     87       0        0        0
10:15:58 AM  vmnic1   81109    9967     121.64   0.66     1499     66       0        0
10:15:58 AM  vmnic5   80496    9540     120.80   0.62     1500     66       0        0
10:15:59 AM  vmnic0   16       64       0.00     0.01     201      268      0        0
10:15:59 AM  vmnic4   9        0        0.00     0.00     82       0        0        0
10:15:59 AM  vmnic2   4        0        0.00     0.00     94       0        0        0
10:15:59 AM  vmnic6   4        0        0.00     0.00     94       0        0        0
10:15:59 AM  vmnic1   81842    9792     122.60   0.65     1498     66       0        0
10:15:59 AM  vmnic5   81698    9849     122.41   0.65     1498     66       0        0

We can see in the first test was that we have data being copied up to 71 MB/sec between interfaces. And in our second test we can see a short sample of the data showing we are at the maximum BW for two vmnics @1G.

I thought I would also show the performance charts from the Vsphere client
vmnic_BW

We can see we have one spike in the chart highlighted in red as this is a 20 second average we can see we have hit peak BW. However we cant tell from this data how many times we were at peak utilisation.

recieve_mb.png

In the above image we can see we are running at max utilisation for the nic’s over the sample period. So where we would have thought that we may have peaked at our max util we can see we were actually running at it for 30 seconds.

So how do we get the data? Well I use the command net-stats where I parse the output and process via awk.


net-stats -A -tc -i 1 -n 300 | sed 's/["|":|:|/|,]//g' |awk -f ./nstat.awk

BEGIN { printf ("%-12s %-8s %-8s %-8s %-8s %-8s %-8s %-8s %-8s %-8s\n","TSTAMP","IFACE","rxpck\/s","txpck\/s","rxMB\/s","txMB\/s","rxsize","txsize","rxeps","txeps");}
 {
 if ( $0 ~ /time / ) { ts=$2;}
 if ( $0 ~ /vmnic/ )
 {
 vmnic=$2;
 found=1;
 }
 if (( $0 ~ /txpps/ ) && ( found == 1))
 {
 printf("%-12s %-8s %-8d %-8d %-8.2f %-8.2f %-8d %-8d %-8d %-8d \n",strftime("%r",ts),vmnic,$10,$2,$12/8,$4/8,$14,$6,$NF,$8);
 found=0;
 }
 }

Enjoy…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s