Overview

In this post we will take a look at Ports the data is gathered by the statport command. We can view different metrics when parsing the data from this command including iops, throughput, latency and queuing.

• Cache 
  • statcmp
• Cpu
  • statcpu
• Hosts
  • statvlun / statvlun -hostsum / -vvsum
• Ports
  • statport -host
• Volumes
  • statvlun -vvsum

Getting started..

How do I run the statport command? Below shows how to setup a passfile so we don’t need to input a password for our commands.

# PATH=$PATH:/opt/hp_3par_cli/bin/
# setpassword -saveonly -file mypass.your3par 
system: your3par 
user:  
password:

Ports

Port performance metrics are very detailed from the statport -host output we have the following header which details what we can utilise. The statport command is for showing host based port details.

00:00:27 09/15/2017 r/w      I/O per second         KBytes per sec      Svt ms     IOSz KB
     Port       D/C        Cur    Avg   Max     Cur     Avg    Max   Cur   Avg   Cur   Avg Qlen

Lets run through what these are:

• Port: Port number
• D/C: Data or Control ( Data – Host based IOPS, Ctrl – VMware VAAI ofload)
• r/w: read/write/total

I/O per second

• Cur: Current IOPS/sec
• Avg: Average IOPS/sec
• Max: Max IOPS/sec

KBytes per sec

• Cur: Current Kbytes/sec
• Avg: Average Kbytes/sec during sample period
• Max: Maximum Kbytes/sec during sample period

Svt ms 

• Cur: Current service time in milliseconds
• Avg: Average service time in milliseconds during the sample period

IOSz  – I/O Size

• Cur: Current I/O size
• Avg: Average I/O size during sample period.
• Qlen: Length of the port Queue

So what metrics are we interested in, well it depends what you are looking for but I take the current IOPS, Kbytes, service time and Qlen as these are reported as totals for the Ports.

*** Monitor QLEN if high you will see high latency *** 

statport command output

Below we have some example output which has been truncated to a single sample.

Command run was statport -sys your3par -host -rw -both -iter 960 -d 15

00:00:27 09/15/2017 r/w      I/O per second         KBytes per sec      Svt ms     IOSz KB
     Port       D/C        Cur    Avg   Max     Cur     Avg    Max   Cur   Avg   Cur   Avg Qlen
    0:4:1      Data   r   3196   3196  3196  119447  119447 119447  1.08  1.08  37.4  37.4    -
    0:4:1      Data   w   3902   3902  3902  127147  127147 127147  3.06  3.06  32.6  32.6    -
    0:4:1      Data   t   7098   7098  7098  246593  246593 246593  2.17  2.17  34.7  34.7   15
    0:4:1      Ctrl   r     37     37    37       3       3      3  0.49  0.49   0.1   0.1    -
    0:4:1      Ctrl   w      0      0     0       0       0      0  0.00  0.00   0.0   0.0    -
    0:4:1      Ctrl   t     37     37    37       3       3      3  0.49  0.49   0.1   0.1    0
    0:4:2      Data   r     85     85    85     304     304    304  0.82  0.82   3.6   3.6    -
    0:4:2      Data   w     59     59    59      53      53     53  3.83  3.83   0.9   0.9    -
    0:4:2      Data   t    143    143   143     357     357    357  2.05  2.05   2.5   2.5    0
    0:4:2      Ctrl   r      8      8     8       0       0      0  0.05  0.05   0.0   0.0    -
    0:4:2      Ctrl   w      0      0     0       0       0      0  0.00  0.00   0.0   0.0    -
    0:4:2      Ctrl   t      8      8     8       0       0      0  0.05  0.05   0.0   0.0    0

In this example we can see two ports (0:4:1/0:4:2) the main thing we are looking for are the Data ports which are used for host access. We can see on port 0:4:1 that we have 3196 read IOPS and 3902 write IOPS with a combined throughput of about 240MB/sec

** What to track High Queuing / Service Times / Bandwidth ( make sure you know the speed of your Array ports and if they are getting closed to maximum)

800MB/sec ~ 8Gb/sec Port speed.** 

A awk based parsing script

So we can parse this data pretty easily using awk .

BEGIN {
  printf("%-8s %-7s %-10s %-5s %-6s %8s\n","Time","Port  T","IOPS","BW_MB","Lat(ms)","Queue");
}

{
 if ($0 ~ /KB/) {time=$1}
 if (($1 ~ /[0-6]:[0-6]:[0-6]/ )  && ( $2 ~ /Data/ ))
 {
    iops[$1" "$3]=$4;
    bw[$1" "$3]=$7;
    svc[$1" "$3]=$10;
    qlen[$1" "$3]=$NF;
 }
 if ( $0 ~ /------/ )
 {
   for (h in iops)
   {
     printf("%-8s %-7s %-10s %-5d %-4.2f %8s\n",time,h,iops[h],int(bw[h])/1024,svc[h],qlen[h]);
     delete iops[h];
     delete bw[h];
     delete svc[h];
     delete qlen[h];
   }
}
}

We can also use this script with a file by doing the following to show a similar output to the above.

awk -f ./awk_port statport-host.out |egrep "IOPS|0:4:1|0:4:2" |head
Time     Port  T IOPS       BW_MB Lat(ms)    Queue
00:00:27 0:4:1 r 3196       116   1.08        -
00:00:27 0:4:1 t 7098       240   2.17       15
00:00:27 0:4:1 w 3902       124   3.06        -
00:00:27 0:4:2 r 85         0     0.82        -
00:00:27 0:4:2 t 143        0     2.05        0
00:00:27 0:4:2 w 59         0     3.83        -
00:00:43 0:4:1 r 3600       138   1.38        -
00:00:43 0:4:1 t 7492       270   2.70       13
00:00:43 0:4:1 w 3892       131   3.93        -

One thing to note is that we only get the queue when we have the total for the port and this is not broken down by reads or writes.

With this data you can now pivot this in excel and draw a nice graph based on type etc.

But in reality we don’t use excel for this we use Grafana, so lets see what that looks like when we put all the data together. First we need a parsing script.  Below is an excerpt from the parsing script for cpu/cache and ports.

# Gather host port details
file=open('statport-host.out', 'rb', 0)
for line in file:
    data=line.split()
    if re.search(b'KBytes',line):
       dte=line.split()
       tsp = data[0] + " " + data[1]
       ts=int(time.mktime(time.strptime(tsp,pattern)))

    if re.search(b'[0-9]:[0-9]:[0-9]',line):
       if re.search(b'Data',line):
          port=data[0]    # port
          cord=data[1]    # Ctrl or Data
          rwt=data[2]     # read write or total
          cio=data[3]     # current IOPS
          aio=data[4]     # average IOPS
          mio=data[5]     # max IOPS
          ckb=data[6]     # current kb
          akb=data[7]     # average kb
          mkb=data[8]     # max kb
          csvc=data[9]    # current service time ms
          asvc=data[10]   # average service time ms
          iosz=data[11]   # io size
          aiosz=data[12]  # average iosize

          if re.match(b't',rwt):
             queue=data[13]
             print("PortIO,PortIO=%s,type=tiops value=%s %d" %(port,cio,ts))
             print("PortQ,PortQ=%s,type=qls value=%s %d" %(port,queue,ts))
             print("PortBW,PortBW=%s,type=tkb value=%s %d" %(port,ckb,ts))
             print("PortRT,PortRT=%s,type=tms value=%s %d" %(port,csvc,ts))
          if re.match(b'r',rwt):
             print("PortIO,PortIO=%s,type=riops value=%s %d" %(port,cio,ts))
             print("PortBW,PortBW=%s,type=rkb value=%s %d" %(port,ckb,ts))
             print("PortRT,PortRT=%s,type=rms value=%s %d" %(port,csvc,ts))
          if re.match(b'w',rwt):
             print("PortIO,PortIO=%s,type=wiops value=%s %d" %(port,cio,ts))
             print("PortBW,PortBW=%s,type=wkb value=%s %d" %(port,ckb,ts))
             print("PortRT,PortRT=%s,type=wms value=%s %d" %(port,csvc,ts))

file.close()

The output will look familiar as with the other scripts we use.

PortIO,PortIO=0:4:1,type=riops value=3196 1505430027
PortBW,PortBW=0:4:1,type=rkb value=119447 1505430027
PortRT,PortRT=0:4:1,type=rms value=1.08 1505430027
PortIO,PortIO=0:4:1,type=wiops value=3902 1505430027
PortBW,PortBW=0:4:1,type=wkb value=127147 1505430027
PortRT,PortRT=0:4:1,type=wms value=3.06 1505430027
PortIO,PortIO=0:4:1,type=tiops value=7098 1505430027
PortQ,PortQ=0:4:1,type=qls value=15 1505430027
PortBW,PortBW=0:4:1,type=tkb value=246593 1505430027
PortRT,PortRT=0:4:1,type=tms value=2.17 1505430027
PortIO,PortIO=0:4:2,type=riops value=85 1505430027
PortBW,PortBW=0:4:2,type=rkb value=304 1505430027
PortRT,PortRT=0:4:2,type=rms value=0.82 1505430027
PortIO,PortIO=0:4:2,type=wiops value=59 1505430027
PortBW,PortBW=0:4:2,type=wkb value=53 1505430027
PortRT,PortRT=0:4:2,type=wms value=3.83 1505430027
PortIO,PortIO=0:4:2,type=tiops value=143 1505430027

Once we have parsed we use curl to input into influx and visualise as follows. (See XIV posts etc on how to enter the data into influx)

Each part below can be opened in a seperate row in Grafana. This is broken down so that we can see the full picture.

Port Queuing and Array total IOPS

** Look for high queuing as this will show where latency is high **

Breakdown of Port IOPS by type.

Breakdown of Port BW by type.

** Look for balance in the totals(IOPS/BW) across the ports zoned to servers to make sure that the workload is balanced and the hosts i/o policy are set to round_robin. **

Breakdown of Port Latency by type.
Cache and CPU – See previous posts..

All Hosts IOPS, Latency & BW Breakdown


*** Look out for high queuing and high latency – look for hosts which are maxing out their read bandwidth (see hosts post ) to find noisy workloads causing queuing ***

CPU Heatmaps and Latency Heatmaps.

** Title say it all lots of data less noise **

Scripts & Grafana dashboards are available here: 3par_performance