버전 비교

  • 이 줄이 추가되었습니다.
  • 이 줄이 삭제되었습니다.
  • 서식이 변경되었습니다.


목차

Host Spec

HostAssetModelCPUMEMNICIP
HOST_A2020024816SMCI/IR110P/1U/200GXeon(R) Gold 6334 CPU @ 3.60GHz 8C 16T96GBConnectX-6 200GbE Dual10.0.1.10
HOST_B2023000636SMCi/IR110P/1U/200GXeon(R) Gold 6346 CPU @ 3.10GHz 16C 32T96GBConnectX-6 200GbE Dual10.0.1.20

Config

OS, SW

코드 블럭
# uname -a
Linux perfsonar5-kisti 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64 GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

# iperf3 --version
iperf 3.17.1 (cJSON 1.7.15)
Linux perfsonar5-kisti 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threads

...

MTU 9000

코드 블럭
linenumberstrue
HOST_A # ping -M do -s 8972 10.0.1.20 -c 5
PING 10.0.1.20 (10.0.1.20) 8972(9000) bytes of data.
8980 bytes from 10.0.1.20: icmp_seq=1 ttl=64 time=0.242 ms
8980 bytes from 10.0.1.20: icmp_seq=2 ttl=64 time=0.268 ms
8980 bytes from 10.0.1.20: icmp_seq=3 ttl=64 time=0.356 ms
8980 bytes from 10.0.1.20: icmp_seq=4 ttl=64 time=0.311 ms
8980 bytes from 10.0.1.20: icmp_seq=5 ttl=64 time=0.315 ms
--- 10.0.1.20 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4076ms
rtt min/avg/max/mdev = 0.242/0.298/0.356/0.039 ms

...

코드 블럭
languagebash
linenumberstrue
#!/bin/bash

SERVER_IP="10.0.1.10"
DURATION=13
OFFSET=3
ITERATIONS=10
TPUT_FILE="tput_results.tsv"
RETRIES_FILE="retries_results.tsv"

echo -e "streams\titer1\titer2\titer3\titer4\titer5\titer6\titer7\titer8\titer9\titer10\tavg" > "$TPUT_FILE"
echo -e "streams\titer1\titer2\titer3\titer4\titer5\titer6\titer7\titer8\titer9\titer10\tavg" > "$RETRIES_FILE"

for P in {1..1210}; do
  TPUT_VALUES=()
  RETRY_VALUES=()
  for ((i=1; i<=ITERATIONS; i++)); do
    JSON_OUTPUT=$(iperf3 -c $SERVER_IP -i 1 -t $DURATION -O $OFFSET -P $P -J)
    THROUGHPUT=$(echo "$JSON_OUTPUT" | jq -r '.end.sum_received.bits_per_second' | awk '{print $1 / 1e9}')
    RETRIES=$(echo "$JSON_OUTPUT" | jq -r '.end.sum_sent.retransmits')
    TPUT_VALUES+=("$THROUGHPUT")
    RETRY_VALUES+=("$RETRIES")
    sleep 10
  done
  TPUT_AVG=$(echo "${TPUT_VALUES[@]}" | awk '{for(i=1;i<=NF;i++) sum+=$i; print sum/NF}')
  RETRY_AVG=$(echo "${RETRY_VALUES[@]}" | awk '{for(i=1;i<=NF;i++) sum+=$i; print sum/NF}')
  echo -e "$P\t${TPUT_VALUES[*]// /\\t}\t$TPUT_AVG" >> "$TPUT_FILE"
  echo -e "$P\t${RETRY_VALUES[*]// /\\t}\t$RETRY_AVG" >> "$RETRIES_FILE"
done

Test

...

Results

Throughput (Gbps)

streamsiter1iter2iter3iter4iter5iter6iter7iter8iter9iter10avg
157.510358.433158.346158.078458.297157.036357.112456.685956.726957.677457.5904
293.336497.559798.075597.892598.147397.746697.58598.152898.230798.722897.5449
3123.778133.295134.464135.621135.088135.191134.999135.302133.645132.244133.363
4167.337137.069166.946167.785167.176166.494166.75167.35168.059166.518164.148
5177.734169.236169.334174.266179.907170.806177.147164.093170.572174.037172.713
6178.636180.729180.96175.383179.595179.805175.311180.81180.798180.058179.209

...

7
streamsiter1iter2iter3iter4iter5iter6iter7iter8iter9iter10avg117399117134245200187198211234179.82782577590225563443549272352468482.131118333330601383517467344394336482.34781710410644450474543611419566560.851313301527848799548503744527467657.7

etc

...

179.966180.08180.729180.097180.975180.72180.387180.238179.946180.262180.34
8179.911178.766179.459179.388179.374179.601179.296179.664180.167179.127179.475
9178.778179.474179.236179.221179.09179.008178.881179.556179.087178.366179.07
10177.788178.057178.302178.367178.58176.807177.586178.144178.389178.45178.047

Retries

streamsiter1iter2iter3iter4iter5iter6iter7iter8iter9iter10avg
117399117134245200187198211234179.8
2782577590225563443549272352468482.1
31118333330601383517467344394336482.3
4781710410644450474543611419566560.8
51313301527848799548503744527467657.7
6890234659271205224740228232181386.4
7291167514205165367772271173315324
852838754639331812491249442339167561.8
9129989211411004481254702392710183802.1
106934181203912064361385254796640216531550.8

Example Full Logs (4 Streams)

HOST_AHOST_B
코드 블럭
linenumberstrue
collapsetrue
# iperf3 -s -i 1
-----------------------------------------------------------
Server listening on 5201 (test #4)
-----------------------------------------------------------
Accepted connection from 10.0.1.20, port 55514
[  5] local 10.0.1.10 port 5201 connected to 10.0.1.20 port 55522
[  8] local 10.0.1.10 port 5201 connected to 10.0.1.20 port 55534
[ 10] local 10.0.1.10 port 5201 connected to 10.0.1.20 port 55548
[ 12] local 10.0.1.10 port 5201 connected to 10.0.1.20 port 55554
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  4.66 GBytes  40.0 Gbits/sec    (omitted)
[  8]   0.00-1.00   sec  4.98 GBytes  42.7 Gbits/sec    (omitted)
[ 10]   0.00-1.06   sec  4.99 GBytes  40.5 Gbits/sec    (omitted)
[ 12]   0.00-1.06   sec  4.94 GBytes  40.1 Gbits/sec    (omitted)
[SUM]   0.00-1.00   sec  19.6 GBytes   168 Gbits/sec    (omitted)
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  4.81 GBytes  41.3 Gbits/sec    (omitted)
[  8]   1.00-2.00   sec  4.75 GBytes  40.8 Gbits/sec    (omitted)
[ 10]   1.06-2.00   sec  4.66 GBytes  42.4 Gbits/sec    (omitted)
[ 12]   1.06-2.00   sec  4.56 GBytes  41.4 Gbits/sec    (omitted)
[SUM]   1.00-2.00   sec  18.8 GBytes   161 Gbits/sec    (omitted)
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  4.76 GBytes  40.9 Gbits/sec    (omitted)
[  8]   2.00-3.00   sec  5.04 GBytes  43.3 Gbits/sec    (omitted)
[ 10]   2.00-3.00   sec  4.94 GBytes  42.5 Gbits/sec    (omitted)
[ 12]   2.00-3.00   sec  4.77 GBytes  41.0 Gbits/sec    (omitted)
[SUM]   2.00-3.00   sec  19.5 GBytes   168 Gbits/sec    (omitted)
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   0.00-1.00   sec  4.78 GBytes  41.0 Gbits/sec
[  8]   0.00-1.00   sec  5.06 GBytes  43.4 Gbits/sec
[ 10]   0.00-1.00   sec  4.96 GBytes  42.5 Gbits/sec
[ 12]   0.00-1.00   sec  4.84 GBytes  41.5 Gbits/sec
[SUM]   0.00-1.00   sec  19.6 GBytes   169 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  4.71 GBytes  40.4 Gbits/sec
[  8]   1.00-2.00   sec  5.10 GBytes  43.8 Gbits/sec
[ 10]   1.00-2.00   sec  4.91 GBytes  42.3 Gbits/sec
[ 12]   1.00-2.00   sec  4.76 GBytes  41.0 Gbits/sec
[SUM]   1.00-2.00   sec  19.5 GBytes   167 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  4.77 GBytes  41.0 Gbits/sec
[  8]   2.00-3.00   sec  5.05 GBytes  43.4 Gbits/sec
[ 10]   2.00-3.00   sec  4.74 GBytes  40.7 Gbits/sec
[ 12]   2.00-3.00   sec  4.47 GBytes  38.4 Gbits/sec
[SUM]   2.00-3.00   sec  19.0 GBytes   163 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  4.77 GBytes  40.9 Gbits/sec
[  8]   3.00-4.00   sec  5.05 GBytes  43.3 Gbits/sec
[ 10]   3.00-4.00   sec  4.89 GBytes  42.0 Gbits/sec
[ 12]   3.00-4.00   sec  4.74 GBytes  40.8 Gbits/sec
[SUM]   3.00-4.00   sec  19.5 GBytes   167 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  4.80 GBytes  41.2 Gbits/sec
[  8]   4.00-5.00   sec  5.06 GBytes  43.4 Gbits/sec
[ 10]   4.00-5.00   sec  4.95 GBytes  42.5 Gbits/sec
[ 12]   4.00-5.00   sec  4.72 GBytes  40.5 Gbits/sec
[SUM]   4.00-5.00   sec  19.5 GBytes   168 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  4.83 GBytes  41.5 Gbits/sec
[  8]   5.00-6.00   sec  5.08 GBytes  43.6 Gbits/sec
[ 10]   5.00-6.00   sec  5.02 GBytes  43.3 Gbits/sec
[ 12]   5.00-6.00   sec  4.74 GBytes  40.9 Gbits/sec
[SUM]   5.00-6.00   sec  19.7 GBytes   169 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  4.71 GBytes  40.5 Gbits/sec
[  8]   6.00-7.00   sec  5.05 GBytes  43.4 Gbits/sec
[ 10]   6.00-7.01   sec  5.03 GBytes  42.9 Gbits/sec
[ 12]   6.00-7.01   sec  4.74 GBytes  40.5 Gbits/sec
[SUM]   6.00-7.00   sec  19.5 GBytes   168 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  4.78 GBytes  41.1 Gbits/sec
[  8]   7.00-8.00   sec  5.05 GBytes  43.4 Gbits/sec
[ 10]   7.01-8.00   sec  5.00 GBytes  43.2 Gbits/sec
[ 12]   7.01-8.00   sec  4.74 GBytes  41.0 Gbits/sec
[SUM]   7.00-8.00   sec  19.6 GBytes   168 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  4.77 GBytes  41.0 Gbits/sec
[  8]   8.00-9.00   sec  5.09 GBytes  43.8 Gbits/sec
[ 10]   8.00-9.00   sec  5.00 GBytes  42.9 Gbits/sec
[ 12]   8.00-9.00   sec  4.75 GBytes  40.8 Gbits/sec
[SUM]   8.00-9.00   sec  19.6 GBytes   168 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  4.83 GBytes  41.5 Gbits/sec
[  8]   9.00-10.00  sec  5.10 GBytes  43.8 Gbits/sec
[ 10]   9.00-10.00  sec  4.99 GBytes  42.8 Gbits/sec
[ 12]   9.00-10.00  sec  4.73 GBytes  40.6 Gbits/sec
[SUM]   9.00-10.00  sec  19.7 GBytes   169 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-11.00  sec  4.80 GBytes  41.3 Gbits/sec
[  8]  10.00-11.00  sec  5.09 GBytes  43.7 Gbits/sec
[ 10]  10.00-11.00  sec  5.00 GBytes  42.9 Gbits/sec
[ 12]  10.00-11.00  sec  4.78 GBytes  41.1 Gbits/sec
[SUM]  10.00-11.00  sec  19.7 GBytes   169 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  11.00-12.00  sec  4.66 GBytes  40.0 Gbits/sec
[  8]  11.00-12.00  sec  4.94 GBytes  42.4 Gbits/sec
[ 10]  11.00-12.00  sec  5.02 GBytes  43.2 Gbits/sec
[ 12]  11.00-12.00  sec  4.77 GBytes  41.0 Gbits/sec
[SUM]  11.00-12.00  sec  19.4 GBytes   167 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  12.00-13.00  sec  4.81 GBytes  41.3 Gbits/sec
[  8]  12.00-13.00  sec  4.86 GBytes  41.7 Gbits/sec
[ 10]  12.00-13.00  sec  4.99 GBytes  42.9 Gbits/sec
[ 12]  12.00-13.00  sec  4.80 GBytes  41.3 Gbits/sec
[SUM]  12.00-13.00  sec  19.5 GBytes   167 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-13.00  sec  62.1 GBytes  41.0 Gbits/sec    receiver
[  8]   0.00-13.00  sec  65.7 GBytes  43.4 Gbits/sec    receiver
[ 10]   0.00-13.00  sec  64.6 GBytes  42.7 Gbits/sec    receiver
[ 12]   0.00-13.00  sec  61.6 GBytes  40.7 Gbits/sec    receiver
[SUM]   0.00-13.00  sec   254 GBytes   168 Gbits/sec    receiver
코드 블럭
linenumberstrue
collapsetrue
# iperf3 -c 10.0.1.10 -i 1 -t 13 -O 3 -P 3
# iperf3 -c 10.0.1.10 -i 1 -t 13 -O 3 -P 4
Connecting to host 10.0.1.10, port 5201
[  5] local 10.0.1.20 port 55522 connected to 10.0.1.10 port 5201
[  7] local 10.0.1.20 port 55534 connected to 10.0.1.10 port 5201
[  9] local 10.0.1.20 port 55548 connected to 10.0.1.10 port 5201
[ 11] local 10.0.1.20 port 55554 connected to 10.0.1.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.68 GBytes  40.1 Gbits/sec  441   12.4 MBytes   (omitted)
[  7]   0.00-1.00   sec  4.99 GBytes  42.7 Gbits/sec  373   10.9 MBytes   (omitted)
[  9]   0.00-1.01   sec  4.75 GBytes  40.6 Gbits/sec  742   11.5 MBytes   (omitted)
[ 11]   0.00-1.01   sec  4.69 GBytes  40.1 Gbits/sec  648   11.7 MBytes   (omitted)
[SUM]   0.00-1.00   sec  19.1 GBytes   164 Gbits/sec  2204         (omitted)
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  4.81 GBytes  41.3 Gbits/sec    0   14.8 MBytes   (omitted)
[  7]   1.00-2.00   sec  5.02 GBytes  43.1 Gbits/sec    0   12.6 MBytes   (omitted)
[  9]   1.01-2.00   sec  4.92 GBytes  42.4 Gbits/sec    0   12.8 MBytes   (omitted)
[ 11]   1.01-2.00   sec  4.86 GBytes  41.8 Gbits/sec    0   13.9 MBytes   (omitted)
[SUM]   1.00-2.00   sec  19.6 GBytes   168 Gbits/sec    0         (omitted)
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  4.78 GBytes  41.0 Gbits/sec  103   10.6 MBytes   (omitted)
[  7]   2.00-3.00   sec  5.04 GBytes  43.3 Gbits/sec    0   14.7 MBytes   (omitted)
[  9]   2.00-3.00   sec  4.94 GBytes  42.4 Gbits/sec    0   13.8 MBytes   (omitted)
[ 11]   2.00-3.00   sec  4.74 GBytes  40.7 Gbits/sec    4   10.2 MBytes   (omitted)
[SUM]   2.00-3.00   sec  19.5 GBytes   168 Gbits/sec  107         (omitted)
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   0.00-1.00   sec  4.78 GBytes  41.0 Gbits/sec    0   13.6 MBytes
[  7]   0.00-1.00   sec  5.01 GBytes  42.9 Gbits/sec    1   8.57 MBytes
[  9]   0.00-1.00   sec  4.91 GBytes  42.0 Gbits/sec    0   14.2 MBytes
[ 11]   0.00-1.00   sec  4.79 GBytes  41.1 Gbits/sec    0   13.2 MBytes
[SUM]   0.00-1.00   sec  19.5 GBytes   167 Gbits/sec    1
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  4.71 GBytes  40.4 Gbits/sec   40   9.52 MBytes
[  7]   1.00-2.00   sec  5.11 GBytes  43.9 Gbits/sec    0   11.4 MBytes
[  9]   1.00-2.00   sec  4.94 GBytes  42.3 Gbits/sec    0   14.2 MBytes
[ 11]   1.00-2.00   sec  4.81 GBytes  41.2 Gbits/sec    2   9.46 MBytes
[SUM]   1.00-2.00   sec  19.6 GBytes   168 Gbits/sec   42
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  4.76 GBytes  40.9 Gbits/sec    0   12.7 MBytes
[  7]   2.00-3.00   sec  5.04 GBytes  43.3 Gbits/sec    0   13.2 MBytes
[  9]   2.00-3.00   sec  4.73 GBytes  40.7 Gbits/sec    2   10.6 MBytes
[ 11]   2.00-3.00   sec  4.44 GBytes  38.2 Gbits/sec    0   12.7 MBytes
[SUM]   2.00-3.00   sec  19.0 GBytes   163 Gbits/sec    2
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  4.78 GBytes  41.1 Gbits/sec   97   8.03 MBytes
[  7]   3.00-4.01   sec  5.10 GBytes  43.4 Gbits/sec    0   15.0 MBytes
[  9]   3.00-4.01   sec  4.96 GBytes  42.2 Gbits/sec    0   13.8 MBytes
[ 11]   3.00-4.01   sec  4.77 GBytes  40.7 Gbits/sec   23   7.94 MBytes
[SUM]   3.00-4.00   sec  19.6 GBytes   168 Gbits/sec  120
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  4.72 GBytes  40.6 Gbits/sec    0   11.6 MBytes
[  7]   4.01-5.00   sec  5.04 GBytes  43.5 Gbits/sec    0   16.7 MBytes
[  9]   4.01-5.01   sec  4.91 GBytes  42.3 Gbits/sec    2   10.6 MBytes
[ 11]   4.01-5.01   sec  4.70 GBytes  40.5 Gbits/sec    0   11.4 MBytes
[SUM]   4.00-5.00   sec  19.4 GBytes   166 Gbits/sec    2
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  4.83 GBytes  41.5 Gbits/sec    0   14.4 MBytes
[  7]   5.00-6.00   sec  5.07 GBytes  43.7 Gbits/sec    0   18.1 MBytes
[  9]   5.01-6.00   sec  5.02 GBytes  43.2 Gbits/sec    0   13.8 MBytes
[ 11]   5.01-6.00   sec  4.74 GBytes  40.8 Gbits/sec    0   14.2 MBytes
[SUM]   5.00-6.00   sec  19.7 GBytes   169 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  4.72 GBytes  40.5 Gbits/sec    7   9.86 MBytes
[  7]   6.00-7.00   sec  5.03 GBytes  43.3 Gbits/sec   27   11.1 MBytes
[  9]   6.00-7.00   sec  4.96 GBytes  42.8 Gbits/sec    5   10.0 MBytes
[ 11]   6.00-7.00   sec  4.65 GBytes  40.1 Gbits/sec   10   10.2 MBytes
[SUM]   6.00-7.00   sec  19.4 GBytes   166 Gbits/sec   49
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  4.80 GBytes  41.2 Gbits/sec    0   13.2 MBytes
[  7]   7.00-8.00   sec  5.10 GBytes  43.6 Gbits/sec    0   13.9 MBytes
[  9]   7.00-8.00   sec  5.06 GBytes  43.2 Gbits/sec    0   12.9 MBytes
[ 11]   7.00-8.01   sec  4.79 GBytes  40.9 Gbits/sec    0   12.0 MBytes
[SUM]   7.00-8.00   sec  19.7 GBytes   170 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  4.74 GBytes  40.7 Gbits/sec    8   9.56 MBytes
[  7]   8.00-9.00   sec  5.07 GBytes  43.7 Gbits/sec    1   9.92 MBytes
[  9]   8.00-9.00   sec  4.98 GBytes  43.0 Gbits/sec    7   9.45 MBytes
[ 11]   8.01-9.00   sec  4.74 GBytes  40.9 Gbits/sec    0   14.1 MBytes
[SUM]   8.00-9.00   sec  19.5 GBytes   168 Gbits/sec   16
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  4.81 GBytes  41.3 Gbits/sec    0   12.8 MBytes
[  7]   9.00-10.00  sec  5.09 GBytes  43.8 Gbits/sec    0   12.8 MBytes
[  9]   9.00-10.00  sec  4.99 GBytes  42.9 Gbits/sec    0   12.9 MBytes
[ 11]   9.00-10.00  sec  4.69 GBytes  40.3 Gbits/sec    3   10.6 MBytes
[SUM]   9.00-10.00  sec  19.6 GBytes   168 Gbits/sec    3
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-11.00  sec  4.82 GBytes  41.4 Gbits/sec    0   15.5 MBytes
[  7]  10.00-11.00  sec  5.09 GBytes  43.8 Gbits/sec    0   15.2 MBytes
[  9]  10.00-11.00  sec  5.00 GBytes  43.0 Gbits/sec    5   9.57 MBytes
[ 11]  10.00-11.00  sec  4.78 GBytes  41.1 Gbits/sec    0   13.5 MBytes
[SUM]  10.00-11.00  sec  19.7 GBytes   169 Gbits/sec    5
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  11.00-12.00  sec  4.64 GBytes  39.9 Gbits/sec   43   11.2 MBytes
[  7]  11.00-12.00  sec  4.96 GBytes  42.6 Gbits/sec   79   10.6 MBytes
[  9]  11.00-12.00  sec  5.04 GBytes  43.2 Gbits/sec    0   13.1 MBytes
[ 11]  11.00-12.00  sec  4.81 GBytes  41.2 Gbits/sec    2   9.61 MBytes
[SUM]  11.00-12.00  sec  19.5 GBytes   167 Gbits/sec  124
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  12.00-13.00  sec  4.79 GBytes  41.2 Gbits/sec    0   14.1 MBytes
[  7]  12.00-13.00  sec  4.84 GBytes  41.6 Gbits/sec    0   13.7 MBytes
[  9]  12.00-13.00  sec  4.97 GBytes  42.9 Gbits/sec    6   9.80 MBytes
[ 11]  12.00-13.00  sec  4.78 GBytes  41.3 Gbits/sec    0   12.8 MBytes
[SUM]  12.00-13.00  sec  19.4 GBytes   167 Gbits/sec    6
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-13.00  sec  62.0 GBytes  41.0 Gbits/sec  195   sender
[  5]   0.00-13.00  sec  62.1 GBytes  41.0 Gbits/sec        receiver
[  7]   0.00-13.00  sec  65.6 GBytes  43.4 Gbits/sec  108   sender
[  7]   0.00-13.00  sec  65.7 GBytes  43.4 Gbits/sec        receiver
[  9]   0.00-13.00  sec  64.5 GBytes  42.6 Gbits/sec   27   sender
[  9]   0.00-13.00  sec  64.6 GBytes  42.7 Gbits/sec        receiver
[ 11]   0.00-13.00  sec  61.6 GBytes  40.7 Gbits/sec   40   sender
[ 11]   0.00-13.00  sec  61.6 GBytes  40.7 Gbits/sec        receiver
[SUM]   0.00-13.00  sec   254 GBytes   168 Gbits/sec  370   sender
[SUM]   0.00-13.00  sec   254 GBytes   168 Gbits/sec        receiver