EPYC 7742 100G NIC 전송 테스트
Date:
Test Environment
Machine
| CPU | AMD EPYC 7742 | Base: 2.25GHz Max: 3.4GHz 64 Core 128 Thread |
|---|---|---|
| MEM | 512GB | DDR4 64GB x 8 |
| NIC | 100GbE | Mellanox MT28908 Family [ConnectX-6] |
| DISK | NVMe RAID 10 | 500G NVMe x 4 |
SW
OS : CentOS 8 Stream
Kernel Parameter: 리눅스 네트워크 튜닝 > 40Gbps 급 이상
MTU : 9000
iperf, scp, GridFTP
Topology
+------+ +-------------------+ +------+ | HOST |---| 100G Switch S4128 |---| HOST | +------+ +-------------------+ +------+
iperf 1 Stream 34 Gbps
serv.sh
iperf3 -s -p 5201
clnt.sh
HN=192.168.100.100 OPT="-t 15" iperf3 -c $HN $OPT -T S1 -p 5201
clnt.sh
# ./clnt.sh Connecting to host 192.168.100.100, port 5201 [ 5] local 134.75.248.5 port 46428 connected to 192.168.100.100 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 4.08 GBytes 35.0 Gbits/sec 13 3.62 MBytes [ 5] 1.00-2.00 sec 3.96 GBytes 34.0 Gbits/sec 0 3.67 MBytes [ 5] 2.00-3.00 sec 3.95 GBytes 34.0 Gbits/sec 0 3.67 MBytes [ 5] 3.00-4.00 sec 3.96 GBytes 34.0 Gbits/sec 0 3.67 MBytes [ 5] 4.00-5.00 sec 3.97 GBytes 34.1 Gbits/sec 0 3.75 MBytes [ 5] 5.00-6.00 sec 3.97 GBytes 34.1 Gbits/sec 0 3.77 MBytes [ 5] 6.00-7.00 sec 3.97 GBytes 34.1 Gbits/sec 0 4.10 MBytes [ 5] 7.00-8.00 sec 3.97 GBytes 34.1 Gbits/sec 0 4.10 MBytes [ 5] 8.00-9.00 sec 3.97 GBytes 34.1 Gbits/sec 0 4.10 MBytes [ 5] 9.00-10.00 sec 3.97 GBytes 34.1 Gbits/sec 0 4.10 MBytes [ 5] 10.00-11.00 sec 3.97 GBytes 34.1 Gbits/sec 0 4.10 MBytes [ 5] 11.00-12.00 sec 3.97 GBytes 34.1 Gbits/sec 0 4.10 MBytes [ 5] 12.00-13.00 sec 3.96 GBytes 34.0 Gbits/sec 0 4.10 MBytes [ 5] 13.00-14.00 sec 3.95 GBytes 33.9 Gbits/sec 0 4.10 MBytes [ 5] 14.00-15.00 sec 3.96 GBytes 34.0 Gbits/sec 0 4.10 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-15.00 sec 59.6 GBytes 34.1 Gbits/sec 13 sender [ 5] 0.00-15.04 sec 59.5 GBytes 34.0 Gbits/sec receiver iperf Done.
iperf 4 Stream 98.2 Gbps
serv.sh
iperf3 -s -p 5201 & iperf3 -s -p 5202 & iperf3 -s -p 5203 & iperf3 -s -p 5204
clnt.sh
HN=192.168.100.100 OPT="-t 15" iperf3 -c $HN $OPT -T S1 -p 5201 & iperf3 -c $HN $OPT -T S2 -p 5202 & iperf3 -c $HN $OPT -T S3 -p 5203 & iperf3 -c $HN $OPT -T S4 -p 5204
# ./clnt.sh S1: Connecting to host 192.168.100.100, port 5201 S2: Connecting to host 192.168.100.100, port 5202 S3: Connecting to host 192.168.100.100, port 5203 S4: Connecting to host 192.168.100.100, port 5204 S1: [ 5] local 134.75.248.5 port 46424 connected to 192.168.100.100 port 5201 S2: [ 5] local 134.75.248.5 port 54262 connected to 192.168.100.100 port 5202 S3: [ 5] local 134.75.248.5 port 55926 connected to 192.168.100.100 port 5203 S4: [ 5] local 134.75.248.5 port 47070 connected to 192.168.100.100 port 5204 S1: [ ID] Interval Transfer Bitrate Retr Cwnd S2: [ ID] Interval Transfer Bitrate Retr Cwnd S3: [ ID] Interval Transfer Bitrate Retr Cwnd S4: [ ID] Interval Transfer Bitrate Retr Cwnd S1: [ 5] 0.00-1.00 sec 2.83 GBytes 24.3 Gbits/sec 157 1.66 MBytes S2: [ 5] 0.00-1.00 sec 3.02 GBytes 25.9 Gbits/sec 182 1.55 MBytes S3: [ 5] 0.00-1.00 sec 2.86 GBytes 24.5 Gbits/sec 207 2.07 MBytes S4: [ 5] 0.00-1.00 sec 2.79 GBytes 24.0 Gbits/sec 135 1.48 MBytes S1: [ 5] 1.00-2.00 sec 3.02 GBytes 25.9 Gbits/sec 932 559 KBytes S2: [ 5] 1.00-2.00 sec 3.05 GBytes 26.2 Gbits/sec 988 664 KBytes S4: [ 5] 1.00-2.00 sec 2.73 GBytes 23.5 Gbits/sec 868 891 KBytes S3: [ 5] 1.00-2.00 sec 2.28 GBytes 19.6 Gbits/sec 1028 839 KBytes S2: [ 5] 2.00-3.00 sec 2.83 GBytes 24.3 Gbits/sec 988 1.31 MBytes S1: [ 5] 2.00-3.00 sec 3.00 GBytes 25.7 Gbits/sec 929 1.46 MBytes S3: [ 5] 2.00-3.00 sec 2.66 GBytes 22.8 Gbits/sec 775 1.76 MBytes S4: [ 5] 2.00-3.00 sec 2.61 GBytes 22.5 Gbits/sec 866 1.58 MBytes S1: [ 5] 3.00-4.00 sec 2.92 GBytes 25.1 Gbits/sec 58 1.64 MBytes S2: [ 5] 3.00-4.00 sec 2.91 GBytes 25.0 Gbits/sec 70 2.06 MBytes S3: [ 5] 3.00-4.00 sec 2.91 GBytes 25.0 Gbits/sec 65 1.60 MBytes S4: [ 5] 3.00-4.00 sec 2.78 GBytes 23.9 Gbits/sec 53 2.64 MBytes S2: [ 5] 4.00-5.00 sec 2.89 GBytes 24.8 Gbits/sec 0 2.06 MBytes S1: [ 5] 4.00-5.00 sec 2.89 GBytes 24.9 Gbits/sec 0 1.64 MBytes S3: [ 5] 4.00-5.00 sec 2.89 GBytes 24.8 Gbits/sec 0 1.60 MBytes S4: [ 5] 4.00-5.00 sec 2.85 GBytes 24.5 Gbits/sec 0 2.64 MBytes S1: [ 5] 5.00-6.00 sec 2.89 GBytes 24.9 Gbits/sec 0 1.64 MBytes S2: [ 5] 5.00-6.00 sec 2.89 GBytes 24.8 Gbits/sec 0 2.06 MBytes S3: [ 5] 5.00-6.00 sec 2.89 GBytes 24.8 Gbits/sec 0 1.60 MBytes S4: [ 5] 5.00-6.00 sec 2.85 GBytes 24.5 Gbits/sec 0 2.64 MBytes S2: [ 5] 6.00-7.00 sec 2.89 GBytes 24.8 Gbits/sec 0 2.06 MBytes S1: [ 5] 6.00-7.00 sec 2.89 GBytes 24.8 Gbits/sec 0 1.64 MBytes S3: [ 5] 6.00-7.00 sec 2.89 GBytes 24.8 Gbits/sec 0 1.60 MBytes S4: [ 5] 6.00-7.00 sec 2.85 GBytes 24.5 Gbits/sec 0 2.64 MBytes S2: [ 5] 7.00-8.00 sec 2.93 GBytes 25.1 Gbits/sec 96 1.66 MBytes S1: [ 5] 7.00-8.00 sec 2.94 GBytes 25.2 Gbits/sec 90 1.66 MBytes S3: [ 5] 7.00-8.00 sec 2.86 GBytes 24.6 Gbits/sec 131 2.01 MBytes S4: [ 5] 7.00-8.00 sec 2.78 GBytes 23.8 Gbits/sec 99 1.81 MBytes S1: [ 5] 8.00-9.00 sec 2.72 GBytes 23.3 Gbits/sec 551 1.68 MBytes S2: [ 5] 8.00-9.00 sec 2.97 GBytes 25.5 Gbits/sec 559 1.72 MBytes S3: [ 5] 8.00-9.00 sec 2.92 GBytes 25.1 Gbits/sec 530 1.62 MBytes S4: [ 5] 8.00-9.00 sec 2.71 GBytes 23.3 Gbits/sec 402 1.73 MBytes S2: [ 5] 9.00-10.00 sec 2.90 GBytes 24.9 Gbits/sec 0 1.75 MBytes S1: [ 5] 9.00-10.00 sec 2.90 GBytes 24.9 Gbits/sec 0 1.68 MBytes S3: [ 5] 9.00-10.00 sec 2.83 GBytes 24.3 Gbits/sec 0 2.31 MBytes S4: [ 5] 9.00-10.00 sec 2.90 GBytes 24.9 Gbits/sec 0 1.73 MBytes S2: [ 5] 10.00-11.00 sec 2.89 GBytes 24.8 Gbits/sec 0 1.75 MBytes S1: [ 5] 10.00-11.00 sec 2.90 GBytes 24.9 Gbits/sec 0 1.68 MBytes S3: [ 5] 10.00-11.00 sec 2.87 GBytes 24.7 Gbits/sec 0 2.36 MBytes S4: [ 5] 10.00-11.00 sec 2.86 GBytes 24.6 Gbits/sec 0 1.73 MBytes S2: [ 5] 11.00-12.00 sec 2.90 GBytes 24.9 Gbits/sec 0 1.77 MBytes S1: [ 5] 11.00-12.00 sec 2.90 GBytes 24.9 Gbits/sec 0 1.68 MBytes S3: [ 5] 11.00-12.00 sec 2.88 GBytes 24.7 Gbits/sec 0 2.36 MBytes S4: [ 5] 11.00-12.00 sec 2.85 GBytes 24.5 Gbits/sec 0 1.73 MBytes S2: [ 5] 12.00-13.00 sec 2.89 GBytes 24.9 Gbits/sec 0 1.77 MBytes S1: [ 5] 12.00-13.00 sec 2.90 GBytes 24.9 Gbits/sec 0 1.68 MBytes S3: [ 5] 12.00-13.00 sec 2.89 GBytes 24.8 Gbits/sec 0 2.36 MBytes S4: [ 5] 12.00-13.00 sec 2.85 GBytes 24.5 Gbits/sec 0 1.73 MBytes S1: [ 5] 13.00-14.00 sec 2.89 GBytes 24.9 Gbits/sec 0 1.68 MBytes S2: [ 5] 13.00-14.00 sec 2.89 GBytes 24.8 Gbits/sec 0 1.77 MBytes S3: [ 5] 13.00-14.00 sec 2.89 GBytes 24.8 Gbits/sec 0 2.36 MBytes S4: [ 5] 13.00-14.00 sec 2.85 GBytes 24.5 Gbits/sec 0 1.73 MBytes S4: [ 5] 14.00-15.00 sec 2.85 GBytes 24.5 Gbits/sec 0 1.73 MBytes S1: [ 5] 14.00-15.00 sec 2.90 GBytes 24.9 Gbits/sec 0 1.68 MBytes S2: [ 5] 14.00-15.00 sec 2.89 GBytes 24.8 Gbits/sec 0 1.77 MBytes S3: [ 5] 14.00-15.00 sec 2.89 GBytes 24.8 Gbits/sec 0 2.36 MBytes S1: - - - - - - - - - - - - - - - - - - - - - - - - - S1: [ ID] Interval Transfer Bitrate Retr S1: [ 5] 0.00-15.00 sec 43.5 GBytes 24.9 Gbits/sec 2717 sender S1: [ 5] 0.00-15.03 sec 43.5 GBytes 24.9 Gbits/sec receiver S1: S1: iperf Done. S2: - - - - - - - - - - - - - - - - - - - - - - - - - S2: [ ID] Interval Transfer Bitrate Retr S2: [ 5] 0.00-15.00 sec 43.7 GBytes 25.1 Gbits/sec 2883 sender S2: [ 5] 0.00-15.03 sec 43.7 GBytes 25.0 Gbits/sec receiver S2: S2: iperf Done. S3: - - - - - - - - - - - - - - - - - - - - - - - - - S3: [ ID] Interval Transfer Bitrate Retr S3: [ 5] 0.00-15.00 sec 42.4 GBytes 24.3 Gbits/sec 2736 sender S3: [ 5] 0.00-15.03 sec 42.4 GBytes 24.2 Gbits/sec receiver S3: S3: iperf Done. S4: - - - - - - - - - - - - - - - - - - - - - - - - - S4: [ ID] Interval Transfer Bitrate Retr S4: [ 5] 0.00-15.00 sec 42.1 GBytes 24.1 Gbits/sec 2423 sender S4: [ 5] 0.00-15.03 sec 42.1 GBytes 24.1 Gbits/sec receiver S4: S4: iperf Done.
SCP 1 stream 7.41 Gbps (927.2 MB/s)
$ scp 134.75.248.5:~/10G.bin 10G.bin 10G.bin 100% 10GB 927.2MB/s 00:11
GridFTP 1 stream 9.56 Gbps (1195.57 MB/s)
TCP 스트림을 1개로 하나 4개로 하나 유의미한 차이가 없음
clnt.sh
globus-url-copy -vb -p 1 sshftp://192.168.100.100/home/gdhong/10G.bin 10G.bin globus-url-copy -vb -p 4 sshftp://192.168.100.100/home/gdhong/10G.bin 10G.bin
clnt.sh
$ globus-url-copy -vb -p 1 sshftp://192.168.100.100/home/gdhong/10G.bin 10G.bin Source: sshftp://192.168.100.100/home/kreonet/ Dest: file:///vm_images/test/ 10G.bin 9809428480 bytes 1169.36 MB/sec avg 1316.85 MB/sec inst $ globus-url-copy -vb -p 4 sshftp://134.75.248.5/home/kreonet/10G.bin 10G.bin Source: sshftp://134.75.248.5/home/kreonet/ Dest: file:///vm_images/test/ 10G.bin 10029367296 bytes 1195.57 MB/sec avg 1339.21 MB/sec inst
GridFTP 10 stream 45.7 Gbps (5712.63 MB/s)
clnt.sh
HN=192.168.100.100 globus-url-copy -vb sshftp://$HN/home/gdhong/10G01.bin 10G01.bin | tee 01.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G02.bin 10G02.bin | tee 02.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G03.bin 10G03.bin | tee 03.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G04.bin 10G04.bin | tee 04.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G05.bin 10G05.bin | tee 05.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G06.bin 10G06.bin | tee 06.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G07.bin 10G07.bin | tee 07.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G08.bin 10G08.bin | tee 08.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G09.bin 10G09.bin | tee 09.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G10.bin 10G10.bin | tee 10.log
$ ./clnt.sh
$ cat *.log | grep avg
9481224192 bytes 645.82 MB/sec avg 688.72 MB/sec inst
9510584320 bytes 647.79 MB/sec avg 690.48 MB/sec inst
9493807104 bytes 646.71 MB/sec avg 689.51 MB/sec inst
9420406784 bytes 641.68 MB/sec avg 686.11 MB/sec inst
9473884160 bytes 645.32 MB/sec avg 688.88 MB/sec inst
10696523776 bytes 637.55 MB/sec avg 694.43 MB/sec inst
9312403456 bytes 634.29 MB/sec avg 661.43 MB/sec inst
9376366592 bytes 638.67 MB/sec avg 681.61 MB/sec inst
9334423552 bytes 635.83 MB/sec avg 673.04 MB/sec inst
10666115072 bytes 635.72 MB/sec avg 702.22 MB/sec inst
$ cat *.log | grep avg | awk '{print $3}' | paste -sd+
579.78+585.27+564.42+571.41+581.91+563.52+567.64+572.81+564.44+561.43
$ cat *.log | grep avg | awk '{print $3}' | paste -sd+ | bc -l
5712.63
GridFTP 20 stream 48.3 Gbps (6040.15 MB/s)
clnt.sh
HN=192.168.100.100 globus-url-copy -vb sshftp://$HN/home/gdhong/10G01.bin 10G01.bin | tee 01.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G02.bin 10G02.bin | tee 02.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G03.bin 10G03.bin | tee 03.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G04.bin 10G04.bin | tee 04.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G05.bin 10G05.bin | tee 05.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G06.bin 10G06.bin | tee 06.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G07.bin 10G07.bin | tee 07.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G08.bin 10G08.bin | tee 08.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G09.bin 10G09.bin | tee 09.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G10.bin 10G10.bin | tee 10.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G01.bin 10G11.bin | tee 11.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G02.bin 10G12.bin | tee 12.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G03.bin 10G13.bin | tee 13.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G04.bin 10G14.bin | tee 14.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G05.bin 10G15.bin | tee 15.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G06.bin 10G16.bin | tee 16.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G07.bin 10G17.bin | tee 17.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G08.bin 10G18.bin | tee 18.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G09.bin 10G19.bin | tee 19.log & globus-url-copy -vb sshftp://$HN/home/gdhong/10G10.bin 10G20.bin | tee 20.log
$ ./clnt.sh
$ cat *.log | grep avg
10291773440 bytes 446.11 MB/sec avg 449.01 MB/sec inst
10681843712 bytes 424.45 MB/sec avg 301.03 MB/sec inst
10200547328 bytes 442.14 MB/sec avg 454.13 MB/sec inst
10299113472 bytes 446.43 MB/sec avg 451.87 MB/sec inst
10342105088 bytes 448.32 MB/sec avg 463.71 MB/sec inst
10379853824 bytes 449.94 MB/sec avg 450.83 MB/sec inst
10619977728 bytes 421.53 MB/sec avg 288.88 MB/sec inst
10345250816 bytes 448.41 MB/sec avg 450.17 MB/sec inst
10636754944 bytes 422.07 MB/sec avg 292.45 MB/sec inst
10647240704 bytes 422.59 MB/sec avg 280.22 MB/sec inst
10656677888 bytes 423.00 MB/sec avg 282.62 MB/sec inst
10305404928 bytes 446.72 MB/sec avg 444.24 MB/sec inst
10697572352 bytes 424.86 MB/sec avg 282.86 MB/sec inst
10454302720 bytes 453.16 MB/sec avg 449.12 MB/sec inst
10165944320 bytes 440.64 MB/sec avg 444.71 MB/sec inst
10256121856 bytes 444.57 MB/sec avg 449.28 MB/sec inst
10517217280 bytes 455.90 MB/sec avg 458.69 MB/sec inst
$ cat *.log | grep avg | awk '{print $3}' | paste -sd+ | bc -l
6040.15