2023 전송성능 벤치마크 (10G 가상머신)

인스턴스 정보

Memo

vCPU

4

AMD EPYC 7742

MEM

16GB

Disk

100G

qcow2 thin provisioning

NFS backend (100Gbps connected)
NVMe RAID5

Hostname

IP

VM Host

Memo

009-u20-dtest

134.xxx.yyy.9

epyc1

서로 다른 호스트에서 실행되는 VM 인스턴스

VM에 스토리지를 제공하는 NFS는 epyc1 에서 실행중

011-u20-dtest

134.xxx.yyy.11

epyc2

디스크 성능

009-u20-dtest

011-u20-dtest

Memo

# ./fio-cdm.py /data -s 1g -n 5 -x
tests: 5, size: 1.0GiB, target: /data 34.0GiB/97.9GiB
Mixed rw: read 70%, write 30%
|Name        |  Read(MB/s)| Write(MB/s)|   Mix(MB/s)|
|------------|------------|------------|------------|
|SEQ1M Q8 T1 |     4462.77|      945.16|     1527.79|
|SEQ1M Q1 T1 |     1551.65|      303.55|      358.20|
|RND4K Q32T16|      261.28|      106.95|      107.39|
|. IOPS      |    63788.18|    26109.78|    26217.99|
|. latency us|     8018.84|    19579.11|    11318.49|
|RND4K Q1 T1 |       44.72|       12.23|       12.59|
|. IOPS      |    10919.02|     2985.40|     3074.17|
|. latency us|       91.15|      332.11|      186.01|

# ./fio-cdm.py /data -s 1g -n 5 -x
tests: 5, size: 1.0GiB, target: /data 34.0GiB/97.9GiB
Mixed rw: read 70%, write 30%
|Name        |  Read(MB/s)| Write(MB/s)|   Mix(MB/s)|
|------------|------------|------------|------------|
|SEQ1M Q8 T1 |     3188.07|      873.60|     1240.28|
|SEQ1M Q1 T1 |     1398.47|      289.44|      373.00|
|RND4K Q32T16|      187.35|      108.71|       87.79|
|. IOPS      |    45738.98|    26540.44|    21433.29|
|. latency us|    11184.81|    19262.66|    13841.49|
|RND4K Q1 T1 |       33.79|       11.76|       10.81|
|. IOPS      |     8248.35|     2870.43|     2640.21|
|. latency us|      120.71|      345.73|      217.52|

네트워크 성능

009-u20-dtest

011-u20-dtest

Memo

# ifconfig enp1s0
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 134.xxx.yyy.9

# iperf -s -i 1
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 134.75.207.9 port 5001 connected with 134.xxx.yyy.009 port 47798
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  2.31 GBytes  19.8 Gbits/sec
[  4]  1.0- 2.0 sec  2.46 GBytes  21.2 Gbits/sec
[  4]  2.0- 3.0 sec  2.42 GBytes  20.8 Gbits/sec
...
[  4] 29.0-30.0 sec  1.97 GBytes  16.9 Gbits/sec
[  4]  0.0-30.0 sec  72.7 GBytes  20.8 Gbits/sec

# ifconfig enp1s0
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 134.xxx.yyy.11

# iperf -c 134.xxx.yyy.9 -i 1 -t 30
------------------------------------------------------------
Client connecting to 134.75.207.9, TCP port 5001
TCP window size: 1.33 MByte (default)
------------------------------------------------------------
[  3] local 134.75.207.11 port 47798 connected with 134.xxx.yyy.9 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  2.31 GBytes  19.9 Gbits/sec
[  3]  1.0- 2.0 sec  2.46 GBytes  21.2 Gbits/sec
[  3]  2.0- 3.0 sec  2.42 GBytes  20.8 Gbits/sec
...
[  3] 29.0-30.0 sec  1.97 GBytes  16.9 Gbits/sec
[  3]  0.0-30.0 sec  72.7 GBytes  20.8 Gbits/sec

전송성능

VM Host 에서 구동중인 다른 인스턴스의 영향을 받을 수 있어서 측정할 때마다 다른 결과값이 나옴

scp

$ scp 134.xxx.yyy.11:/data/10GB.bin /data/10GB.scp.bin
10GB.bin    100%    10GB    209.6MB/s    00:48

hpnscp

$ hpnscp -P 2222 134.xxx.yyy.11:/data/10GB.bin /data/10GB.hpnscp.bin
10GB.bin    100%    10GB    414.3MB/s    446.8MB/s    00:24

GridFTP

$ globus-url-copy -vb -p 1 sshftp://134.xxx.yyy.11/data/10GB.bin /data/10GB.sshftp-p1a.bin
Source: sshftp://134.xxx.yyy.11/data/
Dest:   file:///data/
  10GB.bin  ->  10GB.sshftp-p1a.bin
  10488905728 bytes       625.19 MB/sec avg       485.23 MB/sec inst

$ globus-url-copy -vb -p 2 sshftp://134.xxx.yyy.11/data/10GB.bin /data/10GB.sshftp-p2a.bin
Source: sshftp://134.xxx.yyy.11/data/
Dest:   file:///data/
  10GB.bin  ->  10GB.sshftp-p2a.bin
  10530586624 bytes       717.33 MB/sec avg       597.99 MB/sec inst

$ globus-url-copy -vb -p 4 sshftp://134.xxx.yyy.11/data/10GB.bin /data/10GB.sshftp-p4a.bin
Source: sshftp://134.xxx.yyy.11/data/
Dest:   file:///data/
  10GB.bin  ->  10GB.sshftp-p4a.bin
  10458759168 bytes       623.39 MB/sec avg       478.13 MB/sec inst

globus-url-copy -vb -p 8 sshftp://134.75.207.11/data/10GB.bin /data/10GB.sshftp-p8a.bin
Source: sshftp://134.75.207.11/data/
Dest:   file:///data/
  10GB.bin  ->  10GB.sshftp-p8a.bin
  10487332864 bytes       714.38 MB/sec avg       620.36 MB/sec inst

GridFTP + hpnscp (server side)

$ globus-url-copy -vb -p 1 sshftp://134.xxx.yyy.11:2222/data/10GB.bin /data/10GB.sshgftp-p1b.bin
Source: sshftp://134.xxx.yyy.11:2222/data/
Dest:   file:///data/
  10GB.bin  ->  10GB.sshgftp-p1b.bin
   9668919296 bytes       768.41 MB/sec avg       606.07 MB/sec inst

$ globus-url-copy -vb -p 2 sshftp://134.xxx.yyy.11:2222/data/10GB.bin /data/10GB.sshftp-p2b.bin
Source: sshftp://134.xxx.yyy.11:2222/data/
Dest:   file:///data/
  10GB.bin  ->  10GB.sshftp-p2b.bin
  10454564864 bytes       712.03 MB/sec avg       552.44 MB/sec inst

$ globus-url-copy -vb -p 4 sshftp://134.xxx.yyy.11:2222/data/10GB.bin /data/10GB.sshftp-p4b.bin
Source: sshftp://134.xxx.yyy.11:2222/data/
Dest:   file:///data/
  10GB.bin  ->  10GB.sshftp-p4b.bin
  10525081600 bytes       716.48 MB/sec avg       577.64 MB/sec inst

$ globus-url-copy -vb -p 8 sshftp://134.xxx.yyy.11:2222/data/10GB.bin /data/10GB.sshftp-p8b.bin
Source: sshftp://134.xxx.yyy.11:2222/data/
Dest:   file:///data/
  10GB.bin  ->  10GB.sshftp-p8b.bin
   9481748480 bytes       753.53 MB/sec avg       648.16 MB/sec inst

결론

fio 로 측정한 VM disk의 write performance 는 800~900MB/s

GridFTP + hpnscp 로 전송시 디스크 write 성능의 83~94% 까지 전송 가능

	MB/s	Gbps
fio (로컬 디스크 성능)	800~900 MB/s	6.4~7.2 Gbps
iperf		20.8 Gbps
scp	209.6 MB/s	1.676 Gbps
hpnscp	414.3 MB/s	3.314 Gbps
GridFTP	717.33 MB/s	5.738 Gbps
GridFTP + hpnscp	753.53 MB/s	6.028 Gbps