E3-1220, Single SATA3 SSD, 10G NIC 벤치마크
Environment
Node A
| CPU | Intel(R) Xeon(R) CPU E3-1220 v6 @ 3.00GHz | 4 core |
|---|---|---|
| MEM | Samsung 16GB DDR4 19200 UN/ECC x 2 | |
| M/B | Intel C232 | |
| SSD | Samsung 860 PRO 256GB | SATA3 read_max = 560MB/s |
| NIC | Intel X520-SR2 (SFP+) | |
| Cost | $2,100 / 200만원 / July 2019 |
Node B
same as Node A
Topology
( Node A ) – ( 10G Switch ) – ( Node B )
Xeon E3
C230 칩셋 8 SATA Port
E3 ~ C232 속도
= DMI Gen3 x4 (3.93 GB/s for the CPU–PCH link)
= PCIe 3.0 x4
= 31.44 Gbps
iperf / mtu 1500 / TCP window 64M / 9.40 Gbps
# ifconfig enp1s0f0 mtu 1500 # iperf -s -w 32M ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 MByte (WARNING: requested 32.0 MByte) ------------------------------------------------------------ [ 4] local 203.XXX.YYY.60 port 5001 connected with 203.XXX.YYY.52 port 57040 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 32.9 GBytes 9.40 Gbits/sec | # ifconfig enp1s0f0 mtu 1500 # iperf -c 203.XXX.YYY.60 -i 1 -t 30 -w 32M ------------------------------------------------------------ Client connecting to 203.XXX.YYY.60, TCP port 5001 TCP window size: 64.0 MByte (WARNING: requested 32.0 MByte) ------------------------------------------------------------ [ 3] local 203.XXX.YYY.52 port 57040 connected with 203.XXX.YYY.60 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 1.13 GBytes 9.69 Gbits/sec [ 3] 1.0- 2.0 sec 1.11 GBytes 9.52 Gbits/sec [ 3] 2.0- 3.0 sec 1.08 GBytes 9.27 Gbits/sec [ 3] 3.0- 4.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 4.0- 5.0 sec 1.10 GBytes 9.49 Gbits/sec [ 3] 5.0- 6.0 sec 1.09 GBytes 9.33 Gbits/sec [ 3] 6.0- 7.0 sec 1.10 GBytes 9.43 Gbits/sec [ 3] 7.0- 8.0 sec 1.09 GBytes 9.34 Gbits/sec [ 3] 8.0- 9.0 sec 1.11 GBytes 9.51 Gbits/sec [ 3] 9.0-10.0 sec 1.08 GBytes 9.32 Gbits/sec [ 3] 10.0-11.0 sec 1.10 GBytes 9.46 Gbits/sec [ 3] 11.0-12.0 sec 1.10 GBytes 9.43 Gbits/sec [ 3] 12.0-13.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 13.0-14.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 14.0-15.0 sec 1.09 GBytes 9.33 Gbits/sec [ 3] 15.0-16.0 sec 1.10 GBytes 9.45 Gbits/sec [ 3] 16.0-17.0 sec 1.10 GBytes 9.44 Gbits/sec [ 3] 17.0-18.0 sec 1.09 GBytes 9.33 Gbits/sec [ 3] 18.0-19.0 sec 1.10 GBytes 9.47 Gbits/sec [ 3] 19.0-20.0 sec 1.08 GBytes 9.31 Gbits/sec [ 3] 20.0-21.0 sec 1.11 GBytes 9.56 Gbits/sec [ 3] 21.0-22.0 sec 1.09 GBytes 9.34 Gbits/sec [ 3] 22.0-23.0 sec 1.09 GBytes 9.38 Gbits/sec [ 3] 23.0-24.0 sec 1.10 GBytes 9.45 Gbits/sec [ 3] 24.0-25.0 sec 1.09 GBytes 9.34 Gbits/sec [ 3] 25.0-26.0 sec 1.10 GBytes 9.43 Gbits/sec [ 3] 26.0-27.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 27.0-28.0 sec 1.11 GBytes 9.52 Gbits/sec [ 3] 28.0-29.0 sec 1.08 GBytes 9.26 Gbits/sec [ 3] 29.0-30.0 sec 1.11 GBytes 9.49 Gbits/sec [ 3] 0.0-30.0 sec 32.9 GBytes 9.41 Gbits/sec |
iperf / mtu 9000 / TCP window 64M / 9.89 Gbps
# ifconfig enp1s0f0 mtu 9000 # iperf -s -w 32M ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 MByte (WARNING: requested 32.0 MByte) ------------------------------------------------------------ [ 4] local 203.XXX.YYY.60 port 5001 connected with 203.XXX.YYY.52 port 57042 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 34.6 GBytes 9.89 Gbits/sec | # ifconfig enp1s0f0 mtu 9000 # iperf -c 203.XXX.YYY.60 -i 1 -t 30 -w 32M ------------------------------------------------------------ Client connecting to 203.XXX.YYY.60, TCP port 5001 TCP window size: 64.0 MByte (WARNING: requested 32.0 MByte) ------------------------------------------------------------ [ 3] local 203.XXX.YYY.52 port 57042 connected with 203.XXX.YYY.60 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 1.19 GBytes 10.2 Gbits/sec [ 3] 1.0- 2.0 sec 1.16 GBytes 9.93 Gbits/sec [ 3] 2.0- 3.0 sec 1.15 GBytes 9.87 Gbits/sec [ 3] 3.0- 4.0 sec 1.14 GBytes 9.81 Gbits/sec [ 3] 4.0- 5.0 sec 1.16 GBytes 9.97 Gbits/sec [ 3] 5.0- 6.0 sec 1.16 GBytes 9.98 Gbits/sec [ 3] 6.0- 7.0 sec 1.13 GBytes 9.75 Gbits/sec [ 3] 7.0- 8.0 sec 1.16 GBytes 10.0 Gbits/sec [ 3] 8.0- 9.0 sec 1.14 GBytes 9.78 Gbits/sec [ 3] 9.0-10.0 sec 1.16 GBytes 9.95 Gbits/sec [ 3] 10.0-11.0 sec 1.15 GBytes 9.91 Gbits/sec [ 3] 11.0-12.0 sec 1.16 GBytes 9.92 Gbits/sec [ 3] 12.0-13.0 sec 1.15 GBytes 9.92 Gbits/sec [ 3] 13.0-14.0 sec 1.15 GBytes 9.86 Gbits/sec [ 3] 14.0-15.0 sec 1.15 GBytes 9.92 Gbits/sec [ 3] 15.0-16.0 sec 1.15 GBytes 9.91 Gbits/sec [ 3] 16.0-17.0 sec 1.15 GBytes 9.84 Gbits/sec [ 3] 17.0-18.0 sec 1.15 GBytes 9.87 Gbits/sec [ 3] 18.0-19.0 sec 1.16 GBytes 9.93 Gbits/sec [ 3] 19.0-20.0 sec 1.14 GBytes 9.80 Gbits/sec [ 3] 20.0-21.0 sec 1.16 GBytes 9.98 Gbits/sec [ 3] 21.0-22.0 sec 1.15 GBytes 9.90 Gbits/sec [ 3] 22.0-23.0 sec 1.15 GBytes 9.91 Gbits/sec [ 3] 23.0-24.0 sec 1.16 GBytes 9.95 Gbits/sec [ 3] 24.0-25.0 sec 1.15 GBytes 9.91 Gbits/sec [ 3] 25.0-26.0 sec 1.14 GBytes 9.82 Gbits/sec [ 3] 26.0-27.0 sec 1.16 GBytes 9.94 Gbits/sec [ 3] 27.0-28.0 sec 1.14 GBytes 9.77 Gbits/sec [ 3] 28.0-29.0 sec 1.15 GBytes 9.90 Gbits/sec [ 3] 29.0-30.0 sec 1.17 GBytes 10.0 Gbits/sec [ 3] 0.0-30.0 sec 34.6 GBytes 9.91 Gbits/sec |
dd / write cache enabled / swap on / 897MB/s
# hdparm -W1 /dev/sda setting drive write-caching to 1 (on) write-caching = 1 (on) # dd if=/dev/zero of=/var/www/html/10g.bin bs=100M count=100 100+0 records in 100+0 records out 10485760000 bytes (10 GB) copied, 11.6923 s, 897 MB/s
dd / write cache disabled / swap on / 548MB/s
# hdparm -W0 /dev/sda setting drive write-caching to 0 (off) write-caching = 0 (off) # dd if=/dev/zero of=/var/www/html/10g.bin bs=100M count=100 100+0 records in 100+0 records out 10485760000 bytes (10 GB) copied, 11.6923 s, 548 MB/s
dd / write cache disabled / swap off / 499MB/s
# hdparm -W0 /dev/sda
setting drive write-caching to 0 (off)
write-caching = 0 (off)
# swapoff -a
# free -h
total used free shared buff/cache available
Mem: 31G 2.0G 4.6G 39M 24G 28G
Swap: 0B 0B 0B
# dd if=/dev/zero of=/var/www/html/10g.bin bs=100M count=100
100+0 records in
100+0 records out
10485760000 bytes (10 GB) copied, 21.0164 s, 499 MB/s
HTTP 10GB single stream / write cache enabled / swap on / avg 670MB/s (5.360Gbps) / max 920MB/s (7.360Gbps)
# hdparm -W1 /dev/sda
setting drive write-caching to 1 (on)
write-caching = 1 (on)
# wget http://203.XXX.YYY.60/10g.bin -O ~/10g.bin -o ~/download.log
# awk '{print $8}' ~/download.log | sort | tail -n 20
...
920M
...
# tail -n 2 ~/download.log
2019-10-04 13:05:39 (670 MB/s) - ‘/root/10g.bin’ saved [10485760000/10485760000]
| 커널 블럭 IO 기록 # blktrace -d /dev/sda -o - | blkparse -i - -o /tmp/io.log # ^C 512KB 순차 쓰기 명령 발생
# grep wget io.log | grep Q | tail -n 10 8,0 1 27442 1.697319988 976 Q W 329055424 + 1024 [wget] 8,0 1 27450 1.697536013 976 Q W 329056448 + 1024 [wget] 8,0 1 27458 1.697750478 976 Q W 329057472 + 1024 [wget] 8,0 1 27474 1.698002535 976 Q W 329058496 + 1024 [wget] 8,0 1 27482 1.698221094 976 Q W 329059520 + 1024 [wget] 8,0 1 27490 1.698439973 976 Q W 329060544 + 1024 [wget] 8,0 1 27498 1.698657718 976 Q W 329061568 + 1024 [wget] 8,0 1 27514 1.698902976 976 Q W 329062592 + 1024 [wget] 8,0 1 27522 1.699126703 976 Q W 329063616 + 1024 [wget] 8,0 1 27530 1.699344213 976 Q W 329064640 + 1024 [wget] 스왑 발생
# grep 327201984 io.log 8,0 0 525 0.003899458 976 A W 327201984 + 1024 <- (8,2) 325102784 8,0 0 526 0.003899559 976 Q W 327201984 + 1024 [wget] 8,0 0 527 0.003899860 976 G W 327201984 + 1024 [wget] 8,0 0 528 0.003900468 976 I W 327201984 + 1024 [wget] 8,0 1 329 0.044262917 0 D W 327201984 + 1024 [swapper/1] 8,0 1 330 0.044265884 0 R W 327201984 + 1024 [0] 8,0 1 331 0.044266006 0 I W 327201984 + 1024 [swapper/1] 8,0 1 332 0.044266277 0 D W 327201984 + 1024 [swapper/1] 8,0 1 333 0.044266448 0 R W 327201984 + 1024 [0] 8,0 1 334 0.044266513 0 I W 327201984 + 1024 [swapper/1] 8,0 1 336 0.045243270 0 D W 327201984 + 1024 [swapper/1] 8,0 1 832 0.074691872 0 C W 327201984 + 1024 [0] |
HTTP 10GB single stream / write cache enabled / swap off / avg 709MB/s (5.672Gbps)
# hdparm -W1 /dev/sda
setting drive write-caching to 1 (on)
write-caching = 1 (on)
# swapoff -a
# free -h
total used free shared buff/cache available
Mem: 31G 2.0G 4.6G 39M 24G 28G
Swap: 0B 0B 0B
# wget http://203.XXX.YYY.60/10g.bin -O ~/10g.bin
...
2019-10-04 14:20:21 (709 MB/s) - ‘/root/10g.bin’ saved [10485760000/10485760000]
| 커널 블럭 IO 기록 # blktrace -d /dev/sda -o - | blkparse -i - -o /tmp/io.log # ^C 512KB 순차 쓰기 명령 발생 # grep wget io.log | grep D | tail -n 10 8,0 1 93223 19.518676605 1495 D W 393915424 + 1024 [wget] 8,0 1 93227 19.519645110 1495 D W 393915424 + 1024 [wget] 8,0 1 93228 19.519646224 1495 D W 393916448 + 1024 [wget] 8,0 1 93231 19.519648937 1495 D W 393916448 + 1024 [wget] 8,0 1 93235 19.520615024 1495 D W 393916448 + 1024 [wget] 8,0 1 93236 19.520616783 1495 D W 393917472 + 1024 [wget] 8,0 1 93239 19.520619378 1495 D W 393917472 + 1024 [wget] 8,0 1 93243 19.521467724 1495 D W 393917472 + 1024 [wget] 8,0 1 93244 19.521468858 1495 D W 393918496 + 1024 [wget] 8,0 1 93247 19.521471518 1495 D W 393918496 + 1024 [wget] 스왑 X # grep 393917472 io.log 8,0 3 22102 19.417867154 1157 A W 393917472 + 1024 <- (8,2) 391818272 8,0 3 22103 19.417867263 1157 Q W 393917472 + 1024 [kworker/u8:1] 8,0 3 22104 19.417867637 1157 G W 393917472 + 1024 [kworker/u8:1] 8,0 3 22105 19.417868284 1157 I W 393917472 + 1024 [kworker/u8:1] 8,0 1 93236 19.520616783 1495 D W 393917472 + 1024 [wget] 8,0 1 93237 19.520619072 1495 R W 393917472 + 1024 [0] 8,0 1 93238 19.520619161 1495 I W 393917472 + 1024 [wget] 8,0 1 93239 19.520619378 1495 D W 393917472 + 1024 [wget] 8,0 1 93240 19.520619512 1495 R W 393917472 + 1024 [0] 8,0 1 93241 19.520619570 1495 I W 393917472 + 1024 [wget] 8,0 1 93243 19.521467724 1495 D W 393917472 + 1024 [wget] 8,0 1 93482 19.549841502 0 C W 393917472 + 1024 [0] 8,2 3 38552 21.420915398 1157 A W 393917472 + 1024 <- (253,0) 393915424 8,0 3 38553 21.420915505 1157 A W 396016672 + 1024 <- (8,2) 393917472 8,2 3 38569 21.449524778 1157 A W 393919520 + 1024 <- (253,0) 393917472 |
HTTP 10GB single stream / write cache enabled / swap off / avg 709MB/s (5.672Gbps)
# hdparm -W0 /dev/sda
setting drive write-caching to 0 (off)
write-caching = 0 (off)
# swapoff -a
# free -h
total used free shared buff/cache available
Mem: 31G 2.0G 4.6G 39M 24G 28G
Swap: 0B 0B 0B
# wget http://203.XXX.YYY.60/10g.bin -O ~/10g.bin
...
2019-10-04 14:20:21 (709 MB/s) - ‘/root/10g.bin’ saved [10485760000/10485760000]
| 커널 블럭 IO 기록 # blktrace -d /dev/sda -o - | blkparse -i - -o /tmp/io.log # ^C 512KB 순차 쓰기 명령 발생 # grep wget io.log | grep D | tail -n 10 8,0 1 93223 19.518676605 1495 D W 393915424 + 1024 [wget] 8,0 1 93227 19.519645110 1495 D W 393915424 + 1024 [wget] 8,0 1 93228 19.519646224 1495 D W 393916448 + 1024 [wget] 8,0 1 93231 19.519648937 1495 D W 393916448 + 1024 [wget] 8,0 1 93235 19.520615024 1495 D W 393916448 + 1024 [wget] 8,0 1 93236 19.520616783 1495 D W 393917472 + 1024 [wget] 8,0 1 93239 19.520619378 1495 D W 393917472 + 1024 [wget] 8,0 1 93243 19.521467724 1495 D W 393917472 + 1024 [wget] 8,0 1 93244 19.521468858 1495 D W 393918496 + 1024 [wget] 8,0 1 93247 19.521471518 1495 D W 393918496 + 1024 [wget] 스왑 X # grep 393917472 io.log 8,0 3 22102 19.417867154 1157 A W 393917472 + 1024 <- (8,2) 391818272 8,0 3 22103 19.417867263 1157 Q W 393917472 + 1024 [kworker/u8:1] 8,0 3 22104 19.417867637 1157 G W 393917472 + 1024 [kworker/u8:1] 8,0 3 22105 19.417868284 1157 I W 393917472 + 1024 [kworker/u8:1] 8,0 1 93236 19.520616783 1495 D W 393917472 + 1024 [wget] 8,0 1 93237 19.520619072 1495 R W 393917472 + 1024 [0] 8,0 1 93238 19.520619161 1495 I W 393917472 + 1024 [wget] 8,0 1 93239 19.520619378 1495 D W 393917472 + 1024 [wget] 8,0 1 93240 19.520619512 1495 R W 393917472 + 1024 [0] 8,0 1 93241 19.520619570 1495 I W 393917472 + 1024 [wget] 8,0 1 93243 19.521467724 1495 D W 393917472 + 1024 [wget] 8,0 1 93482 19.549841502 0 C W 393917472 + 1024 [0] 8,2 3 38552 21.420915398 1157 A W 393917472 + 1024 <- (253,0) 393915424 8,0 3 38553 21.420915505 1157 A W 396016672 + 1024 <- (8,2) 393917472 8,2 3 38569 21.449524778 1157 A W 393919520 + 1024 <- (253,0) 393917472 |
HTTP 10GB single stream / write cache disabled / swap on / avg 441MB/s (3.528Gbps) / max 966MB/s (7.728Gbps)
# hdparm -W0 /dev/sda
setting drive write-caching to 0 (off)
write-caching = 0 (off)
# wget http://203.XXX.YYY.60/10g.bin -O ~/10g.bin -o ~/download.log
# awk '{print $8}' ~/download.log | sort | uniq | tail -n 20
...
923M
...
# tail -n 2 ~/download.log
2019-10-04 13:09:47 (441 MB/s) - ‘/root/10g.bin’ saved [10485760000/10485760000]
HTTP 10GB single stream / write cache disabled / swap off / avg 445MB/s (3.560Gbps)
# hdparm -W0 /dev/sda
setting drive write-caching to 0 (off)
write-caching = 0 (off)
# swapoff -a
# free -h
total used free shared buff/cache available
Mem: 31G 2.0G 4.6G 39M 24G 28G
Swap: 0B 0B 0B
# wget http://203.XXX.YYY.60/10g.bin -O ~/10g.bin
...
2019-10-04 14:35:13 (445 MB/s) - ‘/root/10g.bin’ saved [10485760000/10485760000]