100Gbps 네트워크 성능 테스트 서버 사양


(요약)


성능측정데이터 전송
CPU

2.5GHz 이상 8코어

(보통 3.4Ghz 이상) 

2.5GHz 이상 12코어
Disk
NVMe PCIe 3.0  8 개 이상


(2023 구매 예정) 성능측정

  • CPU : Intel Xeon Gold 6334 / 3세대 Xeon / 8C 16T 3.6Ghz (Turbo 3.7Ghz)
  • MEM : 6 x 8GB DDR4 ECC/REG RAM (48G Total)
  • NIC : Mellanox Connect X-5 or X-6
  • 1U short case
  • 가격 : 7백만원 (예상)


(2021 구매) 데이터 전송 노드

  • CPU : Intel Xeon Gold 6246 ( 12C 24T, 3.3GHz, Turbo 4.20 GHz) x 2
  • MEM : 16 x 12GB DDR4 ECC/REG RAM (192G Total)
  • DISK : U.2 NVMe 2TB x 8 (PCIe Gen3 x4)
  • NIC : Mellanox Connect X-5
  • 가격 : 2.5천만원 (2021)


(미국) 100G PS

perfSONAR Node Reference Implementation

Current (~2021) 40/50/100Gbps Capable Design

The total cost of this server was around $4K in mid 2021.  This system is being tested currently, and we be deployed to ESnet in early/mid 2022. Please note that specifics on configuration will be available after full evaluation.  

Configuration Details

Hardware description

  • Base System: Supermicro 5019P-WTR
    • Onboard Dual 10GB RJ45
    • Onboard VGA
    • Redundant Power
    • Onboard IPMI
  • Motherboard: X11SPW-TF
  • CPU: Intel Xeon Silver 4215R 8C 3.2Ghz
  • Storage: 2 x Intel S4610 480GB SSD - 3 DWPD
  • Memory: 6 x 8GB DDR4 ECC/REG RAM (48G Total)
  • Network: Mellanox MCX556A-ECAT Dual Port Connect X5

System Block Diagram

X11SPW TF

https://fasterdata.es.net/performance-testing/perfsonar/perfsonar-node-reference-implementation/


(미국) 100G DTN

100G Experimental (~2018) DTN Design

If you wish to build a 100G DTN, here are the important hardware considerations:

  1. 100G NICs require PCIe Gen3 x16. If you are assuming all transfers are 4 or more parallel streams, any CPU with a clock rate greater than 2.5GHz should be fast enough to push 25Gbps per flow with the standard tuning applied.
  2. 100 Gbps (or 12.5 GBytes/sec) of disk IO is challenging.

Based on reports from colleagues, to get this much disk I/O you'll need either:

  • 10 SSD NVME PCIe Gen 3 x4 drives (e.g.: Samsung 950 Pro with U.2 to M.2 2.5" adapters), or 8 high-end NVME PCIe Gen 3 x4 drives (e.g.: Intel DC P3700 with U.2, 2.5" version that has high endurance).
  • 24 SSD SATA drives (with two PCIe Gen 3 x8 RAID controllers, or one PCIe Gen 3 x16 controller)

Note that both of these configurations will require a special chassis that can hold that many drives, such as:

Sample drives that have been used for 100G DTNs include:

  • SSD NVME drives:
    • Samsung 950 Pro or Intel DC P3700
  • SSD SATA drives:
    • Samsung PM863 or SM863

https://fasterdata.es.net/science-dmz/DTN/100g-dtn/


Current (~2020) 40/50/100Gbps Capable DTN Design

The total cost of this server was around $21K in mid 2019.  This system is being tested currently, and we be deployed to ESnet in mid/late 2020. Please note that specifics on configuration will be available after full evaluation.  Note that this server uses VROC, and requires the purchase of a premium license.  

Configuration Details

Hardware description

  • Base System: Gigabyte R281-NO0 dual socket P 2U server
    • Onboard: VGA, 2 x GbE RJ45 Intel i350, IPMI dedicated LAN
    • 24 x front access U.2 hotswap bays
    • 2 x rear access 2.5” SATA hotswap bays
    • Dual redundant hotswap 1600W PSU

  • 2 x Intel Cascade lake Xeon Gold 6246
    • 12 cores each
    • 3.3GHz (Turbo 4.20 GHz) 165W TDP processor 
  • 12 x 16G DDR4 2933 ECC RDIMM (192G total)
  • 10 x Intel P4610 1.6TB U.2/2.5” PCIe NVMe 3.0 x4 Drives (connect directly to CPU for VROC)
  • 2 x Enterprise 960G 2.5“ SATA SSD (OS, onboard Intel SATA Raid 1)
  • VIntel® Virtual RAID On CPU (VROC) RAID 0, 1, 10, 5
  • Mellanox ConnectX-5 EN MCX516A-CCAT 40/50/100GbE dual-port QSFP28 NIC

https://fasterdata.es.net/science-dmz/DTN/reference-implementation/


(호주) AARnet SDMZ 100G DTN Servers

DTN With Low Storage/Benchmark Server

Hardware Supplier: SuperMicro

web site: http://www.supermicro.com/products/system/2u/2028/SYS-2028U-TN24R4T_.cfm

ModelDescriptionQuantity
SYS-2028U-TN24R4T+SuperServer SYS-2028U-TN24R4T+1
P4X-DPE52643V4-SR2P4CPU E5-2643V4 ( 6C 12T , 3.40 GHz , Turbo 3.70 GHz )2
MEM-DR416L-SL06-ER24Samsung 16GB DDR4-2400 1R x 4 LP ECC REG RoHS8
HDS-2VM-SSDPE2MX012T4Intel P3500 1.2TB, NVMe PCIe 3.0, MLC 2.5" 20nm 0.3DWPD10
HDS-2TD-SSDSC2BB240G7Intel S3520 240GB, SATA 6Gb/s, 3D MLC 2.5" 7.0mm, up to 1DWPD2
SFT-OOB-LICOOB Management Package(per node license)1
EWCSCSMC Warranty- 3 Years Parts & Labor with 1 Year advance replacement1


Network Card

Hardware Supplier: Mellanox

Web site: http://www.mellanox.com/page/products_dyn?product_family=260&mtag=connectx_5_en_card

ModelDescriptionQuantity
MCX515A-CCATConnectX®-5 EN network interface card, 100GbE single-port QSFP28, PCIe3.0 x16, tall bracket, ROHS R61


DTN with Mid Storage Server

Hardware Supplier: SuperMicro

web site: http://www.supermicro.com/products/system/2u/2028/SYS-2028U-TN24R4T_.cfm

ModelDescriptionQuantity
SYS-2028U-TN24R4T+SuperServer SYS-2028U-TN24R4T+1
P4X-DPE52643V4-SR2P4CPU E5-2643V4 ( 6C 12T , 3.40 GHz , Turbo 3.70 GHz )2
MEM-DR416L-SL06-ER24Samsung 16GB DDR4-2400 1R x 4 LP ECC REG RoHS8
HDS-2VM-SSDPE2MX012T4Intel P3500 1.2TB, NVMe PCIe 3.0, MLC 2.5" 20nm 0.3DWPD24
HDS-2TD-SSDSC2BB240G7Intel S3520 240GB, SATA 6Gb/s, 3D MLC 2.5" 7.0mm, up to 1DWPD2
SFT-OOB-LICOOB Management Package(per node license)1
EWCSCSMC Warranty- 3 Years Parts & Labor with 1 Year advance replacement1


Network Card

Hardware Supplier: Mellanox

Web site: http://www.mellanox.com/page/products_dyn?product_family=260&mtag=connectx_5_en_card

ModelDescriptionQuantity
MCX515A-CCATConnectX®-5 EN network interface card, 100GbE single-port QSFP28, PCIe3.0 x16, tall bracket, ROHS R61


https://support.aarnet.edu.au/hc/en-us/articles/115010092388-SDMZ-100G-DTN-Servers