개요

목적

  • 400GbE 패킷 생성 (dpdk, RoCE, iperf3)
  • 400GbE 링크에서 NVMe 기반 Disk to Disk 전송 실험 (NVMe의 MTBF, TBW 낮아도 됨)
  • PCIe 버스를 최대한 Gen5 NVMe로 채우는 구성
  • 실험 종료 후 추후 HDD 추가하여 




레퍼런스Type AType B비고

2023 ESnet6 200G DTN2025 KREONET 400G DTN Intel2025 KREONET 400G DTN AMD
CPU

AMD EPYC 73F3 (2ea) (Total 32C)

  • 16C 24T (per 1ea)
  • Base / Turbo
    : 3.5 / 4.0 GHz
  • L3 256 MB
  • PCIe 4.0 x128
  • EPYC 7003 Series (SP3)
  • TDP 240W
  • Q1'21
  • $3521

Intel Xeon w7-3565X

  • 32C 64T
  • Base / Turbo
    : 2.5 / 4.8 GHz
  • 82.5 MB
  • PCIe 5.0 x112
  • Xeon W (LGA-4677)
  • TDP 335~402W
  • Q3'24
  • $2699 (390만원)

AMD EPYC™ 9355P (1ea)

  • 32C 64T
  • Base / All Turbo / Max Turbo
    : 3.55 / 4.2 / 4.4 GHz
  • L3 256 MB
  • PCIe 5.0 x128
  • EPYC 9005 Series (SP5)
  • TDP 280W
  • Q4'24
  • $2998 (435만원)

싱글소켓 구성 (듀얼소켓 구성 가격 비쌈)

Chassis

SMCi 2124US-TNRP

  • 24 front hot-swap 2.5" NVMe U.2


SMCi > Chassis > 검색조건

  • Global SKU
  • Drive Form Factor 3.5 inch

SuperChassis 846BE1C-R1K23B

  • ATX, E-ATX-, EE-ATX
  • PCIe FHFL 장착 가능

SuperChassis 743AC-1K26B-SQ

  • ATX, E-ATX
  • PCIe FHFL 장착 가능
추후 디스크 추가를 위해 베이가 많은 샤시 선호
Motherboard

SMCi H12DSU-iN

SMCi > Motherboards > 검색조건

  • Form Factor: ATX, E-ATX
  • Socket: LGA-4677

SMCi X13SWA-TF

  • Form Factor: E-ATX
  • LGA-4677
  • Intel W790
  • 6 PCIe 5.0 x16
    • 1 x 400GbE
    • 3 x Hyper M2
    • 1 x Mega RAID
  • Gen5 M.2 onboard
  • VROC key 설치 가능
  • IPMI

SMCi > Motherboards > 검색조건

  • Form Factor: ATX, E-ATX
  • Socket: SP5

SMCi H13SSL-NT

  • Form Factor: ATX
  • EPYC™ 9005 series (SP5)
  • 3 PCIe 5.0 x16
    • 1  x 400GbE
    • 2 x Hyper M2
  • 2 PCIe 5.0 x8
    • 1 x Mega RAID
  • 3 MCIO PCIe 5.0 x8
  • Gen4 M.2 onboard
  • IPMI


RAM

DDR4-3200 16GB x 16ea (256GB)

256GB, 속도 제일 빠른 구성

DDR5-4800 ECC 64GB x 4ea

  • 46만원/1ea (184만원/4ea)

256GB, 속도 제일 빠른 구성

DDR5-4800 ECC 64GB x 4ea

  • 46만원/1ea (184만원/4ea)

OS_M2

Micron 7300 MAX 800G U.2 (2ea)

  • PCIe 3.0 x4

M.2 1ea

M.2 1ea
DATA_NVMe

Micron 9300 MAX 3.2TB U.2 (10ea)

  • PCIe 3.0 x4
  • configured as RAID10

Hynix P51 1TB M.2 (8~12ea)

  • PCIe 5.0 x4
  • 29만원/ea (232만원/8ea, 348만원/12ea)

ASUS Hyper M.2 x16 Gen5 Card (2~3ea)

  • 12만원/1ea (관세 부가 전)

비고

  • 총 비용 240만원/8ea, 400만원/12ea
  • RAID0 묶어서 테스트 할 것

Hynix P51 1TB M.2 (8~12ea)

  • PCIe 5.0 x4
  • 29만원/1ea (232만원/8ea, 348만원/12ea)

ASUS Hyper M.2 x16 Gen5 Card (2~3ea)

  • 12만원/1ea (관세 부가 전)

저렴한 M.2 기반 구성

  • U.2는 국내 일반 마켓에서 구매가 어려워 M.2 선호

Gen4 NVMe 5개, Gen5 NVMe 2~3개면 
레퍼런스와 비슷한 성능을 낼 수 있을 듯

목표는 레퍼런스보다 더 빠른 성능

ASUS Hyper M.2 x16 Gen5 Card

NIC

MCX613106A-VDAT (1ea)

  • PCIe® 4.0 x16
  • 200GbE Dual

MCX75310AAS-NEAT (1ea)

  • PCIe 5.0 x16
  • 400GbE Single
  • RoCE 가능
  • $1700~4000 (250~580만원)

MCX75310AAS-NEAT (1ea)

  • PCIe 5.0 x16
  • 400GbE Single
  • RoCE 가능
400GbE 싱글 포트
PSU1300W (2ea)


구매가$25K in late 20221.5천 / 1대 (목표)1.5천 / 1대 (목표)

CPU

선택 조건

필터

순번필터필터값비고
1base clockover 3.0 GHz너무 느린것 제외
2Pricingunder $ 5500너무 비싼 것 제외
3Num of coresover 15 cores
4Num of PCIe Lanesover 60NVMe 설치 위한 PCIe 레인 확보
5출시일after Jan 2022
6PCIe version5.0최신 NIC 설치

정렬

noorder by
비고
1All Core Turbodesc빠른 순서
2Basedesc
3Pricingasc가격 싼 순서

AMD

https://www.amd.com/ko/products/specifications/server-processor.html

이름PCIeCoresThreadsBaseAll Core TurboTurboL3 캐시Pricing시리즈출시일소켓소켓구성TDP비고
AMD EPYC™ 9275FPCIe® 5.0 x12824484.1 GHz4.5 GHz4.8 GHz256 MB3439EPYC 9005 Series10/10/2024SP51P / 2P320W
AMD EPYC™ 9375FPCIe® 5.0 x12832643.8 GHz4.4 GHz4.8 GHz256 MB5306EPYC 9005 Series10/10/2024SP51P / 2P320W
AMD EPYC™ 9355PPCIe® 5.0 x12832643.55 GHz4.2 GHz4.4 GHz256 MB2998EPYC 9005 Series10/10/2024SP51P280W싱글소켓, 상대적으로 저렴한 가격
AMD EPYC™ 9355PCIe® 5.0 x12832643.55 GHz4.2 GHz4.4 GHz256 MB3694EPYC 9005 Series10/10/2024SP51P / 2P280W
AMD EPYC™ 9335PCIe® 5.0 x12832643 GHz4 GHz4.4 GHz128 MB3178EPYC 9005 Series10/10/2024SP51P / 2P210W
AMD EPYC™ 9455PPCIe® 5.0 x12848963.15 GHz4.1 GHz4.4 GHz256 MB4819EPYC 9005 Series10/10/2024SP51P300W
AMD EPYC™ 9455PCIe® 5.0 x12848963.15 GHz4.1 GHz4.4 GHz256 MB5412EPYC 9005 Series10/10/2024SP51P / 2P300W
AMD EPYC™ 9274FPCIe® 5.0 x12824484.05 GHz4.1 GHz4.3 GHz256 MB3060EPYC 9004 Series11/10/2022SP51P / 2P320W
AMD EPYC™ 9255PCIe® 5.0 x12824483.2 GHz4 GHz4.3 GHz128 MB2495EPYC 9005 Series10/10/2024SP51P / 2P200W
AMD EPYC™ 9374FPCIe® 5.0 x12832643.85 GHz4.1 GHz4.3 GHz256 MB4850EPYC 9004 Series11/10/2022SP51P / 2P320W
AMD EPYC™ 9365PCIe® 5.0 x12836723.4 GHz4.15 GHz4.3 GHz192 MB4341EPYC 9005 Series10/10/2024SP51P / 2P300W
AMD EPYC™ 9354PPCIe® 5.0 x12832643.25 GHz3.75 GHz3.8 GHz256 MB2730EPYC 9004 Series11/10/2022SP51P280W
AMD EPYC™ 9354PCIe® 5.0 x12832643.25 GHz3.75 GHz3.8 GHz256 MB3420EPYC 9004 Series11/10/2022SP51P / 2P280W

Intel

https://www.intel.com/content/www/us/en/ark/featurefilter.html?productType=873&0_ECCMemory=True&3_ClockSpeed-Min=2500&3_ClockSpeedMax-Min=3500&3_CoreCount-Min=16

https://www.intel.com/content/www/us/en/products/compare.html?productIds=240486,240485,233417,236878,240487,233479,233416

SKUNamePCIe_RevPCIe_LanesCoresThreadsBase_GHzTurbo_GHzCachePricingFamilyLaunchLithographySocket_tScalabilityTDPMem_max비고
240486w7-3565X 511232642.54.882.5 MB2699W24Q3Intel 7LGA46771S Only　4 TB1순위
240485w7-3555 511228562.74.875 MB2339W24Q3Intel 7LGA46771S Only　4 TB
233417w7-3465X 511228562.54.875 MB2899W23Q1Intel 7LGA46771S Only　4 TB
236878w7-2595X 56426522.84.848.75 MB2049W24Q3Intel 7LGA46771S Only　2 TB
240487w7-3545 511224482.74.867.5 MB2039W24Q3Intel 7LGA46771S Only　4 TB
233479w7-3455 511224482.54.867.5 MB2489W23Q1Intel 7LGA46771S Only　4 TB
233416w7-2495X 56424482.54.845 MB2199W23Q1Intel 7LGA46771S Only　2 TB
232377Gold 6426Y 58016322.54.137.5 MB1517Scalables 4thQ1'23Intel 7LGA46772S185 W4 TB2소켓
232383Platinum 8462Y+ 58032642.84.160 MB5945Scalables 4thQ1'23Intel 7LGA46772S300 W4 TB
237558Platinum 8562Y+ 58032642.84.160 MB
Scalables 5thQ4'23Intel 7LGA46772S300 W4 TB
237559Gold 6542Y 58024482.94.160 MB
Scalables 5thQ4'23Intel 7LGA46772S250 W4 TB
237564Gold 6548Y+ 58032642.54.160 MB3726Scalables 5thQ4'23Intel 7LGA46772S250 W4 TB1소켓
237566Gold 6558Q 58032643.24.160 MB6416Scalables 5thQ4'23Intel 7LGA46772S350 W4 TB
237567Gold 6548N 58032642.84.160 MB
Scalables 5thQ4'23Intel 7LGA46772S250 W4 TB
237569Gold 6544Y 58016323.64.145 MB
Scalables 5thQ4'23Intel 7LGA46772S270 W4 TB
231749Platinum 8444H 58016322.9445 MB
Scalables 4thQ1'23Intel 7LGA4677S8S270 W4 TB
232380Gold 6442Y 58024482.6460 MB
Scalables 4thQ1'23Intel 7LGA46772S225 W4 TB
232385Gold 6444Y 58016323.6445 MB
Scalables 4thQ1'23Intel 7LGA46772S270 W4 TB
232386Gold 6458Q 58032643.1460 MB
Scalables 4thQ1'23Intel 7LGA46772S350 W4 TB
237259Platinum 8558P 58048962.74260 MB
Scalables 5thQ4'23Intel 7LGA46772S350 W4 TB
237560Gold 6526Y 58016322.83.937.5 MB
Scalables 5thQ4'23Intel 7LGA46772S195 W4 TB
2407756960P 596721442.73.9432 MB
Xeon 6Q3'24Intel 3LGA75292S500 W3 TB
231742Platinum 8458P 58044882.73.882.5 MB
Scalables 4thQ1'23Intel 7LGA46772S350 W4 TB
232597CPU Max 9462 58032642.73.575 MB
CPU MaxQ1'23Intel 7LGA46772S350 W4 TB

Chassis

SMCi AS-2015CS-TNR

CPUMotherboardChassis

AMD EPYC™ 9355P

SMCi H13SSW

SMCi AS-2015CS-TNR

SMCi AS-2015CS-TNR (PDF)






1x16 Full

ASUS Hyper M.2 x16 Gen5 Card

4 Hynix P51 1TB M.2

2x8 Full


3x16 Half
4x8 Full


5x16 Full

ASUS Hyper M.2 x16 Gen5 Card

2 Hynix P51 1TB M.2

6x16 Half

NVMe




NVMe 성능


Hynix P31 1TBHynix P41 1TBKIOXIA Plus G4 1TBPhison X200P 1.92TBPhison D200P 1.92TBHynix P51 1TB비고
가격 (만원)141517707929
PCIe3.04.05.05.05.05.0

M.2

M.2

M.2

U.2

E1.S

M.2


성능

순차읽기: 3,500MB/s 

순차쓰기: 3,200MB/s

읽기IOPS: 570K

쓰기IOPS: 600K

순차읽기: 7,000MB/s

순차쓰기: 6,500MB/s

읽기IOPS: 1,400K

쓰기IOPS: 1,300K

순차읽기: 10,000MB/s

순차쓰기: 7,900MB/s

읽기IOPS: 1,300K

쓰기IOPS: 1,400K

순차읽기: 14,800MB/s

순차쓰기: 4,300MB/s

읽기IOPS: 2,400K

쓰기IOPS: 170K

순차읽기: 14,000MB/s

순차쓰기: 8,600MB/s

읽기IOPS: 3,000K

쓰기IOPS: 400K

순차읽기: 14,700MB/s

순차쓰기: 13,400MB/s

읽기IOPS: 2,300K

쓰기IOPS: 2,400K

PCIe 버전 오를 때마다
약 2배씩 성능 향상

NVMe 개수 적게 장착하려면
무조건 최신 PCIe 지원되는 CPU/메인보드 구매

Hyper M.2 x16 Gen5 Card

https://www.asus.com/motherboards-components/motherboards/accessories/hyper-m-2-x16-gen5-card/

Compatibility - https://www.asus.com/support/faq/1037507/

NIC

ConnectX-6 (QSFP56)

https://docs.nvidia.com/networking/display/connectx6vpi

NVIDIA SKU

Legacy OPN

Marketing Description

Lifecycle

900-9X603-0016-DT0

MCX653105A-EFAT

ConnectX®-6 InfiniBand/Ethernet adapter card,  100Gb/s  (HDR100, EDR IB and 100GbE),  single-port QSFP56,  PCIe3.0/4.0 Socket Direct 2x8 in a row, tall bracket

Mass Production

900-9X603-0056-DT0

MCX653106A-EFAT

ConnectX®-6 InfiniBand/Ethernet adapter card,  100Gb/s  (HDR100, EDR IBand100GbE),  dual-port QSFP56,  PCIe3.0/4.0 Socket Direct 2x8 in a row, tall bracket

Mass Production

900-9X628-0016-ST0

MCX651105A-EDAT

ConnectX®-6 InfiniBand/Ethernet adapter card,  100Gb/s  (HDR100, EDR IB and 100GbE,  single-port QSFP56,  PCIe4.0 x8, tall bracket

Mass Production

900-9X6AF-0016-ST1

MCX653105A-ECAT

ConnectX®-6 InfiniBand/Ethernet adapter card,  100Gb/s  (HDR100, EDR IB and 100GbE),  single-port QSFP56,  PCIe3.0/4.0 x16, tall bracket

Mass Production

900-9X6AF-0056-MT1

MCX653106A-ECAT

ConnectX®-6 InfiniBand/Ethernet adapter card,  100Gb/s  (HDR100, EDR IB and 100GbE),  dual-port QSFP56,  PCIe3.0/4.0 x16, tall bracket

Mass Production

900-9X6AF-0018-MT2

MCX653105A-HDAT

ConnectX®-6 InfiniBand/Ethernet adapter card,  HDR IB  (200Gb/s) and 200GbE,  single-port QSFP56,  PCIe3.0/4.0 x16, tall bracket

Mass Production

900-9X6AF-0058-ST1

MCX653106A-HDAT

ConnectX®-6 InfiniBand/Ethernet adapter card,  HDR IB  (200Gb/s) and 200GbE,  dual-port QSFP56,  PCIe3.0/4.0 x16, tall bracket

Mass Production

900-9X6B4-0058-DT0

MCX654106A-HCAT

ConnectX®-6 InfiniBand/Ethernet adapter card,  HDR IB  (200Gb/s) and 200GbE,  dual-port QSFP56,  Socket Direct 2x PCIe3.0/4.0x16, tall bracket

Mass Production

900-9X6AF-0018-SS0

MCX653105A-HDAL

ConnectX®-6 InfiniBand/Ethernet adapter card,  HDR IB  (200Gb/s) and 200GbE,  single-port QSFP56,  PCIe3.0/4.0 x16,  cold plate for liquid-cooled Intel® Server System D50TNP platforms, tall bracket, ROHS R6

Mass Production

900-9X6AF-0058-SS0

MCX653106A-HDAL

ConnectX®-6 InfiniBand/Ethernet adapter card,  HDR IB  (200Gb/s) and 200GbE,  dual-port QSFP56,  PCIe3.0/4.0 x16,  cold plate for liquid-cooled Intel® Server System D50TNP platforms, tall bracket, ROHS R6

Mass Production

900-9X0BC-001H-ST1

MCX683105AN-HDAT

ConnectX®-6 DE adapter card,  HDR IB  (200Gb/s),  single-port QSFP,  PCIe4.0 x16,  no crypto, tall bracket

Mass Production

ConnectX-7 (OSFP)

https://docs.nvidia.com/networking/display/connectx7vpi

NVIDIA SKU

Legacy OPN

Form Factor

Data Transmission Rate

No. of Ports and Type

PCIe Support

Secure Boot

Crypto

Bracket Type

Lifecycle

900-9X766-003N-SQ0

MCX75310AAS-NEAT

PCIe Half Height, Half Length

2.71 in. x 6.6 in. (68.90mm x 167.65 mm)

InfiniBand: NDR 400Gb/s (Default speed)

Ethernet: 400GbE

Single-port OSFP

PCIe x16 Gen 4.0/5.0 @ SERDES 16GT/s/32GT/s

✔

-

Tall Bracket

Mass Production

900-9X766-003N-SR0

MCX75310AAC-NEAT

PCIe Half Height, Half Length

2.71 in. x 6.6 in. (68.90mm x 167.65 mm)

InfiniBand: NDR 400Gb/s (Default speed)

Ethernet: 400GbE

Single-port OSFP

PCIe x16 Gen 4.0/5.0 @ SERDES 16GT/s/32GT/s

✔

✔

Tall Bracket

Mass Production

900-9X766-003N-ST0

MCX75310AAS-HEAT

PCIe Half Height, Half Length

2.71 in. x 6.6 in. (68.90mm x 167.65 mm)

InfiniBand: NDR200 200Gb/s (Default speed)

Ethernet: 200GbE

Single-port OSFP

PCIe x16 Gen 4.0/5.0 @ SERDES 16GT/s/32GT/s

✔

-

Tall Bracket

Mass Production

900-9X7AO-00C3-STZ

MCX713104AC-ADAT

PCIe Half Height, Half Length

2.71 in. x 6.6 in. (68.90mm x 167.65 mm)

Ethernet: 50/25GbE

Quad-port SFP56

PCIe x16 Gen 4.0 @ SERDES 16GT/s

✔

✔

Tall Bracket

Mass Production

900-9X7AO-0003-ST0

MCX713104AS-ADAT

PCIe Half Height, Half Length

2.71 in. x 6.6 in. (68.90mm x 167.65 mm)

Ethernet: 50/25GbE

Quad-port SFP56

PCIe x16 Gen 4.0 @ SERDES 16GT/s

✔

-

Tall Bracket

Mass Production

OSFP - QSFP 연결

OSFP-SR 트랜시버 - OM4 케이블 - QSFP-DD SR 트랜시버 연결

https://www.fibermall.com/ko/blog/qsfp-dd-400g-sr4-choice-for-hpc.htm

OSFP-QDD-CV

https://www.fibermall.com/ko/sale-461130-800g-osfp-to-qsfp-dd-adapter-module.htm

  • 레이블 없음