Goal

Find minimum hardward requirement of 100G perfSonar


Method

Change num of CPU, freq of CPU → test


Ref

Spec of 100G PS using VM

3.5GHz 2 core, 16GB RAM

http://ps-100g-hpr01.stanford.edu/


Change CPU (num of cpu, on/off multithreading, freq)

# lscpu
CPU(s):                24
On-line CPU(s) list:   0-23
Model name:            Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz

# cat /proc/cpuinfo | grep processor | wc -l
24

# echo 0 > /sys/devices/system/cpu/cpu{20,21,22,23}/online

# lscpu
CPU(s):                24
On-line CPU(s) list:   0-19
Off-line CPU(s) list:  20-23

# cat /proc/cpuinfo | grep processor | wc -l
20

# echo 1 > /sys/devices/system/cpu/cpu{20,21,22,23}/online


# lscpu
CPU(s):                24
On-line CPU(s) list:   0-23

# cat /sys/devices/system/cpu/smt/active
1

# echo off > /sys/devices/system/cpu/smt/control

# lscpu
CPU(s):                24
On-line CPU(s) list:   0-11
Off-line CPU(s) list:  12-23

# cat /proc/cpuinfo | grep processor | wc -l
12


# echo off > /sys/devices/system/cpu/smt/control

# lscpu
CPU(s):                16
On-line CPU(s) list:   0-15
Off-line CPU(s) list:  8-15
Model name:            Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
CPU MHz:               1243.188
CPU min MHz:           1200.0000
CPU max MHz:           3300.0000

# cpupower frequency-info
analyzing CPU 0:
  driver: intel_pstate
  hardware limits: 1.20 GHz - 3.30 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 1.20 GHz and 3.30 GHz.
                  The governor "powersave" may decide which speed to use  within this range.
  current CPU frequency: 1.20 GHz (asserted by call to hardware)
  boost state support:
    Supported: yes
    Active: yes
    3200 MHz max turbo 4 active cores
    3200 MHz max turbo 3 active cores
    3300 MHz max turbo 2 active cores
    3300 MHz max turbo 1 active cores

# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
...
Setting cpu: 7

# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | sort -u
performance


# vi /etc/default/grub
# Append "intel_pstate=disable" to GRUB_CMDLINE_LINUX= option
# update grub via some commands
# reboot

# modprobe cpufreq_userspace
# cpupower frequency-set --governor userspace
# cpupower --cpu all frequency-set --freq 1.5GHz


# ethtool p6p1
Settings for p6p1:
	Supported ports: [ FIBRE ]
    Supported link modes: ... 100000baseCR4/Full

## check NUMA locality
# cat /sys/class/net/p6p1/device/numa_node
0

## if single socket system, no need to be set
# numactl -N 0 iperf3 -s
# watch -n 1 grep MHz /proc/cpuinfo
cpu	MHz									:	1199.960	
...
cpu	MHz									:	3700.000	
...	

https://fasterdata.es.net/assets/Papers-and-Publications/100G-Tuning-TechEx2016.tierney.pdf



Disable hyperthreading from within Linux (no access to BIOS), https://serverfault.com/questions/235825

Programmatically disable CPU core, https://stackoverflow.com/questions/44907731

How to on/offline CPU on RHEL(CentOS), https://blog.seabow.pe.kr/?p=7303

(star) 100g Network Adapter Tuning - https://srcc.stanford.edu/100g-network-adapter-tuning

(star) (2016) Recent Linux TCP Updates, and how to tune your 100G host - https://fasterdata.es.net/assets/Papers-and-Publications/100G-Tuning-TechEx2016.tierney.pdf

(star) (2016) Network Performance Tuning, Senior Software Maintenance Engineer Red Hat, Inc. - https://sysadmin.miniconf.org/2016/lca2016-jamie_bainbridge-network_performance_tuning.html#1

Red Hat > Monitoring and managing system status and performanceChapter > 17. Tuning CPU frequency to optimize energy consumption - https://access.redhat.com/documentation/enus/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/tuning-cpu-frequency-to-optimize-energy-consumption_monitoring-and-managing-system-status-and-performance

How to change and check cpu frequency governors - https://docs.altibase.com/pages/viewpage.action?pageId=13436485

CPU Frequency Scaling On Linux With cpupower - https://krimson8.github.io/linux/2019/03/13/Configuring-CPU-frequency.html

How to solve the cpufreqset errors - https://stackoverflow.com/questions/23526671

How to set init.d (rc.d) for permanent settings - https://docs.altibase.com/pages/viewpage.action?pageId=13436485 / https://kwonnam.pe.kr/wiki/linux/cpufreq

Optimize iperf Performance of External Loopback in Dual-port Design - https://github.com/corundum/corundum/issues/33



Ref spec for 100G PS

perfSONAR Deployment Examples > 100G Configuration - https://docs.perfsonar.net/deployment_examples.html

  • CPU: 2 x Intel Xeon Haswell E5-2643V3 3.4GHz 6 Cores (Total 12 Threads each)
  • Memory: 8 x 16GB DDR4-2133MHz RAM ECC/REG (128GB total)
  • NetworkMellanox MCX455A-ECAT 100Gbps NIC

perfSONAR Node Reference Implementation - https://fasterdata.es.net/performance-testing/perfsonar/perfsonar-node-reference-implementation

  • Current (~2021/22) 40/50/100Gbps Capable Design 1
    • The total cost of this server was around $4K in mid 2021.
    • CPU: AMD EPYC 7302P 16-Core Processor (3.0GHz base clock, 3.3GHz max boost)
    • Memory: 8 x 8GB DDR4 ECC/REG RAM (64G Total)
    • Network: Mellanox MCX556A-ECAT Dual Port Connect X5
  • Current (~2021/22) 40/50/100Gbps Capable Design 2
    • We also tested and liked the following system, but due to supply chain issues, went with the above system.
    • CPU: Intel Xeon Silver 4215R 8C 3.2Ghz
    • Memory: 6 x 8GB DDR4 ECC/REG RAM (48G Total)
    • Network: Mellanox MCX556A-ECAT Dual Port Connect X5


  • No labels