Overview

Purpose (KR)

100GbE PS와 10GbE PS를 하나의 머신에서 운영

PS(perfSONAR) 운영시 대역폭 테스트 머신과 지연(latency) 테스트 머신을 분리하는 것이 좋음

100GbE 머신에서 지연(latency) 테스트 머신을 VM으로 운영


Purpose (EN)

Operate 100GbE PS and 10GbE PS on same baremetal server

Highly recommended to split throughput and latency test machine

Launch latency test machine as VM on throughput test machine


Host ( 100G perfSONAR + VM Host )

VendorSupermicro
Model6028r
CPUIntel Xeon E5-2643 v4 @ 3.40GHz6C 12T x 2ea
MEMDDR4 128GB
NIC1Intel 10GbE dual port
NIC2Mellanox ConnectX-5 100GbE single port
Hostnameps-amst.kreonet2.net
OSCentOS 7.9.2009
perfSONAR4.4.4-1.el7

VM Guest ( 10G perfSONAR )

CPU

host-passthrough

vCPU : 4C

MEMDDR4 16GB
NIC1

pci-passthrogh

Intel 10GbE single port
Hostname

ps-amst-v1.kreonet2.net


OSCentOS 7.9.2009
perfSONAR4.4.4-1.el7


Install utils and cockpit-machines

# sudo yum -y install tmux tree git zsh sqlite net-tools ipmitool lshw inxi htop hwloc-gui

# sudo yum -y install cockpit cockpit-machines libosinfo qemu-kvm libvirt libvirt-python libguestfs-tools virt-install

# sudo firewall-cmd --add-service=cockpit
# sudo firewall-cmd --add-service=cockpit --permanent

# sudo systemctl enable --now cockpit.socket

Check Hardware

# lstopo --ignore misc hwconfig.png

# sudo lshw -short -c net
H/W path         Device      Class          Description
=======================================================
/0/100/1/0       ens1f0      network        82599ES 10-Gigabit SFI/SFP+ Network Connection
/0/100/1/0.1     ens1f1      network        82599ES 10-Gigabit SFI/SFP+ Network Connection
/0/2/0           ens6        network        MT27800 Family [ConnectX-5]

# sudo ls -al /sys/class/net/ens1f0/device/driver
/sys/class/net/ens1f0/device/driver -> ../../../../bus/pci/drivers/ixgbe

# sudo ls -al /sys/bus/pci/drivers/ixgbe
0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
...

Unload NIC driver

[Unit]
Description=Script to enable SR-IOV on boot

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/ixgbe/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:00.1 > /sys/bus/pci/drivers/ixgbe/unbind'

[Install]
WantedBy=multi-user.target


# sudo ifconfig ens1f0 down
# sudo ifconfig ens1f1 down

# sudo systemctl daemon-reload
# sudo systemctl enable sriov-vf.service
# sudo systemctl start sriov-vf.service

# lshw -short -c net
H/W path         Device      Class          Description
=======================================================
/0/100/1/0                   network        82599ES 10-Gigabit SFI/SFP+ Network Connection
/0/100/1/0.1                 network        82599ES 10-Gigabit SFI/SFP+ Network Connection
/0/2/0           ens6        network        MT27800 Family [ConnectX-5]

Enable SR-IOV

2-1. SR-IOV config for Dell BIOS

2-2. SR-IOV config for AMI BIOS

3. SR-IOV config for Linux Kernel Boot

Launch VM

# scp ps-amst-v1.qcow2 root@HOST:/var/lib/libvirt/images
# scp ps-amst-v1.xml root@HOST:/var/lib/libvirt/images


# sudo virsh define ps-amst-v1.xml
# sudo virsh edit ps-amst-v1
# sudo virsh start ps-amst-v1

Etc


To do

Libvirt NUMA Tuning > vCPU Pinning

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-numa-numa_and_libvirt#sect-Virtualization_Tuning_Optimization_Guide-NUMA-VCPU_Pinning