100GbE PS와 10GbE PS를 하나의 머신에서 운영
PS(perfSONAR) 운영시 대역폭 테스트 머신과 지연(latency) 테스트 머신을 분리하는 것이 좋음
100GbE 머신에서 지연(latency) 테스트 머신을 VM으로 운영
Operate 100GbE PS and 10GbE PS on same baremetal server
Highly recommended to split throughput and latency test machine
Launch latency test machine as VM on throughput test machine
Vendor | Supermicro | |
---|---|---|
Model | 6028r | |
CPU | Intel Xeon E5-2643 v4 @ 3.40GHz | 6C 12T x 2ea |
MEM | DDR4 128GB | |
NIC1 | Intel 10GbE dual port | |
NIC2 | Mellanox ConnectX-5 100GbE single port | |
Hostname | ps-amst.kreonet2.net | |
OS | CentOS 7.9.2009 | |
perfSONAR | 4.4.4-1.el7 |
CPU | host-passthrough | vCPU : 4C |
---|---|---|
MEM | DDR4 16GB | |
NIC1 | pci-passthrogh | Intel 10GbE single port |
Hostname | ps-amst-v1.kreonet2.net | |
OS | CentOS 7.9.2009 | |
perfSONAR | 4.4.4-1.el7 |
# sudo yum -y install tmux tree git zsh sqlite net-tools ipmitool lshw inxi htop hwloc-gui # sudo yum -y install cockpit cockpit-machines libosinfo qemu-kvm libvirt libvirt-python libguestfs-tools virt-install # sudo firewall-cmd --add-service=cockpit # sudo firewall-cmd --add-service=cockpit --permanent # sudo systemctl enable --now cockpit.socket |
# lstopo --ignore misc hwconfig.png |
# sudo lshw -short -c net H/W path Device Class Description ======================================================= /0/100/1/0 ens1f0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection /0/100/1/0.1 ens1f1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection /0/2/0 ens6 network MT27800 Family [ConnectX-5] # sudo ls -al /sys/class/net/ens1f0/device/driver /sys/class/net/ens1f0/device/driver -> ../../../../bus/pci/drivers/ixgbe # sudo ls -al /sys/bus/pci/drivers/ixgbe 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 ... |
[Unit] Description=Script to enable SR-IOV on boot [Service] Type=oneshot ExecStart=/usr/bin/bash -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/ixgbe/unbind' ExecStart=/usr/bin/bash -c 'echo 0000:01:00.1 > /sys/bus/pci/drivers/ixgbe/unbind' [Install] WantedBy=multi-user.target |
# sudo ifconfig ens1f0 down # sudo ifconfig ens1f1 down # sudo systemctl daemon-reload # sudo systemctl enable sriov-vf.service # sudo systemctl start sriov-vf.service # lshw -short -c net H/W path Device Class Description ======================================================= /0/100/1/0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection /0/100/1/0.1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection /0/2/0 ens6 network MT27800 Family [ConnectX-5] |
2-1. SR-IOV config for Dell BIOS
2-2. SR-IOV config for AMI BIOS
3. SR-IOV config for Linux Kernel Boot
# scp ps-amst-v1.qcow2 root@HOST:/var/lib/libvirt/images # scp ps-amst-v1.xml root@HOST:/var/lib/libvirt/images |
# sudo virsh define ps-amst-v1.xml # sudo virsh edit ps-amst-v1 # sudo virsh start ps-amst-v1 |
Libvirt NUMA Tuning > vCPU Pinning