Overview
Host ( 100G perfSONAR + VM Host )
| Vendor | Supermicro | |
|---|---|---|
| Model | 6028r | |
| CPU | Intel Xeon E5-2643 v4 @ 3.40GHz | 6C 12T x 2ea |
| MEM | DDR4 128GB | |
| NIC1 | Intel 10GbE dual port | |
| NIC2 | Mellanox ConnectX-5 100GbE single port | |
| Hostname | ps-amst.kreonet2.net | |
| OS | CentOS 7.9.2009 | |
| perfSONAR | 4.4.4-1.el7 |
VM Guest ( 10G perfSONAR )
| CPU | host-passthrough | vCPU : 4C |
|---|---|---|
| MEM | DDR4 16GB | |
| NIC1 | pci-passthrogh | Intel 10GbE single port |
| Hostname | ps-amst-v1.kreonet2.net | |
| OS | CentOS 7.9.2009 | |
| perfSONAR | 4.4.4-1.el7 |
Install utils and cockpit-machines
# sudo yum -y install tmux tree git zsh sqlite net-tools ipmitool lshw inxi htop hwloc-gui # sudo yum -y install cockpit cockpit-machines libosinfo qemu-kvm libvirt libvirt-python libguestfs-tools virt-install # sudo firewall-cmd --add-service=cockpit # sudo firewall-cmd --add-service=cockpit --permanent # sudo systemctl enable --now cockpit.socket
Check Hardware
# lstopo --ignore misc hwconfig.png
# sudo lshw -short -c net H/W path Device Class Description ======================================================= /0/100/1/0 ens1f0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection /0/100/1/0.1 ens1f1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection /0/2/0 ens6 network MT27800 Family [ConnectX-5] # sudo ls -al /sys/class/net/ens1f0/device/driver /sys/class/net/ens1f0/device/driver -> ../../../../bus/pci/drivers/ixgbe # sudo ls -al /sys/bus/pci/drivers/ixgbe 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 ...
Unload NIC driver
/etc/systemd/system/sriov-vf.service
[Unit] Description=Script to enable SR-IOV on boot [Service] Type=oneshot ExecStart=/usr/bin/bash -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/ixgbe/unbind' ExecStart=/usr/bin/bash -c 'echo 0000:01:00.1 > /sys/bus/pci/drivers/ixgbe/unbind' [Install] WantedBy=multi-user.target
# sudo ifconfig ens1f0 down # sudo ifconfig ens1f1 down # sudo systemctl daemon-reload # sudo systemctl enable sriov-vf.service # sudo systemctl start sriov-vf.service # lshw -short -c net H/W path Device Class Description ======================================================= /0/100/1/0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection /0/100/1/0.1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection /0/2/0 ens6 network MT27800 Family [ConnectX-5]
Enable SR-IOV
2-1. SR-IOV config for Dell BIOS
2-2. SR-IOV config for AMI BIOS
3. SR-IOV config for Linux Kernel Boot
Launch VM
# scp ps-amst-v1.qcow2 root@HOST:/var/lib/libvirt/images # scp ps-amst-v1.xml root@HOST:/var/lib/libvirt/images
# sudo virsh define ps-amst-v1.xml # sudo virsh edit ps-amst-v1 # sudo virsh start ps-amst-v1
Etc
