Pre-req: Install mlxofed pkg with --dpdk and --upstream-lib
| 코드 블럭 |
|---|
/mnt/mlnxofedinstall --without-fw-update --dpdk --upstream-libs --force
/etc/init.d/openibd restart |
Pre-req: Enable IOMMU for virtualization
| 코드 블럭 |
|---|
dmesg | grep -i -e DMAR -e IOMMU
# For rocky/centos
sudo nano /etc/default/grub
GRUB_CMDLINE_LINUX="..... intel_iommu=on iommu=pt pci=realloc"
sudo grub-mkconfig -o /boot/grub/grub.cfg
# For ubuntu
GRUB_CMDLINE_LINUX_DEFAULT="...... intel_iommu=on"
sudo grub-mkconfig -o /boot/grub/grub.cfg
reboot
|
If you want to create virtual function, first check in the firmware SRIOV_EN is 'enabled' or 'disabled'. Enable it if it isnt.
| 코드 블럭 |
|---|
mlxconfig -d /dev/mst/mt4119_pciconf0 q | grep SRIOV_EN
# Set the Number of VFs in the firmware
# Run mst start.
# Using the IDs obtained in the previous task, type the following:
sudo mlxconfig -d /dev/mst/mt4119_pciconf0set SRIOV_EN=1 NUM_OF_VFS=<number between 0-127>
sudo mlxconfig -d /dev/mst/mt4119_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
sudo mlxconfig -d /dev/mst/mt4119_pciconf0 query
sudo mlxconfig -d /dev/mst/mt4119_pciconf0 query
#Update the firmware
sudo mst start
sudo mst status -v
sudo flint -d /dev/mst/mt4119_pciconf0 -i fw-download-from-official-website-based-on-psid-of-your-nic.bin burn
# Reboot for the settings to take effect.
cat /sys/class/net/enp10s0np0/device/mlx5_num_vfs
echo 8 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs
or
echo 8 > /sys/class/net/ens785f0/device/sriov_numvfs
or
echo 8 > /sys/class/net/ens785f0/device/mlx5_num_vfs |
Build 'igb_uio' driver if needed
| 정보 |
|---|
If you wanna build 'igb_uio', follow below. In my case, it did not work for Mellanox Connect X5 (mlx_core driver was enough) |
...
| 코드 블럭 |
|---|
export RTE_SDK=/root/dpdk-23.03
export RTE_TARGET=x86_64-native-linux-gcc
wget https://git.dpdk.org/apps/pktgen-dpdk/snapshot/pktgen-dpdk-pktgen-23.06.1.tar.xz
tar xJf pktgen-dpdk-pktgen-23.06.1.tar.xz
|
| 코드 블럭 |
|---|
335 mlxconfig -d /dev/mst/mt4119_pciconf0 set ZERO_TOUCH_TUNING_ENABLE=1
336 mlxconfig -d /dev/mst/mt4119_pciconf0 set CQE_COMPRESSION=1\n
337 mlxconfig -d /dev/mst/mt4119_pciconf0 s PCI_WR_ORDERING=1
338 reboot |
Testing between two servers. We are using 8 virtual function in each server but not utilizing all of them.
...