Pre-req: Install dependencies
# For Ubuntu apt install build-essential apt install python3-pyelftools apt-get install -y libarchive-dev apt-get install -y libelf-dev apt install cmake apt-get install -y pkg-config apt-get install -y libdpdk-dev apt-get install -y libnuma-dev apt-get install git libpcap-dev apt install python3-pip pip3 install meson ninja pip3 install meson ninja # For Rocky/RHEL dnf groupinstall "Development Tools" dnf install numactl-devel pip3 install meson ninja pip3 install pyelftools # For CentOS 7 yum -y install kernel-devel.x86_64 yum -y install numactl-devel.x86_64 yum -y install elfutils-libelf-devel yum -y install libpcap-devel
Pre-req: Install mlxofed pkg with --dpdk and --upstream-lib
/mnt/mlnxofedinstall --without-fw-update --dpdk --upstream-libs --force /etc/init.d/openibd restart
Pre-req: Enable IOMMU for virtualization
dmesg | grep -i -e DMAR -e IOMMU # For rocky/centos sudo nano /etc/default/grub GRUB_CMDLINE_LINUX="..... intel_iommu=on iommu=pt pci=realloc" sudo grub-mkconfig -o /boot/grub/grub.cfg # For ubuntu GRUB_CMDLINE_LINUX_DEFAULT="...... intel_iommu=on" sudo grub-mkconfig -o /boot/grub/grub.cfg reboot
If you want to create virtual function, first check in the firmware SRIOV_EN is 'enabled' or 'disabled'. Enable it if it isnt.
mlxconfig -d /dev/mst/mt4119_pciconf0 q | grep SRIOV_EN # Set the Number of VFs in the firmware # Run mst start. # Using the IDs obtained in the previous task, type the following: sudo mlxconfig -d /dev/mst/mt4119_pciconf0set SRIOV_EN=1 NUM_OF_VFS=<number between 0-127> sudo mlxconfig -d /dev/mst/mt4119_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8 sudo mlxconfig -d /dev/mst/mt4119_pciconf0 query sudo mlxconfig -d /dev/mst/mt4119_pciconf0 query #Update the firmware sudo mst start sudo mst status -v sudo flint -d /dev/mst/mt4119_pciconf0 -i fw-download-from-official-website-based-on-psid-of-your-nic.bin burn # Reboot for the settings to take effect. cat /sys/class/net/enp10s0np0/device/mlx5_num_vfs echo 8 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs or echo 8 > /sys/class/net/ens785f0/device/sriov_numvfs or echo 8 > /sys/class/net/ens785f0/device/mlx5_num_vfs
Build 'igb_uio' driver if needed
If you wanna build 'igb_uio', follow below. In my case, it did not work for Mellanox Connect X5 (mlx_core driver was enough)
git clone git://dpdk.org/dpdk-kmods
cp -r dpdk-kmods/linux/igb_uio dpdk-23.03/kernel/linux/
cd dpdk-23.03/kernel/linux/igb_uio
vi meson.build
""""""""""
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017 Intel Corporation
mkfile = custom_target('igb_uio_makefile',
output: 'Makefile',
command: ['touch', '@OUTPUT@'])
kernel_dir = '/lib/modules/' + kernel_version
custom_target('igb_uio',
input: ['igb_uio.c', 'Kbuild'],
output: 'igb_uio.ko',
command: ['make', '-C', kernel_dir + '/build',
'M=' + meson.current_build_dir(),
'src=' + meson.current_source_dir(),
'EXTRA_CFLAGS=-I' + meson.current_source_dir() +
'/../../../lib/librte_eal/include',
'modules'],
depends: mkfile,
install: true,
install_dir: kernel_dir + '/extra/dpdk',
build_by_default: get_option('enable_kmods'))
""""""
cd .. (in meson.build -> dpdk-23.03/kernel/linux)
add the following:
subdirs = ['kni', 'igb_uio']
Download dpdk from official website
wget https://fast.dpdk.org/rel/dpdk-23.03.tar.xz tar xJf dpdk-23.03.tar.xz
Build dpdk and install using meson and ninja
cd dpdk-23.03 meson setup build cd build ninja ninja install ldconfig
Set hugepage
Set the hugepage to 1G and number of page as per your requirements (just increase 'nr_hugepages' if the eval mem allocation failed). more details: https://haryachyy.wordpress.com/2019/04/17/learning-dpdk-huge-pages/
There are three ways to do it as follows.
# 1. Using dpdk based python program /usr/local/bin/dpdk-hugepages.py -p 1G --setup 32G # 2. Insert 'nr_hugepages' echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages # 3. The permanent option (but I won't recommend this, this makes the the grub penic attack in rocky linux and goes to emergency mode) vi /etc/fstab nodev /mnt/huge hugetlbfs pagesize=1GB 0 0
Download pktgen-dpdk from official website (make sure, the version of pktgen-dpdk is higher than dpdk → it's very fragile)
export RTE_SDK=/root/dpdk-23.03 export RTE_TARGET=x86_64-native-linux-gcc wget https://git.dpdk.org/apps/pktgen-dpdk/snapshot/pktgen-dpdk-pktgen-23.06.1.tar.xz tar xJf pktgen-dpdk-pktgen-23.06.1.tar.xz
Following configuration can be applied if the expected bandwidth is not showing up for mellanox X5 (ref: https://stackoverflow.com/questions/72345569/why-does-dpdk-mellanox-connectx5-process-128b-packets-much-faster-than-other-s/72393527#72393527)
systemctl stop irqbalance.service systemctl disable irqbalance.service systemctl stop wpa_supplicant systemctl disable wpa_supplicant ./set_irq_affinity_cpulist.sh [non dpdk cores] [desired NIC] mlxconfig -d [pcie device id] set SRIOV_EN=0 mlx_tune -r ifconfig [NIC] txqueuelen 20000 ethtool -G [NIC] rx 8192 tx 8192 ethtool -A [NIC] rx off tx off mlxconfig -d [pcie address] set ZERO_TOUCH_TUNING_ENABLE=1 mlxconfig -d [pcie address] set CQE_COMPRESSION=1 mlxconfig -d [pcie address] s PCI_WR_ORDERING=1
Testing between two servers. We are using 8 virtual function in each server but not utilizing all of them.
Sender: /usr/local/bin/pktgen --huge-dir /dev/hugepages -l 1-10 -n 2 -- -P -m '[0:2-8].2-6' -T | Receiver: /usr/local/bin/pktgen --huge-dir /dev/hugepages -l 1-10 -n 2 -- -P -m '[2-5:6-10].0' -T -s 2:output.pcap |
Pktgen:/> set 2-6 src ip 10.10.11.0/24 Pktgen:/> set 2-6 dst ip 10.0.10.3 Pktgen:/> set 2-6 dst mac fe:63:23:fc:5e:36 Pktgen:/> set 2-6 size 512 Pktgen:/> start 2-6 |
|
Some helpful config in pkt-gen: https://stackoverflow.com/questions/73218563/dpdk-pktgen-how-to-change-rate-of-packet-transmission