SR-IOV with ConnectX-5

Proxmox 설치 후 Mellanox ConnectX-5 의 SR-IOV를 활성화 하는 법

SR-IOV 활성화

# apt install -y infiniband-diags ibutils rdma-core rdmacm-utils mstflint

# lshw -short -c net
H/W path         Device      Class          Description
=======================================================
/0/100/1/0       ens1f0      network        82599ES 10-Gigabit SFI/SFP+ Network Connection
/0/100/1/0.1     ens1f1      network        82599ES 10-Gigabit SFI/SFP+ Network Connection
/0/100/3.2/0     eno1        network        I350 Gigabit Network Connection
/0/100/3.2/0.1   enp4s0f1    network        I350 Gigabit Network Connection
/0/2/0           ens6np0     network        MT27800 Family [ConnectX-5]

# lspci | grep -i mellanox
81:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

# mstconfig -d 81:00.0 query
Device #1:
----------
Device type:    ConnectX5
Name:           MCX555A-ECA_Ax_Bx
Description:    ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE;...
Device:         81:00.0
Configurations:                              Next Boot
         NUM_OF_VFS                          0
         NUM_OF_PF                           1
         SRIOV_EN                            False(0)

# mstconfig -d 81:00.0 set SRIOV_EN=1 NUM_OF_VFS=4

# reboot

VF 생성 서비스 등록

/etc/systemd/system/sriov-vf.service

[Unit]
Description=Script to enable SR-IOV on boot
Before=networking.service
RequiredBy=networking.service

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c 'echo 4 > /sys/class/net/ens6np0/device/sriov_numvfs'

[Install]
WantedBy=multi-user.target

# systemctl enable --now sriov-vf.service

VF 할당

VF의 PCI 주소를 확인하고 VM에 할당하는 방법입니다.

# lshw -short -c net
H/W path         Device      Class          Description
=======================================================
/0/2/0           ens6np0     network        MT27800 Family [ConnectX-5]
/0/2/0.1         ens6v0      network        MT27800 Family [ConnectX-5 Virtual Function]
/0/2/0.2         ens6v1      network        MT27800 Family [ConnectX-5 Virtual Function]
/0/2/0.3         ens6v2      network        MT27800 Family [ConnectX-5 Virtual Function]
/0/2/0.4         ens6v3      network        MT27800 Family [ConnectX-5 Virtual Function]
...

# ls -al /sys/class/net
ens6np0  -> ../../devices/pci0000:80/0000:80:02.0/0000:81:00.0/net/ens6np0
ens6v0   -> ../../devices/pci0000:80/0000:80:02.0/0000:81:00.1/net/ens6v0
ens6v1   -> ../../devices/pci0000:80/0000:80:02.0/0000:81:00.2/net/ens6v1
ens6v2   -> ../../devices/pci0000:80/0000:80:02.0/0000:81:00.3/net/ens6v2
ens6v3   -> ../../devices/pci0000:80/0000:80:02.0/0000:81:00.4/net/ens6v3
...

ens6v1 는 0000:81:00.2 가 PCI 주소입니다.

다음과 같이 PCI 장치를 선택해 pass through 로 장착합니다.

Proxmox 웹 로그인 → 특정 VM 선택 → Hardware → Add → PCI Device → 장치 선택

파일을 직접 수정해도 됩니다.

/etc/pve/nodes/<prox-hostname>/qemu-server/<vm-id>.conf

...
hostpci0: 0000:81:00.2,rombar=0
...