멜라녹스(Mellanox) NIC 의 SR-IOV 활성화
Mellanox NIC 의 경우 기본값이 Infiniband 로 설정되어 있다.
| 코드 블럭 |
|---|
| language | bash |
|---|
| theme | Emacs |
|---|
| linenumbers | true |
|---|
|
# ifconfig -a | grep -i UP
eno1np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
eno2np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
ib0: flags=4099<UP,BROADCAST,MULTICAST> mtu 4092
ib1: flags=4099<UP,BROADCAST,MULTICAST> mtu 4092
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
# lspci -v | grep -i mel
81:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
81:00.1 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6] |
OS에 맞는 Mellanox Firmware Tool 을 공식 홈페이지에서 다운로드 하여 설치한다.
...
| 코드 블럭 |
|---|
| language | bash |
|---|
| theme | Emacs |
|---|
| linenumbers | true |
|---|
|
wget https://www.mellanox.com/downloads/MFT/mft-4.17.0-106-x86_64-rpm.tgz
tar zxvf mft-4.17.0-106-x86_64-rpm.tgz
cd mft-4.17.0-106-x86_64-rpm
./install.sh |
LINE 1 : MFT 도구 실행 / LINE 2 : 장비 타입 변경 (infiniband → ethernet) / LINE 3~4 : SR-IOV 설정 / LINE 5 : 설정 확인
...
| 코드 블럭 |
|---|
| language | bash |
|---|
| theme | Emacs |
|---|
| linenumbers | true |
|---|
|
# mst start
# ls /dev/mst
mt4123_pciconf0 mt4123_pciconf0.1
# mlxconfig -d /dev/mst/mt4123_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
# mlxconfig -d /dev/mst/mt4123_pciconf0 set SRIOV_EN=1
# mlxconfig -d /dev/mst/mt4123_pciconf0 set NUM_OF_VFS=16
# mlxconfig -d /dev/mst/mt4123_pciconf0 query | grep -E 'TYPE|SRIOV|VFS'
NUM_OF_VFS 16
SRIOV_EN True(1)
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
SRIOV_IB_ROUTING_MODE_P1 LID(1)
SRIOV_IB_ROUTING_MODE_P2 LID(1)
# reboot |
| 코드 블럭 |
|---|
| language | bash |
|---|
| theme | Emacs |
|---|
| linenumbers | true |
|---|
|
ifconfig -a | grep -i UP
lspci -v | grep -i mel
lspci -v | grep -i mel -A 30
cat /sys/class/net/enp129s0f0/device/sriov_totalvfs |
LINE 24 에서 SR-IOV 기능이 활성화 된 것을 확인할 수 있다.
LINE 29-30 에서 멜라녹스의 커널 드라이버 모듈이 bnxt_en 임을 확인할 수 있다.
LINE 34-35 에서 VF (Virtual Function) 개수가 위에서 NUM_OF_VFS=16 으로 설정한 대로 설정되었다.
| 코드 블럭 |
|---|
| language | bash |
|---|
| theme | Emacs |
|---|
| linenumbers | true |
|---|
|
# ifconfig -a | grep -i UP
eno1np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
eno2np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000...
enp129s0f0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
enp129s0f1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
# lspci -v | grep -i mel
81:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
81:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
# lspci -v | grep -i mel -A 30
81:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
Subsystem: Mellanox Technologies Device 0008
Flags: bus master, fast devsel, latency 0, IRQ 701, IOMMU group 13
Memory at 2001a000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at f0100000 [disabled] [size=1M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1c0] Secondary PCI Express
Capabilities: [230] Access Control Services
Capabilities: [320] Lane Margining at the Receiver <?>
Capabilities: [370] Physical Layer 16.0 GT/s <?>
Capabilities: [420] Data Link Feature <?>
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
...
# cat /sys/class/net/enp129s0f0/device/sriov_totalvfs
16 |
...