enricorossi.org

Enrico Rossi

Intel sr-iov on Debian Stretch

Posted on in blog • 450 words • 3 minute read
Last updated on

Wrap-up about the sr-iov, some passage may be forgotten.

Expose the SR-IOV virtual network interfaces 1

Search for the pci interfaces (here the server has a 4 port intel network card).

root@srv:~# find /sys -name sriov_numvfs

/sys/devices/pci0000:00/0000:00:1c.0/0000:06:00.2/sriov_numvfs
/sys/devices/pci0000:00/0000:00:1c.0/0000:06:00.0/sriov_numvfs
/sys/devices/pci0000:00/0000:00:1c.0/0000:06:00.3/sriov_numvfs
/sys/devices/pci0000:00/0000:00:1c.0/0000:06:00.1/sriov_numvfs

find maximum numbers of VFs

root@srv:~# cat /sys/devices/pci0000:00/0000:00:1c.0/0000:06:00.0/sriov_totalvfs
7

expose 7 virtual interface on the 1st ethernet card and check the result.

root@srv:~# echo 7 > /sys/devices/pci0000:00/0000:00:1c.0/0000:06:00.0/sriov_numvfs
root@srv:~# ip link show
[...]
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
[...]

note: the old way of loading the igb modules with options is now depracated

Make it permanently after boot

How devices are handled is managed by udev. First discover some info about the network card needed later.

root@srv:~# lspci -s 06:00.0 -n
06:00.0 0200: 8086:1521 (rev 01)

Create the file:

/etc/udev/rules.d/99-sriov.rules

# Activare the SR-IOV VF on all interfaces of the ethernet card

KERNEL=="0000:06:00.[0-9]", SUBSYSTEM=="pci", DRIVER=="igb", ATTR{vendor}=="0x8086", ATTR{device}=="0x1521", ATTR{sriov_numvfs}="7"

Play with the interfaces

Suppose you want to create some interfaces to be used on the server with VLANs.

# setup the mac addresses and vlan tag
# ip link set <pf> vf <vf_index> mac <mac-id>
ip link set eno1 up
ip link set eno1 vf 0 mac 52:54:00:4d:8b:bc
ip link set eno1 vf 0 vlan 100
ip link set eno1 vf 1 mac 52:54:00:5e:3b:e2
ip link set eno1 vf 1 vlan 200

note: This is NOT the way you should use the SR-IOV with libvirt.

Activate the iommu for libvirt

To use all this in libvirt or VMs, the iommu have to be active.

/etc/default/grub

GRUB_CMDLINE_LINUX="intel_iommu=on"

and reboot!

and finally about libvirt 2

When the sr-iov is present, the recommended setup is to pass all the VFs as a network pool in libvirt config, then let each VM’s configuration setup MAC address and (eventually) the VLAN tag. In this way VMs can be moved or migrated without any particular network setup on each host. When started the VM pick a virtual network interface from the pool and configure it.

## Network Pool
#
# SR-IOV
# https://www.suse.com/documentation/sles-12/book_virt/data/sec_libvirt_config_io.html
#
# create a file

<network>
  <name>sr-iov</name>
  <forward mode='hostdev' managed='yes'>
    <pf dev='eno1'/>
  </forward>
</network>

# then define it

virsh # net-define virsh_net_define.xml
Network sr-iov defined from virsh_net_define.xml

virsh # net-autostart sr-iov
Network sr-iov marked as autostarted

virsh # net-start sr-iov
Network sr-iov started

virsh # net-list 
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 sr-iov               active     yes           yes

  1. https://communities.intel.com/thread/53409 ↩︎

  2. https://www.suse.com/documentation/sles-12/book_virt/data/sec_libvirt_config_io.html ↩︎