Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
.. _sriov:
=====
SRIOV
=====
Neutron SRIOV
~~~~~~~~~~~~~
Preparation and deployment
--------------------------
SRIOV requires specific NIC and BIOS configuration and is not supported on all
platforms. Consult NIC and platform specific documentation for instructions
on enablement.
Modify the ``/etc/kolla/globals.yml`` file as the following example shows:
.. code-block:: yaml
enable_neutron_sriov: "yes"
Modify the ``/etc/kolla/config/neutron/ml2_conf.ini`` file and add
``sriovnicswitch`` to the ``mechanism_drivers``. Also, the provider
networks used by SRIOV should be configured. Both flat and VLAN are configured
with the same physical network name in this example:
.. path /etc/kolla/config/neutron/ml2_conf.ini
.. code-block:: ini
[ml2]
mechanism_drivers = openvswitch,l2population,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = sriovtenant1:1000:1009
[ml2_type_flat]
flat_networks = sriovtenant1
Add ``PciPassthroughFilter`` to scheduler_default_filters
The ``PciPassthroughFilter``, which is required by Nova Scheduler service
on the Controller, should be added to ``scheduler_default_filters``
Modify the ``/etc/kolla/config/nova.conf`` file and add
``PciPassthroughFilter`` to ``scheduler_default_filters``. this filter is
required by The Nova Scheduler service on the controller node.
.. path /etc/kolla/config/nova.conf
.. code-block:: ini
[DEFAULT]
scheduler_default_filters = <existing filters>, PciPassthroughFilter
scheduler_available_filters = nova.scheduler.filters.all_filters
Edit the ``/etc/kolla/config/nova.conf`` file and add PCI device whitelisting.
this is needed by OpenStack Compute service(s) on the Compute.
.. path /etc/kolla/config/nova.conf
.. code-block:: ini
[pci]
passthrough_whitelist = [{"devname": "ens785f0", "physical_network": "sriovtenant1"}]
Modify the ``/etc/kolla/config/neutron/sriov_agent.ini`` file. Add physical
network to interface mapping. Specific VFs can also be excluded here. Leaving
blank means to enable all VFs for the interface:
.. path /etc/kolla/config/neutron/sriov_agent.ini
.. code-block:: ini
[sriov_nic]
physical_device_mappings = sriovtenant1:ens785f0
exclude_devices =
Run deployment.
Verification
------------
Check that VFs were created on the compute node(s). VFs will appear in the
output of both ``lspci`` and ``ip link show``. For example:
.. code-block:: console
# lspci | grep net
05:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
# ip -d link show ens785f0
4: ens785f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
link/ether 90:e2:ba:ba:fb:20 brd ff:ff:ff:ff:ff:ff promiscuity 1
openvswitch_slave addrgenmode eui64
vf 0 MAC 52:54:00:36:57:e0, spoof checking on, link-state auto, trust off
vf 1 MAC 52:54:00:00:62:db, spoof checking on, link-state auto, trust off
vf 2 MAC fa:16:3e:92:cf:12, spoof checking on, link-state auto, trust off
vf 3 MAC fa:16:3e:00:a3:01, vlan 1000, spoof checking on, link-state auto, trust off
Verify the SRIOV Agent container is running on the compute node(s):
.. code-block:: console
# docker ps --filter name=neutron_sriov_agent
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b03a8f4c0b80 10.10.10.10:4000/registry/centos-source-neutron-sriov-agent:17.04.0 "kolla_start" 18 minutes ago Up 18 minutes neutron_sriov_agent
Verify the SRIOV Agent service is present and UP:
.. code-block:: console
# openstack network agent list
+--------------------------------------+--------------------+-------------+-------------------+-------+-------+---------------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+--------------------+-------------+-------------------+-------+-------+---------------------------+
| 7c06bda9-7b87-487e-a645-cc6c289d9082 | NIC Switch agent | av09-18-wcp | None | :-) | UP | neutron-sriov-nic-agent |
+--------------------------------------+--------------------+-------------+-------------------+-------+-------+---------------------------+
Create a new provider network. Set ``provider-physical-network`` to the
physical network name that was configured in ``/etc/kolla/config/nova.conf``.
Set ``provider-network-type`` to the desired type. If using VLAN, ensure
``provider-segment`` is set to the correct VLAN ID. This example uses ``VLAN``
network type:
.. code-block:: console
# openstack network create --project=admin \
--provider-network-type=vlan \
--provider-physical-network=sriovtenant1 \
--provider-segment=1000 \
sriovnet1
Create a subnet with a DHCP range for the provider network:
.. code-block:: console
# openstack subnet create --network=sriovnet1 \
--subnet-range=11.0.0.0/24 \
--allocation-pool start=11.0.0.5,end=11.0.0.100 \
sriovnet1_sub1
Create a port on the provider network with ``vnic_type`` set to ``direct``:
.. code-block:: console
# openstack port create --network sriovnet1 --vnic-type=direct sriovnet1-port1
Start a new instance with the SRIOV port assigned:
.. code-block:: console
# openstack server create --flavor flavor1 \
--image fc-26 \
--nic port-id=`openstack port list | grep sriovnet1-port1 | awk '{print $2}'` \
vm1
Verify the instance boots with the SRIOV port. Verify VF assignment by running
dmesg on the compute node where the instance was placed.
.. code-block:: console
# dmesg
[ 2896.849970] ixgbe 0000:05:00.0: setting MAC fa:16:3e:00:a3:01 on VF 3
[ 2896.850028] ixgbe 0000:05:00.0: Setting VLAN 1000, QOS 0x0 on VF 3
[ 2897.403367] vfio-pci 0000:05:10.4: enabling device (0000 -> 0002)
For more information see `OpenStack SRIOV documentation <https://docs.openstack.org/neutron/pike/admin/config-sriov.html>`_.
Nova SRIOV
~~~~~~~~~~
Preparation and deployment
--------------------------
Nova provides a separate mechanism to attach PCI devices to instances that
is independent from Neutron. Using the PCI alias configuration option in
nova.conf, any PCI device (PF or VF) that supports passthrough can be attached
to an instance. One major drawback to be aware of when using this method is
that the PCI alias option uses a device's product id and vendor id only,
so in environments that have NICs with multiple ports configured for SRIOV,
it is impossible to specify a specific NIC port to pull VFs from.
Modify the file ``/etc/kolla/config/nova.conf``. The Nova Scheduler service
on the control node requires the ``PciPassthroughFilter`` to be added to the
list of filters and the Nova Compute service(s) on the compute node(s) need
PCI device whitelisting. The Nova API service on the control node and the Nova
Compute service on the compute node also require the ``alias`` option under the
``[pci]`` section. The alias can be configured as 'type-VF' to pass VFs or
'type-PF' to pass the PF. Type-VF is shown in this example:
.. path /etc/kolla/config/nova.conf
.. code-block:: ini
[DEFAULT]
scheduler_default_filters = <existing filters>, PciPassthroughFilter
scheduler_available_filters = nova.scheduler.filters.all_filters
[pci]
passthrough_whitelist = [{"vendor_id": "8086", "product_id": "10fb"}]
alias = [{"vendor_id":"8086", "product_id":"10ed", "device_type":"type-VF", "name":"vf1"}]
Run deployment.
Verification
------------
Create (or use an existing) flavor, and then configure it to request one PCI
device from the PCI alias:
.. code-block:: console
# openstack flavor set sriov-flavor --property "pci_passthrough:alias"="vf1:1"
Start a new instance using the flavor:
.. code-block:: console
# openstack server create --flavor sriov-flavor --image fc-26 vm2
Verify VF devices were created and the instance starts successfully as in
the Neutron SRIOV case.
For more information see `OpenStack PCI passthrough documentation <https://docs.openstack.org/nova/pike/admin/pci-passthrough.html>`_.