RHOSP13 가이드대로 SR-IOV VFs 템플릿 구성하여 배포하였으나 VF구성이 되지 않았습니다.
그래서 어찌어찌 찾다보니 TripleO 사이트에서 가이드하는 대로 템플릿을 구성하여 성공하였고, 그래서 TripleO 가이드방법을 통한 SR-IOV VF 자동 배포방법을 정리하였습니다.
## RHOSP13 SR-IOV 가이드
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/network_functions_virtualization_planning_and_configuration_guide/part-sriov-nfv-configuration
## TripleO 가이드 (SR-IOV 및 VF 생성방법)
https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/sriov_deployment.html
1. RHOSP13 가이드 형태로 compute 노드의 extra 부분에 SR-IOV VF 구성 후 배포
ComputeDpdkParameters:
KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=180 intel_iommu=on intel_iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 nmi_watchdog=0 transparent_hugepage=never intel_idle.max_cstate=0 processor.max_cstate=1 idle=mwait nohpet nosoftlockup isolcpus=4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70"
TunedProfileName: "cpu-partitioning"
IsolCpusList: "4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70"
NovaVcpuPinSet: ['4','6','8','10','12','14','16','18','20','22','24','26','28','30','32','34','40','42','44','46','48','50','52','54','56','58','60','62','64','66','68','70']
NovaComputeCpuSharedSet: "0,2,36,38,1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71"
NovaReservedHostMemory: 16384
NovaLibvirtRxQueueSize: 1024
NovaLibvirtTxQueueSize: 1024
NeutronBridgeMappings:
- physnet-az2-dell-mgmt0:br-mgmt0
- physnet-az2-dell-svc0:br-service0
ExtraSysctlSettings:
kernel.sysrq:
value: 1
net.netfilter.nf_conntrack_max:
value: 1000000
net.nf_conntrack_max:
value: 1000000
net.ipv6.conf.all.disable_ipv6:
value: 1
NeutronPhysicalDevMappings: "physnet-az2-svc-sriov1:p7p1,physnet-az2-svc-sriov2:p7p2"
NeutronSriovNumVFs:
- "p7p1:4"
- "p7p2:4"
NovaPCIPassthrough:
- devname: "p7p1"
physical_network: "physnet-az2-svc-sriov1"
- devname: "p7p2"
physical_network: "physnet-az2-svc-sriov2"
- vendor_id: "8086"
product_id: "158b"
address: "0000:86:00.0"
- vendor_id: "8086"
product_id: "158b"
address: "0000:86:00.1"
결과 : VF 관련 설정 적용이 된 듯한데, 실제 p7p1 및 p7p2 NIC에 VF가 생성되지 않음 (호스트 재기동해도 VF 생성안됨)
[root@rhosp-comp-dell-2-1 sysconfig]# cat allocate_vfs
#!/bin/bash
[ "p7p1" == "$1" ] && echo 4 > /sys/class/net/p7p1/device/sriov_numvfs
[ "p7p2" == "$1" ] && echo 4 > /sys/class/net/p7p2/device/sriov_numvfs
[root@rhosp-comp-dell-2-1 rules.d]# cat 70-tripleo-reset-sriov.rules
KERNEL=="p7p1", RUN+="/etc/sysconfig/allocate_vfs %k"
KERNEL=="p7p2", RUN+="/etc/sysconfig/allocate_vfs %k"
[root@rhosp-comp-dell-2-1 device]# cat sriov_numvfs
0
2. 위 1번 설정 기반에 compute.yaml에 내용을 tripleo 가이드 형태로 Interface에 vf 설정후 배포
테스트를 위해 p7p1에 대한 설정만 진행함
- type: sriov_pf
name: p7p1
mtu: 1500
numvfs: 4
use_dhcp: false
defroute: false
nm_controlled: true
promisc: false
결과 : 성공
수동 설정이 아닌 배포를 통해서 해당 NIC에 자동으로 VF 설정을 할 수 있음
6: p7p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 40:a6:b7:0a:60:80 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
24: p7p1_0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 6e:67:ba:3e:2a:5a brd ff:ff:ff:ff:ff:ff
25: p7p1_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether fa:df:e4:fa:97:c8 brd ff:ff:ff:ff:ff:ff
26: p7p1_2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 5a:ed:93:2c:97:85 brd ff:ff:ff:ff:ff:ff
27: p7p1_3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 56:d3:ac:25:a8:55 brd ff:ff:ff:ff:ff:ff
[root@rhosp-comp-dell-2-1 device]# cat sriov_numvfs
4
3. SR-IOV 네트워크 생성 및 Port 생성 샘플
## SR-IOV 네트워크 생성
openstack network create sriov_101 --provider-physical-network physnet-az2-svc-sriov1 --provider-network-type vlan --provider-segment 101
## SR-IOV Subnet 생성
openstack subnet create --subnet-range 10.0.101.0/24 --network sriov_101 sriov_101_subnet --no-dhcp
## SR-IOV Port 생성
openstack port create --network sriov_101 --fixed-ip subnet=sriov_101_subnet,ip-address=10.0.101.150 --vnic-type direct sriov_port_1
## SR-IOV VM 생성
openstack server create --flavor hk-sriov --image rhel7 --port sriov_port_1 --availability-zone AZ-2-COMP-DELL:rhosp-comp-dell-2.test.net sriov