반응형
약 2년전에 구축했던 RHOSP 16.1 배포에 대한 템플릿 내용입니다.
stack 계정을 만드는 등.. 설치 방법은 레드햇 사이트 등등에 많이 공유 되어있 있어 따로 기재하진 않았고 사용했던 template 정보만 공유합니다.
환경 정보 :
1. Director VM x1 , Controller VM x3, Compute VM x2, RHEL Repo VM x 1 on KVM-Host (DPDK 미 구축)
2. OS Version: RHEL 8.2
3. RHEL Repository : Offline / Container Image : Online 을 통해 배포
4. NFS를 통한 Glance Image, Volume, Volume-backup 구성 (RHEL Repo에 NFS-Server 구성)
5. Single Stack 구성
6. KVM-Host에 VBMC 사용
아래 github에서 다운 받을 수 있습니다.
https://github.com/hkjeon/rhosp16.1
1. Undercloud templates
1.1 undercloud.conf
undercloud 설치에 필요한 config 파일 내용입니다.
[DEFAULT]
local_interface = ens3
local_ip = 55.55.157.170/24
subnets = AZ-0-OSC-CP0
local_subnet = AZ-0-OSC-CP0
generate_service_certificate = false
enable_tempest = false
enable_telemetry = false
enable_ui = true
enable_validation = true
enable_routed_networks = true
container_images_file = containers-prepare-parameter.yaml
undercloud_admin_host = 55.55.157.170
undercloud_debug = true
undercloud_hostname = hk-director.hk.com
undercloud_ntp_servers = 55.55.157.4
undercloud_public_host = 55.55.152.170
[AZ-0-OSC-CP0]
cidr = 55.55.157.0/24
dhcp_start = 55.55.157.171
dhcp_end = 55.55.157.179
gateway = 55.55.157.1
inspection_iprange = 55.55.157.161,55.55.157.168
masquerade = false
1.2 containers-prepare-parameter.yaml
RHOSP16 (train) 부터 undercloud 및 overcloud에서 모두 container를 사용합니다.
container 준비를 위한 registry 주소, tag 정보, Registry 접근을 위한 RedHat 계정 정보등을 구성합니다.
# Generated with the following on 2021-01-26T15:35:47.981831
#
# openstack tripleo container image prepare default --local-push-destination --output-env-file containers-prepare-parameter.yaml
#
parameter_defaults:
ContainerImageRegistryLogin: true
ContainerImagePrepare:
- push_destination: true
set:
ceph_alertmanager_image: ose-prometheus-alertmanager
ceph_alertmanager_namespace: registry.redhat.io/openshift4
ceph_alertmanager_tag: 4.1
ceph_grafana_image: rhceph-4-dashboard-rhel8
ceph_grafana_namespace: registry.redhat.io/rhceph
ceph_grafana_tag: 4
ceph_image: rhceph-4-rhel8
ceph_namespace: registry.redhat.io/rhceph
ceph_node_exporter_image: ose-prometheus-node-exporter
ceph_node_exporter_namespace: registry.redhat.io/openshift4
ceph_node_exporter_tag: v4.1
ceph_prometheus_image: ose-prometheus
ceph_prometheus_namespace: registry.redhat.io/openshift4
ceph_prometheus_tag: 4.1
ceph_tag: latest
name_prefix: openstack-
name_suffix: ''
namespace: registry.redhat.io/rhosp-rhel8
neutron_driver: ovs
rhel_containers: false
tag: '16.1'
tag_from_label: '{version}-{release}'
ContainerImageRegistryCredentials:
registry.redhat.io:
{redhat ID}: "{redhat password}"
1.3 /home/stack/instackenv/instack.yaml
Baremetal node 등록에 필요한 Openstack Controller의 HW 정보를 구성합니다.
nodes:
- "name": "hk-osc-1"
"pm_addr": "55.55.157.4"
"mac": ["52:54:00:aa:8c:5a"]
"pm_type": "ipmi"
"pm_user": "admin"
"pm_password": "hpinvent"
"pm_port": "6451"
"capabilities": "node:controller-0,boot_option:local"
"root_device": {
"name":"/dev/vda"
}
- "name": "hk-osc-2"
"pm_addr": "55.55.157.4"
"mac": ["52:54:00:de:24:a1"]
"pm_type": "ipmi"
"pm_user": "admin"
"pm_password": "hpinvent"
"pm_port": "6452"
"capabilities": "node:controller-1,boot_option:local"
"root_device": {
"name":"/dev/vda"
}
- "name": "hk-osc-3"
"pm_addr": "55.55.157.4"
"mac": ["52:54:00:b8:d3:2b"]
"pm_type": "ipmi"
"pm_user": "admin"
"pm_password": "hpinvent"
"pm_port": "6453"
"capabilities": "node:controller-2,boot_option:local"
"root_device": {
"name":"/dev/vda"
}
1.4 /home/stack/instackenv/instack-ovs.yaml
Baremetal node 등록에 필요한 Openstack Compute노드의 HW 정보를 구성합니다.
nodes:
- "name": "hk-comp-1"
"pm_addr": "55.55.157.4"
"mac": ["52:54:00:8c:bb:a0"]
"pm_type": "ipmi"
"pm_user": "admin"
"pm_password": "hpinvent"
"pm_port": "6454"
"capabilities": "node:compute-ovs-0,boot_option:local"
"root_device": {
"name":"/dev/vda"
}
- "name": "hk-comp-2"
"pm_addr": "55.55.157.4"
"mac": ["52:54:00:e0:b8:8e"]
"pm_type": "ipmi"
"pm_user": "admin"
"pm_password": "hpinvent"
"pm_port": "6455"
"capabilities": "node:compute-ovs-1,boot_option:local"
"root_device": {
"name":"/dev/vda"
}
- "name": "hk-comp-3"
"pm_addr": "55.55.157.4"
"mac": ["52:54:00:ec:e1:c9"]
"pm_type": "ipmi"
"pm_user": "admin"
"pm_password": "hpinvent"
"pm_port": "6456"
"capabilities": "node:compute-ovs-2,boot_option:local"
"root_device": {
"name":"/dev/vda"
}
2. Overcloud templates
2.1 network_data.yaml
아래 파일들은 /home/stack/templates/ 에 파일 내용입니다.
# --------------------------------------------------------------- #
- name: InternalApi
name_lower: internal_api
vip: true
ip_subnet: '55.55.155.0/24'
allocation_pools: [{'start': '55.55.155.181', 'end': '55.55.155.189'}]
# --------------------------------------------------------------- #
- name: External
vip: true
name_lower: external
ip_subnet: '55.55.152.0/24'
allocation_pools: [{'start': '55.55.152.181', 'end': '55.55.152.189'}]
gateway_ip: '55.55.152.1'
2.2 node-info.yaml
parameter_defaults:
OvercloudControllerFlavor: baremetal
ControllerCount: 3
ControllerSchedulerHints:
'capabilities:node': 'controller-%index%'
# --------------------------------------------------------------- #
OvercloudComputeFlavor: baremetal
ComputeOvsCount: 3
ComputeOvsSchedulerHints:
'capabilities:node': 'compute-ovs-%index%'
# --------------------------------------------------------------- #
OvercloudComputeDpdkFlavor: baremetal
ComputeDpdkCount: 0
ComputeDpdkSchedulerHints:
'capabilities:node': 'compute-dpdk-%index%'
# --------------------------------------------------------------- #
HostnameMap:
overcloud-controller-0: hk-osc-1
overcloud-controller-1: hk-osc-2
overcloud-controller-2: hk-osc-3
#-----------------------------------------------------------#
# Leaf0 #
#-----------------------------------------------------------#
overcloud-compute-ovs-0: hk-comp-1
overcloud-compute-ovs-1: hk-comp-2
overcloud-compute-ovs-2: hk-comp-3
overcloud-compute-dpdk-0: hk-dpdk-1
2.3 roles-data.yaml
###############################################################################
# Role: Controller #
###############################################################################
- name: Controller
description: |
Controller role that has all the controler services loaded and handles
Database, Messaging and Network functions.
CountDefault: 3
tags:
- primary
- controller
networks:
External:
subnet: external_subnet
InternalApi:
subnet: internal_api_subnet
HostnameFormatDefault: '%stackname%-controller-%index%'
uses_deprecated_params: True
deprecated_param_extraconfig: 'controllerExtraConfig'
deprecated_param_flavor: 'OvercloudControlFlavor'
deprecated_param_image: 'controllerImage'
deprecated_nic_config_name: 'controller.yaml'
ServicesDefault:
- OS::TripleO::Services::Aide
- OS::TripleO::Services::AuditD
- OS::TripleO::Services::BootParams
- OS::TripleO::Services::CACerts
- OS::TripleO::Services::CertmongerUser
- OS::TripleO::Services::CinderApi
- OS::TripleO::Services::CinderBackup
- OS::TripleO::Services::CinderScheduler
- OS::TripleO::Services::CinderVolume
- OS::TripleO::Services::Clustercheck
- OS::TripleO::Services::ContainerImagePrepare
- OS::TripleO::Services::Docker
- OS::TripleO::Services::Ec2Api
- OS::TripleO::Services::Etcd
- OS::TripleO::Services::GlanceApi
- OS::TripleO::Services::HAproxy
- OS::TripleO::Services::HeatApi
- OS::TripleO::Services::HeatApiCloudwatch
- OS::TripleO::Services::HeatApiCfn
- OS::TripleO::Services::HeatEngine
- OS::TripleO::Services::Horizon
- OS::TripleO::Services::Ipsec
- OS::TripleO::Services::Iscsid
- OS::TripleO::Services::Keepalived
- OS::TripleO::Services::Kernel
- OS::TripleO::Services::Keystone
- OS::TripleO::Services::LoginDefs
- OS::TripleO::Services::Memcached
- OS::TripleO::Services::MetricsQdr
- OS::TripleO::Services::MySQL
- OS::TripleO::Services::MySQLClient
- OS::TripleO::Services::NeutronApi
- OS::TripleO::Services::NeutronCorePlugin
# - OS::TripleO::Services::NeutronOvsAgent
- OS::TripleO::Services::NovaApi
- OS::TripleO::Services::NovaConductor
- OS::TripleO::Services::NovaIronic
- OS::TripleO::Services::NovaMetadata
- OS::TripleO::Services::NovaScheduler
- OS::TripleO::Services::NovaVncProxy
- OS::TripleO::Services::ContainersLogrotateCrond
- OS::TripleO::Services::OpenStackClients
- OS::TripleO::Services::Pacemaker
- OS::TripleO::Services::PlacementApi
- OS::TripleO::Services::OsloMessagingRpc
- OS::TripleO::Services::OsloMessagingNotify
- OS::TripleO::Services::Podman
- OS::TripleO::Services::Rear
- OS::TripleO::Services::Redis
- OS::TripleO::Services::Rsyslog
- OS::TripleO::Services::RsyslogSidecar
- OS::TripleO::Services::Securetty
- OS::TripleO::Services::Snmp
- OS::TripleO::Services::Sshd
- OS::TripleO::Services::Timezone
- OS::TripleO::Services::TripleoFirewall
- OS::TripleO::Services::TripleoPackages
- OS::TripleO::Services::Tuned
###############################################################################
# Role: ComputeOvs #
###############################################################################
- name: ComputeOvs
description: |
Compute OvS Role
CountDefault: 0
networks:
InternalApi:
subnet: internal_api_subnet
HostnameFormatDefault: '%stackname%-compute-ovs-%index%'
RoleParametersDefault:
TunedProfileName: "virtual-guest"
deprecated_nic_config_name: 'compute-ovs.yaml'
ServicesDefault:
- OS::TripleO::Services::Aide
- OS::TripleO::Services::AuditD
- OS::TripleO::Services::CACerts
- OS::TripleO::Services::CertmongerUser
- OS::TripleO::Services::ComputeNeutronCorePlugin
- OS::TripleO::Services::ComputeNeutronL3Agent
- OS::TripleO::Services::ComputeNeutronMetadataAgent
- OS::TripleO::Services::ComputeNeutronOvsAgent
# - OS::TripleO::Services::ComputeNeutronOvsDpdk
- OS::TripleO::Services::Docker
- OS::TripleO::Services::Iscsid
- OS::TripleO::Services::Kernel
- OS::TripleO::Services::LoginDefs
- OS::TripleO::Services::MySQLClient
- OS::TripleO::Services::NeutronDhcpAgent
- OS::TripleO::Services::NeutronMetadataAgent
- OS::TripleO::Services::NovaAZConfig
- OS::TripleO::Services::NovaCompute
- OS::TripleO::Services::NovaLibvirt
- OS::TripleO::Services::NovaMigrationTarget
- OS::TripleO::Services::ContainersLogrotateCrond
- OS::TripleO::Services::OvsDpdkNetcontrold
- OS::TripleO::Services::Podman
- OS::TripleO::Services::Rear
- OS::TripleO::Services::Securetty
- OS::TripleO::Services::Snmp
- OS::TripleO::Services::Sshd
- OS::TripleO::Services::Timezone
- OS::TripleO::Services::TripleoFirewall
- OS::TripleO::Services::TripleoPackages
- OS::TripleO::Services::Tuned
###############################################################################
# Role: ComputeDpdk #
###############################################################################
- name: ComputeDpdk
description: |
Compute Dpdk Role
CountDefault: 0
networks:
InternalApi:
subnet: internal_api_subnet
HostnameFormatDefault: '%stackname%-compute-dpdk-%index%'
RoleParametersDefault:
TunedProfileName: "cpu-partitioning"
deprecated_nic_config_name: 'compute-dpdk.yaml'
ServicesDefault:
- OS::TripleO::Services::Aide
- OS::TripleO::Services::AuditD
- OS::TripleO::Services::CACerts
- OS::TripleO::Services::CertmongerUser
- OS::TripleO::Services::ComputeNeutronCorePlugin
- OS::TripleO::Services::ComputeNeutronL3Agent
- OS::TripleO::Services::ComputeNeutronMetadataAgent
#- OS::TripleO::Services::ComputeNeutronOvsAgent
- OS::TripleO::Services::ComputeNeutronOvsDpdk
- OS::TripleO::Services::Docker
- OS::TripleO::Services::Iscsid
- OS::TripleO::Services::Kernel
- OS::TripleO::Services::LoginDefs
- OS::TripleO::Services::MySQLClient
- OS::TripleO::Services::NeutronDhcpAgent
- OS::TripleO::Services::NeutronMetadataAgent
- OS::TripleO::Services::NovaAZConfig
- OS::TripleO::Services::NovaCompute
- OS::TripleO::Services::NovaLibvirt
- OS::TripleO::Services::NovaMigrationTarget
- OS::TripleO::Services::ContainersLogrotateCrond
- OS::TripleO::Services::OvsDpdkNetcontrold
- OS::TripleO::Services::Podman
- OS::TripleO::Services::Rear
- OS::TripleO::Services::Securetty
- OS::TripleO::Services::Snmp
- OS::TripleO::Services::Sshd
- OS::TripleO::Services::Timezone
- OS::TripleO::Services::TripleoFirewall
- OS::TripleO::Services::TripleoPackages
- OS::TripleO::Services::Tuned
3. Overcloud templates
3.1 1-commons-parameters.yaml
아래 파일들은 /home/stack/templates/environment/ 에 파일 내용입니다.
parameter_defaults:
# TimeZone for the environment
TimeZone: "Asia/Seoul"
# DNS and NTP Server
DnsServers: ["8.8.8.8"]
NtpServers: ["55.55.157.4"]
# Public API Domain Name
CloudDomain: hk.com
CloudName: overcloud.hk.com
CloudNameCtlplane: overcloud.ctlplane.hk.com
CloudNameInternal: overcloud.internalapi.hk.com
#CloudNameStorage: overcloud.storage.hk.com
#CloudNameStorageManagement: overcloud.storagemgmt.hk.com
# Admin Keystone user password
AdminPassword: 'test1234'
# Custom SSH Config to allow password and Root login
SshServerOptions:
UseDNS: 'yes'
PasswordAuthentication: 'yes'
PermitRootLogin: 'no'
PermitEmptyPasswords: 'no'
3.2 10-endpoint.yaml
parameter_defaults:
ServiceNetMap:
KeystoneAdminApiNetwork: internal_api
3.3 10-neutron-custom-configs.yaml
parameter_defaults:
# List of Plugins enabled in Neutron
NeutronPluginExtensions: 'qos,port_security'
# Metadata through Neutron DHCP
NeutronEnableIsolatedMetadata: true
NeutronEnableForceMetadata: true
# Neutron allowed Network types
NeutronNetworkType: 'vlan,flat'
NeutronTunnelTypes: ''
# Neutron VLAN range per bridge for OS Controller
NeutronNetworkVLANRanges: 'physnet,physnet-az0-ovs-svc0:100:110,physnet-az1-bare-dpdk:1000:1100'
NeutronML2PhysicalNetworkMtus: "physnet:1500,physnet-az0-ovs-svc0:1500,physnet-az1-bare-dpdk:1500"
# Neutron Flat Network
NeutronFlatNetworks: ''
# Default global MTU for the platform
# In ExtraConfig specific override for overlay
NeutronGlobalPhysnetMtu: 9000
NeutronMechanismDrivers: ['openvswitch', 'sriovnicswitch']
NeutronOVSFirewallDriver: noop # before - openvswitch
NeutronDhcpAgentsPerNetwork: 2
NumDpdkInterfaceRxQueues: 2
3.4 10-nova-custom-configs.yaml
parameter_defaults:
NovaSchedulerDefaultFilters:
- RetryFilter
- AvailabilityZoneFilter
- ComputeFilter
- ImagePropertiesFilter
- ServerGroupAntiAffinityFilter
- ServerGroupAffinityFilter
- ComputeCapabilitiesFilter
- PciPassthroughFilter
- NUMATopologyFilter
- AggregateInstanceExtraSpecsFilter
NovaSchedulerAvailableFilters: ["nova.scheduler.filters.all_filters","nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter"]
NovaComputeLibvirtType: 'qemu' ## if compute node is vm
3.5 10-telemetry-custom-configs.yaml
parameter_defaults:
# Gnocchi backend can be either 'rbd' (Ceph), 'swift' or 'file'.
GnocchiBackend: file
3.6 2-network-environment.yaml
resource_registry:
# NIC Configuration files for all the roles
OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml
OS::TripleO::ComputeOvs::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute-ovs.yaml
OS::TripleO::ComputeDpdk::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute-dpdk.yaml
# VIP Ports
OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external.yaml
OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api.yaml
OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/vip.yaml
# IP for Controller
OS::TripleO::Controller::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external_from_pool.yaml
OS::TripleO::Controller::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
# IP for Compute-Ovs
OS::TripleO::ComputeOvs::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
# IP for Compute-Dpdk
OS::TripleO::ComputeDpdk::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
parameter_defaults:
# VIP Address
ControlFixedIPs: [{'ip_address':'55.55.157.180'}]
PublicVirtualFixedIPs: [{'ip_address':'55.55.152.180'}]
InternalApiVirtualFixedIPs: [{'ip_address':'55.55.155.180'}]
RedisVirtualFixedIPs: [{'ip_address':'55.55.155.179'}]
#-----------------------------------------------------------------------------------#
ControlPlaneSubnet: AZ-0-OSC-CP0
ControllerControlPlaneSubnet: AZ-0-OSC-CP0
ComputeOvsControlPlaneSubnet: AZ-0-OSC-CP0
ComputeDpdkControlPlaneSubnet: AZ-0-OSC-CP0
#-----------------------------------------------------------------------------------#
# Add in configuration for the Control Plane
ControlPlaneSubnetCidr: "24"
ControlPlaneDefaultRoute: 55.55.157.1
AZ-0-OSC-CP0EC2MetadataIp: 55.55.157.170
#-----------------------------------------------------------------------------------#
# Public Endpoint
ExternalNetCidr: 55.55.152.0/24
ExternalAllocationPools: [{'start': '55.55.152.181', 'end': '55.55.152.189'}]
ExternalNetworkVlanID: 202
#ExternalInterfaceDefaultRoute: 55.55.152.1
#-----------------------------------------------------------------------------------#
# admin, internal Endpoint
InternalApiNetCidr: 55.55.155.0/24
InternalApiAllocationPools: [{'start': '55.55.155.181', 'end': '55.55.155.189'}]
InternalApiNetworkVlanID: 205
#-----------------------------------------------------------------------------------#
# To refresh the overcloud nodes network configuration at every Overcloud Update
# ControllerNetworkDeploymentActions: ['CREATE','UPDATE']
# ComputeOvsNetworkDeploymentActions: ['CREATE','UPDATE']
# ComputeDpdkNetworkDeploymentActions: ['CREATE','UPDATE']
#-----------------------------------------------------------------------------------#
# Static IP for Controller
ControllerIPs:
ctlplane:
- 55.55.157.181
- 55.55.157.182
- 55.55.157.183
external:
- 55.55.152.181
- 55.55.152.182
- 55.55.152.183
internal_api:
- 55.55.155.181
- 55.55.155.182
- 55.55.155.183
# Static IP for ComputeOvs
ComputeOvsIPs:
ctlplane:
- 55.55.157.185
- 55.55.157.186
- 55.55.157.187
internal_api:
- 55.55.155.185
- 55.55.155.186
- 55.55.155.187
# Static IP for ComputeDpdk
ComputeDpdkIPs:
ctlplane:
- 55.55.157.188
internal_api:
- 55.55.155.188
3.7 21-os-extraconfigs.yaml
resource_registry:
# FirstBoot Script for initial configuration (i.e. Disk Wipe, Root Password, SSH Root Access)
OS::TripleO::NodeUserData: ../firstboot/first-boot.yaml
# tuned Script for Openstack Nodes
# OS::TripleO::NodeExtraConfigPost: ../extraconfig/31-tuned.yaml
parameter_defaults:
# ------------------------------------------------------- #
# !!! Warning !!! Node Replacement and Reuse Option
# ------------------------------------------------------- #
# ControllerRemovalPolicies: []
# ControllerRemovalPoliciesMode: update
# ComputeOvsRemovalPolicies: []
# ComputeOvsRemovalPoliciesMode: update
# ComputeDpdkRemovalPolicies: []
# ComputeDpdkRemovalPoliciesMode: update
# ------------------------------------------------------- #
ControllerExtraConfig:
nova::ram_allocation_ratio: 1.0
nova::api::allow_resize_to_same_host: true
nova::disk_allocation_ratio: 1.0
nova::cpu_allocation_ratio: 1.0
glance::api::limit_param_default: 100
nova::pci::aliases:
- name: "pcipt25g01"
device_type: "type-PF"
vendor_id: "8086"
product_id: "158b"
# ------------------------------------------------------- #
neutron::agents::ml2::ovs::enable_security_group: true
neutron::plugins::ml2::enable_security_group: true
neutron::plugins::ml2::path_mtu: 1500
# ------------------------------------------------------- #
neutron::server::network_scheduler_driver: neutron.scheduler.dhcp_agent_scheduler.AZAwareWeightScheduler
neutron::server::router_scheduler_driver: neutron.scheduler.l3_agent_scheduler.AZLeastRoutersScheduler
neutron::server::dhcp_load_type: networks
# ------------------------------------------------------- #
neutron::server::default_availability_zones:
- 'AZ-0-COMP-OVS'
- 'AZ-1-COMP'
# ------------------------------------------------------- #
# Keystone token expiration set at 10 hours
TokenExpiration: 36000
# ------------------------------------------------------- #
ComputeOvsParameters:
NeutronBridgeMappings:
- physnet:br-ex
- physnet-az0-ovs-svc0:br-service0
ExtraSysctlSettings:
net.netfilter.nf_conntrack_max:
value: 1000000
net.nf_conntrack_max:
value: 1000000
net.ipv6.conf.all.disable_ipv6:
value: 1
kernel.sysrq:
value: 1
# ------------------------------------------------------- #
# **** ComputeOvsApp Extraconfig ****
# ------------------------------------------------------- #
# ComputeOvsApp Leaf0 ExtraConfig
# ------------------------------------------------------- #
ComputeOvsExtraConfig:
neutron::plugins::ml2::path_mtu: 1500
neutron::plugins::ml2::physical_network_mtus: ['physnet-az0-ovs-svc0:1500']
neutron::plugins::ml2::network_vlan_ranges: ['physnet-az0-ovs-svc0']
neutron::agents::ml2::ovs::enable_security_group: true
neutron::plugins::ml2::enable_security_group: true
nova::compute::allow_resize_to_same_host: true
nova::compute::force_config_drive: true
nova::compute::resume_guests_state_on_host_boot: true
nova::compute::libvirt::libvirt_inject_password: true
nova::compute::libvirt::libvirt_inject_key: true
nova::compute::libvirt::libvirt_inject_partition: -1
nova::compute::libvirt::vncserver_listen: "%{hiera('internal_api')}"
nova::compute::vncserver_proxyclient_address: "%{hiera('internal_api')}"
cold_migration_ssh_inbound_addr: "%{hiera('internal_api')}"
live_migration_ssh_inbound_addr: "%{hiera('internal_api')}"
nova::migration::libvirt::live_migration_inbound_addr: "%{hiera('internal_api')}"
nova::my_ip: "%{hiera('internal_api')}"
tripleo::profile::base::database::mysql::client::mysql_client_bind_address: "%{hiera('internal_api')}"
# ---- Add availability zone Neutron ---- #
neutron::agents::dhcp::availability_zone: AZ-0-COMP-OVS
# ---- Add availability zone Neutron ---- #
# ------------------------------------------------------- #
3.8 3-storage-environment.yaml
parameter_defaults:
CinderEnableIscsiBackend: false
CinderEnableRbdBackend: false
ExtraConfig:
horizon::cinder_options: { enable_backup: true }
NovaEnableRbdBackend: false
GlanceBackend: file
# Config NFS for glance images.
GlanceNfsEnabled: true
GlanceNfsShare: '55.55.157.253:/root/data/nfs/glance-images'
GlanceNfsOptions: 'rw,sync,context=system_u:object_r:svirt_sandbox_file_t:s0'
# Config NFS for Cinder Volumes.
CinderEnableNfsBackend: true
CinderNfsMountOptions: 'nfsvers=3,rw,sync,intr,bg'
CinderNfsServers: '55.55.157.253:/root/data/nfs/cinder-volumes'
# Config NFS for Cinder Backups
CinderBackupBackend: nfs
CinderBackupNfsMountOptions: 'nfsvers=3,rw,sync,intr,bg'
CinderBackupNfsShare: '55.55.157.253:/root/data/nfs/cinder-backups'
3.9 31-ovs-dpdk-sriov.yaml
resource_registry:
OS::TripleO::Services::ComputeNeutronOvsDpdk: /usr/share/openstack-tripleo-heat-templates/deployment/neutron/neutron-ovs-dpdk-agent-container-puppet.yaml
OS::TripleO::Services::NeutronSriovAgent: /usr/share/openstack-tripleo-heat-templates/deployment/neutron/neutron-sriov-agent-container-puppet.yaml
OS::TripleO::Services::NeutronSriovHostConfig: /usr/share/openstack-tripleo-heat-templates/deployment/deprecated/neutron/neutron-sriov-host-config.yaml
parameter_defaults:
ComputeDpdkParameters:
# GRUB Kernel CLI
KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=30 intel_iommu=on intel_iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 nmi_watchdog=0 transparent_hugepage=never intel_idle.max_cstate=0 processor.max_cstate=1 idle=mwait nohpet nosoftlockup isolcpus=1-9,11-19"
# Tuned profile
TunedProfileName: "cpu-partitioning"
# Isolated pCPU from Kernel, NMI, IRQ, and userland
IsolCpusList: "2-9,12-19"
# Nova pCPU to be used by the vCPU
NovaVcpuPinSet: ['2-9,12-19']
# OVS housekeeper cores
OvsDpdkCoreList: "1,11"
# CPU Share for the QEMU Emulator Threads
NovaComputeCpuSharedSet: "0,10"
# OVS-DPDK PMD Threads pCPU
OvsPmdCoreList: "1,11"
# Reserved memory for the hypervisor
NovaReservedHostMemory: 4096
# OVS-DPDK DPC config (DPC1 here)
OvsDpdkMemoryChannels: "4"
# OVS-DPDK Memory size for NUMA node
OvsDpdkSocketMemory: "1024,1024"
# OvsDpdkSocketMemory: "4096,4096"
NovaLibvirtRxQueueSize: 1024
NovaLibvirtTxQueueSize: 1024
# VHU Socket Group
VhostuserSocketGroup: "hugetlbfs"
NeutronBridgeMappings:
- physnet-az1-bare-dpdk:br-dpdkbond0
ExtraSysctlSettings:
net.netfilter.nf_conntrack_max:
value: 1000000
net.nf_conntrack_max:
value: 1000000
net.ipv6.conf.all.disable_ipv6:
value: 1
kernel.sysrq:
value: 1
# Neutron Physical mapping between a custom name and physical devices
NeutronPhysicalDevMappings:
- physnet-az1-sriov1:p7p1
- physnet-az1-sriov2:p7p2
# Mapping of SR-IOV PF interface to neutron physical_network.
NovaPCIPassthrough:
- devname: "p7p1"
physical_network: "physnet-az1-sriov1"
- devname: "p7p2"
physical_network: "physnet-az1-sriov2"
- vendor_id: "8086"
product_id: "158b"
address: "0000:86:00.0"
- vendor_id: "8086"
product_id: "158b"
address: "0000:86:00.1"
# ------------------------------------------------------- #
# **** ComputeDpdk Extraconfig ****
# ------------------------------------------------------- #
# ComputeDpdk Leaf0 ExtraConfig
# ------------------------------------------------------- #
ComputeDpdkExtraConfig:
neutron::agents::ml2::ovs::enable_security_group: true
neutron::plugins::ml2::enable_security_group: true
nova::compute::allow_resize_to_same_host: true
nova::compute::force_config_drive: true
nova::compute::resume_guests_state_on_host_boot: true
nova::compute::libvirt::libvirt_inject_password: true
nova::compute::libvirt::libvirt_inject_key: true
nova::compute::libvirt::libvirt_inject_partition: -1
nova::compute::libvirt::vncserver_listen: "%{hiera('internal_api')}"
nova::compute::vncserver_proxyclient_address: "%{hiera('internal_api')}"
cold_migration_ssh_inbound_addr: "%{hiera('internal_api')}"
live_migration_ssh_inbound_addr: "%{hiera('internal_api')}"
nova::migration::libvirt::live_migration_inbound_addr: "%{hiera('internal_api')}"
nova::my_ip: "%{hiera('internal_api')}"
tripleo::profile::base::database::mysql::client::mysql_client_bind_address: "%{hiera('internal_api')}"
# ---- Add availability zone Neutron ---- #
neutron::agents::dhcp::availability_zone: AZ-0-COMP-OVS
# ---- Add availability zone Neutron ---- #
# ------------------------------------------------------- #
3.10 4-controller-fencing.yaml
parameter_defaults:
EnableFencing: true
FencingConfig:
devices:
# ------------------------------------------------------- #
# Controller #1 Stonigh Fencing
# ------------------------------------------------------- #
- agent: fence_ipmilan
host_mac: "52:54:00:aa:8c:5a"
params:
login: admin
passwd: hpinvent
ipaddr: 55.55.157.4
ipport: 6451
lanplus: true
pcmk_host_list: hk-osc-1
# ------------------------------------------------------- #
# Controller #2 Stonigh Fencing
# ------------------------------------------------------- #
- agent: fence_ipmilan
host_mac: "52:54:00:de:24:a1"
params:
login: admin
passwd: hpinvent
ipaddr: 55.55.157.4
ipport: 6452
lanplus: true
pcmk_host_list: hk-osc-2
# ------------------------------------------------------- #
# Controller #3 Stonigh Fencing
# ------------------------------------------------------- #
- agent: fence_ipmilan
host_mac: "52:54:00:b8:d3:2b"
params:
login: admin
passwd: hpinvent
ipaddr: 55.55.157.4
ipport: 6453
lanplus: true
pcmk_host_list: hk-osc-3
# ------------------------------------------------------- #
3.11 41-disable-swift.yaml
resource_registry:
OS::TripleO::Services::SwiftProxy: OS::Heat::None
OS::TripleO::Services::SwiftStorage: OS::Heat::None
OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None
3.12 42-disable-ceph.yaml
resource_registry:
OS::TripleO::Services::CephMgr: OS::Heat::None
OS::TripleO::Services::CephMon: OS::Heat::None
OS::TripleO::Services::CephOSD: OS::Heat::None
OS::TripleO::Services::CephClient: OS::Heat::None
3.13 99-server-blacklist.yaml
parameter_defaults:
DeploymentServerBlacklist: []
# - overcloud-compute-ovs-0
4. Overcloud templates
4.1 31-tuned.yaml
아래 파일들은 /home/stack/templates/extraconfig/ 에 파일 내용입니다.
heat_template_version: 2014-10-16
description: >
Extra configuration
parameters:
servers:
type: json
DeployIdentifier:
type: string
EndpointMap:
default: {}
type: json
resources:
ExtraConfig:
type: OS::Heat::SoftwareConfig
properties:
group: script
inputs:
- name: deploy_identifier
config: |
#!/bin/sh
set -x
function tuned_service_dependency() {
tuned_service=/usr/lib/systemd/system/tuned.service
grep -q "network.target" $tuned_service
if [ "$?" -eq 0 ]; then
sed -i '/After=.*/s/network.target//g' $tuned_service
fi
grep -q "Before=.*network.target" $tuned_service
if [ ! "$?" -eq 0 ]; then
grep -q "Before=.*" $tuned_service
if [ "$?" -eq 0 ]; then
sed -i 's/^\(Before=.*\)/\1 network.target openvswitch.service/g' $tuned_service
else
sed -i '/After/i Before=network.target openvswitch.service' $tuned_service
fi
fi
}
if hiera -c /etc/puppet/hiera.yaml service_names | grep -q neutron_ovs_dpdk_agent; then
tuned_service_dependency
fi
if [[ `hostname` = *"controller"* ]]
then
hostname
fi
dnf install -y sysstat vim bash-completion tcpdump
ExtraDeployments:
type: OS::Heat::SoftwareDeployments
properties:
name: ExtraDeployments
servers: {get_param: servers}
config: {get_resource: ExtraConfig}
actions: ['CREATE','UPDATE']
input_values:
deploy_identifier: {get_param: DeployIdentifier}
4.2 32-extra-firewallrule.yaml
parameter_defaults:
ExtraConfig:
tripleo::firewall::firewall_rules:
'300 allow Zabbix agent':
port: 150
proto: tcp
action: accept
'301 allow Zabbix agent':
port: 151
proto: tcp
action: accept
'302 allow udagent':
port: 38
proto: tcp
action: accept
'303 allow Zabbix agent':
port: 255
proto: tcp
action: accept
4.3 postconfig.yaml
resource_registry:
OS::TripleO::NodeExtraConfigPost: /home/stack/templates/extraconfig/31-tuned.yaml
5. Overcloud templates
5.1 first-boot.yaml
아래 파일들은 /home/stack/templates/firstboot/ 에 파일 내용입니다.
heat_template_version: rocky
description: >
Extra hostname configuration
resources:
userdata:
type: OS::Heat::MultipartMime
properties:
parts:
- config: {get_resource: nameserver_config}
nameserver_config:
type: OS::Heat::SoftwareConfig
properties:
config: |
#!/bin/bash
cat > /etc/yum.repos.d/rhosp16.repo << EOF
[rhel-8-for-x86_64-baseos-rpms]
name=rhel-8-for-x86_64-baseos-rpms
baseurl=http://55.55.157.253/rhel-8-for-x86_64-baseos-rpms
enabled=1
gpgcheck=0
[rhel-8-for-x86_64-appstream-rpms]
name=rhel-8-for-x86_64-appstream-rpms
baseurl=http://55.55.157.253/rhel-8-for-x86_64-appstream-rpms
enabled=1
gpgcheck=0
[rhel-8-for-x86_64-highavailability-rpms]
name=rhel-8-for-x86_64-highavailability-rpms
baseurl=http://55.55.157.253/rhel-8-for-x86_64-highavailability-rpms
enabled=1
gpgcheck=0
[ansible-2.9-for-rhel-8-x86_64-rpms]
name=ansible-2.9-for-rhel-8-x86_64-rpms
baseurl=http://55.55.157.253/ansible-2.9-for-rhel-8-x86_64-rpms
enabled=1
gpgcheck=0
[satellite-tools-6.5-for-rhel-8-x86_64-rpms]
name=satellite-tools-6.5-for-rhel-8-x86_64-rpms
baseurl=http://55.55.157.253/satellite-tools-6.5-for-rhel-8-x86_64-rpms
enabled=1
gpgcheck=0
[openstack-16.1-for-rhel-8-x86_64-rpms]
name=openstack-16.1-for-rhel-8-x86_64-rpms
baseurl=http://55.55.157.253/openstack-16.1-for-rhel-8-x86_64-rpms
enabled=1
gpgcheck=0
[fast-datapath-for-rhel-8-x86_64-rpms]
name=fast-datapath-for-rhel-8-x86_64-rpms
baseurl=http://55.55.157.253/fast-datapath-for-rhel-8-x86_64-rpms
enabled=1
gpgcheck=0
EOF
echo 'hk!234' | passwd --stdin root
dnf install -y sysstat vim bash-completion tcpdump
# ZAP Disk : /dev/vdb nova instance data partition
if [[ "$(hostname)" =~ "comp" ]]; then
echo "Number of disks detected: $(lsblk -no NAME,TYPE,MOUNTPOINT | grep "disk" | awk '{print $1}' | wc -l)"
for DEVICE in `lsblk -no NAME,TYPE,MOUNTPOINT | grep "disk" | awk '{print $1}'`
do
ROOTFOUND=0
echo "Checking /dev/${DEVICE}..."
echo "Number of partitions on /dev/${DEVICE}: $(expr $(lsblk -n /dev/${DEVICE} | awk '{print $7}' | wc -l) - 1)"
for MOUNTS in `lsblk -n /dev/${DEVICE} | awk '{print $7}'`
do
if [[ "$MOUNTS" = "/" ]]
then
ROOTFOUND=1
fi
done
if [[ $ROOTFOUND = 0 ]]
then
echo "Root not found in /dev/${DEVICE}"
echo "Wiping disk /dev/${DEVICE}"
sgdisk -Z /dev/${DEVICE}
sgdisk -g /dev/${DEVICE}
else
echo "Root found in /dev/${DEVICE}"
fi
done
fi
if [[ "$(hostname)" =~ "comp" ]]; then
# LVM parted
parted -s /dev/vdb mklabel gpt
parted -s /dev/vdb mkpart primary 0 100%
pvcreate /dev/vdb1
vgcreate vg-compute /dev/vdb1
lvcreate -y -l 100%FREE -n hostvolume vg-compute
mkfs.xfs -L hostvolume /dev/vg-compute/hostvolume
echo "/dev/vg-compute/hostvolume /var/lib/nova/instances xfs defaults,_netdev 0 0" >> /etc/fstab
mount -a
chown -R nova:nova /var/lib/nova/instances
restorecon -RF /var/lib/nova/instances
fi
outputs:
OS::stack_id:
value: {get_resource: userdata}
6. Overcloud templates
6.1 controller.yaml
아래 파일들은 /home/stack/templates/nic-configs/ 에 파일 내용입니다.
heat_template_version: rocky
parameters:
#------------------------------------------------------------------------------------#
ControlPlaneIp:
type: string
ControlPlaneSubnetCidr:
type: string
ControlPlaneDefaultRoute:
type: string
ControlPlaneStaticRoutes:
type: json
ControlPlaneMtu:
type: number
#------------------------------------------------------------------------------------#
InternalApiIpSubnet:
type: string
InternalApiMtu:
type: number
InternalApiInterfaceRoutes:
type: json
#------------------------------------------------------------------------------------#
ExternalIpSubnet:
type: string
ExternalMtu:
type: number
ExternalInterfaceRoutes:
type: json
AZ-0-OSC-CP0EC2MetadataIp:
type: string
DnsServers: # Override this via parameter_defaults
type: comma_delimited_list
#------------------------------------------------------------------------------------#
resources:
OsNetConfigImpl:
type: OS::Heat::SoftwareConfig
properties:
group: script
config:
str_replace:
template:
get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
params:
$network_config:
network_config:
#------------------------------------------------------------------------------------#
- type: interface
name: ens3
use_dhcp: false
dns_servers:
get_param: DnsServers
addresses:
- ip_netmask:
list_join:
- /
- - get_param: ControlPlaneIp
- get_param: ControlPlaneSubnetCidr
routes:
- ip_netmask: 169.254.169.254/32
next_hop:
get_param: ControlPlaneDefaultRoute
#------------------------------------------------------------------------------------#
- type: interface
name: ens4
use_dhcp: false
addresses:
- ip_netmask:
get_param: ExternalIpSubnet
routes:
- default: true
next_hop: 55.55.152.1
#------------------------------------------------------------------------------------#
- type: interface
name: ens5
use_dhcp: false
addresses:
- ip_netmask:
get_param: InternalApiIpSubnet
#------------------------------------------------------------------------------------#
outputs:
OS::stack_id:
description: The OsNetConfigImpl resource.
value:
get_resource: OsNetConfigImpl
6.2 compute-ovs.yaml
heat_template_version: rocky
parameters:
#------------------------------------------------------------------------------------#
ControlPlaneIp:
type: string
ControlPlaneSubnetCidr:
type: string
ControlPlaneDefaultRoute:
type: string
ControlPlaneStaticRoutes:
type: json
ControlPlaneMtu:
type: number
#------------------------------------------------------------------------------------#
InternalApiIpSubnet:
type: string
InternalApiMtu:
type: number
InternalApiInterfaceRoutes:
type: json
AZ-0-OSC-CP0EC2MetadataIp:
type: string
DnsServers: # Override this via parameter_defaults
type: comma_delimited_list
#------------------------------------------------------------------------------------#
resources:
OsNetConfigImpl:
type: OS::Heat::SoftwareConfig
properties:
group: script
config:
str_replace:
template:
get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
params:
$network_config:
network_config:
#------------------------------------------------------------------------------------#
- type: interface
name: ens3
use_dhcp: false
dns_servers:
get_param: DnsServers
addresses:
- ip_netmask:
list_join:
- /
- - get_param: ControlPlaneIp
- get_param: ControlPlaneSubnetCidr
routes:
- ip_netmask: 169.254.169.254/32
next_hop:
get_param: ControlPlaneDefaultRoute
#------------------------------------------------------------------------------------#
- type: interface
name: ens4
use_dhcp: false
addresses:
- ip_netmask:
get_param: InternalApiIpSubnet
routes:
- default: true
next_hop: 55.55.155.1
#------------------------------------------------------------------------------------#
- type: ovs_bridge
name: br-service0
use_dhcp: false
mtu: 1500
members:
- type: interface
name: ens5
mtu: 1500
#------------------------------------------------------------------------------------#
outputs:
OS::stack_id:
description: The OsNetConfigImpl resource.
value:
get_resource: OsNetConfigImpl
6.3 compute-dpdk.yaml
heat_template_version: rocky
parameters:
#------------------------------------------------------------------------------------#
ControlPlaneIp:
type: string
ControlPlaneSubnetCidr:
type: string
ControlPlaneDefaultRoute:
type: string
ControlPlaneStaticRoutes:
type: json
ControlPlaneMtu:
type: number
#------------------------------------------------------------------------------------#
InternalApiIpSubnet:
type: string
InternalApiMtu:
type: number
InternalApiInterfaceRoutes:
type: json
AZ-0-OSC-CP0EC2MetadataIp:
type: string
DnsServers: # Override this via parameter_defaults
type: comma_delimited_list
#------------------------------------------------------------------------------------#
resources:
OsNetConfigImpl:
type: OS::Heat::SoftwareConfig
properties:
group: script
config:
str_replace:
template:
get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
params:
$network_config:
network_config:
#------------------------------------------------------------------------------------#
- type: interface
name: ens3
use_dhcp: false
dns_servers:
get_param: DnsServers
addresses:
- ip_netmask:
list_join:
- /
- - get_param: ControlPlaneIp
- get_param: ControlPlaneSubnetCidr
routes:
- ip_netmask: 169.254.169.254/32
next_hop:
get_param: ControlPlaneDefaultRoute
#------------------------------------------------------------------------------------#
- type: interface
name: ens4
use_dhcp: false
addresses:
- ip_netmask:
get_param: InternalApiIpSubnet
routes:
- default: true
next_hop: 55.55.155.1
#------------------------------------------------------------------------------------#
- type: ovs_user_bridge
name: br-dpdkbond0
use_dhcp: false
mtu: 1500
members:
- type: ovs_dpdk_bond
name: dpdkbond0
rx_queue: 2
use_dhcp: false
mtu: 1500
ovs_options: "bond_mode=active-backup other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100"
members:
- type: ovs_dpdk_port
name: dpdk0
members:
- type: interface
name: ens5
mtu: 1500
#------------------------------------------------------------------------------------#
outputs:
OS::stack_id:
description: The OsNetConfigImpl resource.
value:
get_resource: OsNetConfigImpl
7. deploy.sh-http
#!/bin/bash
source /home/stack/stackrc
tripleo="/usr/share/openstack-tripleo-heat-templates"
templates="/home/stack/templates"
#exit
starttime=$(date);
time openstack overcloud deploy --templates \
-n ${templates}/network_data.yaml \
-r ${templates}/roles-data.yaml \
-e ${templates}/node-info.yaml \
-e ${tripleo}/environments/sshd-banner.yaml \
-e ${tripleo}/environments/network-isolation.yaml \
-e ${tripleo}/environments/services/neutron-ovs.yaml \
-e ${tripleo}/environments/services/cinder-backup.yaml \
-e ${tripleo}/environments/host-config-and-reboot.yaml \
-e ${tripleo}/environments/disable-swift.yaml \
-e ~/containers-prepare-parameter.yaml \
-e ${templates}/enviroments/1-commons-parameters.yaml \
-e ${templates}/enviroments/2-network-environment.yaml \
-e ${templates}/enviroments/3-storage-environment.yaml \
-e ${templates}/enviroments/4-controller-fencing.yaml \
-e ${templates}/enviroments/10-endpoint.yaml \
-e ${templates}/enviroments/10-neutron-custom-configs.yaml \
-e ${templates}/enviroments/10-nova-custom-configs.yaml \
-e ${templates}/enviroments/10-telemetry-custom-configs.yaml \
-e ${templates}/enviroments/21-os-extraconfigs.yaml \
-e ${templates}/enviroments/31-ovs-dpdk-sriov.yaml \
-e ${templates}/extraconfig/postconfig.yaml \
-e ${templates}/extraconfig/32-extra-firewallrule.yaml \
-e ${templates}/enviroments/41-disable-swift.yaml \
-e ${templates}/enviroments/42-disable-ceph.yaml \
-e ${templates}/enviroments/99-server-blacklist.yaml
echo $starttime;
date728x90
