반응형
본 내용은 사내 TestBed - KVM 환경에서 OpenShift 4.8.14 설치방법입니다.
빠르게 Redhat OpenShift Container Platform 설치를 위한 용도로 인터넷이 되는 환경에서의 방법이며, 간략한 TestBed 구성이 목적임에 따라 Bastion 1 / Master 3 / Worker 2대로 설치하는 방법입니다.
간략 기능 설명 :
- Bastion : DNS, LB 역할, 설치를 위한 HTTP 서버
- Bootstrap : RHOCP Cluster 구성을 위해 필요한 노드 (마스터 구성 이후 사용 안함)
- Worker : Pod 들이 동작하는 노드
# RHOSP 4.8.14 배포를 위한 VM 사양 정보는 다음과 같습니다.
# 인프라 구성정보
55.55.34.0 : Public
55.55.35.0: OCP MGMT
1. RHOCP (RedHat OpenShift Container Platform) 4.8.14 설치 및 환경 구성
1.1 Bastion 설치
1) 편의성 패키지 설치
[root@bastion ~]#dnf install -y vim bash-completion tcpdump tar wget
2) 호스트네임 설정
[root@bastion ~]#hostnamectl set-hostname bastion.ocp-dc.hk.com
3) 방화벽 및 SELINUX 설정
[root@bastion ~]#cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
[root@hk-bastion ~]# systemctl disable firewalld --now
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
4) HAProxy 설치 및 구성, 시작
[root@bastion ~]#yum install -y haproxy
[root@bastion ~]#cat /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
# utilize system-wide crypto-policies
ssl-default-bind-ciphers PROFILE=SYSTEM
ssl-default-server-ciphers PROFILE=SYSTEM
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen stats
bind :9000
mode http
stats enable
stats uri /
monitor-uri /bastion-test
stats refresh 5s
#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
balance roundrobin
server static 127.0.0.1:4331 check
#---------------------------------------------------------------------
# RHOCP-API
#---------------------------------------------------------------------
frontend openshift-api-server
bind *:6443
default_backend openshift-api-server
mode tcp
option tcplog
backend openshift-api-server
balance source
mode tcp
server bootstrap 55.55.35.205:6443 check
server master1 55.55.35.211:6443 check
server master2 55.55.35.212:6443 check
server master3 55.55.35.213:6443 check
#---------------------------------------------------------------------
# RHOCP-machine-config-server
#---------------------------------------------------------------------
frontend machine-config-server
bind *:22623
default_backend machine-config-server
mode tcp
option tcplog
backend machine-config-server
balance source
mode tcp
server bootstrap 55.55.35.205:22623 check
server master1 55.55.35.211:22623 check
server master2 55.55.35.212:22623 check
server master3 55.55.35.213:22623 check
#---------------------------------------------------------------------
# RHOCP ingress-http (80)
#---------------------------------------------------------------------
frontend ingress-http
bind *:80
default_backend ingress-http
mode tcp
option tcplog
backend ingress-http
balance source
mode tcp
server worker1 55.55.35.216:80 check
server worker2 55.55.35.217:80 check
#---------------------------------------------------------------------
# RHOCP ingress-httpi (443)
#---------------------------------------------------------------------
frontend ingress-https
bind *:443
default_backend ingress-https
mode tcp
option tcplog
backend ingress-https
balance source
mode tcp
server worker1 55.55.35.216:443 check
server worker2 55.55.35.217:443 check
[root@bastion ~]#systemctl enable haproxy --now
5) HTTP 서버 설치 및 구성, 시작
HTTP 설치 후 기본 80 포트 구성을 8080으로 변경한 후 서비스를 시작합니다.
[root@bastion haproxy]#dnf install -y httpd
[root@bastion ~]#cat /etc/httpd/conf/httpd.conf | grep Listen
# Listen: Allows you to bind Apache to specific IP addresses and/or
# Change this to Listen on specific IP addresses as shown below to
#Listen 12.34.56.78:80
Listen 8080
[root@bastion conf]#systemctl enable httpd --now
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.
6) RHOCP 설치를 위한 이미지 및 설정 파일 다운로드
[root@bastion ~]#wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.8/4.8.14/rhcos-4.8.14-x86_64-live-initramfs.x86_64.img
[root@bastion ~]#wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.8/4.8.14/rhcos-4.8.14-x86_64-live-kernel-x86_64
[root@bastion ~]#wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.8/4.8.14/rhcos-4.8.14-x86_64-metal.x86_64.raw.gz
[root@bastion ~]#wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.8/4.8.14/rhcos-4.8.14-x86_64-live-rootfs.x86_64.img
[root@bastion ~]#mv rhcos-4.8.14-x86_64-live-initramfs.x86_64.img /var/www/html/
[root@bastion ~]#mv rhcos-4.8.14-x86_64-live-rootfs.x86_64.img /var/www/html/
[root@bastion ~]#mv rhcos-4.8.14-x86_64-live-kernel-x86_64 /var/www/html/
[root@bastion ~]#mv rhcos-4.8.14-x86_64-metal.x86_64.raw.gz /var/www/html/
[root@bastion ~]#mv /var/www/html/rhcos-4.8.14-x86_64-metal.x86_64.raw.gz /var/www/html/rhcos.tar.gz
[root@bastion ~]#systemctl restart httpd
7) DNS 서버 설치 및 구성
[root@bastion ~]#dnf install -y bind bind-utils
[root@bastion ~]#systemctl enable named --now
[root@bastion ~]#cat /etc/named.conf
options {
listen-on port 53 { any; };
listen-on-v6 port 53 { none; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
secroots-file "/var/named/data/named.secroots";
recursing-file "/var/named/data/named.recursing";
allow-query { any; };
/*
- If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
- If you are building a RECURSIVE (caching) DNS server, you need to enable
recursion.
- If your recursive DNS server has a public IP address, you MUST enable access
control to limit queries to your legitimate users. Failing to do so will
cause your server to become part of large scale DNS amplification
attacks. Implementing BCP38 within your network would greatly
reduce such attack surface
*/
recursion yes;
dnssec-enable yes;
dnssec-validation yes;
managed-keys-directory "/var/named/dynamic";
pid-file "/run/named/named.pid";
session-keyfile "/run/named/session.key";
/* https://fedoraproject.org/wiki/Changes/CryptoPolicy */
include "/etc/crypto-policies/back-ends/bind.config";
};
logging {
channel default_debug {
file "data/named.run";
severity dynamic;
};
};
zone "." IN {
type hint;
file "named.ca";
};
include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
# rfc1912 zone 파일 수정
[root@bastion ~]#cat /etc/named.rfc1912.zones
zone "localhost.localdomain" IN {
type master;
file "named.localhost";
allow-update { none; };
};
zone "localhost" IN {
type master;
file "named.localhost";
allow-update { none; };
};
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
type master;
file "named.loopback";
allow-update { none; };
};
zone "1.0.0.127.in-addr.arpa" IN {
type master;
file "named.loopback";
allow-update { none; };
};
zone "0.in-addr.arpa" IN {
type master;
file "named.empty";
allow-update { none; };
};
zone "hk.com" IN {
type master;
file "/var/named/ocp-dc.hk.com.zone";
allow-update { none; };
};
zone "35.55.55.in-addr.arpa" IN {
type master;
file "/var/named/35.55.55.in-addr.rev";
allow-update { none; };
};
# Zone 파일 설정 (DNS 정방향)
[root@bastion ~]#cat /var/named/ocp-dc.hk.com.zone
$TTL 1D
@ IN SOA @ ns.hk.com. (
0 ; serial
1D ; refresh
1H ; retry
1W ; expire
3H ) ; minimum
@ IN NS ns.hk.com.
@ IN A 55.55.35.200 ;
; Ancillary services
lb.ocp-dc IN A 55.55.35.200
; Bastion or Jumphost
ns IN A 55.55.35.200 ;
; OCP Cluster
bootstrap.ocp-dc IN A 55.55.35.205
master1.ocp-dc IN A 55.55.35.211
master2.ocp-dc IN A 55.55.35.212
master3.ocp-dc IN A 55.55.35.213
api.ocp-dc IN A 55.55.35.200
api-int.ocp-dc IN A 55.55.35.200
*.apps.ocp-dc IN A 55.55.35.200
# Zone 파일 설정 (DNS 역방향)
[root@bastion ~]#cat /var/named/35.55.55.in-addr.rev
$TTL 1D
@ IN SOA hk.com. ns.hk.com. (
0 ; serial
1D ; refresh
1H ; retry
1W ; expire
3H ) ; minimum
@ IN NS ns.
200 IN PTR ns.
200 IN PTR bastion.ocp-dc.hk.com.
205 IN PTR bootstrap.ocp-dc.hk.com.
211 IN PTR master1.ocp-dc.hk.com.
212 IN PTR master2.ocp-dc.hk.com.
213 IN PTR master3.ocp-dc.hk.com.
200 IN PTR api.ocp-dc.hk.com.
200 IN PTR api-int.ocp-dc.hk.com.
# zone 파일 권한 설정
[root@bastion named]#cd /var/named
[root@bastion named]#chown root:named 35.55.55.in-addr.rev
[root@bastion named]#chown root:named ocp-dc.hk.com.zone
[root@bastion named]#systemctl restart named
1.2 RHOCP 배포 준비
RHOCP 4.8.14 배포는 openshift-install 을 이용하여 진행합니다.
각 노드들에 맞는 ignition 파일을 생성해서 http 웹서버에서 서비스가 되도록 설정합니다.
1) ssh-key 생성
[root@bastion ~]#ssh-keygen -t rsa -b 4096 -N ''
2) oc 바이너리와 openshift-install 바이너리 다운로드
[root@bastion ~]#wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.8.14/openshift-client-linux.tar.gz
[root@bastion ~]#wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.8.14/openshift-install-linux.tar.gz
[root@bastion ~]# tar xvf openshift-install-linux.tar.gz -C /usr/local/bin/
[root@bastion ~]# tar xvf openshift-client-linux.tar.gz -C /usr/local/bin/
3) install-config.yaml 파일 생성 및 백업
[root@bastion ~]#mkdir -p /root/ocp/config
[root@bastion ~]#cd /root/ocp/config/
[root@bastion config]#cat install-config.yaml
apiVersion: v1
baseDomain: hk.com
compute:
- hyperthreading: Enabled
name: worker
replicas: 0
controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: ocp-dc #cluster name
networking:
clusterNetworks:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
pullSecret: '{pull secret 정보 필요}'
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDLN6gJubKTHWbQL0QRUcpIcpTJaJiGjttKPDbR71WJkgvp41ia+ZX1Y7iJPU7rDx8E2bDtBRc+be+Q0X361n1RxVH2cRro2eCPLu/kPu7rXeWSJxFutCXzJ9E/9SCmBqNBQ1Iv78ysuSFhkOxGsMCt0nqsO3nJVeVr9TZRUx7CpO4adkCBKAKjLuBYjT1xR3QEY5KKLBPQpxEG7z4xWx1xuRNnqPzDTBy/CvoHSxKH+AgXEdOoSBE8d+zUDCTWCYiMA+NZjOlWMJUKtGJrOU6hIUfMj+OIH7xBl27MWZ0hso4l7Qu8SE3ptGH27+AijQUBnreRYzodPmxbWtKD5wqENNMyIZocEIOV+OAnevrJZwdpmG7rtX8d2f4PaK8PX3P6ChlKOszn1P/AkqoqQlJODIvVJe+qr4bYVyy1Lck1i36wMU9R1z80zA2LhaCTnwf2wB28B6ngwL4XlukyPwfbacBRMkArkSaIUOHVHRKt8RKeL7l1qBI9FsKEWJNWTZt55rGXbRnvcuOuoVTS3UaxcNQP4WZbW72B6BTel9lDWkAlCKcfgnRK9xs+OdTkhZD+EZP9gM8AmaGgDRmpsFgeRAgaEUhMFO+k/NCwiTR6CXxxeRjwvyXDx8MCG8N5A3YZ+M/keLFPHf7wg2agOURAjJ8a8WJvlmFtDbC1bBhulw== root@bastion.ocp-dc.hk.com'
# pulsect : https://console.redhat.com/openshift/create
# sshKey -> cat /root/,ssh/id_rsa.pub 값을 추가
[root@bastion config]#mkdir backup
[root@bastion config]#cp install-config.yaml ./backup/install-config.yaml
1.3 manifest 및 ignition 파일 생성
openshift-install 바이너리를 이용하여 RHOCP Cluster 설치를 위한 manifest와 ignition 파일을 생성합니다.
1) manifests 파일 생성
[root@bastion config]#openshift-install create manifests --dir=/root/ocp/config/
INFO Consuming Install Config from target directory
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
INFO Manifests created in: /root/ocp/config/manifests and /root/ocp/config/openshift
[root@bastion config]#cd openshift/
[root@bastion openshift]#ls
99_kubeadmin-password-secret.yaml 99_openshift-machineconfig_99-master-ssh.yaml
99_openshift-cluster-api_master-user-data-secret.yaml 99_openshift-machineconfig_99-worker-ssh.yaml
99_openshift-cluster-api_worker-user-data-secret.yaml openshift-install-manifests.yaml
2) ignition 파일 생성
[root@bastion config]#openshift-install create ignition-configs --dir /root/ocp/config
INFO Consuming Master Machines from target directory
INFO Consuming Worker Machines from target directory
INFO Consuming Common Manifests from target directory
INFO Consuming Openshift Manifests from target directory
INFO Consuming OpenShift Install (Manifests) from target directory
INFO Ignition-Configs created in: /root/ocp/config and /root/ocp/config/auth
[root@bastion config]#ls /root/ocp/config/
auth backup bootstrap.ign master.ign metadata.json worker.ign
[root@bastion config]#mkdir /var/www/html/ign
[root@bastion config]#cp *.ign /var/www/html/ign/
[root@bastion config]#chmod 777 /var/www/html/ign/*.ign
[root@bastion config]#systemctl restart httpd
2. RHOCP 배포
2.1 bootstap 생성
CoreOS를 설치하기 위해 4.8.14 (rhcos-4.8.14-live.x86_64.iso) iso 파일로 부팅하여 설치했습니다.
설치 방식은 GRUB 모드에서 커널 파라미터에서 설정하여 설치를 할 수도 있지만, 아래 가이드는 live iso 파일을 통해 부팅한 후 cli 명령으로 설치도 가능합니다.
커널 파라미터 수정보다 live iso 파일을 통해 설치하는 방식이 편해 해당 방법으로 구성하였습니다.
bootstrap / master / worker 노드 모두 부트스트랩 방식으로 설치하였으며, ign 파일명과 IP 설정만 변경하여 설치하여야 합니다.
1) bootstrap 설치를 위한 iso 마운트
RHEL CoreOS 선택
IP 및 nameserver 설정
bootstrap 설치 후 reboot
# CoreOS 노드별 설치 명령어
# bootstrap 설치 명령어
sudo coreos-installer install /dev/vda --insecure-ignition -I http://55.55.35.200:8080/ign/bootstrap.ign -n
# master 설치 명령어
sudo coreos-installer install /dev/vda --insecure-ignition -I http://55.55.35.200:8080/ign/master.ign -n
# worker 설치 명령어
sudo coreos-installer install /dev/vda --insecure-ignition -I http://55.55.35.200:8080/ign/worker.ign -n
2) master 노드 설치
bootstrap 설치한 방식과 동일하게 master 3대 노드를 설치합니다.
master 노드 설치전에 bootstrap 설치에 시간이 걸릴 수 있는데 (노드 성능에 따라..) 아래 출력 로그가 확인이 되면 마스터 노드 설치를 진행합니다.
[출력 로그]
API v1.21.1 up
[root@bastion ~]#openshift-install --dir=/root/ocp/config wait-for bootstrap-complete --log-level=info
3) cluster 구성 확인
master 3대를 설치를 완료하였다면, 아래 로그 정보가 나타나는데, "It is now safe to remove the bootstrap resources" 해당 메시지가 출력되면 master3대를 통해 정상적으로 Cluster가 구성되었다는 것으로 이 이후에는 bootstrap 노드가 필요하지 않습니다.
[root@bastion ~]#openshift-install --dir=/root/ocp/config/ wait-for bootstrap-complete --log-level=info
INFO Waiting up to 20m0s for the Kubernetes API at https://api.ocp-dc.hk.com:6443...
INFO API v1.21.1+a620f50 up
INFO Waiting up to 30m0s for bootstrapping to complete...
INFO It is now safe to remove the bootstrap resources
INFO Time elapsed: 0s
4) HAProxy 설정 변경
RHOCP 클러스터가 정상적으로 구성되었기 때문에, HAProxy에서 bootstrap으로 LB되지 않도록 수정 후 서비스를 재시작합니다.
backend openshift-api-server
balance source
mode tcp
#server bootstrap 55.55.35.205:6443 check
backend machine-config-server
balance source
mode tcp
#server bootstrap 55.55.35.205:22623 check
[root@bastion ~]#systemctl restart haproxy
3. RHOCP 로그인 및 확인
3.1 RHOCP 노드 정보 확인
1) kubeadmin export 설정
[root@bastion ~]#cat ~/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
export KUBECONFIG=/root/ocp/config/auth/kubeconfig <- 추가
2) 노드 정보 확인
[root@bastion ~]#oc get node
NAME STATUS ROLES AGE VERSION
master1.ocp-dc.hk.com Ready master,worker 2d20h v1.21.1+a620f50
master2.ocp-dc.hk.com Ready master,worker 2d20h v1.21.1+a620f50
master3.ocp-dc.hk.com Ready master,worker 2d20h v1.21.1+a620f50
3) RHOCP Cluster Component 정상 동작 확인
모든 Cluster Operator가 True / False / False 여야 정상입니다.
4) RHOCP 4.8.14 Cluster 설치 완료 확인
[root@bastion ~]#openshift-install --dir=/root/ocp/config wait-for install-complete
INFO Waiting up to 40m0s for the cluster at https://api.ocp-dc.hk.com:6443 to initialize...
INFO Waiting up to 10m0s for the openshift-console route to be created...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/ocp/config/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp-dc.hk.com
INFO Login to the console with user: "kubeadmin", and password: "xxxxxxxxx"
INFO Time elapsed: 0s
5) (Option) worker 노드 추가
상단 "2. RHOCP 배포"에 정리한 대로 worker 노드를 설치하여 worker 노드를 추가할 수 있습니다.
현재 master 노드의 롤이 master, worker인데 롤을 분류하여 master는 master의 역할만 worker에는 worker 롤로 분류할 수 있습니다. (롤 분류 설정 추가 예정)
[root@bastion ~]#oc get node
NAME STATUS ROLES AGE VERSION
master1.ocp-dc.hk.com Ready master,worker 2d20h v1.21.1+a620f50
master2.ocp-dc.hk.com Ready master,worker 2d20h v1.21.1+a620f50
master3.ocp-dc.hk.com Ready master,worker 2d20h v1.21.1+a620f50
worker1.ocp-dc.hk.com Ready worker 2d20h v1.21.1+a620f50
728x90
'Cloud > RedHat - OpenShift' 카테고리의 다른 글
| [RHOCP4] RHEL7 worker 노드 추가 (0) | 2021.11.09 |
|---|---|
| [RHOCP4] openshift4 Pod 관리 (0) | 2021.11.09 |
| [RHOCP4] openshift4 node 관리 (0) | 2021.11.09 |
| [RHOCP4] Openshift - PXE를 통한 RHCOS work node 설치 (0) | 2021.09.13 |
| RHOCP (Openshift) 개요 (0) | 2021.08.27 |
