[k8s] k8s cluster install guide (설치 가이드)

Kubenetes Cluster Install Guide입니다. 

본 가이드는 Master 3 노드 및 Worker 2노드 기준이며, HAProxy 및 Keepliaved를 통한 HA 환경입니다.

내용도 많아 설치 및 설정 값 위주이며, 현재 RHOCP 구성으로 인해 k8s cluster가 없어 당시에 정리했던 설치가이드를 공유하는 차원이라 구성 정보 조회 내용은 빠져있습니다.

 

 

VM 구성 정보

 

1. HAProxy 설치 및 설정 (Master 전체)

dnf install -y haproxy

vi /etc/haproxy/haproxy.cfg

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
	
frontend k8s-api
  bind 0.0.0.0:8443
  mode tcp
  option tcplog
  default_backend k8s-api

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------

backend k8s-api
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  server hk-k8s-master1 88.88.88.11:6443 check fall 3 rise 2
  server hk-k8s-master2 88.88.88.12:6443 check fall 3 rise 2
  server hk-k8s-master3 88.88.88.13:6443 check fall 3 rise 2


listen stats
    mode http
    bind *:80
    log global

    stats enable
    stats refresh 10s
    stats show-node
    stats uri /haproxy



systemctl enable haproxy --now

 

 

2. keepalived 설치 및 설정 (Master 전체)

dnf install -y keepalived

vi /etc/keepalived/keepalived.conf

## master node 1
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 103
    advert_int 1
    virtual_ipaddress {
        88.88.88.10/24 dev eth0
    }
}

systemctl enable keepalived --now

## master node 2
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    virtual_ipaddress {
        88.88.88.10/24 dev eth0
    }
}

systemctl enable keepalived --now

## master node 3
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 102
    advert_int 1
    virtual_ipaddress {
        88.88.88.10/24 dev eth0
    }
}

systemctl enable keepalived --now

 

 

3. Master 및 Worker 노드 설정

3.1 기본 환경 설정

hosts 설정 추가

vi /etc/hosts 설정 추가
# k8s-master VIP
88.88.88.10 hk-k8s-master

# k8s-master
88.88.88.11 hk-k8s-master1
88.88.88.12 hk-k8s-master2
88.88.88.13 hk-k8s-master3

# k8s-worker
88.88.88.21 hk-k8s-worker1
88.88.88.22 hk-k8s-worker2

 

패키지 업그레이드

dnf -y upgrade

 

SELINUX disable

setenforce 0

sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

 

Masquerading enable 및 방화벽 정책 추가

modprobe br_netfilter

firewall-cmd --add-masquerade --permanent

firewall-cmd --reload

 

iptables rule 설정

cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl --system

swapoff -a

또는

vi /etc/fstab 에서 swap 부분 삭제

 

 

4. docker 설치

4.1 docker 설치 및 구성

## Repository 추가
dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo


## docker 및 containerd.io 설치

dnf list docker-ce --showduplicates | sort -r

dnf install -y docker-ce-20.10.5 docker-ce-cli-20.10.5 containerd.io

systemctl enable docker --now

docker version

docker images (아무것도 출력 X)



## daemon.json 추가
vi /etc/docker/daemon.json

{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}

systemctl enable docker --now

 

 

6. Master 노드 구성

6.1 k8s 설정

## kubeadm 구성
kubeadm config images pull


## 방화벽 포트 허용
firewall-cmd --zone=public --permanent --add-port={6443,2379,2380,10250,10251,10252}/tcp


## docker 컨테이너에 Host access 허용
firewall-cmd --zone=public --permanent --add-rich-rule 'rule family=ipv4 source address=88.88.88.0/24 accept'

firewall-cmd --reload


## k8s의 CNI plugin install (Master node 1)
kubeadm init --control-plane-endpoint "88.88.88.10:8443" --upload-certs --pod-network-cidr "192.168.1.0/24"

... (중략)

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 88.88.88.10:8443 --token f4iqm8.ablgmd4fsgmxrpg1 --discovery-token-ca-cert-hash sha256:3d54851c8ce7a39c83aea562044e6bb47eefc12a3a2961dc2d6457d1275557e3  --control-plane --certificate-key b8fc486a79e710d29498eaa3e4244d33cbff6e33d8ae223d099691156b03bcef

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 88.88.88.10:8443 --token f4iqm8.ablgmd4fsgmxrpg1 --discovery-token-ca-cert-hash sha256:3d54851c8ce7a39c83aea562044e6bb47eefc12a3a2961dc2d6457d1275557e3

 

6.2 certs-config 복사

[root@hk-k8s-master1 ~]# cat deploy_certs_conf.sh
#!/bin/bash
set -e

EXITCODE=0
MASTERS="hk-k8s-master2 hk-k8s-master3"
CERTS=$(find /etc/kubernetes/pki/ -maxdepth 1 -name '*ca.*' -o -name '*sa.*')
ETCD_CERTS=$(find /etc/kubernetes/pki/etcd/ -maxdepth 1 -name '*ca.*')
for MASTER in $MASTERS; do
  ssh $MASTER mkdir -p /etc/kubernetes/pki/etcd
  scp $CERTS $MASTER:/etc/kubernetes/pki/
  scp $ETCD_CERTS $MASTER:/etc/kubernetes/pki/etcd/
  scp /etc/kubernetes/admin.conf $MASTER:/etc/kubernetes
done

exit $EXITCODE

chmod 777 deploy_certs_conf.sh

./deploy_certs_conf.sh

 

6.3 Master node 2 cluster join

kubeadm join 88.88.88.10:8443 --token f4iqm8.ablgmd4fsgmxrpg1 --discovery-token-ca-cert-hash sha256:3d54851c8ce7a39c83aea562044e6bb47eefc12a3a2961dc2d6457d1275557e3  --control-plane --certificate-key b8fc486a79e710d29498eaa3e4244d33cbff6e33d8ae223d099691156b03bcef

 

6.4 Master node 3 cluster join

kubeadm join 88.88.88.10:8443 --token f4iqm8.ablgmd4fsgmxrpg1 --discovery-token-ca-cert-hash sha256:3d54851c8ce7a39c83aea562044e6bb47eefc12a3a2961dc2d6457d1275557e3  --control-plane --certificate-key b8fc486a79e710d29498eaa3e4244d33cbff6e33d8ae223d099691156b03bcef

 

6.5 k8s node 정보 확인

kubectl get nodes

# coreDNS pending
CNI (Container Network Interface)가 설치되어있지 않아 pending 이므로 Pod Network 설치 후 Ready가 됨

 

6.6 Pod Network Plugin 설치

Network 종류별 설치 방법입니다.

# Flannel network
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml kubectl apply -f ~/k8s-images/kube-flannel.yml
kubectl get pods -o wide -w -n kube-system kubectl -n kube-system get pods -l app=flannel

# Weave network
export kubever=$(kubectl version | base64 | tr -d '\n')
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever" 
kubectl get pods --all-namespaces
kubectl apply -f ~/k8s-images/weave.yml
sudo curl -L git.io/weave -o /usr/local/bin/weave 
sudo chmod a+x /usr/local/bin/weave

# calico
kubeadm init --pod-network-cidr=192.168.1.0/24 --apiserver-advertise-address={ip address}
kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml kubectl taint nodes --all node-role.kubernetes.io/master-

 

 

7. worker 노드 구성

7.1 k8s 설정

## 방화벽 포트 허용
firewall-cmd --zone=public --permanent --add-port={10250,30000-32767}/tcp

firewall-cmd --reload


## cluster에 join
kubeadm join 88.88.88.10:8443 --token f4iqm8.ablgmd4fsgmxrpg1 --discovery-token-ca-cert-hash sha256:3d54851c8ce7a39c83aea562044e6bb47eefc12a3a2961dc2d6457d1275557e3

 

 

 

7.2 node 상태 확인

kubectl get nodes