Cree un clúster de Kubernetes multi-master con Centos 7 en la plataforma de nube de Google

4. Habilite chrony para los servicios NTP con este comando.

systemctl enable --now chronyd

2. Verifique el estado con este comando.

systemctl status chronyd

3. Verifique los orígenes NTP con este comando.

chronyc sources -v

Servidor Bastion

Paso 1. Compruebe las actualizaciones.

sudo yum check-update

Paso 2. Instalar actualizaciones si no están actualizadas.

yum update -y

Paso 3. Instale yum utilities.

yum -y install yum-utils

Paso 4. Instale el paquete haproxy.

yum install haproxy -y

Paso 5. Agregue esta configuración en valores predeterminados en /etc/haproxy/haproxy.cfg :

frontend kubernetes
    bind 10.160.x.19:6443
    option tcplog
    mode tcp
    default_backend kubernetes-master-nodes
frontend http_front
    mode http
    bind 10.160.x.19:80
    default_backend http_back
frontend https_front
    mode http
    bind 10.160.x.19:443
    default_backend https_back
backend kubernetes-master-nodes
    mode tcp
    balance roundrobin
    option tcp-check
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2
backend http_back
    mode http
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2
backend https_back
    mode http
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2

listen stats
    bind 10.160.x.19:8080
    mode http
    stats enable
    stats uri /

Paso 6. Verifique el archivo de configuración para que se vea como este comando vi /etc/haproxy/haproxy.cfg:

---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend kubernetes
    bind 10.160.x.19:6443
    option tcplog
    mode tcp
    default_backend kubernetes-master-nodes
frontend http_front
    mode http
    bind 10.160.x.19:80
    default_backend http_back
frontend https_front
    mode http
    bind 10.160.x.19:443
    default_backend https_back
backend kubernetes-master-nodes
    mode tcp
    balance roundrobin
    option tcp-check
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2
backend http_back
    mode http
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2
backend https_back
    mode http
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2

listen stats
    bind 10.160.x.19:8080
    mode http
    stats enable
    stats uri /

Paso 7. Verifique el estado de haproxy:

[root@k8-loadbalancer vapadala]# systemctl status haproxy 
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-10-26 08:33:17 UTC; 6s ago
 Main PID: 30985 (haproxy-systemd)
   CGroup: /system.slice/haproxy.service
           ├─30985 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           ├─30986 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           └─30987 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Oct 26 08:33:17 k8-loadbalancer systemd[1]: Started HAProxy Load Balancer.
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /r...pid -Ds
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: [WARNING] 298/083317 (30986) : config : 'option forwardfor' ignored for frontend 'kube...P mode.
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: [WARNING] 298/083317 (30986) : config : 'option forwardfor' ignored for backend 'kuber...P mode.
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8-loadbalancer vapadala]# 

Posibles errores

1. El servicio HAProxy se encuentra en estado de error después de realizar cambios de configuración en haproxy.cfg. Por ejemplo;

[root@k8-loadbalancer vapadala]# systemctl status haproxy 
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2022-10-26 08:29:23 UTC; 3min 44s ago
  Process: 30951 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid $OPTIONS (code=exited, status=1/FAILURE)
 Main PID: 30951 (code=exited, status=1/FAILURE)

Oct 26 08:29:23 k8-loadbalancer systemd[1]: Started HAProxy Load Balancer.
Oct 26 08:29:23 k8-loadbalancer haproxy-systemd-wrapper[30951]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /r...pid -Ds
Oct 26 08:29:23 k8-loadbalancer haproxy-systemd-wrapper[30951]: [WARNING] 298/082923 (30952) : config : 'option forwardfor' ignored for frontend 'kube...P mode.
Oct 26 08:29:23 k8-loadbalancer haproxy-systemd-wrapper[30951]: [WARNING] 298/082923 (30952) : config : 'option forwardfor' ignored for backend 'kuber...P mode.
Oct 26 08:29:23 k8-loadbalancer haproxy-systemd-wrapper[30951]: [ALERT] 298/082923 (30952) : Starting frontend kubernetes: cannot bind socket [10.160.x.19:6443].
Oct 26 08:29:23 k8-loadbalancer haproxy-systemd-wrapper[30951]: haproxy-systemd-wrapper: exit, haproxy RC=1.
Oct 26 08:29:23 k8-loadbalancer systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE.
Oct 26 08:29:23 k8-loadbalancer systemd[1]: Unit haproxy.service entered failed state.
Oct 26 08:29:23 k8-loadbalancer systemd[1]: haproxy.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

Resolución

1. Set the boolean value for haproxy_connect_any to true.
2. Restart the haproxy service.
3. Verify the status.

Ejecución:

[root@k8-loadbalancer vapadala]# setsebool -P haproxy_connect_any=1
[root@k8-loadbalancer vapadala]# systemctl restart haproxy 
[root@k8-loadbalancer vapadala]# systemctl status haproxy 
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-10-26 08:33:17 UTC; 6s ago
 Main PID: 30985 (haproxy-systemd)
   CGroup: /system.slice/haproxy.service
           ├─30985 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           ├─30986 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           └─30987 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Oct 26 08:33:17 k8-loadbalancer systemd[1]: Started HAProxy Load Balancer.
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /r...pid -Ds.
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: [WARNING] 298/083317 (30986) : config : 'option forwardfor' ignored for frontend 'kube...P mode.
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: [WARNING] 298/083317 (30986) : config : 'option forwardfor' ignored for backend 'kuber...P mode.
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8-loadbalancer vapadala]# 

Instalar Docker en nodos maestros y de trabajo

Paso 1. Compruebe las actualizaciones.

sudo yum check-update

Paso 2. Instalar actualizaciones si no están actualizadas.

yum update -y

Paso 3. Instale yum utilities.

yum -y install yum-utils

Paso 4. Instale Docker.

curl -fsSL https://get.docker.com/ | sh

Paso 5. Habilite docker.

systemctl enable --now docker

Paso 6. Inicie el servicio de acoplamiento.

sudo systemctl start docker

Paso 7. Compruebe el estado del acoplador.

sudo systemctl status docker

Salida:

[root@kube-master1 vapadala]# sudo systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-10-25 10:44:28 UTC; 40s ago
     Docs: https://docs.docker.com
 Main PID: 4275 (dockerd)
    Tasks: 21
   Memory: 35.2M
   CGroup: /system.slice/docker.service
           └─4275 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Oct 25 10:44:26 kube-master1 dockerd[4275]: time="2022-10-25T10:44:26.128233809Z" level=info msg="scheme \"unix\" not registered, fallback to defaul...dule=grpc.
Oct 25 10:44:26 kube-master1 dockerd[4275]: time="2022-10-25T10:44:26.128251910Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:/...dule=grpc.
Oct 25 10:44:26 kube-master1 dockerd[4275]: time="2022-10-25T10:44:26.128260953Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc.
Oct 25 10:44:27 kube-master1 dockerd[4275]: time="2022-10-25T10:44:27.920336293Z" level=info msg="Loading containers: start."
Oct 25 10:44:28 kube-master1 dockerd[4275]: time="2022-10-25T10:44:28.104357517Z" level=info msg="Default bridge (docker0) is assigned with an IP ad... address".
Oct 25 10:44:28 kube-master1 dockerd[4275]: time="2022-10-25T10:44:28.163830549Z" level=info msg="Loading containers: done."
Oct 25 10:44:28 kube-master1 dockerd[4275]: time="2022-10-25T10:44:28.182833703Z" level=info msg="Docker daemon" commit=03df974 graphdriver(s)=overl...=20.10.20.
Oct 25 10:44:28 kube-master1 dockerd[4275]: time="2022-10-25T10:44:28.182939545Z" level=info msg="Daemon has completed initialization".
Oct 25 10:44:28 kube-master1 systemd[1]: Started Docker Application Container Engine.
Oct 25 10:44:28 kube-master1 dockerd[4275]: time="2022-10-25T10:44:28.208492147Z" level=info msg="API listen on /var/run/docker.sock".
Hint: Some lines were ellipsized, use -l to show in full.
[root@kube-master1 vapadala]# 

Instalar Kubernetes en nodos maestros y de trabajo

Paso 1. Agregar repositorio de Kubernetes.

cat <
      
      

"gpgcheck = 0" will not verify the authenticity of the package if unsigned. Production environment it is recommended to set "gpgcheck = 1"

Paso 2. inhabilitar SELinux.

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Paso 3. Instalar Kubernetes.

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Paso 4. Habilitar Kubelet.

sudo systemctl enable --now kubelet

Paso 5. Configurar Pod Network.

kubeadm init --control-plane-endpoint "10.160.x.19:6443" --upload-certs

Paso 6. Generación de token:

Salida para el maestro (plano de control) y para los nodos de trabajo.

Nodo maestro

Token: El plano de control de Kubernetes se ha inicializado correctamente.

Para utilizar su cluster, ejecute esto como un usuario regular:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternativamente, si usted es el usuario root, puede ejecutar.

export KUBECONFIG=/etc/kubernetes/admin.conf

Ahora puede implementar una red de grupo de dispositivos en el clúster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Ahora puede unir cualquier número de nodos del plano de control o nodos maestros que ejecuten este comando en cada uno como raíz.

Referencia: flujo de trabajo de incorporación de kubeadm

kubeadm join 10.160.0.19:6443 --token 5fv6ce.h07kyelronuy0mw2 \
        --discovery-token-ca-cert-hash sha256:b5509b6fe784561f3435bf6b909dc8877e567c70921b21e35c464eb61d9527d0 \
        --control-plane --certificate-key 66773b960199ef4530461ef4014e1432066902d4a3dee01669d8622579731701

Nota: Tenga en cuenta que la clave del certificado da acceso a los datos confidenciales del clúster, ¡manténgala en secreto!

Como medida de seguridad, los certificados cargados se eliminan en dos horas; si es necesario, puede utilizarlos.

"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

A continuación, puede unirse a cualquier número de nodos de trabajo y ejecutarlo en cada uno como raíz.

kubeadm join 10.160.0.19:6443 --token 5fv6ce.h07kyelronuy0mw2 \
        --discovery-token-ca-cert-hash sha256:b5509b6fe784561f3435bf6b909dc8877e567c70921b21e35c464eb61d9527d0 

Paso 7. Verifique el servicio DNS principal.

[root@kube-master1 vapadala]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS    RESTARTS         AGE
kube-system   calico-kube-controllers-59697b644f-v2f22   1/1     Running   0                32m
kube-system   calico-node-gdwr6                          1/1     Running   0                5m54s
kube-system   calico-node-zszbc                          1/1     Running   11 (5m22s ago)   32m
kube-system   calico-typha-6944f58589-q9jxf              1/1     Running   0                32m
kube-system   coredns-565d847f94-cwgj8                   1/1     Running   0                58m
kube-system   coredns-565d847f94-tttpq                   1/1     Running   0                58m
kube-system   etcd-kube-master1                          1/1     Running   0                59m
kube-system   kube-apiserver-kube-master1                1/1     Running   0                59m
kube-system   kube-controller-manager-kube-master1       1/1     Running   0                59m
kube-system   kube-proxy-flgvq                           1/1     Running   0                5m54s
kube-system   kube-proxy-hf5qv                           1/1     Running   0                58m
kube-system   kube-scheduler-kube-master1                1/1     Running   0                59m
[root@kube-master1 vapadala]# 

Paso 8. Compruebe el estado del nodo maestro.

[root@kube-master1 vapadala]# kubectl get nodes
NAME           STATUS     ROLES           AGE   VERSION
kube-master1   Ready   control-plane   11m   v1.25.3

Para unir varios nodos maestros, este comando join se ejecuta en los nodos maestros.

kubeadm join 10.160.x.19:6443 --token 5fv6ce.h07kyelronuy0mw2 \
        --discovery-token-ca-cert-hash sha256:b5509b6fe784561f3435bf6b909dc8877e567c70921b21e35c464eb61d9527d0 \
        --control-plane --certificate-key 66773b960199ef4530461ef4014e1432066902d4a3dee01669d8622579731701

Posibles errores encontrados en el momento de la generación del token

Error CRI

[root@kube-master1 vapadala]# kubeadm init --control-plane-endpoint "10.160.x.19:6443" --upload-certs
[init] Using Kubernetes version: v1.25.3
[preflight] Running pre-flight checks
        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2022-10-25T08:56:42Z" level=fatal msg="unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/containerd/containerd.sock: connect: no such file or directory\"", error: exit status 1.
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

Resolución CRI de error

Paso 1. Quite el archivo config.toml y reinicie el contenedor.

rm -rf /etc/containerd/config.toml
systemctl restart containerd
kubeadm init

Paso 2. Instalar contenido:
Instale el paquete containerd.io desde los repositorios Docker oficiales con estos comandos.

yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo.
yum install -y containerd.io

Paso 3. Configurar contenido:

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

Paso 4. Reiniciar contenido:

systemctl restart containerd

Error FileContent-proc-sys-net-ipv4-ip_forward

[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

Error FileContent-proc-sys-net-ipv4-ip_forward Resolution

Establezca el valor ip_forward en 1.

echo 1 > /proc/sys/net/ipv4/ip_forward

El servicio DNS principal no se ejecuta

[root@kube-master1 vapadala]# kubectl get nodes
NAME           STATUS     ROLES           AGE   VERSION
kube-master1   NotReady   control-plane   11m   v1.25.3
[root@kube-master1 vapadala]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-565d847f94-cwgj8                       0/1         Pending   0                12m
kube-system   coredns-565d847f94-tttpq                        0/1         Pending   0                12m
kube-system   etcd-kube-master1                                    1/1         Running   0                12m
kube-system   kube-apiserver-kube-master1                   1/1         Running   0                12m
kube-system   kube-controller-manager-kube-master1  1/1         Running   0                12m
kube-system   kube-proxy-hf5qv                                      1/1        Running   0                 12m
kube-system   kube-scheduler-kube-master1                  1/1        Running   0                 12m
[root@kube-master1 vapadala]# 

Resolución

El DNS principal está pendiente, lo que indica un problema con la red. Por lo tanto, Calico necesita ser instalado.

Referencia: Calico

Ejecute estos dos comandos:

curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.3/manifests/calico-typha.yaml -o calico.yaml
kubectl apply -f calico.yaml

Nodo de trabajo

En el nodo de trabajo cuando el token se obtiene del maestro:

Paso 1. Habilitar servicio kubelet.

sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl restart kubelet
systemctl enable kubelet.service

Paso 2. Unir todos los nodos de trabajo con el nodo maestro con este comando join.

kubeadm join 10.160.x.19:6443 --token 5fv6ce.h07kyelronuy0mw2 \
        --discovery-token-ca-cert-hash sha256:b5509b6fe784561f3435bf6b909dc8877e567c70921b21e35c464eb61d9527d0 

Paso 3. Si se encuentran errores relacionados con archivos o puertos, restablezca el kubeadm con este comando.

kubeadm reset

Después de restablecer, introduzca el token del nodo maestro.

kubeadm join 10.160.x.19:6443 --token 5fv6ce.h07kyelronuy0mw2 \
        --discovery-token-ca-cert-hash sha256:b5509b6fe784561f3435bf6b909dc8877e567c70921b21e35c464eb61d9527d0

Resultado final

El clúster está ahora formado, verifíquelo con el comando kubectl get nodes.

resultado de clúster de kubernetes de alta disponibilidad

Nota: En el momento de la formación del cluster, sólo se configura el control desde los nodos maestros. El host bastión no está configurado como servidor centralizado para monitorear todos los Kube del clúster.