MOSIP v3 On-Prem set up issues

Developer
1

Hi all.

I’m trying to set up MOSIP v3 On-Prem. and I’m facing issues:

Personal computer : Mac OS Ventura (13.5)

K8’s infra repo with tag : 1.2.0.1-B3
mosip-infra with tag : 1.2.0.1-B3

kubectl version
Client Version: v1.28.3
Server Version: v1.26.8

helm version : v3.13.1
istio :1.19.3

Running RKE version: v1.4.10
ansible [core 2.15.5]

I’ve encountered different issues, below:

  1. Missing PV provisioner support responsible for dynamically creating and managing PVs on the infrastructure’s storage resources when Persistent Volume Claims (PVCs) are made by applications.

    I was installing and PVCs were stuck in pending state, Until I had to manually create PVs. I’m thinking that is not how it should be.

  2. While going through Modules deployment list and installing each one by one. I face 2 errors on almost every module installation:

    1. Error: sed: 1: “copy_cm.sh”: command c expects \ followed by text
      In every install.sh file, I ended up having to add ‘’ after -i for it to run. Like : sed -i ‘’ ‘s/\r$//’ copy_cm.sh

    2. Second error I’m encountering installing modules is that they are stuck waiting for deployment rollout to finish. below is the installation logs messages

       kubectl -n packetmanager rollout status deployment.apps/packetmanager
 Waiting for deployment "packetmanager" rollout to finish: 0 of 1 updated replicas are available...



 kubectl -n datashare rollout status deployment.apps/datashare
 Waiting for deployment "datashare" rollout to finish: 0 of 1 updated replicas are available...


 Installing prereg rate-control Envoyfilter
 envoyfilter.networking.istio.io/local-rate-limit created
 kubectl -n prereg rollout status deployment.apps/prereg-application
 Waiting for deployment "prereg-application" rollout to finish: 0 of 1 updated replicas are available...


 kubectl -n pms rollout status deployment.apps/pmp-ui
 Waiting for deployment "pmp-ui" rollout to finish: 0 of 1 updated replicas are available...
 deployment "pmp-ui" successfully rolled out
 kubectl -n pms rollout status deployment.apps/pms-partner
 Waiting for deployment "pms-partner" rollout to finish: 0 of 1

Kindly provide me with guidance that would help me with these issues.
Thank you.

2

Hi @yahlife_nation

Please @ckm007 look into this !

Best Regards,
Team MOSIP

3

@yahlife_nation we are still testing with latest rke version 1.4.10 .
Our last release i.e. 1.2.0.1-B3 is released with:

  1. rke: 1.3.10
  2. k8 version: 1.22.9
  3. istio : 1.15

Check this:

1 Like
4

@ckm007 rke version v1.3.10 does not suppport docker version 24.0.5 which installed by default using ansible-playbook -i hosts.ini docker.yaml

kubectl version: v1.22.9
istioctl version: 1.15.0
rke version : v1.3.10
ansible [core 2.13.13]
Personal pc : ubuntu 20.04 LTS

Observation cluster 2 VMs running : Ubuntu 20.04 LTS

Full logs below :

rke up
INFO[0000] Running RKE version: v1.3.10
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.132.0.18]
INFO[0000] [dialer] Setup tunnel for host [10.132.0.17]
WARN[0002] [state] can’t fetch legacy cluster state from Kubernetes: Unsupported Docker version found [24.0.5] on host [10.132.0.18], supported versions are [1.13.x 17.03.x 17.06.x 17.09.x 18.06.x 18.09.x 19.03.x 20.10.x]
INFO[0002] [certificates] Generating CA kubernetes certificates
INFO[0002] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates
INFO[0002] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0002] [certificates] Generating Kubernetes API server certificates
INFO[0003] [certificates] Generating Service account token key
INFO[0003] [certificates] Generating Kube Controller certificates
INFO[0003] [certificates] Generating Kube Scheduler certificates
INFO[0004] [certificates] Generating Kube Proxy certificates
INFO[0004] [certificates] Generating Node certificate
INFO[0004] [certificates] Generating admin certificates and kubeconfig
INFO[0004] [certificates] Generating Kubernetes API server proxy client certificates
INFO[0004] [certificates] Generating kube-etcd-10-132-0-17 certificate and key
INFO[0005] [certificates] Generating kube-etcd-10-132-0-18 certificate and key
INFO[0005] Successfully Deployed state file at [./cluster.rkestate]
INFO[0005] Building Kubernetes cluster
INFO[0005] [dialer] Setup tunnel for host [10.132.0.18]
INFO[0005] [dialer] Setup tunnel for host [10.132.0.17]
FATA[0007] Unsupported Docker version found [24.0.5] on host [10.132.0.18], supported versions are [1.13.x 17.03.x 17.06.x 17.09.x 18.06.x 18.09.x 19.03.x 20.10.x]














As a workaround I tried rke up --ignore-docker-version but still doesnt work. it give this error :

rke up --ignore-docker-version
INFO[0000] Running RKE version: v1.3.10
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0000] [certificates] Generating admin certificates and kubeconfig
INFO[0000] Successfully Deployed state file at [./cluster.rkestate]
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.132.0.17]
INFO[0000] [dialer] Setup tunnel for host [10.132.0.18]
INFO[0002] [network] Deploying port listener containers
INFO[0002] Pulling image [rancher/rke-tools:v0.1.80] on host [10.132.0.17], try #1
INFO[0002] Pulling image [rancher/rke-tools:v0.1.80] on host [10.132.0.18], try #1
INFO[0009] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0010] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0012] Starting container [rke-etcd-port-listener] on host [10.132.0.17], try #1
INFO[0012] Starting container [rke-etcd-port-listener] on host [10.132.0.18], try #1
INFO[0012] [network] Successfully started [rke-etcd-port-listener] container on host [10.132.0.17]
INFO[0013] [network] Successfully started [rke-etcd-port-listener] container on host [10.132.0.18]
INFO[0013] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0013] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0014] Starting container [rke-cp-port-listener] on host [10.132.0.17], try #1
INFO[0014] Starting container [rke-cp-port-listener] on host [10.132.0.18], try #1
INFO[0014] [network] Successfully started [rke-cp-port-listener] container on host [10.132.0.17]
INFO[0014] [network] Successfully started [rke-cp-port-listener] container on host [10.132.0.18]
INFO[0014] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0014] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0015] Starting container [rke-worker-port-listener] on host [10.132.0.18], try #1
INFO[0015] Starting container [rke-worker-port-listener] on host [10.132.0.17], try #1
INFO[0016] [network] Successfully started [rke-worker-port-listener] container on host [10.132.0.18]
INFO[0016] [network] Successfully started [rke-worker-port-listener] container on host [10.132.0.17]
INFO[0016] [network] Port listener containers deployed successfully
INFO[0016] [network] Running etcd ↔ etcd port checks
INFO[0016] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379 2380], try #1
INFO[0016] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379 2380], try #1
INFO[0016] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0016] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0017] Starting container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0017] Starting container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0017] [network] Successfully started [rke-port-checker] container on host [10.132.0.17]
INFO[0017] [network] Successfully started [rke-port-checker] container on host [10.132.0.18]
INFO[0017] Removing container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0017] Removing container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0017] [network] Running control plane → etcd port checks
INFO[0017] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379], try #1
INFO[0017] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379], try #1
INFO[0018] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0018] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0018] Starting container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0018] Starting container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0019] [network] Successfully started [rke-port-checker] container on host [10.132.0.17]
INFO[0019] [network] Successfully started [rke-port-checker] container on host [10.132.0.18]
INFO[0019] Removing container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0019] Removing container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0019] [network] Running control plane → worker port checks
INFO[0019] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [10250], try #1
INFO[0019] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [10250], try #1
INFO[0019] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0019] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0020] Starting container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0020] Starting container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0020] [network] Successfully started [rke-port-checker] container on host [10.132.0.17]
INFO[0020] [network] Successfully started [rke-port-checker] container on host [10.132.0.18]
INFO[0021] Removing container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0021] Removing container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0021] [network] Running workers → control plane port checks
INFO[0021] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [6443], try #1
INFO[0021] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [6443], try #1
INFO[0021] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0021] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0022] Starting container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0022] Starting container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0022] [network] Successfully started [rke-port-checker] container on host [10.132.0.17]
INFO[0022] [network] Successfully started [rke-port-checker] container on host [10.132.0.18]
INFO[0022] Removing container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0022] Removing container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0022] [network] Checking KubeAPI port Control Plane hosts
INFO[0023] [network] Removing port listener containers
INFO[0023] Removing container [rke-etcd-port-listener] on host [10.132.0.17], try #1
INFO[0023] Removing container [rke-etcd-port-listener] on host [10.132.0.18], try #1
INFO[0023] [remove/rke-etcd-port-listener] Successfully removed container on host [10.132.0.18]
INFO[0023] [remove/rke-etcd-port-listener] Successfully removed container on host [10.132.0.17]
INFO[0023] Removing container [rke-cp-port-listener] on host [10.132.0.18], try #1
INFO[0023] Removing container [rke-cp-port-listener] on host [10.132.0.17], try #1
INFO[0024] [remove/rke-cp-port-listener] Successfully removed container on host [10.132.0.18]
INFO[0024] [remove/rke-cp-port-listener] Successfully removed container on host [10.132.0.17]
INFO[0024] Removing container [rke-worker-port-listener] on host [10.132.0.17], try #1
INFO[0024] Removing container [rke-worker-port-listener] on host [10.132.0.18], try #1
INFO[0024] [remove/rke-worker-port-listener] Successfully removed container on host [10.132.0.17]
INFO[0024] [remove/rke-worker-port-listener] Successfully removed container on host [10.132.0.18]
INFO[0024] [network] Port listener containers removed successfully
INFO[0024] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0024] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1
INFO[0024] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1
INFO[0024] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0024] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0025] Starting container [cert-deployer] on host [10.132.0.17], try #1
INFO[0025] Starting container [cert-deployer] on host [10.132.0.18], try #1
INFO[0026] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1
INFO[0026] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1
INFO[0031] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1
INFO[0031] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1
INFO[0031] Removing container [cert-deployer] on host [10.132.0.17], try #1
INFO[0031] Removing container [cert-deployer] on host [10.132.0.18], try #1
INFO[0031] [reconcile] Rebuilding and updating local kube config
INFO[0031] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
WARN[0061] [reconcile] host [10.132.0.17] is a control plane node without reachable Kubernetes API endpoint in the cluster
INFO[0061] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
WARN[0091] [reconcile] host [10.132.0.18] is a control plane node without reachable Kubernetes API endpoint in the cluster
WARN[0091] [reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found
INFO[0091] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0091] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.132.0.17]
INFO[0093] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0094] Starting container [file-deployer] on host [10.132.0.17], try #1
INFO[0094] Successfully started [file-deployer] container on host [10.132.0.17]
INFO[0094] Waiting for [file-deployer] container to exit on host [10.132.0.17]
INFO[0094] Waiting for [file-deployer] container to exit on host [10.132.0.17]
INFO[0095] Removing container [file-deployer] on host [10.132.0.17], try #1
INFO[0095] [remove/file-deployer] Successfully removed container on host [10.132.0.17]
INFO[0095] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.132.0.18]
INFO[0097] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0097] Starting container [file-deployer] on host [10.132.0.18], try #1
INFO[0098] Successfully started [file-deployer] container on host [10.132.0.18]
INFO[0098] Waiting for [file-deployer] container to exit on host [10.132.0.18]
INFO[0098] Waiting for [file-deployer] container to exit on host [10.132.0.18]
INFO[0098] Removing container [file-deployer] on host [10.132.0.18], try #1
INFO[0098] [remove/file-deployer] Successfully removed container on host [10.132.0.18]
INFO[0098] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes
INFO[0098] [reconcile] Reconciling cluster state
INFO[0098] [reconcile] This is newly generated cluster
INFO[0098] Pre-pulling kubernetes images
INFO[0098] Pulling image [rancher/hyperkube:v1.22.9-rancher1] on host [10.132.0.17], try #1
INFO[0098] Pulling image [rancher/hyperkube:v1.22.9-rancher1] on host [10.132.0.18], try #1
INFO[0166] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17]
INFO[0169] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18]
INFO[0169] Kubernetes images pulled successfully
INFO[0169] [etcd] Building up etcd plane…
INFO[0169] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0175] Starting container [etcd-fix-perm] on host [10.132.0.17], try #1
INFO[0175] Successfully started [etcd-fix-perm] container on host [10.132.0.17]
INFO[0175] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.17]
INFO[0175] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.17]
INFO[0175] Removing container [etcd-fix-perm] on host [10.132.0.17], try #1
INFO[0176] [remove/etcd-fix-perm] Successfully removed container on host [10.132.0.17]
INFO[0176] Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.132.0.17], try #1
INFO[0183] Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.132.0.17]
INFO[0185] Starting container [etcd] on host [10.132.0.17], try #1
INFO[0185] [etcd] Successfully started [etcd] container on host [10.132.0.17]
INFO[0185] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.132.0.17]
INFO[0185] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0186] Starting container [etcd-rolling-snapshots] on host [10.132.0.17], try #1
INFO[0187] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.132.0.17]
INFO[0192] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0193] Starting container [rke-bundle-cert] on host [10.132.0.17], try #1
INFO[0193] [certificates] Successfully started [rke-bundle-cert] container on host [10.132.0.17]
INFO[0193] Waiting for [rke-bundle-cert] container to exit on host [10.132.0.17]
INFO[0193] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.132.0.17]
INFO[0193] Removing container [rke-bundle-cert] on host [10.132.0.17], try #1
INFO[0194] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0195] Starting container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0195] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.17]
INFO[0195] Removing container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0196] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17]
INFO[0196] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0197] Starting container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0197] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.17]
INFO[0197] Removing container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0198] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17]
INFO[0198] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0199] Starting container [etcd-fix-perm] on host [10.132.0.18], try #1
INFO[0200] Successfully started [etcd-fix-perm] container on host [10.132.0.18]
INFO[0200] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.18]
INFO[0200] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.18]
INFO[0200] Removing container [etcd-fix-perm] on host [10.132.0.18], try #1
INFO[0200] [remove/etcd-fix-perm] Successfully removed container on host [10.132.0.18]
INFO[0200] Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.132.0.18], try #1
INFO[0208] Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.132.0.18]
INFO[0210] Starting container [etcd] on host [10.132.0.18], try #1
INFO[0210] [etcd] Successfully started [etcd] container on host [10.132.0.18]
INFO[0210] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.132.0.18]
INFO[0210] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0211] Starting container [etcd-rolling-snapshots] on host [10.132.0.18], try #1
INFO[0211] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.132.0.18]
INFO[0217] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0217] Starting container [rke-bundle-cert] on host [10.132.0.18], try #1
INFO[0218] [certificates] Successfully started [rke-bundle-cert] container on host [10.132.0.18]
INFO[0218] Waiting for [rke-bundle-cert] container to exit on host [10.132.0.18]
INFO[0218] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.132.0.18]
INFO[0218] Removing container [rke-bundle-cert] on host [10.132.0.18], try #1
INFO[0219] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0220] Starting container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0220] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.18]
INFO[0220] Removing container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0220] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18]
INFO[0221] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0222] Starting container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0222] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.18]
INFO[0222] Removing container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0222] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18]
INFO[0222] [etcd] Successfully started etcd plane… Checking etcd cluster health
INFO[0225] [etcd] etcd host [10.132.0.17] reported healthy=true
INFO[0225] [controlplane] Building up Controller Plane…
INFO[0225] Checking if container [service-sidekick] is running on host [10.132.0.17], try #1
INFO[0225] Checking if container [service-sidekick] is running on host [10.132.0.18], try #1
INFO[0225] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0225] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0226] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18]
INFO[0226] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17]
INFO[0226] Starting container [kube-apiserver] on host [10.132.0.18], try #1
INFO[0226] Starting container [kube-apiserver] on host [10.132.0.17], try #1
INFO[0226] [controlplane] Successfully started [kube-apiserver] container on host [10.132.0.18]
INFO[0226] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.132.0.18]
INFO[0226] [controlplane] Successfully started [kube-apiserver] container on host [10.132.0.17]
INFO[0226] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.132.0.17]
INFO[0236] [healthcheck] service [kube-apiserver] on host [10.132.0.17] is healthy
INFO[0236] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0236] [healthcheck] service [kube-apiserver] on host [10.132.0.18] is healthy
INFO[0237] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0237] Starting container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0238] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17]
INFO[0238] Starting container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0238] Removing container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0238] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17]
INFO[0238] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17]
INFO[0238] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18]
INFO[0238] Starting container [kube-controller-manager] on host [10.132.0.17], try #1
INFO[0238] Removing container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0239] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18]
INFO[0239] [controlplane] Successfully started [kube-controller-manager] container on host [10.132.0.17]
INFO[0239] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.132.0.17]
INFO[0239] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18]
INFO[0239] Starting container [kube-controller-manager] on host [10.132.0.18], try #1
INFO[0239] [controlplane] Successfully started [kube-controller-manager] container on host [10.132.0.18]
INFO[0239] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.132.0.18]
INFO[0241] [healthcheck] service [kube-controller-manager] on host [10.132.0.17] is healthy
INFO[0241] [healthcheck] service [kube-controller-manager] on host [10.132.0.18] is healthy
INFO[0241] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0242] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0242] Starting container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0243] Starting container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0243] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17]
INFO[0243] Removing container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0243] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17]
INFO[0243] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18]
INFO[0243] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17]
INFO[0243] Removing container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0243] Starting container [kube-scheduler] on host [10.132.0.17], try #1
INFO[0243] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18]
INFO[0244] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18]
INFO[0244] [controlplane] Successfully started [kube-scheduler] container on host [10.132.0.17]
INFO[0244] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.132.0.17]
INFO[0244] Starting container [kube-scheduler] on host [10.132.0.18], try #1
INFO[0244] [controlplane] Successfully started [kube-scheduler] container on host [10.132.0.18]
INFO[0244] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.132.0.18]
INFO[0246] [healthcheck] service [kube-scheduler] on host [10.132.0.17] is healthy
INFO[0246] [healthcheck] service [kube-scheduler] on host [10.132.0.18] is healthy
INFO[0246] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0246] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0247] Starting container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0247] Starting container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0247] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17]
INFO[0247] Removing container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0248] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17]
INFO[0248] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18]
INFO[0248] Removing container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0248] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18]
INFO[0248] [controlplane] Successfully started Controller Plane…
INFO[0248] [authz] Creating rke-job-deployer ServiceAccount
FATA[0423] Failed to apply the ServiceAccount needed for job execution: Post “https://10.132.0.18:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=30s”: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

5

Hi @yahlife_nation

I have shared your issue with our deployment team they will look into this tomorrow due to Sat/Sun the team members were not available.

On Monday they will provide you with a solution so can move ahead and setup On-Prem.

6

@yahlife_nation

  • Run rke remove
  • Install docker with specific version 20.10.xxx on all cluster nodes.
  • Run rke up

and

7

@syed.salman @sanchi-singh24

I did that using

  • rke version you provided me v1.3.10
  • docker version 20.10.10

and rke up give this error :

INFO[0000] Running RKE version: v1.3.10
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0000] [certificates] Generating admin certificates and kubeconfig
INFO[0000] Successfully Deployed state file at [./cluster.rkestate]
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.132.0.17]
INFO[0000] [dialer] Setup tunnel for host [10.132.0.18]
INFO[0002] [network] Deploying port listener containers
INFO[0002] Pulling image [rancher/rke-tools:v0.1.80] on host [10.132.0.17], try #1
INFO[0002] Pulling image [rancher/rke-tools:v0.1.80] on host [10.132.0.18], try #1
INFO[0009] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0010] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0012] Starting container [rke-etcd-port-listener] on host [10.132.0.17], try #1
INFO[0012] Starting container [rke-etcd-port-listener] on host [10.132.0.18], try #1
INFO[0012] [network] Successfully started [rke-etcd-port-listener] container on host [10.132.0.17]
INFO[0013] [network] Successfully started [rke-etcd-port-listener] container on host [10.132.0.18]
INFO[0013] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0013] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0014] Starting container [rke-cp-port-listener] on host [10.132.0.17], try #1
INFO[0014] Starting container [rke-cp-port-listener] on host [10.132.0.18], try #1
INFO[0014] [network] Successfully started [rke-cp-port-listener] container on host [10.132.0.17]
INFO[0014] [network] Successfully started [rke-cp-port-listener] container on host [10.132.0.18]
INFO[0014] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0014] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0015] Starting container [rke-worker-port-listener] on host [10.132.0.18], try #1
INFO[0015] Starting container [rke-worker-port-listener] on host [10.132.0.17], try #1
INFO[0016] [network] Successfully started [rke-worker-port-listener] container on host [10.132.0.18]
INFO[0016] [network] Successfully started [rke-worker-port-listener] container on host [10.132.0.17]
INFO[0016] [network] Port listener containers deployed successfully
INFO[0016] [network] Running etcd ↔ etcd port checks
INFO[0016] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379 2380], try #1
INFO[0016] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379 2380], try #1
INFO[0016] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0016] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0017] Starting container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0017] Starting container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0017] [network] Successfully started [rke-port-checker] container on host [10.132.0.17]
INFO[0017] [network] Successfully started [rke-port-checker] container on host [10.132.0.18]
INFO[0017] Removing container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0017] Removing container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0017] [network] Running control plane → etcd port checks
INFO[0017] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379], try #1
INFO[0017] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379], try #1
INFO[0018] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0018] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0018] Starting container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0018] Starting container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0019] [network] Successfully started [rke-port-checker] container on host [10.132.0.17]
INFO[0019] [network] Successfully started [rke-port-checker] container on host [10.132.0.18]
INFO[0019] Removing container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0019] Removing container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0019] [network] Running control plane → worker port checks
INFO[0019] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [10250], try #1
INFO[0019] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [10250], try #1
INFO[0019] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0019] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0020] Starting container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0020] Starting container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0020] [network] Successfully started [rke-port-checker] container on host [10.132.0.17]
INFO[0020] [network] Successfully started [rke-port-checker] container on host [10.132.0.18]
INFO[0021] Removing container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0021] Removing container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0021] [network] Running workers → control plane port checks
INFO[0021] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [6443], try #1
INFO[0021] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [6443], try #1
INFO[0021] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0021] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0022] Starting container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0022] Starting container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0022] [network] Successfully started [rke-port-checker] container on host [10.132.0.17]
INFO[0022] [network] Successfully started [rke-port-checker] container on host [10.132.0.18]
INFO[0022] Removing container [rke-port-checker] on host [10.132.0.17], try #1
INFO[0022] Removing container [rke-port-checker] on host [10.132.0.18], try #1
INFO[0022] [network] Checking KubeAPI port Control Plane hosts
INFO[0023] [network] Removing port listener containers
INFO[0023] Removing container [rke-etcd-port-listener] on host [10.132.0.17], try #1
INFO[0023] Removing container [rke-etcd-port-listener] on host [10.132.0.18], try #1
INFO[0023] [remove/rke-etcd-port-listener] Successfully removed container on host [10.132.0.18]
INFO[0023] [remove/rke-etcd-port-listener] Successfully removed container on host [10.132.0.17]
INFO[0023] Removing container [rke-cp-port-listener] on host [10.132.0.18], try #1
INFO[0023] Removing container [rke-cp-port-listener] on host [10.132.0.17], try #1
INFO[0024] [remove/rke-cp-port-listener] Successfully removed container on host [10.132.0.18]
INFO[0024] [remove/rke-cp-port-listener] Successfully removed container on host [10.132.0.17]
INFO[0024] Removing container [rke-worker-port-listener] on host [10.132.0.17], try #1
INFO[0024] Removing container [rke-worker-port-listener] on host [10.132.0.18], try #1
INFO[0024] [remove/rke-worker-port-listener] Successfully removed container on host [10.132.0.17]
INFO[0024] [remove/rke-worker-port-listener] Successfully removed container on host [10.132.0.18]
INFO[0024] [network] Port listener containers removed successfully
INFO[0024] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0024] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1
INFO[0024] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1
INFO[0024] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0024] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0025] Starting container [cert-deployer] on host [10.132.0.17], try #1
INFO[0025] Starting container [cert-deployer] on host [10.132.0.18], try #1
INFO[0026] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1
INFO[0026] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1
INFO[0031] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1
INFO[0031] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1
INFO[0031] Removing container [cert-deployer] on host [10.132.0.17], try #1
INFO[0031] Removing container [cert-deployer] on host [10.132.0.18], try #1
INFO[0031] [reconcile] Rebuilding and updating local kube config
INFO[0031] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
WARN[0061] [reconcile] host [10.132.0.17] is a control plane node without reachable Kubernetes API endpoint in the cluster
INFO[0061] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
WARN[0091] [reconcile] host [10.132.0.18] is a control plane node without reachable Kubernetes API endpoint in the cluster
WARN[0091] [reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found
INFO[0091] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0091] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.132.0.17]
INFO[0093] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0094] Starting container [file-deployer] on host [10.132.0.17], try #1
INFO[0094] Successfully started [file-deployer] container on host [10.132.0.17]
INFO[0094] Waiting for [file-deployer] container to exit on host [10.132.0.17]
INFO[0094] Waiting for [file-deployer] container to exit on host [10.132.0.17]
INFO[0095] Removing container [file-deployer] on host [10.132.0.17], try #1
INFO[0095] [remove/file-deployer] Successfully removed container on host [10.132.0.17]
INFO[0095] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.132.0.18]
INFO[0097] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0097] Starting container [file-deployer] on host [10.132.0.18], try #1
INFO[0098] Successfully started [file-deployer] container on host [10.132.0.18]
INFO[0098] Waiting for [file-deployer] container to exit on host [10.132.0.18]
INFO[0098] Waiting for [file-deployer] container to exit on host [10.132.0.18]
INFO[0098] Removing container [file-deployer] on host [10.132.0.18], try #1
INFO[0098] [remove/file-deployer] Successfully removed container on host [10.132.0.18]
INFO[0098] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes
INFO[0098] [reconcile] Reconciling cluster state
INFO[0098] [reconcile] This is newly generated cluster
INFO[0098] Pre-pulling kubernetes images
INFO[0098] Pulling image [rancher/hyperkube:v1.22.9-rancher1] on host [10.132.0.17], try #1
INFO[0098] Pulling image [rancher/hyperkube:v1.22.9-rancher1] on host [10.132.0.18], try #1
INFO[0166] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17]
INFO[0169] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18]
INFO[0169] Kubernetes images pulled successfully
INFO[0169] [etcd] Building up etcd plane…
INFO[0169] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0175] Starting container [etcd-fix-perm] on host [10.132.0.17], try #1
INFO[0175] Successfully started [etcd-fix-perm] container on host [10.132.0.17]
INFO[0175] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.17]
INFO[0175] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.17]
INFO[0175] Removing container [etcd-fix-perm] on host [10.132.0.17], try #1
INFO[0176] [remove/etcd-fix-perm] Successfully removed container on host [10.132.0.17]
INFO[0176] Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.132.0.17], try #1
INFO[0183] Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.132.0.17]
INFO[0185] Starting container [etcd] on host [10.132.0.17], try #1
INFO[0185] [etcd] Successfully started [etcd] container on host [10.132.0.17]
INFO[0185] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.132.0.17]
INFO[0185] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0186] Starting container [etcd-rolling-snapshots] on host [10.132.0.17], try #1
INFO[0187] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.132.0.17]
INFO[0192] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0193] Starting container [rke-bundle-cert] on host [10.132.0.17], try #1
INFO[0193] [certificates] Successfully started [rke-bundle-cert] container on host [10.132.0.17]
INFO[0193] Waiting for [rke-bundle-cert] container to exit on host [10.132.0.17]
INFO[0193] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.132.0.17]
INFO[0193] Removing container [rke-bundle-cert] on host [10.132.0.17], try #1
INFO[0194] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0195] Starting container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0195] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.17]
INFO[0195] Removing container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0196] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17]
INFO[0196] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0197] Starting container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0197] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.17]
INFO[0197] Removing container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0198] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17]
INFO[0198] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0199] Starting container [etcd-fix-perm] on host [10.132.0.18], try #1
INFO[0200] Successfully started [etcd-fix-perm] container on host [10.132.0.18]
INFO[0200] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.18]
INFO[0200] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.18]
INFO[0200] Removing container [etcd-fix-perm] on host [10.132.0.18], try #1
INFO[0200] [remove/etcd-fix-perm] Successfully removed container on host [10.132.0.18]
INFO[0200] Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.132.0.18], try #1
INFO[0208] Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.132.0.18]
INFO[0210] Starting container [etcd] on host [10.132.0.18], try #1
INFO[0210] [etcd] Successfully started [etcd] container on host [10.132.0.18]
INFO[0210] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.132.0.18]
INFO[0210] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0211] Starting container [etcd-rolling-snapshots] on host [10.132.0.18], try #1
INFO[0211] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.132.0.18]
INFO[0217] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0217] Starting container [rke-bundle-cert] on host [10.132.0.18], try #1
INFO[0218] [certificates] Successfully started [rke-bundle-cert] container on host [10.132.0.18]
INFO[0218] Waiting for [rke-bundle-cert] container to exit on host [10.132.0.18]
INFO[0218] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.132.0.18]
INFO[0218] Removing container [rke-bundle-cert] on host [10.132.0.18], try #1
INFO[0219] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0220] Starting container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0220] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.18]
INFO[0220] Removing container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0220] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18]
INFO[0221] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0222] Starting container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0222] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.18]
INFO[0222] Removing container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0222] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18]
INFO[0222] [etcd] Successfully started etcd plane… Checking etcd cluster health
INFO[0225] [etcd] etcd host [10.132.0.17] reported healthy=true
INFO[0225] [controlplane] Building up Controller Plane…
INFO[0225] Checking if container [service-sidekick] is running on host [10.132.0.17], try #1
INFO[0225] Checking if container [service-sidekick] is running on host [10.132.0.18], try #1
INFO[0225] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0225] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0226] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18]
INFO[0226] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17]
INFO[0226] Starting container [kube-apiserver] on host [10.132.0.18], try #1
INFO[0226] Starting container [kube-apiserver] on host [10.132.0.17], try #1
INFO[0226] [controlplane] Successfully started [kube-apiserver] container on host [10.132.0.18]
INFO[0226] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.132.0.18]
INFO[0226] [controlplane] Successfully started [kube-apiserver] container on host [10.132.0.17]
INFO[0226] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.132.0.17]
INFO[0236] [healthcheck] service [kube-apiserver] on host [10.132.0.17] is healthy
INFO[0236] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0236] [healthcheck] service [kube-apiserver] on host [10.132.0.18] is healthy
INFO[0237] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0237] Starting container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0238] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17]
INFO[0238] Starting container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0238] Removing container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0238] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17]
INFO[0238] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17]
INFO[0238] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18]
INFO[0238] Starting container [kube-controller-manager] on host [10.132.0.17], try #1
INFO[0238] Removing container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0239] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18]
INFO[0239] [controlplane] Successfully started [kube-controller-manager] container on host [10.132.0.17]
INFO[0239] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.132.0.17]
INFO[0239] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18]
INFO[0239] Starting container [kube-controller-manager] on host [10.132.0.18], try #1
INFO[0239] [controlplane] Successfully started [kube-controller-manager] container on host [10.132.0.18]
INFO[0239] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.132.0.18]
INFO[0241] [healthcheck] service [kube-controller-manager] on host [10.132.0.17] is healthy
INFO[0241] [healthcheck] service [kube-controller-manager] on host [10.132.0.18] is healthy
INFO[0241] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0242] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0242] Starting container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0243] Starting container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0243] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17]
INFO[0243] Removing container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0243] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17]
INFO[0243] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18]
INFO[0243] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17]
INFO[0243] Removing container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0243] Starting container [kube-scheduler] on host [10.132.0.17], try #1
INFO[0243] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18]
INFO[0244] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18]
INFO[0244] [controlplane] Successfully started [kube-scheduler] container on host [10.132.0.17]
INFO[0244] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.132.0.17]
INFO[0244] Starting container [kube-scheduler] on host [10.132.0.18], try #1
INFO[0244] [controlplane] Successfully started [kube-scheduler] container on host [10.132.0.18]
INFO[0244] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.132.0.18]
INFO[0246] [healthcheck] service [kube-scheduler] on host [10.132.0.17] is healthy
INFO[0246] [healthcheck] service [kube-scheduler] on host [10.132.0.18] is healthy
INFO[0246] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17]
INFO[0246] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18]
INFO[0247] Starting container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0247] Starting container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0247] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17]
INFO[0247] Removing container [rke-log-linker] on host [10.132.0.17], try #1
INFO[0248] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17]
INFO[0248] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18]
INFO[0248] Removing container [rke-log-linker] on host [10.132.0.18], try #1
INFO[0248] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18]
INFO[0248] [controlplane] Successfully started Controller Plane…
INFO[0248] [authz] Creating rke-job-deployer ServiceAccount
FATA[0423] Failed to apply the ServiceAccount needed for job execution: Post “https://10.132.0.18:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=30s”: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

The only versions that worked was when I used latest RKE v1.4.10 with latest docker 24.0.5. thats when it was successful.

1 Like
8

@yahlife_nation Can you follow the instructions mentioned below:

  • Run rke remove to remove the cluster
  • Remove docker from all the nodes and rke components:
  • Clean each host from the directories left by the services:
    • /etc/kubernetes/ssl
    • /var/lib/etcd
    • /etc/cni
    • /opt/cni
    • /var/run/calico
  • Take the latest pull from mosip/k8s-infra’s main branch
  • Ensure rke tool version is v1.3.10.
  • Run ansible docker.yaml to install docker with 20.10.x version.
  • Run rke up
1 Like