# MOSIP v3 On-Prem set up issues **URL:** \ **Category:** Developer\ **Created:** [November 7, 2023, 3:20pm UTC](https://community.mosip.io/t/mosip-v3-on-prem-set-up-issues/901 "2023-11-07T15:20:58Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![yahlife\_nation](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/yahlife_nation/32/850_2.png) [@yahlife\_nation](https://community.mosip.io/u/yahlife_nation)\ **Post date:** [November 7, 2023, 3:20pm UTC](https://community.mosip.io/t/mosip-v3-on-prem-set-up-issues/901/1 "2023-11-07T15:20:58Z")
Hi all. I’m trying to set up MOSIP v3 On-Prem. and I’m facing issues: Personal computer : Mac OS Ventura (13.5) K8’s infra repo with tag : 1.2.0.1-B3 mosip-infra with tag : 1.2.0.1-B3 kubectl version Client Version: v1.28.3 Server Version: v1.26.8 helm version : v3.13.1 istio :1.19.3 Running RKE version: v1.4.10 ansible [core 2.15.5] I’ve encountered different issues, below: 1. Missing PV provisioner support responsible for dynamically creating and managing PVs on the infrastructure’s storage resources when Persistent Volume Claims (PVCs) are made by applications. 2. While going through Modules deployment list and installing each one by one. I face 2 errors on almost every module installation: Kindly provide me with guidance that would help me with these issues. Thank you. ---
**Author:** ![sanchi-singh24](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/sanchi-singh24/32/538_2.png) [@sanchi-singh24](https://community.mosip.io/u/sanchi-singh24)\ **Post date:** [November 16, 2023, 12:22pm UTC](https://community.mosip.io/t/mosip-v3-on-prem-set-up-issues/901/2 "2023-11-16T12:22:05Z")
Hi @yahlife_nation Please @ckm007 look into this ! Best Regards, Team MOSIP ---
**Author:** ![ckm007](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/ckm007/32/264_2.png) [@ckm007](https://community.mosip.io/u/ckm007)\ **Post date:** [November 16, 2023, 12:26pm UTC](https://community.mosip.io/t/mosip-v3-on-prem-set-up-issues/901/3 "2023-11-16T12:26:09Z")
@yahlife_nation we are still testing with latest rke version 1.4.10 . Our last release i.e. 1.2.0.1-B3 is released with: 1. rke: 1.3.10 2. k8 version: 1.22.9 3. istio : 1.15 Check this: > **[On-Prem Installation Guidelines](https://docs.mosip.io/1.2.0/deploymentnew/v3-installation/on-prem-installation-guidelines#pre-requisites)** ---
**Author:** ![yahlife\_nation](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/yahlife_nation/32/850_2.png) [@yahlife\_nation](https://community.mosip.io/u/yahlife_nation)\ **Post date:** [November 17, 2023, 10:52am UTC](https://community.mosip.io/t/mosip-v3-on-prem-set-up-issues/901/4 "2023-11-17T10:52:28Z")
@ckm007 rke version v1.3.10 does not suppport docker version 24.0.5 which installed by default using `ansible-playbook -i hosts.ini docker.yaml` kubectl version: v1.22.9 istioctl version: 1.15.0 rke version : v1.3.10 ansible [core 2.13.13] Personal pc : ubuntu 20.04 LTS Observation cluster 2 VMs running : Ubuntu 20.04 LTS Full logs below : rke up INFO[0000] Running RKE version: v1.3.10 INFO[0000] Initiating Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [10.132.0.18] INFO[0000] [dialer] Setup tunnel for host [10.132.0.17] WARN[0002] [state] can’t fetch legacy cluster state from Kubernetes: Unsupported Docker version found [24.0.5] on host [10.132.0.18], supported versions are [1.13.x 17.03.x 17.06.x 17.09.x 18.06.x 18.09.x 19.03.x 20.10.x] INFO[0002] [certificates] Generating CA kubernetes certificates INFO[0002] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates INFO[0002] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates INFO[0002] [certificates] Generating Kubernetes API server certificates INFO[0003] [certificates] Generating Service account token key INFO[0003] [certificates] Generating Kube Controller certificates INFO[0003] [certificates] Generating Kube Scheduler certificates INFO[0004] [certificates] Generating Kube Proxy certificates INFO[0004] [certificates] Generating Node certificate INFO[0004] [certificates] Generating admin certificates and kubeconfig INFO[0004] [certificates] Generating Kubernetes API server proxy client certificates INFO[0004] [certificates] Generating kube-etcd-10-132-0-17 certificate and key INFO[0005] [certificates] Generating kube-etcd-10-132-0-18 certificate and key INFO[0005] Successfully Deployed state file at [./cluster.rkestate] INFO[0005] Building Kubernetes cluster INFO[0005] [dialer] Setup tunnel for host [10.132.0.18] INFO[0005] [dialer] Setup tunnel for host [10.132.0.17] FATA[0007] Unsupported Docker version found [24.0.5] on host [10.132.0.18], supported versions are [1.13.x 17.03.x 17.06.x 17.09.x 18.06.x 18.09.x 19.03.x 20.10.x] As a workaround I tried `rke up --ignore-docker-version` but still doesnt work. it give this error : rke up --ignore-docker-version INFO[0000] Running RKE version: v1.3.10 INFO[0000] Initiating Kubernetes cluster INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates INFO[0000] [certificates] Generating admin certificates and kubeconfig INFO[0000] Successfully Deployed state file at [./cluster.rkestate] INFO[0000] Building Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [10.132.0.17] INFO[0000] [dialer] Setup tunnel for host [10.132.0.18] INFO[0002] [network] Deploying port listener containers INFO[0002] Pulling image [rancher/rke-tools:v0.1.80] on host [10.132.0.17], try #1 INFO[0002] Pulling image [rancher/rke-tools:v0.1.80] on host [10.132.0.18], try #1 INFO[0009] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0010] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0012] Starting container [rke-etcd-port-listener] on host [10.132.0.17], try #1 INFO[0012] Starting container [rke-etcd-port-listener] on host [10.132.0.18], try #1 INFO[0012] [network] Successfully started [rke-etcd-port-listener] container on host [10.132.0.17] INFO[0013] [network] Successfully started [rke-etcd-port-listener] container on host [10.132.0.18] INFO[0013] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0013] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0014] Starting container [rke-cp-port-listener] on host [10.132.0.17], try #1 INFO[0014] Starting container [rke-cp-port-listener] on host [10.132.0.18], try #1 INFO[0014] [network] Successfully started [rke-cp-port-listener] container on host [10.132.0.17] INFO[0014] [network] Successfully started [rke-cp-port-listener] container on host [10.132.0.18] INFO[0014] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0014] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0015] Starting container [rke-worker-port-listener] on host [10.132.0.18], try #1 INFO[0015] Starting container [rke-worker-port-listener] on host [10.132.0.17], try #1 INFO[0016] [network] Successfully started [rke-worker-port-listener] container on host [10.132.0.18] INFO[0016] [network] Successfully started [rke-worker-port-listener] container on host [10.132.0.17] INFO[0016] [network] Port listener containers deployed successfully INFO[0016] [network] Running etcd ↔ etcd port checks INFO[0016] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379 2380], try #1 INFO[0016] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379 2380], try #1 INFO[0016] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0016] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0017] Starting container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0017] Starting container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0017] [network] Successfully started [rke-port-checker] container on host [10.132.0.17] INFO[0017] [network] Successfully started [rke-port-checker] container on host [10.132.0.18] INFO[0017] Removing container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0017] Removing container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0017] [network] Running control plane → etcd port checks INFO[0017] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379], try #1 INFO[0017] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379], try #1 INFO[0018] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0018] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0018] Starting container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0018] Starting container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0019] [network] Successfully started [rke-port-checker] container on host [10.132.0.17] INFO[0019] [network] Successfully started [rke-port-checker] container on host [10.132.0.18] INFO[0019] Removing container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0019] Removing container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0019] [network] Running control plane → worker port checks INFO[0019] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [10250], try #1 INFO[0019] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [10250], try #1 INFO[0019] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0019] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0020] Starting container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0020] Starting container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0020] [network] Successfully started [rke-port-checker] container on host [10.132.0.17] INFO[0020] [network] Successfully started [rke-port-checker] container on host [10.132.0.18] INFO[0021] Removing container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0021] Removing container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0021] [network] Running workers → control plane port checks INFO[0021] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [6443], try #1 INFO[0021] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [6443], try #1 INFO[0021] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0021] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0022] Starting container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0022] Starting container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0022] [network] Successfully started [rke-port-checker] container on host [10.132.0.17] INFO[0022] [network] Successfully started [rke-port-checker] container on host [10.132.0.18] INFO[0022] Removing container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0022] Removing container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0022] [network] Checking KubeAPI port Control Plane hosts INFO[0023] [network] Removing port listener containers INFO[0023] Removing container [rke-etcd-port-listener] on host [10.132.0.17], try #1 INFO[0023] Removing container [rke-etcd-port-listener] on host [10.132.0.18], try #1 INFO[0023] [remove/rke-etcd-port-listener] Successfully removed container on host [10.132.0.18] INFO[0023] [remove/rke-etcd-port-listener] Successfully removed container on host [10.132.0.17] INFO[0023] Removing container [rke-cp-port-listener] on host [10.132.0.18], try #1 INFO[0023] Removing container [rke-cp-port-listener] on host [10.132.0.17], try #1 INFO[0024] [remove/rke-cp-port-listener] Successfully removed container on host [10.132.0.18] INFO[0024] [remove/rke-cp-port-listener] Successfully removed container on host [10.132.0.17] INFO[0024] Removing container [rke-worker-port-listener] on host [10.132.0.17], try #1 INFO[0024] Removing container [rke-worker-port-listener] on host [10.132.0.18], try #1 INFO[0024] [remove/rke-worker-port-listener] Successfully removed container on host [10.132.0.17] INFO[0024] [remove/rke-worker-port-listener] Successfully removed container on host [10.132.0.18] INFO[0024] [network] Port listener containers removed successfully INFO[0024] [certificates] Deploying kubernetes certificates to Cluster nodes INFO[0024] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1 INFO[0024] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1 INFO[0024] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0024] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0025] Starting container [cert-deployer] on host [10.132.0.17], try #1 INFO[0025] Starting container [cert-deployer] on host [10.132.0.18], try #1 INFO[0026] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1 INFO[0026] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1 INFO[0031] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1 INFO[0031] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1 INFO[0031] Removing container [cert-deployer] on host [10.132.0.17], try #1 INFO[0031] Removing container [cert-deployer] on host [10.132.0.18], try #1 INFO[0031] [reconcile] Rebuilding and updating local kube config INFO[0031] Successfully Deployed local admin kubeconfig at [./kube\_config\_cluster.yml] WARN[0061] [reconcile] host [10.132.0.17] is a control plane node without reachable Kubernetes API endpoint in the cluster INFO[0061] Successfully Deployed local admin kubeconfig at [./kube\_config\_cluster.yml] WARN[0091] [reconcile] host [10.132.0.18] is a control plane node without reachable Kubernetes API endpoint in the cluster WARN[0091] [reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found INFO[0091] [certificates] Successfully deployed kubernetes certificates to Cluster nodes INFO[0091] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.132.0.17] INFO[0093] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0094] Starting container [file-deployer] on host [10.132.0.17], try #1 INFO[0094] Successfully started [file-deployer] container on host [10.132.0.17] INFO[0094] Waiting for [file-deployer] container to exit on host [10.132.0.17] INFO[0094] Waiting for [file-deployer] container to exit on host [10.132.0.17] INFO[0095] Removing container [file-deployer] on host [10.132.0.17], try #1 INFO[0095] [remove/file-deployer] Successfully removed container on host [10.132.0.17] INFO[0095] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.132.0.18] INFO[0097] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0097] Starting container [file-deployer] on host [10.132.0.18], try #1 INFO[0098] Successfully started [file-deployer] container on host [10.132.0.18] INFO[0098] Waiting for [file-deployer] container to exit on host [10.132.0.18] INFO[0098] Waiting for [file-deployer] container to exit on host [10.132.0.18] INFO[0098] Removing container [file-deployer] on host [10.132.0.18], try #1 INFO[0098] [remove/file-deployer] Successfully removed container on host [10.132.0.18] INFO[0098] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes INFO[0098] [reconcile] Reconciling cluster state INFO[0098] [reconcile] This is newly generated cluster INFO[0098] Pre-pulling kubernetes images INFO[0098] Pulling image [rancher/hyperkube:v1.22.9-rancher1] on host [10.132.0.17], try #1 INFO[0098] Pulling image [rancher/hyperkube:v1.22.9-rancher1] on host [10.132.0.18], try #1 INFO[0166] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17] INFO[0169] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18] INFO[0169] Kubernetes images pulled successfully INFO[0169] [etcd] Building up etcd plane… INFO[0169] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0175] Starting container [etcd-fix-perm] on host [10.132.0.17], try #1 INFO[0175] Successfully started [etcd-fix-perm] container on host [10.132.0.17] INFO[0175] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.17] INFO[0175] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.17] INFO[0175] Removing container [etcd-fix-perm] on host [10.132.0.17], try #1 INFO[0176] [remove/etcd-fix-perm] Successfully removed container on host [10.132.0.17] INFO[0176] Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.132.0.17], try #1 INFO[0183] Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.132.0.17] INFO[0185] Starting container [etcd] on host [10.132.0.17], try #1 INFO[0185] [etcd] Successfully started [etcd] container on host [10.132.0.17] INFO[0185] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.132.0.17] INFO[0185] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0186] Starting container [etcd-rolling-snapshots] on host [10.132.0.17], try #1 INFO[0187] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.132.0.17] INFO[0192] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0193] Starting container [rke-bundle-cert] on host [10.132.0.17], try #1 INFO[0193] [certificates] Successfully started [rke-bundle-cert] container on host [10.132.0.17] INFO[0193] Waiting for [rke-bundle-cert] container to exit on host [10.132.0.17] INFO[0193] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.132.0.17] INFO[0193] Removing container [rke-bundle-cert] on host [10.132.0.17], try #1 INFO[0194] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0195] Starting container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0195] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.17] INFO[0195] Removing container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0196] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17] INFO[0196] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0197] Starting container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0197] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.17] INFO[0197] Removing container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0198] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17] INFO[0198] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0199] Starting container [etcd-fix-perm] on host [10.132.0.18], try #1 INFO[0200] Successfully started [etcd-fix-perm] container on host [10.132.0.18] INFO[0200] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.18] INFO[0200] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.18] INFO[0200] Removing container [etcd-fix-perm] on host [10.132.0.18], try #1 INFO[0200] [remove/etcd-fix-perm] Successfully removed container on host [10.132.0.18] INFO[0200] Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.132.0.18], try #1 INFO[0208] Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.132.0.18] INFO[0210] Starting container [etcd] on host [10.132.0.18], try #1 INFO[0210] [etcd] Successfully started [etcd] container on host [10.132.0.18] INFO[0210] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.132.0.18] INFO[0210] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0211] Starting container [etcd-rolling-snapshots] on host [10.132.0.18], try #1 INFO[0211] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.132.0.18] INFO[0217] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0217] Starting container [rke-bundle-cert] on host [10.132.0.18], try #1 INFO[0218] [certificates] Successfully started [rke-bundle-cert] container on host [10.132.0.18] INFO[0218] Waiting for [rke-bundle-cert] container to exit on host [10.132.0.18] INFO[0218] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.132.0.18] INFO[0218] Removing container [rke-bundle-cert] on host [10.132.0.18], try #1 INFO[0219] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0220] Starting container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0220] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.18] INFO[0220] Removing container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0220] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18] INFO[0221] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0222] Starting container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0222] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.18] INFO[0222] Removing container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0222] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18] INFO[0222] [etcd] Successfully started etcd plane… Checking etcd cluster health INFO[0225] [etcd] etcd host [10.132.0.17] reported healthy=true INFO[0225] [controlplane] Building up Controller Plane… INFO[0225] Checking if container [service-sidekick] is running on host [10.132.0.17], try #1 INFO[0225] Checking if container [service-sidekick] is running on host [10.132.0.18], try #1 INFO[0225] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0225] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0226] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18] INFO[0226] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17] INFO[0226] Starting container [kube-apiserver] on host [10.132.0.18], try #1 INFO[0226] Starting container [kube-apiserver] on host [10.132.0.17], try #1 INFO[0226] [controlplane] Successfully started [kube-apiserver] container on host [10.132.0.18] INFO[0226] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.132.0.18] INFO[0226] [controlplane] Successfully started [kube-apiserver] container on host [10.132.0.17] INFO[0226] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.132.0.17] INFO[0236] [healthcheck] service [kube-apiserver] on host [10.132.0.17] is healthy INFO[0236] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0236] [healthcheck] service [kube-apiserver] on host [10.132.0.18] is healthy INFO[0237] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0237] Starting container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0238] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17] INFO[0238] Starting container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0238] Removing container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0238] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17] INFO[0238] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17] INFO[0238] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18] INFO[0238] Starting container [kube-controller-manager] on host [10.132.0.17], try #1 INFO[0238] Removing container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0239] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18] INFO[0239] [controlplane] Successfully started [kube-controller-manager] container on host [10.132.0.17] INFO[0239] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.132.0.17] INFO[0239] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18] INFO[0239] Starting container [kube-controller-manager] on host [10.132.0.18], try #1 INFO[0239] [controlplane] Successfully started [kube-controller-manager] container on host [10.132.0.18] INFO[0239] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.132.0.18] INFO[0241] [healthcheck] service [kube-controller-manager] on host [10.132.0.17] is healthy INFO[0241] [healthcheck] service [kube-controller-manager] on host [10.132.0.18] is healthy INFO[0241] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0242] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0242] Starting container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0243] Starting container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0243] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17] INFO[0243] Removing container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0243] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17] INFO[0243] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18] INFO[0243] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17] INFO[0243] Removing container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0243] Starting container [kube-scheduler] on host [10.132.0.17], try #1 INFO[0243] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18] INFO[0244] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18] INFO[0244] [controlplane] Successfully started [kube-scheduler] container on host [10.132.0.17] INFO[0244] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.132.0.17] INFO[0244] Starting container [kube-scheduler] on host [10.132.0.18], try #1 INFO[0244] [controlplane] Successfully started [kube-scheduler] container on host [10.132.0.18] INFO[0244] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.132.0.18] INFO[0246] [healthcheck] service [kube-scheduler] on host [10.132.0.17] is healthy INFO[0246] [healthcheck] service [kube-scheduler] on host [10.132.0.18] is healthy INFO[0246] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0246] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0247] Starting container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0247] Starting container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0247] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17] INFO[0247] Removing container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0248] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17] INFO[0248] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18] INFO[0248] Removing container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0248] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18] INFO[0248] [controlplane] Successfully started Controller Plane… INFO[0248] [authz] Creating rke-job-deployer ServiceAccount FATA[0423] Failed to apply the ServiceAccount needed for job execution: Post “[https://10.132.0.18:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=30s](https://10.132.0.18:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=30s)”: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ---
**Author:** ![sanchi-singh24](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/sanchi-singh24/32/538_2.png) [@sanchi-singh24](https://community.mosip.io/u/sanchi-singh24)\ **Post date:** [November 19, 2023, 4:49pm UTC](https://community.mosip.io/t/mosip-v3-on-prem-set-up-issues/901/5 "2023-11-19T16:49:20Z")
Hi @yahlife_nation I have shared your issue with our deployment team they will look into this tomorrow due to Sat/Sun the team members were not available. On Monday they will provide you with a solution so can move ahead and setup On-Prem. ---
**Author:** ![syed.salman](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/syed.salman/32/25_2.png) [@syed.salman](https://community.mosip.io/u/syed.salman)\ **Post date:** [November 20, 2023, 6:14am UTC](https://community.mosip.io/t/mosip-v3-on-prem-set-up-issues/901/6 "2023-11-20T06:14:21Z")
@yahlife_nation - Run `rke remove` - Install docker with specific version `20.10.xxx` on all cluster nodes. - Run `rke up` and ---
**Author:** ![yahlife\_nation](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/yahlife_nation/32/850_2.png) [@yahlife\_nation](https://community.mosip.io/u/yahlife_nation)\ **Post date:** [November 20, 2023, 1:46pm UTC](https://community.mosip.io/t/mosip-v3-on-prem-set-up-issues/901/7 "2023-11-20T13:46:56Z")
@syed.salman @sanchi-singh24 I did that using - rke version you provided me v1.3.10 - docker version `20.10.10` and `rke up` give this error : INFO[0000] Running RKE version: v1.3.10 INFO[0000] Initiating Kubernetes cluster INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates INFO[0000] [certificates] Generating admin certificates and kubeconfig INFO[0000] Successfully Deployed state file at [./cluster.rkestate] INFO[0000] Building Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [10.132.0.17] INFO[0000] [dialer] Setup tunnel for host [10.132.0.18] INFO[0002] [network] Deploying port listener containers INFO[0002] Pulling image [rancher/rke-tools:v0.1.80] on host [10.132.0.17], try #1 INFO[0002] Pulling image [rancher/rke-tools:v0.1.80] on host [10.132.0.18], try #1 INFO[0009] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0010] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0012] Starting container [rke-etcd-port-listener] on host [10.132.0.17], try #1 INFO[0012] Starting container [rke-etcd-port-listener] on host [10.132.0.18], try #1 INFO[0012] [network] Successfully started [rke-etcd-port-listener] container on host [10.132.0.17] INFO[0013] [network] Successfully started [rke-etcd-port-listener] container on host [10.132.0.18] INFO[0013] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0013] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0014] Starting container [rke-cp-port-listener] on host [10.132.0.17], try #1 INFO[0014] Starting container [rke-cp-port-listener] on host [10.132.0.18], try #1 INFO[0014] [network] Successfully started [rke-cp-port-listener] container on host [10.132.0.17] INFO[0014] [network] Successfully started [rke-cp-port-listener] container on host [10.132.0.18] INFO[0014] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0014] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0015] Starting container [rke-worker-port-listener] on host [10.132.0.18], try #1 INFO[0015] Starting container [rke-worker-port-listener] on host [10.132.0.17], try #1 INFO[0016] [network] Successfully started [rke-worker-port-listener] container on host [10.132.0.18] INFO[0016] [network] Successfully started [rke-worker-port-listener] container on host [10.132.0.17] INFO[0016] [network] Port listener containers deployed successfully INFO[0016] [network] Running etcd ↔ etcd port checks INFO[0016] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379 2380], try #1 INFO[0016] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379 2380], try #1 INFO[0016] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0016] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0017] Starting container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0017] Starting container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0017] [network] Successfully started [rke-port-checker] container on host [10.132.0.17] INFO[0017] [network] Successfully started [rke-port-checker] container on host [10.132.0.18] INFO[0017] Removing container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0017] Removing container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0017] [network] Running control plane → etcd port checks INFO[0017] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379], try #1 INFO[0017] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [2379], try #1 INFO[0018] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0018] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0018] Starting container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0018] Starting container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0019] [network] Successfully started [rke-port-checker] container on host [10.132.0.17] INFO[0019] [network] Successfully started [rke-port-checker] container on host [10.132.0.18] INFO[0019] Removing container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0019] Removing container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0019] [network] Running control plane → worker port checks INFO[0019] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [10250], try #1 INFO[0019] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [10250], try #1 INFO[0019] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0019] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0020] Starting container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0020] Starting container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0020] [network] Successfully started [rke-port-checker] container on host [10.132.0.17] INFO[0020] [network] Successfully started [rke-port-checker] container on host [10.132.0.18] INFO[0021] Removing container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0021] Removing container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0021] [network] Running workers → control plane port checks INFO[0021] [network] Checking if host [10.132.0.18] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [6443], try #1 INFO[0021] [network] Checking if host [10.132.0.17] can connect to host(s) [10.132.0.17 10.132.0.18] on port(s) [6443], try #1 INFO[0021] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0021] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0022] Starting container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0022] Starting container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0022] [network] Successfully started [rke-port-checker] container on host [10.132.0.17] INFO[0022] [network] Successfully started [rke-port-checker] container on host [10.132.0.18] INFO[0022] Removing container [rke-port-checker] on host [10.132.0.17], try #1 INFO[0022] Removing container [rke-port-checker] on host [10.132.0.18], try #1 INFO[0022] [network] Checking KubeAPI port Control Plane hosts INFO[0023] [network] Removing port listener containers INFO[0023] Removing container [rke-etcd-port-listener] on host [10.132.0.17], try #1 INFO[0023] Removing container [rke-etcd-port-listener] on host [10.132.0.18], try #1 INFO[0023] [remove/rke-etcd-port-listener] Successfully removed container on host [10.132.0.18] INFO[0023] [remove/rke-etcd-port-listener] Successfully removed container on host [10.132.0.17] INFO[0023] Removing container [rke-cp-port-listener] on host [10.132.0.18], try #1 INFO[0023] Removing container [rke-cp-port-listener] on host [10.132.0.17], try #1 INFO[0024] [remove/rke-cp-port-listener] Successfully removed container on host [10.132.0.18] INFO[0024] [remove/rke-cp-port-listener] Successfully removed container on host [10.132.0.17] INFO[0024] Removing container [rke-worker-port-listener] on host [10.132.0.17], try #1 INFO[0024] Removing container [rke-worker-port-listener] on host [10.132.0.18], try #1 INFO[0024] [remove/rke-worker-port-listener] Successfully removed container on host [10.132.0.17] INFO[0024] [remove/rke-worker-port-listener] Successfully removed container on host [10.132.0.18] INFO[0024] [network] Port listener containers removed successfully INFO[0024] [certificates] Deploying kubernetes certificates to Cluster nodes INFO[0024] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1 INFO[0024] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1 INFO[0024] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0024] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0025] Starting container [cert-deployer] on host [10.132.0.17], try #1 INFO[0025] Starting container [cert-deployer] on host [10.132.0.18], try #1 INFO[0026] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1 INFO[0026] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1 INFO[0031] Checking if container [cert-deployer] is running on host [10.132.0.17], try #1 INFO[0031] Checking if container [cert-deployer] is running on host [10.132.0.18], try #1 INFO[0031] Removing container [cert-deployer] on host [10.132.0.17], try #1 INFO[0031] Removing container [cert-deployer] on host [10.132.0.18], try #1 INFO[0031] [reconcile] Rebuilding and updating local kube config INFO[0031] Successfully Deployed local admin kubeconfig at [./kube\_config\_cluster.yml] WARN[0061] [reconcile] host [10.132.0.17] is a control plane node without reachable Kubernetes API endpoint in the cluster INFO[0061] Successfully Deployed local admin kubeconfig at [./kube\_config\_cluster.yml] WARN[0091] [reconcile] host [10.132.0.18] is a control plane node without reachable Kubernetes API endpoint in the cluster WARN[0091] [reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found INFO[0091] [certificates] Successfully deployed kubernetes certificates to Cluster nodes INFO[0091] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.132.0.17] INFO[0093] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0094] Starting container [file-deployer] on host [10.132.0.17], try #1 INFO[0094] Successfully started [file-deployer] container on host [10.132.0.17] INFO[0094] Waiting for [file-deployer] container to exit on host [10.132.0.17] INFO[0094] Waiting for [file-deployer] container to exit on host [10.132.0.17] INFO[0095] Removing container [file-deployer] on host [10.132.0.17], try #1 INFO[0095] [remove/file-deployer] Successfully removed container on host [10.132.0.17] INFO[0095] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.132.0.18] INFO[0097] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0097] Starting container [file-deployer] on host [10.132.0.18], try #1 INFO[0098] Successfully started [file-deployer] container on host [10.132.0.18] INFO[0098] Waiting for [file-deployer] container to exit on host [10.132.0.18] INFO[0098] Waiting for [file-deployer] container to exit on host [10.132.0.18] INFO[0098] Removing container [file-deployer] on host [10.132.0.18], try #1 INFO[0098] [remove/file-deployer] Successfully removed container on host [10.132.0.18] INFO[0098] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes INFO[0098] [reconcile] Reconciling cluster state INFO[0098] [reconcile] This is newly generated cluster INFO[0098] Pre-pulling kubernetes images INFO[0098] Pulling image [rancher/hyperkube:v1.22.9-rancher1] on host [10.132.0.17], try #1 INFO[0098] Pulling image [rancher/hyperkube:v1.22.9-rancher1] on host [10.132.0.18], try #1 INFO[0166] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17] INFO[0169] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18] INFO[0169] Kubernetes images pulled successfully INFO[0169] [etcd] Building up etcd plane… INFO[0169] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0175] Starting container [etcd-fix-perm] on host [10.132.0.17], try #1 INFO[0175] Successfully started [etcd-fix-perm] container on host [10.132.0.17] INFO[0175] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.17] INFO[0175] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.17] INFO[0175] Removing container [etcd-fix-perm] on host [10.132.0.17], try #1 INFO[0176] [remove/etcd-fix-perm] Successfully removed container on host [10.132.0.17] INFO[0176] Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.132.0.17], try #1 INFO[0183] Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.132.0.17] INFO[0185] Starting container [etcd] on host [10.132.0.17], try #1 INFO[0185] [etcd] Successfully started [etcd] container on host [10.132.0.17] INFO[0185] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.132.0.17] INFO[0185] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0186] Starting container [etcd-rolling-snapshots] on host [10.132.0.17], try #1 INFO[0187] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.132.0.17] INFO[0192] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0193] Starting container [rke-bundle-cert] on host [10.132.0.17], try #1 INFO[0193] [certificates] Successfully started [rke-bundle-cert] container on host [10.132.0.17] INFO[0193] Waiting for [rke-bundle-cert] container to exit on host [10.132.0.17] INFO[0193] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.132.0.17] INFO[0193] Removing container [rke-bundle-cert] on host [10.132.0.17], try #1 INFO[0194] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0195] Starting container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0195] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.17] INFO[0195] Removing container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0196] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17] INFO[0196] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0197] Starting container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0197] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.17] INFO[0197] Removing container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0198] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17] INFO[0198] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0199] Starting container [etcd-fix-perm] on host [10.132.0.18], try #1 INFO[0200] Successfully started [etcd-fix-perm] container on host [10.132.0.18] INFO[0200] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.18] INFO[0200] Waiting for [etcd-fix-perm] container to exit on host [10.132.0.18] INFO[0200] Removing container [etcd-fix-perm] on host [10.132.0.18], try #1 INFO[0200] [remove/etcd-fix-perm] Successfully removed container on host [10.132.0.18] INFO[0200] Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.132.0.18], try #1 INFO[0208] Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.132.0.18] INFO[0210] Starting container [etcd] on host [10.132.0.18], try #1 INFO[0210] [etcd] Successfully started [etcd] container on host [10.132.0.18] INFO[0210] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.132.0.18] INFO[0210] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0211] Starting container [etcd-rolling-snapshots] on host [10.132.0.18], try #1 INFO[0211] [etcd] Successfully started [etcd-rolling-snapshots] container on host [10.132.0.18] INFO[0217] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0217] Starting container [rke-bundle-cert] on host [10.132.0.18], try #1 INFO[0218] [certificates] Successfully started [rke-bundle-cert] container on host [10.132.0.18] INFO[0218] Waiting for [rke-bundle-cert] container to exit on host [10.132.0.18] INFO[0218] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.132.0.18] INFO[0218] Removing container [rke-bundle-cert] on host [10.132.0.18], try #1 INFO[0219] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0220] Starting container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0220] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.18] INFO[0220] Removing container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0220] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18] INFO[0221] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0222] Starting container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0222] [etcd] Successfully started [rke-log-linker] container on host [10.132.0.18] INFO[0222] Removing container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0222] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18] INFO[0222] [etcd] Successfully started etcd plane… Checking etcd cluster health INFO[0225] [etcd] etcd host [10.132.0.17] reported healthy=true INFO[0225] [controlplane] Building up Controller Plane… INFO[0225] Checking if container [service-sidekick] is running on host [10.132.0.17], try #1 INFO[0225] Checking if container [service-sidekick] is running on host [10.132.0.18], try #1 INFO[0225] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0225] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0226] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18] INFO[0226] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17] INFO[0226] Starting container [kube-apiserver] on host [10.132.0.18], try #1 INFO[0226] Starting container [kube-apiserver] on host [10.132.0.17], try #1 INFO[0226] [controlplane] Successfully started [kube-apiserver] container on host [10.132.0.18] INFO[0226] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.132.0.18] INFO[0226] [controlplane] Successfully started [kube-apiserver] container on host [10.132.0.17] INFO[0226] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.132.0.17] INFO[0236] [healthcheck] service [kube-apiserver] on host [10.132.0.17] is healthy INFO[0236] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0236] [healthcheck] service [kube-apiserver] on host [10.132.0.18] is healthy INFO[0237] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0237] Starting container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0238] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17] INFO[0238] Starting container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0238] Removing container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0238] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17] INFO[0238] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17] INFO[0238] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18] INFO[0238] Starting container [kube-controller-manager] on host [10.132.0.17], try #1 INFO[0238] Removing container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0239] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18] INFO[0239] [controlplane] Successfully started [kube-controller-manager] container on host [10.132.0.17] INFO[0239] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.132.0.17] INFO[0239] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18] INFO[0239] Starting container [kube-controller-manager] on host [10.132.0.18], try #1 INFO[0239] [controlplane] Successfully started [kube-controller-manager] container on host [10.132.0.18] INFO[0239] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.132.0.18] INFO[0241] [healthcheck] service [kube-controller-manager] on host [10.132.0.17] is healthy INFO[0241] [healthcheck] service [kube-controller-manager] on host [10.132.0.18] is healthy INFO[0241] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0242] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0242] Starting container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0243] Starting container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0243] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17] INFO[0243] Removing container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0243] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17] INFO[0243] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18] INFO[0243] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.17] INFO[0243] Removing container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0243] Starting container [kube-scheduler] on host [10.132.0.17], try #1 INFO[0243] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18] INFO[0244] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [10.132.0.18] INFO[0244] [controlplane] Successfully started [kube-scheduler] container on host [10.132.0.17] INFO[0244] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.132.0.17] INFO[0244] Starting container [kube-scheduler] on host [10.132.0.18], try #1 INFO[0244] [controlplane] Successfully started [kube-scheduler] container on host [10.132.0.18] INFO[0244] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.132.0.18] INFO[0246] [healthcheck] service [kube-scheduler] on host [10.132.0.17] is healthy INFO[0246] [healthcheck] service [kube-scheduler] on host [10.132.0.18] is healthy INFO[0246] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.17] INFO[0246] Image [rancher/rke-tools:v0.1.80] exists on host [10.132.0.18] INFO[0247] Starting container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0247] Starting container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0247] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.17] INFO[0247] Removing container [rke-log-linker] on host [10.132.0.17], try #1 INFO[0248] [remove/rke-log-linker] Successfully removed container on host [10.132.0.17] INFO[0248] [controlplane] Successfully started [rke-log-linker] container on host [10.132.0.18] INFO[0248] Removing container [rke-log-linker] on host [10.132.0.18], try #1 INFO[0248] [remove/rke-log-linker] Successfully removed container on host [10.132.0.18] INFO[0248] [controlplane] Successfully started Controller Plane… INFO[0248] [authz] Creating rke-job-deployer ServiceAccount FATA[0423] Failed to apply the ServiceAccount needed for job execution: Post “[https://10.132.0.18:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=30s”:](https://10.132.0.18:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=30s%E2%80%9D:) net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) # The only versions that worked was when I used latest RKE v1.4.10 with latest docker 24.0.5. thats when it was successful. ---
**Author:** ![syed.salman](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/syed.salman/32/25_2.png) [@syed.salman](https://community.mosip.io/u/syed.salman)\ **Post date:** [November 21, 2023, 11:19am UTC](https://community.mosip.io/t/mosip-v3-on-prem-set-up-issues/901/8 "2023-11-21T11:19:28Z")
@yahlife_nation Can you follow the instructions mentioned below: - Run `rke remove` to remove the cluster - Remove docker from all the nodes and rke components: - Clean each host from the directories left by the services: - /etc/kubernetes/ssl - /var/lib/etcd - /etc/cni - /opt/cni - /var/run/calico - Take the latest pull from `mosip/k8s-infra`’s `main` branch - Ensure rke tool version is `v1.3.10`. - Run ansible docker.yaml to install docker with 20.10.x version. - Run `rke up`