I am presently trying to set up mosip v2 using centos 9 and I keep getting this error:
fatal: [console.sb]: FAILED! => {“changed”: false, “msg”: “Failed to download metadata for repo ‘kubernetes’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried”, “rc”: 1, “results”: }
Dear @Johnny
Thank you for reaching out, as you’re already aware about CentOS having been deprecated and the latest versions of MOSIP is not tested with CentOS, I will advise you to not to try it with CentOS.
Please refer to our On-Prem Installation Guidelines | MOSIP Docs 1.2.0.
Best Regards
Team MOSIP
Hi @svahsek,
Thanks for your previous response, I am now working with On-Prem Insatallation 1.20 using ubuntu and I need your assistant, I was able to successful deploy the RKE Cluster, which I check by executing kubectl get nodes and kubectl get pods -A and get a response showing nodes ready or pod runnning but executing the command in an hour again shows connection refused. find attached the two scenario
Dear Johny,
Than you for reaching out, One of our experts will respond to you on this.
Regards
Team MOSIP
Hello @Johnny,
Could you please provide more details about the deployment?
Copy the kubeconfig file:
cp kube_config_cluster.yml $HOME/.kube/<cluster_name>_config
chmod 400 $HOME/.kube/<cluster_name>_config
To access the cluster using the kubeconfig file
export KUBECONFIG="$HOME/.kube/<cluster_name>_config"
Please try the above steps and let us know if you face any issues.
Regards
Team MOSIP
Hi Praful_Rakhade,
Thank you very much for your response.
I am presently facing an issue with RKE and Docker deployment. I deploy Docker version 28.1.1 and whenever I execute the deployment of rke up --config cluster.yml(rke version v1.3.10), I get this error:
FATA[0006] Unsupported Docker version found [28.1.1] on host [172.16.2.156], supported versions are [1.13.x 17.03.x 17.06.x 17.09.x 18.06.x 18.09.x 19.03.x 20.10.x]
I have tried to downgrade the docker to 20.10.24, it keeps giving error. Please, can you help with any known solution. Thank you
Hello @Johnny,
Please remove any existing Docker installation from your virtual machines and reinstall it using our official guide:
MOSIP On-Prem Setup – Rancher Guide
Using the recommended Docker version and setup ensures compatibility with the MOSIP infrastructure.
Regards
Team MOSIP
Hi Praful_Rakhade,
Thank you for your quick response, I follow the official and this is the error:
john@bastion:~/k8s-infra/rancher/on-prem$ ansible-playbook -i hosts.ini docker.yaml
[WARNING]: Could not match supplied host pattern, ignoring: cluster
PLAY [cluster] *************************************************************************************************************************************************
skipping: no hosts matched
PLAY RECAP *****************************************************************************************************************************************************
john@bastion:~/k8s-infra/rancher/on-prem$ ^C
john@bastion:~/k8s-infra/rancher/on-prem$ vi hosts.ini
john@bastion:~/k8s-infra/rancher/on-prem$ ansible-playbook -i hosts.ini docker.yaml
[WARNING]: * Failed to parse /home/john/k8s-infra/rancher/on-prem/hosts.ini with ini plugin: /home/john/k8s-infra/rancher/on-prem/hosts.ini:17: Section
[docker_nodes:children] includes undefined group: mosip_nodes
[WARNING]: Unable to parse /home/john/k8s-infra/rancher/on-prem/hosts.ini as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match ‘all’
PLAY [cluster] *************************************************************************************************************************************************
TASK [Gathering Facts] *****************************************************************************************************************************************
[WARNING]: Platform linux on host mosip-node2 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python
interpreter could change the meaning of that path. See Interpreter Discovery — Ansible Core Documentation for more
information.
ok: [mosip-node2]
[WARNING]: Platform linux on host mosip-node5 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python
interpreter could change the meaning of that path. See Interpreter Discovery — Ansible Core Documentation for more
information.
ok: [mosip-node5]
[WARNING]: Platform linux on host mosip-node1 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python
interpreter could change the meaning of that path. See Interpreter Discovery — Ansible Core Documentation for more
information.
ok: [mosip-node1]
[WARNING]: Platform linux on host mosip-node4 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python
interpreter could change the meaning of that path. See Interpreter Discovery — Ansible Core Documentation for more
information.
ok: [mosip-node4]
[WARNING]: Platform linux on host mosip-node3 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python
interpreter could change the meaning of that path. See Interpreter Discovery — Ansible Core Documentation for more
information.
ok: [mosip-node3]
[WARNING]: Platform linux on host mosip-node6 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python
interpreter could change the meaning of that path. See Interpreter Discovery — Ansible Core Documentation for more
information.
ok: [mosip-node6]
TASK [Install docker.io for Debian] ****************************************************************************************************************************
fatal: [mosip-node5]: FAILED! => {“cache_update_time”: 1747392418, “cache_updated”: true, “changed”: false, “msg”: “no available installation candidate for docker.io=20.10."}
fatal: [mosip-node1]: FAILED! => {“cache_update_time”: 1747392418, “cache_updated”: true, “changed”: false, “msg”: "no available installation candidate for docker.io=20.10.”}
fatal: [mosip-node2]: FAILED! => {“cache_update_time”: 1747392418, “cache_updated”: true, “changed”: false, “msg”: “no available installation candidate for docker.io=20.10."}
fatal: [mosip-node3]: FAILED! => {“cache_update_time”: 1747392418, “cache_updated”: true, “changed”: false, “msg”: "no available installation candidate for docker.io=20.10.”}
fatal: [mosip-node4]: FAILED! => {“cache_update_time”: 1747392418, “cache_updated”: true, “changed”: false, “msg”: “no available installation candidate for docker.io=20.10."}
fatal: [mosip-node6]: FAILED! => {“cache_update_time”: 1747392426, “cache_updated”: true, “changed”: false, “msg”: "no available installation candidate for docker.io=20.10.”}
PLAY RECAP *****************************************************************************************************************************************************
mosip-node1 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
mosip-node2 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
mosip-node3 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
mosip-node4 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
mosip-node5 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
mosip-node6 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
Kindly assist
Hello @Johnny,
Could you please share your host.ini file?
This will help us better understand your environment and assist you more effectively.
Thanks!
Regards
Team MOSIP
Hi Prafu_Rakhade,
I have been able to go beyond that part but I am now having error at the point of setting up Mosip Cluster. I get this error when I am trying to deploy the RKE cluster:
john@bastion:~/k8s-infra/mosip/on-prem$ rke up --config cluster.yml
INFO[0000] Running RKE version: v1.3.10
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0000] [certificates] Generating admin certificates and kubeconfig
INFO[0000] Successfully Deployed state file at [./cluster.rkestate]
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [172.16.2.151]
INFO[0000] [dialer] Setup tunnel for host [172.16.2.154]
INFO[0000] [dialer] Setup tunnel for host [172.16.2.153]
INFO[0000] [dialer] Setup tunnel for host [172.16.2.152]
INFO[0000] [dialer] Setup tunnel for host [172.16.2.150]
INFO[0000] [dialer] Setup tunnel for host [172.16.2.156]
INFO[0001] [network] No hosts added existing cluster, skipping port check
INFO[0001] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.154], try #1
INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.156], try #1
INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.153], try #1
INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.152], try #1
INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.151], try #1
INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.150], try #1
INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156]
INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153]
INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154]
INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152]
INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150]
INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151]
INFO[0001] Starting container [cert-deployer] on host [172.16.2.156], try #1
INFO[0002] Starting container [cert-deployer] on host [172.16.2.151], try #1
INFO[0002] Checking if container [cert-deployer] is running on host [172.16.2.156], try #1
INFO[0002] Starting container [cert-deployer] on host [172.16.2.152], try #1
INFO[0002] Starting container [cert-deployer] on host [172.16.2.150], try #1
INFO[0003] Checking if container [cert-deployer] is running on host [172.16.2.151], try #1
INFO[0003] Checking if container [cert-deployer] is running on host [172.16.2.152], try #1
INFO[0003] Checking if container [cert-deployer] is running on host [172.16.2.150], try #1
INFO[0003] Starting container [cert-deployer] on host [172.16.2.153], try #1
INFO[0003] Checking if container [cert-deployer] is running on host [172.16.2.153], try #1
INFO[0003] Starting container [cert-deployer] on host [172.16.2.154], try #1
INFO[0004] Checking if container [cert-deployer] is running on host [172.16.2.154], try #1
INFO[0007] Checking if container [cert-deployer] is running on host [172.16.2.156], try #1
INFO[0007] Removing container [cert-deployer] on host [172.16.2.156], try #1
INFO[0008] Checking if container [cert-deployer] is running on host [172.16.2.151], try #1
INFO[0008] Removing container [cert-deployer] on host [172.16.2.151], try #1
INFO[0008] Checking if container [cert-deployer] is running on host [172.16.2.152], try #1
INFO[0008] Removing container [cert-deployer] on host [172.16.2.152], try #1
INFO[0008] Checking if container [cert-deployer] is running on host [172.16.2.150], try #1
INFO[0008] Removing container [cert-deployer] on host [172.16.2.150], try #1
INFO[0008] Checking if container [cert-deployer] is running on host [172.16.2.153], try #1
INFO[0008] Removing container [cert-deployer] on host [172.16.2.153], try #1
INFO[0009] Checking if container [cert-deployer] is running on host [172.16.2.154], try #1
INFO[0009] Removing container [cert-deployer] on host [172.16.2.154], try #1
INFO[0009] [reconcile] Rebuilding and updating local kube config
INFO[0009] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0009] [reconcile] host [172.16.2.156] is a control plane node with reachable Kubernetes API endpoint in the cluster
INFO[0009] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0009] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.16.2.156]
INFO[0009] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156]
INFO[0009] Starting container [file-deployer] on host [172.16.2.156], try #1
INFO[0010] Successfully started [file-deployer] container on host [172.16.2.156]
INFO[0010] Waiting for [file-deployer] container to exit on host [172.16.2.156]
INFO[0010] Waiting for [file-deployer] container to exit on host [172.16.2.156]
INFO[0010] Container [file-deployer] is still running on host [172.16.2.156]: stderr: , stdout:
INFO[0011] Waiting for [file-deployer] container to exit on host [172.16.2.156]
INFO[0011] Removing container [file-deployer] on host [172.16.2.156], try #1
INFO[0011] [remove/file-deployer] Successfully removed container on host [172.16.2.156]
INFO[0011] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.16.2.154]
INFO[0011] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154]
INFO[0011] Starting container [file-deployer] on host [172.16.2.154], try #1
INFO[0011] Successfully started [file-deployer] container on host [172.16.2.154]
INFO[0011] Waiting for [file-deployer] container to exit on host [172.16.2.154]
INFO[0011] Waiting for [file-deployer] container to exit on host [172.16.2.154]
INFO[0011] Container [file-deployer] is still running on host [172.16.2.154]: stderr: , stdout:
INFO[0012] Waiting for [file-deployer] container to exit on host [172.16.2.154]
INFO[0012] Removing container [file-deployer] on host [172.16.2.154], try #1
INFO[0012] [remove/file-deployer] Successfully removed container on host [172.16.2.154]
INFO[0012] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.16.2.153]
INFO[0012] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153]
INFO[0013] Starting container [file-deployer] on host [172.16.2.153], try #1
INFO[0013] Successfully started [file-deployer] container on host [172.16.2.153]
INFO[0013] Waiting for [file-deployer] container to exit on host [172.16.2.153]
INFO[0013] Waiting for [file-deployer] container to exit on host [172.16.2.153]
INFO[0013] Container [file-deployer] is still running on host [172.16.2.153]: stderr: , stdout:
INFO[0014] Waiting for [file-deployer] container to exit on host [172.16.2.153]
INFO[0014] Removing container [file-deployer] on host [172.16.2.153], try #1
INFO[0014] [remove/file-deployer] Successfully removed container on host [172.16.2.153]
INFO[0014] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes
INFO[0014] [reconcile] Reconciling cluster state
INFO[0014] [reconcile] Check etcd hosts to be deleted
INFO[0014] [reconcile] Check etcd hosts to be added
INFO[0014] [reconcile] Rebuilding and updating local kube config
INFO[0014] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0014] [reconcile] host [172.16.2.156] is a control plane node with reachable Kubernetes API endpoint in the cluster
INFO[0014] [reconcile] Reconciled cluster state successfully
INFO[0014] max_unavailable_worker got rounded down to 0, resetting to 1
INFO[0014] Setting maxUnavailable for worker nodes to: 1
INFO[0014] Setting maxUnavailable for controlplane nodes to: 1
INFO[0014] Pre-pulling kubernetes images
INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.153]
INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.156]
INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.151]
INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.154]
INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.150]
INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.152]
INFO[0014] Kubernetes images pulled successfully
INFO[0014] [etcd] Building up etcd plane..
INFO[0014] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156]
INFO[0015] Starting container [etcd-fix-perm] on host [172.16.2.156], try #1
INFO[0015] Successfully started [etcd-fix-perm] container on host [172.16.2.156]
INFO[0015] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.156]
INFO[0015] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.156]
INFO[0015] Container [etcd-fix-perm] is still running on host [172.16.2.156]: stderr: , stdout:
INFO[0016] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.156]
INFO[0016] Removing container [etcd-fix-perm] on host [172.16.2.156], try #1
INFO[0016] [remove/etcd-fix-perm] Successfully removed container on host [172.16.2.156]
INFO[0016] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [172.16.2.156]
INFO[0016] Removing container [etcd-rolling-snapshots] on host [172.16.2.156], try #1
INFO[0026] [remove/etcd-rolling-snapshots] Successfully removed container on host [172.16.2.156]
INFO[0026] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156]
INFO[0027] Starting container [etcd-rolling-snapshots] on host [172.16.2.156], try #1
INFO[0027] [etcd] Successfully started [etcd-rolling-snapshots] container on host [172.16.2.156]
INFO[0032] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156]
INFO[0032] Starting container [rke-bundle-cert] on host [172.16.2.156], try #1
INFO[0033] [certificates] Successfully started [rke-bundle-cert] container on host [172.16.2.156]
INFO[0033] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.156]
INFO[0033] Container [rke-bundle-cert] is still running on host [172.16.2.156]: stderr: , stdout:
INFO[0034] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.156]
INFO[0034] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [172.16.2.156]
INFO[0034] Removing container [rke-bundle-cert] on host [172.16.2.156], try #1
INFO[0034] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156]
INFO[0034] Starting container [rke-log-linker] on host [172.16.2.156], try #1
INFO[0035] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.156]
INFO[0035] Removing container [rke-log-linker] on host [172.16.2.156], try #1
INFO[0035] [remove/rke-log-linker] Successfully removed container on host [172.16.2.156]
INFO[0035] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156]
INFO[0035] Starting container [rke-log-linker] on host [172.16.2.156], try #1
INFO[0035] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.156]
INFO[0035] Removing container [rke-log-linker] on host [172.16.2.156], try #1
INFO[0036] [remove/rke-log-linker] Successfully removed container on host [172.16.2.156]
INFO[0036] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154]
INFO[0036] Starting container [etcd-fix-perm] on host [172.16.2.154], try #1
INFO[0036] Successfully started [etcd-fix-perm] container on host [172.16.2.154]
INFO[0036] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.154]
INFO[0036] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.154]
INFO[0036] Container [etcd-fix-perm] is still running on host [172.16.2.154]: stderr: , stdout:
INFO[0037] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.154]
INFO[0037] Removing container [etcd-fix-perm] on host [172.16.2.154], try #1
INFO[0037] [remove/etcd-fix-perm] Successfully removed container on host [172.16.2.154]
INFO[0037] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [172.16.2.154]
INFO[0037] Removing container [etcd-rolling-snapshots] on host [172.16.2.154], try #1
INFO[0037] [remove/etcd-rolling-snapshots] Successfully removed container on host [172.16.2.154]
INFO[0037] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154]
INFO[0038] Starting container [etcd-rolling-snapshots] on host [172.16.2.154], try #1
INFO[0038] [etcd] Successfully started [etcd-rolling-snapshots] container on host [172.16.2.154]
INFO[0043] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154]
INFO[0043] Starting container [rke-bundle-cert] on host [172.16.2.154], try #1
INFO[0044] [certificates] Successfully started [rke-bundle-cert] container on host [172.16.2.154]
INFO[0044] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.154]
INFO[0044] Container [rke-bundle-cert] is still running on host [172.16.2.154]: stderr: , stdout:
INFO[0045] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.154]
INFO[0045] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [172.16.2.154]
INFO[0045] Removing container [rke-bundle-cert] on host [172.16.2.154], try #1
INFO[0045] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154]
INFO[0045] Starting container [rke-log-linker] on host [172.16.2.154], try #1
INFO[0045] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.154]
INFO[0045] Removing container [rke-log-linker] on host [172.16.2.154], try #1
INFO[0046] [remove/rke-log-linker] Successfully removed container on host [172.16.2.154]
INFO[0046] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154]
INFO[0046] Starting container [rke-log-linker] on host [172.16.2.154], try #1
INFO[0046] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.154]
INFO[0046] Removing container [rke-log-linker] on host [172.16.2.154], try #1
INFO[0046] [remove/rke-log-linker] Successfully removed container on host [172.16.2.154]
INFO[0046] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153]
INFO[0047] Starting container [etcd-fix-perm] on host [172.16.2.153], try #1
INFO[0047] Successfully started [etcd-fix-perm] container on host [172.16.2.153]
INFO[0047] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.153]
INFO[0047] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.153]
INFO[0047] Container [etcd-fix-perm] is still running on host [172.16.2.153]: stderr: , stdout:
INFO[0048] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.153]
INFO[0048] Removing container [etcd-fix-perm] on host [172.16.2.153], try #1
INFO[0048] [remove/etcd-fix-perm] Successfully removed container on host [172.16.2.153]
INFO[0048] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [172.16.2.153]
INFO[0048] Removing container [etcd-rolling-snapshots] on host [172.16.2.153], try #1
WARN[0058] Can’t remove Docker container [etcd-rolling-snapshots] for host [172.16.2.153]: Error response from daemon: Could not kill running container 11c3576d3281877a2dfee21bc16be73d9bbaf942bf658a542236712c43c1fd0b, cannot remove - container 11c3576d3281 PID 122516 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes
INFO[0058] Removing container [etcd-rolling-snapshots] on host [172.16.2.153], try #2
INFO[0058] [remove/etcd-rolling-snapshots] Successfully removed container on host [172.16.2.153]
INFO[0058] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153]
INFO[0058] Starting container [etcd-rolling-snapshots] on host [172.16.2.153], try #1
INFO[0059] [etcd] Successfully started [etcd-rolling-snapshots] container on host [172.16.2.153]
INFO[0064] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153]
INFO[0064] Starting container [rke-bundle-cert] on host [172.16.2.153], try #1
INFO[0064] [certificates] Successfully started [rke-bundle-cert] container on host [172.16.2.153]
INFO[0064] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.153]
INFO[0064] Container [rke-bundle-cert] is still running on host [172.16.2.153]: stderr: , stdout:
INFO[0065] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.153]
INFO[0065] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [172.16.2.153]
INFO[0065] Removing container [rke-bundle-cert] on host [172.16.2.153], try #1
INFO[0065] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153]
INFO[0066] Starting container [rke-log-linker] on host [172.16.2.153], try #1
INFO[0066] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.153]
INFO[0066] Removing container [rke-log-linker] on host [172.16.2.153], try #1
INFO[0066] [remove/rke-log-linker] Successfully removed container on host [172.16.2.153]
INFO[0066] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153]
INFO[0067] Starting container [rke-log-linker] on host [172.16.2.153], try #1
INFO[0067] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.153]
INFO[0067] Removing container [rke-log-linker] on host [172.16.2.153], try #1
INFO[0067] [remove/rke-log-linker] Successfully removed container on host [172.16.2.153]
INFO[0067] [etcd] Successfully started etcd plane.. Checking etcd cluster health
INFO[0068] [etcd] etcd host [172.16.2.156] reported healthy=true
INFO[0068] [controlplane] Now checking status of node mosip-node1, try #1
INFO[0068] [controlplane] Now checking status of node mosip-node2, try #1
INFO[0068] [controlplane] Now checking status of node mosip-node3, try #1
INFO[0068] [controlplane] Processing controlplane hosts for upgrade 1 at a time
INFO[0068] Processing controlplane host mosip-node1
INFO[0068] [controlplane] Now checking status of node mosip-node1, try #1
INFO[0068] [controlplane] Getting list of nodes for upgrade
INFO[0068] Upgrade not required for controlplane and worker components of host mosip-node1
INFO[0068] Processing controlplane host mosip-node2
INFO[0068] [controlplane] Now checking status of node mosip-node2, try #1
INFO[0068] [controlplane] Getting list of nodes for upgrade
INFO[0068] Upgrade not required for controlplane and worker components of host mosip-node2
INFO[0068] Processing controlplane host mosip-node3
INFO[0068] [controlplane] Now checking status of node mosip-node3, try #1
INFO[0068] [controlplane] Getting list of nodes for upgrade
INFO[0068] Upgrade not required for controlplane and worker components of host mosip-node3
INFO[0068] [controlplane] Successfully upgraded Controller Plane..
INFO[0068] [authz] Creating rke-job-deployer ServiceAccount
INFO[0068] [authz] rke-job-deployer ServiceAccount created successfully
INFO[0068] [authz] Creating system:node ClusterRoleBinding
INFO[0068] [authz] system:node ClusterRoleBinding created successfully
INFO[0068] [authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding
INFO[0068] [authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully
INFO[0068] Successfully Deployed state file at [./cluster.rkestate]
INFO[0068] [state] Saving full cluster state to Kubernetes
INFO[0068] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state
INFO[0068] [worker] Now checking status of node mosip-node4, try #1
INFO[0093] [worker] Now checking status of node mosip-node5, try #1
INFO[0119] [worker] Now checking status of node mosip-node6, try #1
INFO[0144] Attempting upgrade of worker components on following hosts in NotReady status: mosip-node4,mosip-node5,mosip-node6
INFO[0144] [worker] Building up Worker Plane..
INFO[0144] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150]
INFO[0144] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152]
INFO[0144] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151]
INFO[0145] Starting container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0145] Starting container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0145] Starting container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0145] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150]
INFO[0145] Removing container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0145] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152]
INFO[0145] Removing container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0145] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151]
INFO[0145] Removing container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0145] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150]
INFO[0145] Checking if container [service-sidekick] is running on host [172.16.2.150], try #1
INFO[0145] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152]
INFO[0145] Checking if container [service-sidekick] is running on host [172.16.2.152], try #1
INFO[0145] [sidekick] Sidekick container already created on host [172.16.2.150]
INFO[0145] Restarting container [kubelet] on host [172.16.2.150], try #1
INFO[0145] [sidekick] Sidekick container already created on host [172.16.2.152]
INFO[0145] Restarting container [kubelet] on host [172.16.2.152], try #1
INFO[0145] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151]
INFO[0145] Checking if container [service-sidekick] is running on host [172.16.2.151], try #1
INFO[0145] [sidekick] Sidekick container already created on host [172.16.2.151]
INFO[0145] Restarting container [kubelet] on host [172.16.2.151], try #1
INFO[0145] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.150]
INFO[0146] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.151]
INFO[0146] [healthcheck] service [kubelet] on host [172.16.2.150] is healthy
INFO[0146] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150]
INFO[0146] [healthcheck] service [kubelet] on host [172.16.2.151] is healthy
INFO[0146] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151]
INFO[0146] Starting container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0146] Starting container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0146] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.152]
INFO[0146] [healthcheck] service [kubelet] on host [172.16.2.152] is healthy
INFO[0146] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152]
INFO[0146] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150]
INFO[0146] Removing container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0146] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151]
INFO[0146] Removing container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0147] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150]
INFO[0147] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.150]
INFO[0147] [healthcheck] service [kube-proxy] on host [172.16.2.150] is healthy
INFO[0147] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151]
INFO[0147] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150]
INFO[0147] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.151]
INFO[0147] Starting container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0147] [healthcheck] service [kube-proxy] on host [172.16.2.151] is healthy
INFO[0147] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151]
INFO[0147] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150]
INFO[0147] Removing container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0148] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150]
INFO[0148] Starting container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0148] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152]
INFO[0148] Removing container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0148] Starting container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0148] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152]
INFO[0148] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.152]
INFO[0148] [healthcheck] service [kube-proxy] on host [172.16.2.152] is healthy
INFO[0148] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152]
INFO[0148] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151]
INFO[0148] Removing container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0149] Starting container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0149] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151]
INFO[0149] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152]
INFO[0149] Removing container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0149] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152]
INFO[0149] [worker] Successfully started Worker Plane..
INFO[0149] [worker] Now checking status of node mosip-node4, try #1
ERRO[0174] Host mosip-node4 failed to report Ready status with error: [worker] Error getting node mosip-node4: “mosip-node4” not found
INFO[0174] [worker] Now checking status of node mosip-node5, try #1
ERRO[0199] Host mosip-node5 failed to report Ready status with error: [worker] Error getting node mosip-node5: “mosip-node5” not found
INFO[0199] [worker] Now checking status of node mosip-node6, try #1
ERRO[0224] Host mosip-node6 failed to report Ready status with error: [worker] Error getting node mosip-node6: “mosip-node6” not found
INFO[0224] [worker] Upgrading Worker Plane..
INFO[0299] Now checking and upgrading worker components on nodes with only worker role 1 at a time
INFO[0299] [workerplane] Processing host mosip-node4
INFO[0300] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152]
INFO[0300] Starting container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0301] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152]
INFO[0301] Removing container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0301] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152]
INFO[0301] Checking if container [service-sidekick] is running on host [172.16.2.152], try #1
INFO[0301] [sidekick] Sidekick container already created on host [172.16.2.152]
INFO[0301] Restarting container [kubelet] on host [172.16.2.152], try #1
INFO[0301] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.152]
INFO[0302] [healthcheck] service [kubelet] on host [172.16.2.152] is healthy
INFO[0302] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152]
INFO[0302] Starting container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0302] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152]
INFO[0302] Removing container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0302] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152]
INFO[0302] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.152]
INFO[0303] [healthcheck] service [kube-proxy] on host [172.16.2.152] is healthy
INFO[0303] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152]
INFO[0303] Starting container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0303] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152]
INFO[0303] Removing container [rke-log-linker] on host [172.16.2.152], try #1
INFO[0303] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152]
INFO[0303] [workerplane] Processing host mosip-node5
INFO[0304] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151]
INFO[0304] Starting container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0305] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151]
INFO[0305] Removing container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0305] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151]
INFO[0305] Checking if container [service-sidekick] is running on host [172.16.2.151], try #1
INFO[0305] [sidekick] Sidekick container already created on host [172.16.2.151]
INFO[0305] Restarting container [kubelet] on host [172.16.2.151], try #1
INFO[0305] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.151]
INFO[0306] [healthcheck] service [kubelet] on host [172.16.2.151] is healthy
INFO[0306] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151]
INFO[0306] Starting container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0306] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151]
INFO[0306] Removing container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0306] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151]
INFO[0306] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.151]
INFO[0307] [healthcheck] service [kube-proxy] on host [172.16.2.151] is healthy
INFO[0307] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151]
INFO[0307] Starting container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0307] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151]
INFO[0307] Removing container [rke-log-linker] on host [172.16.2.151], try #1
INFO[0307] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151]
INFO[0307] [workerplane] Processing host mosip-node6
INFO[0308] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150]
INFO[0308] Starting container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0309] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150]
INFO[0309] Removing container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0309] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150]
INFO[0309] Checking if container [service-sidekick] is running on host [172.16.2.150], try #1
INFO[0309] [sidekick] Sidekick container already created on host [172.16.2.150]
INFO[0309] Restarting container [kubelet] on host [172.16.2.150], try #1
INFO[0309] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.150]
INFO[0310] [healthcheck] service [kubelet] on host [172.16.2.150] is healthy
INFO[0310] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150]
INFO[0310] Starting container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0310] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150]
INFO[0310] Removing container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0310] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150]
INFO[0310] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.150]
INFO[0311] [healthcheck] service [kube-proxy] on host [172.16.2.150] is healthy
INFO[0311] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150]
INFO[0311] Starting container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0311] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150]
INFO[0311] Removing container [rke-log-linker] on host [172.16.2.150], try #1
INFO[0311] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150]
INFO[0311] [worker] Successfully upgraded Worker Plane..
INFO[0311] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152]
INFO[0311] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150]
INFO[0311] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151]
INFO[0311] Starting container [rke-log-cleaner] on host [172.16.2.150], try #1
INFO[0311] Starting container [rke-log-cleaner] on host [172.16.2.152], try #1
INFO[0312] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.150]
INFO[0312] Removing container [rke-log-cleaner] on host [172.16.2.150], try #1
INFO[0312] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.152]
INFO[0312] Removing container [rke-log-cleaner] on host [172.16.2.152], try #1
INFO[0312] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156]
INFO[0312] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153]
INFO[0312] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.150]
INFO[0312] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.152]
INFO[0312] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154]
INFO[0314] Starting container [rke-log-cleaner] on host [172.16.2.156], try #1
INFO[0314] Starting container [rke-log-cleaner] on host [172.16.2.151], try #1
INFO[0314] Starting container [rke-log-cleaner] on host [172.16.2.153], try #1
INFO[0314] Starting container [rke-log-cleaner] on host [172.16.2.154], try #1
INFO[0315] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.156]
INFO[0315] Removing container [rke-log-cleaner] on host [172.16.2.156], try #1
INFO[0315] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.154]
INFO[0315] Removing container [rke-log-cleaner] on host [172.16.2.154], try #1
INFO[0315] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.153]
INFO[0315] Removing container [rke-log-cleaner] on host [172.16.2.153], try #1
INFO[0315] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.151]
INFO[0315] Removing container [rke-log-cleaner] on host [172.16.2.151], try #1
INFO[0315] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.153]
INFO[0315] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.151]
INFO[0315] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.154]
INFO[0316] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.156]
INFO[0316] [sync] Syncing nodes Labels and Taints
FATA[0478] [ “mosip-node6” not found]
The error are in bold
The content of Hosts.ini:
[mosip_nodes]
mosip-node1 ansible_host=172.16.2.156 ansible_user=john ansible_ssh_private_key_file=~/.ssh/id_ed25519 ansible_become=true
mosip-node2 ansible_host=172.16.2.154 ansible_user=john ansible_ssh_private_key_file=~/.ssh/id_ed25519 ansible_become=true
mosip-node3 ansible_host=172.16.2.153 ansible_user=john ansible_ssh_private_key_file=~/.ssh/id_ed25519 ansible_become=true
mosip-node4 ansible_host=172.16.2.152 ansible_user=john ansible_ssh_private_key_file=~/.ssh/id_ed25519 ansible_become=true
mosip-node5 ansible_host=172.16.2.151 ansible_user=john ansible_ssh_private_key_file=~/.ssh/id_ed25519 ansible_become=true
mosip-node6 ansible_host=172.16.2.150 ansible_user=john ansible_ssh_private_key_file=~/.ssh/id_ed25519 ansible_become=true
[etcd_nodes]
mosip-node1 ansible_host=172.16.2.156
mosip-node2 ansible_host=172.16.2.154
mosip-node3 ansible_host=172.16.2.153
Hi @Johnny,
Please follow the official documentation here:
MOSIP On-Prem Installation Guide and try again.
Note: Your hosts.ini file format appears to be incorrect — kindly review and update it as per the structure mentioned in the documentation.
