# Cannot download repomd.xml **URL:** \ **Category:** Support\ **Created:** [April 23, 2025, 11:51am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020 "2025-04-23T11:51:47Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [April 23, 2025, 11:51am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/1 "2025-04-23T11:51:47Z")
![image](https://europe1.discourse-cdn.com/flex017/uploads/mosip1/original/2X/3/3cdf3bd74e8a62a3d9185c76f716b6db61edda63.png) I am presently trying to set up mosip v2 using centos 9 and I keep getting this error: fatal: [console.sb]: FAILED! =\> {“changed”: false, “msg”: “Failed to download metadata for repo ‘kubernetes’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried”, “rc”: 1, “results”: } ---
**Author:** ![svahsek](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/svahsek/32/1098_2.png) [@svahsek](https://community.mosip.io/u/svahsek)\ **Post date:** [April 24, 2025, 6:27am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/2 "2025-04-24T06:27:27Z")
Dear @Johnny Thank you for reaching out, as you’re already aware about CentOS having been deprecated and the latest versions of MOSIP is not tested with CentOS, I will advise you to not to try it with CentOS. Please refer to our [On-Prem Installation Guidelines | MOSIP Docs 1.2.0](https://docs.mosip.io/1.2.0/setup/deploymentnew/v3-installation/on-prem-installation-guidelines). Best Regards Team MOSIP ---
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [May 14, 2025, 7:19pm UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/3 "2025-05-14T19:19:29Z")
Hi @svahsek, Thanks for your previous response, I am now working with On-Prem Insatallation 1.20 using ubuntu and I need your assistant, I was able to successful deploy the RKE Cluster, which I check by executing kubectl get nodes and kubectl get pods -A and get a response showing nodes ready or pod runnning but executing the command in an hour again shows connection refused. find attached the two scenario ![kubectl](https://europe1.discourse-cdn.com/flex017/uploads/mosip1/original/2X/1/156f95007b61f20e02520c5a8746eb14582351b8.jpeg) ![kubectl2](https://europe1.discourse-cdn.com/flex017/uploads/mosip1/original/2X/0/02cf18e97fb0c6a4363ec6d25acc7b667a70c449.jpeg) ---
**Author:** ![svahsek](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/svahsek/32/1098_2.png) [@svahsek](https://community.mosip.io/u/svahsek)\ **Post date:** [May 15, 2025, 5:37am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/4 "2025-05-15T05:37:00Z")
Dear Johny, Than you for reaching out, One of our experts will respond to you on this. Regards Team MOSIP ---
**Author:** ![Praful\_Rakhade](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/praful_rakhade/32/1065_2.png) [@Praful\_Rakhade](https://community.mosip.io/u/Praful_Rakhade)\ **Post date:** [May 15, 2025, 7:02am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/5 "2025-05-15T07:02:14Z")
Hello @Johnny, Could you please provide more details about the deployment? 1. Are you planning to create an observation cluster? 2. Have you verified that all nodes are up and running? 3. If yes, please follow the steps below: Copy the kubeconfig file: `cp kube_config_cluster.yml $HOME/.kube/_config` `chmod 400 $HOME/.kube/_config` To access the cluster using the kubeconfig file `export KUBECONFIG="$HOME/.kube/_config"` Please try the above steps and let us know if you face any issues. Regards Team MOSIP ---
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [May 16, 2025, 9:39am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/6 "2025-05-16T09:39:27Z")
Hi Praful\_Rakhade, Thank you very much for your response. I am presently facing an issue with RKE and Docker deployment. I deploy Docker version 28.1.1 and whenever I execute the deployment of rke up --config cluster.yml(rke version v1.3.10), I get this error: FATA[0006] Unsupported Docker version found [28.1.1] on host [172.16.2.156], supported versions are [1.13.x 17.03.x 17.06.x 17.09.x 18.06.x 18.09.x 19.03.x 20.10.x] I have tried to downgrade the docker to 20.10.24, it keeps giving error. Please, can you help with any known solution. Thank you ---
**Author:** ![Praful\_Rakhade](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/praful_rakhade/32/1065_2.png) [@Praful\_Rakhade](https://community.mosip.io/u/Praful_Rakhade)\ **Post date:** [May 16, 2025, 10:19am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/7 "2025-05-16T10:19:50Z")
Hello @Johnny, Please remove any existing Docker installation from your virtual machines and reinstall it using our official guide: 👉 [MOSIP On-Prem Setup – Rancher Guide](https://github.com/mosip/k8s-infra/tree/main/rancher/on-prem) Using the recommended Docker version and setup ensures compatibility with the MOSIP infrastructure. Regards Team MOSIP ---
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [May 16, 2025, 10:50am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/8 "2025-05-16T10:50:48Z")
Hi Praful\_Rakhade, Thank you for your quick response, I follow the official and this is the error: john@bastion:~/k8s-infra/rancher/on-prem$ ansible-playbook -i hosts.ini docker.yaml [WARNING]: Could not match supplied host pattern, ignoring: cluster PLAY [cluster] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* skipping: no hosts matched PLAY RECAP \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* john@bastion:~/k8s-infra/rancher/on-prem$ ^C john@bastion:~/k8s-infra/rancher/on-prem$ vi hosts.ini john@bastion:~/k8s-infra/rancher/on-prem$ ansible-playbook -i hosts.ini docker.yaml [WARNING]: \* Failed to parse /home/john/k8s-infra/rancher/on-prem/hosts.ini with ini plugin: /home/john/k8s-infra/rancher/on-prem/hosts.ini:17: Section [docker\_nodes:children] includes undefined group: mosip\_nodes [WARNING]: Unable to parse /home/john/k8s-infra/rancher/on-prem/hosts.ini as an inventory source [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match ‘all’ PLAY [cluster] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* TASK [Gathering Facts] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* [WARNING]: Platform linux on host mosip-node2 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python interpreter could change the meaning of that path. See [Interpreter Discovery — Ansible Core Documentation](https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html) for more information. ok: [mosip-node2] [WARNING]: Platform linux on host mosip-node5 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python interpreter could change the meaning of that path. See [Interpreter Discovery — Ansible Core Documentation](https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html) for more information. ok: [mosip-node5] [WARNING]: Platform linux on host mosip-node1 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python interpreter could change the meaning of that path. See [Interpreter Discovery — Ansible Core Documentation](https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html) for more information. ok: [mosip-node1] [WARNING]: Platform linux on host mosip-node4 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python interpreter could change the meaning of that path. See [Interpreter Discovery — Ansible Core Documentation](https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html) for more information. ok: [mosip-node4] [WARNING]: Platform linux on host mosip-node3 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python interpreter could change the meaning of that path. See [Interpreter Discovery — Ansible Core Documentation](https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html) for more information. ok: [mosip-node3] [WARNING]: Platform linux on host mosip-node6 is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python interpreter could change the meaning of that path. See [Interpreter Discovery — Ansible Core Documentation](https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html) for more information. ok: [mosip-node6] TASK [Install [docker.io](http://docker.io) for Debian] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* fatal: [mosip-node5]: FAILED! =\> {“cache\_update\_time”: 1747392418, “cache\_updated”: true, “changed”: false, “msg”: “no available installation candidate for docker.io=20.10._"} fatal: [mosip-node1]: FAILED! =\> {“cache\_update\_time”: 1747392418, “cache\_updated”: true, “changed”: false, “msg”: "no available installation candidate for docker.io=20.10._”} fatal: [mosip-node2]: FAILED! =\> {“cache\_update\_time”: 1747392418, “cache\_updated”: true, “changed”: false, “msg”: “no available installation candidate for docker.io=20.10._"} fatal: [mosip-node3]: FAILED! =\> {“cache\_update\_time”: 1747392418, “cache\_updated”: true, “changed”: false, “msg”: "no available installation candidate for docker.io=20.10._”} fatal: [mosip-node4]: FAILED! =\> {“cache\_update\_time”: 1747392418, “cache\_updated”: true, “changed”: false, “msg”: “no available installation candidate for docker.io=20.10._"} fatal: [mosip-node6]: FAILED! =\> {“cache\_update\_time”: 1747392426, “cache\_updated”: true, “changed”: false, “msg”: "no available installation candidate for docker.io=20.10._”} PLAY RECAP \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* mosip-node1 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 mosip-node2 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 mosip-node3 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 mosip-node4 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 mosip-node5 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 mosip-node6 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Kindly assist ---
**Author:** ![Praful\_Rakhade](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/praful_rakhade/32/1065_2.png) [@Praful\_Rakhade](https://community.mosip.io/u/Praful_Rakhade)\ **Post date:** [May 19, 2025, 7:23am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/9 "2025-05-19T07:23:43Z")
Hello @Johnny, Could you please share your `host.ini` file? This will help us better understand your environment and assist you more effectively. Thanks! Regards Team MOSIP ---
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [May 19, 2025, 8:50am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/10 "2025-05-19T08:50:26Z")
Hi Prafu\_Rakhade, I have been able to go beyond that part but I am now having error at the point of setting up Mosip Cluster. I get this error when I am trying to deploy the RKE cluster: john@bastion:~/k8s-infra/mosip/on-prem$ rke up --config cluster.yml INFO[0000] Running RKE version: v1.3.10 INFO[0000] Initiating Kubernetes cluster INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates INFO[0000] [certificates] Generating admin certificates and kubeconfig INFO[0000] Successfully Deployed state file at [./cluster.rkestate] INFO[0000] Building Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [172.16.2.151] INFO[0000] [dialer] Setup tunnel for host [172.16.2.154] INFO[0000] [dialer] Setup tunnel for host [172.16.2.153] INFO[0000] [dialer] Setup tunnel for host [172.16.2.152] INFO[0000] [dialer] Setup tunnel for host [172.16.2.150] INFO[0000] [dialer] Setup tunnel for host [172.16.2.156] INFO[0001] [network] No hosts added existing cluster, skipping port check INFO[0001] [certificates] Deploying kubernetes certificates to Cluster nodes INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.154], try #1 INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.156], try #1 INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.153], try #1 INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.152], try #1 INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.151], try #1 INFO[0001] Checking if container [cert-deployer] is running on host [172.16.2.150], try #1 INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156] INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153] INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154] INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152] INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150] INFO[0001] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151] INFO[0001] Starting container [cert-deployer] on host [172.16.2.156], try #1 INFO[0002] Starting container [cert-deployer] on host [172.16.2.151], try #1 INFO[0002] Checking if container [cert-deployer] is running on host [172.16.2.156], try #1 INFO[0002] Starting container [cert-deployer] on host [172.16.2.152], try #1 INFO[0002] Starting container [cert-deployer] on host [172.16.2.150], try #1 INFO[0003] Checking if container [cert-deployer] is running on host [172.16.2.151], try #1 INFO[0003] Checking if container [cert-deployer] is running on host [172.16.2.152], try #1 INFO[0003] Checking if container [cert-deployer] is running on host [172.16.2.150], try #1 INFO[0003] Starting container [cert-deployer] on host [172.16.2.153], try #1 INFO[0003] Checking if container [cert-deployer] is running on host [172.16.2.153], try #1 INFO[0003] Starting container [cert-deployer] on host [172.16.2.154], try #1 INFO[0004] Checking if container [cert-deployer] is running on host [172.16.2.154], try #1 INFO[0007] Checking if container [cert-deployer] is running on host [172.16.2.156], try #1 INFO[0007] Removing container [cert-deployer] on host [172.16.2.156], try #1 INFO[0008] Checking if container [cert-deployer] is running on host [172.16.2.151], try #1 INFO[0008] Removing container [cert-deployer] on host [172.16.2.151], try #1 INFO[0008] Checking if container [cert-deployer] is running on host [172.16.2.152], try #1 INFO[0008] Removing container [cert-deployer] on host [172.16.2.152], try #1 INFO[0008] Checking if container [cert-deployer] is running on host [172.16.2.150], try #1 INFO[0008] Removing container [cert-deployer] on host [172.16.2.150], try #1 INFO[0008] Checking if container [cert-deployer] is running on host [172.16.2.153], try #1 INFO[0008] Removing container [cert-deployer] on host [172.16.2.153], try #1 INFO[0009] Checking if container [cert-deployer] is running on host [172.16.2.154], try #1 INFO[0009] Removing container [cert-deployer] on host [172.16.2.154], try #1 INFO[0009] [reconcile] Rebuilding and updating local kube config INFO[0009] Successfully Deployed local admin kubeconfig at [./kube\_config\_cluster.yml] INFO[0009] [reconcile] host [172.16.2.156] is a control plane node with reachable Kubernetes API endpoint in the cluster INFO[0009] [certificates] Successfully deployed kubernetes certificates to Cluster nodes INFO[0009] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.16.2.156] INFO[0009] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156] INFO[0009] Starting container [file-deployer] on host [172.16.2.156], try #1 INFO[0010] Successfully started [file-deployer] container on host [172.16.2.156] INFO[0010] Waiting for [file-deployer] container to exit on host [172.16.2.156] INFO[0010] Waiting for [file-deployer] container to exit on host [172.16.2.156] INFO[0010] Container [file-deployer] is still running on host [172.16.2.156]: stderr: , stdout: INFO[0011] Waiting for [file-deployer] container to exit on host [172.16.2.156] INFO[0011] Removing container [file-deployer] on host [172.16.2.156], try #1 ---
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [May 19, 2025, 8:52am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/11 "2025-05-19T08:52:50Z")
INFO[0011] [remove/file-deployer] Successfully removed container on host [172.16.2.156] INFO[0011] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.16.2.154] INFO[0011] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154] INFO[0011] Starting container [file-deployer] on host [172.16.2.154], try #1 INFO[0011] Successfully started [file-deployer] container on host [172.16.2.154] INFO[0011] Waiting for [file-deployer] container to exit on host [172.16.2.154] INFO[0011] Waiting for [file-deployer] container to exit on host [172.16.2.154] INFO[0011] Container [file-deployer] is still running on host [172.16.2.154]: stderr: , stdout: INFO[0012] Waiting for [file-deployer] container to exit on host [172.16.2.154] INFO[0012] Removing container [file-deployer] on host [172.16.2.154], try #1 INFO[0012] [remove/file-deployer] Successfully removed container on host [172.16.2.154] INFO[0012] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.16.2.153] INFO[0012] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153] INFO[0013] Starting container [file-deployer] on host [172.16.2.153], try #1 INFO[0013] Successfully started [file-deployer] container on host [172.16.2.153] INFO[0013] Waiting for [file-deployer] container to exit on host [172.16.2.153] INFO[0013] Waiting for [file-deployer] container to exit on host [172.16.2.153] INFO[0013] Container [file-deployer] is still running on host [172.16.2.153]: stderr: , stdout: INFO[0014] Waiting for [file-deployer] container to exit on host [172.16.2.153] INFO[0014] Removing container [file-deployer] on host [172.16.2.153], try #1 INFO[0014] [remove/file-deployer] Successfully removed container on host [172.16.2.153] INFO[0014] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes INFO[0014] [reconcile] Reconciling cluster state INFO[0014] [reconcile] Check etcd hosts to be deleted INFO[0014] [reconcile] Check etcd hosts to be added INFO[0014] [reconcile] Rebuilding and updating local kube config INFO[0014] Successfully Deployed local admin kubeconfig at [./kube\_config\_cluster.yml] INFO[0014] [reconcile] host [172.16.2.156] is a control plane node with reachable Kubernetes API endpoint in the cluster INFO[0014] [reconcile] Reconciled cluster state successfully INFO[0014] max\_unavailable\_worker got rounded down to 0, resetting to 1 INFO[0014] Setting maxUnavailable for worker nodes to: 1 INFO[0014] Setting maxUnavailable for controlplane nodes to: 1 INFO[0014] Pre-pulling kubernetes images INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.153] INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.156] INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.151] INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.154] INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.150] INFO[0014] Image [rancher/hyperkube:v1.22.9-rancher1] exists on host [172.16.2.152] INFO[0014] Kubernetes images pulled successfully INFO[0014] [etcd] Building up etcd plane.. INFO[0014] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156] INFO[0015] Starting container [etcd-fix-perm] on host [172.16.2.156], try #1 INFO[0015] Successfully started [etcd-fix-perm] container on host [172.16.2.156] INFO[0015] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.156] INFO[0015] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.156] INFO[0015] Container [etcd-fix-perm] is still running on host [172.16.2.156]: stderr: , stdout: INFO[0016] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.156] INFO[0016] Removing container [etcd-fix-perm] on host [172.16.2.156], try #1 INFO[0016] [remove/etcd-fix-perm] Successfully removed container on host [172.16.2.156] INFO[0016] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [172.16.2.156] INFO[0016] Removing container [etcd-rolling-snapshots] on host [172.16.2.156], try #1 INFO[0026] [remove/etcd-rolling-snapshots] Successfully removed container on host [172.16.2.156] INFO[0026] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156] INFO[0027] Starting container [etcd-rolling-snapshots] on host [172.16.2.156], try #1 INFO[0027] [etcd] Successfully started [etcd-rolling-snapshots] container on host [172.16.2.156] INFO[0032] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156] INFO[0032] Starting container [rke-bundle-cert] on host [172.16.2.156], try #1 INFO[0033] [certificates] Successfully started [rke-bundle-cert] container on host [172.16.2.156] INFO[0033] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.156] INFO[0033] Container [rke-bundle-cert] is still running on host [172.16.2.156]: stderr: , stdout: INFO[0034] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.156] INFO[0034] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [172.16.2.156] INFO[0034] Removing container [rke-bundle-cert] on host [172.16.2.156], try #1 INFO[0034] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156] INFO[0034] Starting container [rke-log-linker] on host [172.16.2.156], try #1 INFO[0035] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.156] INFO[0035] Removing container [rke-log-linker] on host [172.16.2.156], try #1 INFO[0035] [remove/rke-log-linker] Successfully removed container on host [172.16.2.156] INFO[0035] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156] INFO[0035] Starting container [rke-log-linker] on host [172.16.2.156], try #1 INFO[0035] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.156] INFO[0035] Removing container [rke-log-linker] on host [172.16.2.156], try #1 INFO[0036] [remove/rke-log-linker] Successfully removed container on host [172.16.2.156] INFO[0036] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154] INFO[0036] Starting container [etcd-fix-perm] on host [172.16.2.154], try #1 INFO[0036] Successfully started [etcd-fix-perm] container on host [172.16.2.154] INFO[0036] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.154] INFO[0036] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.154] INFO[0036] Container [etcd-fix-perm] is still running on host [172.16.2.154]: stderr: , stdout: INFO[0037] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.154] INFO[0037] Removing container [etcd-fix-perm] on host [172.16.2.154], try #1 INFO[0037] [remove/etcd-fix-perm] Successfully removed container on host [172.16.2.154] INFO[0037] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [172.16.2.154] INFO[0037] Removing container [etcd-rolling-snapshots] on host [172.16.2.154], try #1 INFO[0037] [remove/etcd-rolling-snapshots] Successfully removed container on host [172.16.2.154] INFO[0037] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154] INFO[0038] Starting container [etcd-rolling-snapshots] on host [172.16.2.154], try #1 INFO[0038] [etcd] Successfully started [etcd-rolling-snapshots] container on host [172.16.2.154] INFO[0043] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154] INFO[0043] Starting container [rke-bundle-cert] on host [172.16.2.154], try #1 INFO[0044] [certificates] Successfully started [rke-bundle-cert] container on host [172.16.2.154] INFO[0044] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.154] INFO[0044] Container [rke-bundle-cert] is still running on host [172.16.2.154]: stderr: , stdout: INFO[0045] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.154] INFO[0045] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [172.16.2.154] INFO[0045] Removing container [rke-bundle-cert] on host [172.16.2.154], try #1 INFO[0045] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154] INFO[0045] Starting container [rke-log-linker] on host [172.16.2.154], try #1 INFO[0045] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.154] INFO[0045] Removing container [rke-log-linker] on host [172.16.2.154], try #1 INFO[0046] [remove/rke-log-linker] Successfully removed container on host [172.16.2.154] INFO[0046] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154] INFO[0046] Starting container [rke-log-linker] on host [172.16.2.154], try #1 INFO[0046] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.154] INFO[0046] Removing container [rke-log-linker] on host [172.16.2.154], try #1 INFO[0046] [remove/rke-log-linker] Successfully removed container on host [172.16.2.154] INFO[0046] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153] INFO[0047] Starting container [etcd-fix-perm] on host [172.16.2.153], try #1 INFO[0047] Successfully started [etcd-fix-perm] container on host [172.16.2.153] INFO[0047] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.153] INFO[0047] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.153] INFO[0047] Container [etcd-fix-perm] is still running on host [172.16.2.153]: stderr: , stdout: INFO[0048] Waiting for [etcd-fix-perm] container to exit on host [172.16.2.153] INFO[0048] Removing container [etcd-fix-perm] on host [172.16.2.153], try #1 INFO[0048] [remove/etcd-fix-perm] Successfully removed container on host [172.16.2.153] INFO[0048] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [172.16.2.153] INFO[0048] Removing container [etcd-rolling-snapshots] on host [172.16.2.153], try #1 WARN[0058] Can’t remove Docker container [etcd-rolling-snapshots] for host [172.16.2.153]: Error response from daemon: Could not kill running container 11c3576d3281877a2dfee21bc16be73d9bbaf942bf658a542236712c43c1fd0b, cannot remove - container 11c3576d3281 PID 122516 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes INFO[0058] Removing container [etcd-rolling-snapshots] on host [172.16.2.153], try #2 INFO[0058] [remove/etcd-rolling-snapshots] Successfully removed container on host [172.16.2.153] INFO[0058] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153] INFO[0058] Starting container [etcd-rolling-snapshots] on host [172.16.2.153], try #1 INFO[0059] [etcd] Successfully started [etcd-rolling-snapshots] container on host [172.16.2.153] INFO[0064] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153] INFO[0064] Starting container [rke-bundle-cert] on host [172.16.2.153], try #1 INFO[0064] [certificates] Successfully started [rke-bundle-cert] container on host [172.16.2.153] INFO[0064] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.153] INFO[0064] Container [rke-bundle-cert] is still running on host [172.16.2.153]: stderr: , stdout: INFO[0065] Waiting for [rke-bundle-cert] container to exit on host [172.16.2.153] INFO[0065] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [172.16.2.153] INFO[0065] Removing container [rke-bundle-cert] on host [172.16.2.153], try #1 INFO[0065] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153] INFO[0066] Starting container [rke-log-linker] on host [172.16.2.153], try #1 INFO[0066] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.153] INFO[0066] Removing container [rke-log-linker] on host [172.16.2.153], try #1 INFO[0066] [remove/rke-log-linker] Successfully removed container on host [172.16.2.153] INFO[0066] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153] INFO[0067] Starting container [rke-log-linker] on host [172.16.2.153], try #1 INFO[0067] [etcd] Successfully started [rke-log-linker] container on host [172.16.2.153] INFO[0067] Removing container [rke-log-linker] on host [172.16.2.153], try #1 INFO[0067] [remove/rke-log-linker] Successfully removed container on host [172.16.2.153] INFO[0067] [etcd] Successfully started etcd plane.. Checking etcd cluster health INFO[0068] [etcd] etcd host [172.16.2.156] reported healthy=true INFO[0068] [controlplane] Now checking status of node mosip-node1, try #1 INFO[0068] [controlplane] Now checking status of node mosip-node2, try #1 INFO[0068] [controlplane] Now checking status of node mosip-node3, try #1 INFO[0068] [controlplane] Processing controlplane hosts for upgrade 1 at a time INFO[0068] Processing controlplane host mosip-node1 INFO[0068] [controlplane] Now checking status of node mosip-node1, try #1 INFO[0068] [controlplane] Getting list of nodes for upgrade INFO[0068] Upgrade not required for controlplane and worker components of host mosip-node1 INFO[0068] Processing controlplane host mosip-node2 INFO[0068] [controlplane] Now checking status of node mosip-node2, try #1 INFO[0068] [controlplane] Getting list of nodes for upgrade INFO[0068] Upgrade not required for controlplane and worker components of host mosip-node2 INFO[0068] Processing controlplane host mosip-node3 INFO[0068] [controlplane] Now checking status of node mosip-node3, try #1 INFO[0068] [controlplane] Getting list of nodes for upgrade INFO[0068] Upgrade not required for controlplane and worker components of host mosip-node3 INFO[0068] [controlplane] Successfully upgraded Controller Plane.. INFO[0068] [authz] Creating rke-job-deployer ServiceAccount INFO[0068] [authz] rke-job-deployer ServiceAccount created successfully INFO[0068] [authz] Creating system:node ClusterRoleBinding INFO[0068] [authz] system:node ClusterRoleBinding created successfully INFO[0068] [authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding INFO[0068] [authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully INFO[0068] Successfully Deployed state file at [./cluster.rkestate] INFO[0068] [state] Saving full cluster state to Kubernetes INFO[0068] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state INFO[0068] [worker] Now checking status of node mosip-node4, try #1 INFO[0093] [worker] Now checking status of node mosip-node5, try #1 INFO[0119] [worker] Now checking status of node mosip-node6, try #1 INFO[0144] Attempting upgrade of worker components on following hosts in NotReady status: mosip-node4,mosip-node5,mosip-node6 INFO[0144] [worker] Building up Worker Plane.. INFO[0144] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150] INFO[0144] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152] INFO[0144] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151] INFO[0145] Starting container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0145] Starting container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0145] Starting container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0145] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150] INFO[0145] Removing container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0145] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152] INFO[0145] Removing container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0145] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151] INFO[0145] Removing container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0145] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150] INFO[0145] Checking if container [service-sidekick] is running on host [172.16.2.150], try #1 INFO[0145] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152] INFO[0145] Checking if container [service-sidekick] is running on host [172.16.2.152], try #1 INFO[0145] [sidekick] Sidekick container already created on host [172.16.2.150] INFO[0145] Restarting container [kubelet] on host [172.16.2.150], try #1 INFO[0145] [sidekick] Sidekick container already created on host [172.16.2.152] INFO[0145] Restarting container [kubelet] on host [172.16.2.152], try #1 INFO[0145] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151] INFO[0145] Checking if container [service-sidekick] is running on host [172.16.2.151], try #1 INFO[0145] [sidekick] Sidekick container already created on host [172.16.2.151] INFO[0145] Restarting container [kubelet] on host [172.16.2.151], try #1 INFO[0145] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.150] INFO[0146] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.151] INFO[0146] [healthcheck] service [kubelet] on host [172.16.2.150] is healthy INFO[0146] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150] INFO[0146] [healthcheck] service [kubelet] on host [172.16.2.151] is healthy INFO[0146] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151] INFO[0146] Starting container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0146] Starting container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0146] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.152] INFO[0146] [healthcheck] service [kubelet] on host [172.16.2.152] is healthy INFO[0146] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152] INFO[0146] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150] INFO[0146] Removing container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0146] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151] INFO[0146] Removing container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0147] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150] INFO[0147] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.150] INFO[0147] [healthcheck] service [kube-proxy] on host [172.16.2.150] is healthy INFO[0147] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151] INFO[0147] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150] INFO[0147] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.151] INFO[0147] Starting container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0147] [healthcheck] service [kube-proxy] on host [172.16.2.151] is healthy INFO[0147] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151] INFO[0147] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150] INFO[0147] Removing container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0148] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150] INFO[0148] Starting container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0148] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152] INFO[0148] Removing container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0148] Starting container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0148] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152] INFO[0148] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.152] INFO[0148] [healthcheck] service [kube-proxy] on host [172.16.2.152] is healthy INFO[0148] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152] INFO[0148] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151] INFO[0148] Removing container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0149] Starting container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0149] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151] INFO[0149] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152] INFO[0149] Removing container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0149] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152] INFO[0149] [worker] Successfully started Worker Plane.. INFO[0149] [worker] Now checking status of node mosip-node4, try #1 **ERRO[0174] Host mosip-node4 failed to report Ready status with error: [worker] Error getting node mosip-node4: “mosip-node4” not found** INFO[0174] [worker] Now checking status of node mosip-node5, try #1 **ERRO[0199] Host mosip-node5 failed to report Ready status with error: [worker] Error getting node mosip-node5: “mosip-node5” not found** INFO[0199] [worker] Now checking status of node mosip-node6, try #1 **ERRO[0224] Host mosip-node6 failed to report Ready status with error: [worker] Error getting node mosip-node6: “mosip-node6” not found** INFO[0224] [worker] Upgrading Worker Plane.. INFO[0299] Now checking and upgrading worker components on nodes with only worker role 1 at a time INFO[0299] [workerplane] Processing host mosip-node4 INFO[0300] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152] INFO[0300] Starting container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0301] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152] INFO[0301] Removing container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0301] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152] INFO[0301] Checking if container [service-sidekick] is running on host [172.16.2.152], try #1 INFO[0301] [sidekick] Sidekick container already created on host [172.16.2.152] INFO[0301] Restarting container [kubelet] on host [172.16.2.152], try #1 INFO[0301] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.152] INFO[0302] [healthcheck] service [kubelet] on host [172.16.2.152] is healthy INFO[0302] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152] INFO[0302] Starting container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0302] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152] INFO[0302] Removing container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0302] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152] INFO[0302] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.152] INFO[0303] [healthcheck] service [kube-proxy] on host [172.16.2.152] is healthy INFO[0303] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152] INFO[0303] Starting container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0303] [worker] Successfully started [rke-log-linker] container on host [172.16.2.152] INFO[0303] Removing container [rke-log-linker] on host [172.16.2.152], try #1 INFO[0303] [remove/rke-log-linker] Successfully removed container on host [172.16.2.152] INFO[0303] [workerplane] Processing host mosip-node5 INFO[0304] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151] INFO[0304] Starting container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0305] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151] INFO[0305] Removing container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0305] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151] INFO[0305] Checking if container [service-sidekick] is running on host [172.16.2.151], try #1 INFO[0305] [sidekick] Sidekick container already created on host [172.16.2.151] INFO[0305] Restarting container [kubelet] on host [172.16.2.151], try #1 INFO[0305] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.151] INFO[0306] [healthcheck] service [kubelet] on host [172.16.2.151] is healthy INFO[0306] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151] INFO[0306] Starting container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0306] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151] INFO[0306] Removing container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0306] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151] INFO[0306] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.151] INFO[0307] [healthcheck] service [kube-proxy] on host [172.16.2.151] is healthy INFO[0307] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151] INFO[0307] Starting container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0307] [worker] Successfully started [rke-log-linker] container on host [172.16.2.151] INFO[0307] Removing container [rke-log-linker] on host [172.16.2.151], try #1 INFO[0307] [remove/rke-log-linker] Successfully removed container on host [172.16.2.151] INFO[0307] [workerplane] Processing host mosip-node6 INFO[0308] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150] INFO[0308] Starting container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0309] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150] INFO[0309] Removing container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0309] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150] INFO[0309] Checking if container [service-sidekick] is running on host [172.16.2.150], try #1 INFO[0309] [sidekick] Sidekick container already created on host [172.16.2.150] INFO[0309] Restarting container [kubelet] on host [172.16.2.150], try #1 INFO[0309] [healthcheck] Start Healthcheck on service [kubelet] on host [172.16.2.150] INFO[0310] [healthcheck] service [kubelet] on host [172.16.2.150] is healthy INFO[0310] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150] INFO[0310] Starting container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0310] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150] INFO[0310] Removing container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0310] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150] INFO[0310] [healthcheck] Start Healthcheck on service [kube-proxy] on host [172.16.2.150] INFO[0311] [healthcheck] service [kube-proxy] on host [172.16.2.150] is healthy INFO[0311] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150] INFO[0311] Starting container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0311] [worker] Successfully started [rke-log-linker] container on host [172.16.2.150] INFO[0311] Removing container [rke-log-linker] on host [172.16.2.150], try #1 INFO[0311] [remove/rke-log-linker] Successfully removed container on host [172.16.2.150] INFO[0311] [worker] Successfully upgraded Worker Plane.. INFO[0311] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.152] INFO[0311] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.150] INFO[0311] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.151] INFO[0311] Starting container [rke-log-cleaner] on host [172.16.2.150], try #1 INFO[0311] Starting container [rke-log-cleaner] on host [172.16.2.152], try #1 INFO[0312] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.150] INFO[0312] Removing container [rke-log-cleaner] on host [172.16.2.150], try #1 INFO[0312] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.152] INFO[0312] Removing container [rke-log-cleaner] on host [172.16.2.152], try #1 INFO[0312] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.156] INFO[0312] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.153] INFO[0312] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.150] INFO[0312] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.152] INFO[0312] Image [rancher/rke-tools:v0.1.80] exists on host [172.16.2.154] INFO[0314] Starting container [rke-log-cleaner] on host [172.16.2.156], try #1 INFO[0314] Starting container [rke-log-cleaner] on host [172.16.2.151], try #1 INFO[0314] Starting container [rke-log-cleaner] on host [172.16.2.153], try #1 INFO[0314] Starting container [rke-log-cleaner] on host [172.16.2.154], try #1 INFO[0315] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.156] INFO[0315] Removing container [rke-log-cleaner] on host [172.16.2.156], try #1 INFO[0315] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.154] INFO[0315] Removing container [rke-log-cleaner] on host [172.16.2.154], try #1 INFO[0315] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.153] INFO[0315] Removing container [rke-log-cleaner] on host [172.16.2.153], try #1 INFO[0315] [cleanup] Successfully started [rke-log-cleaner] container on host [172.16.2.151] INFO[0315] Removing container [rke-log-cleaner] on host [172.16.2.151], try #1 INFO[0315] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.153] INFO[0315] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.151] INFO[0315] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.154] INFO[0316] [remove/rke-log-cleaner] Successfully removed container on host [172.16.2.156] INFO[0316] [sync] Syncing nodes Labels and Taints **FATA[0478] [“mosip-node6” not found]** The error are in bold ---
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [May 19, 2025, 8:54am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/12 "2025-05-19T08:54:21Z")
The content of Hosts.ini: [mosip\_nodes] mosip-node1 ansible\_host=172.16.2.156 ansible\_user=john ansible\_ssh\_private\_key\_file=~/.ssh/id\_ed25519 ansible\_become=true mosip-node2 ansible\_host=172.16.2.154 ansible\_user=john ansible\_ssh\_private\_key\_file=~/.ssh/id\_ed25519 ansible\_become=true mosip-node3 ansible\_host=172.16.2.153 ansible\_user=john ansible\_ssh\_private\_key\_file=~/.ssh/id\_ed25519 ansible\_become=true mosip-node4 ansible\_host=172.16.2.152 ansible\_user=john ansible\_ssh\_private\_key\_file=~/.ssh/id\_ed25519 ansible\_become=true mosip-node5 ansible\_host=172.16.2.151 ansible\_user=john ansible\_ssh\_private\_key\_file=~/.ssh/id\_ed25519 ansible\_become=true mosip-node6 ansible\_host=172.16.2.150 ansible\_user=john ansible\_ssh\_private\_key\_file=~/.ssh/id\_ed25519 ansible\_become=true [etcd\_nodes] mosip-node1 ansible\_host=172.16.2.156 mosip-node2 ansible\_host=172.16.2.154 mosip-node3 ansible\_host=172.16.2.153 ---
**Author:** ![Praful\_Rakhade](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/praful_rakhade/32/1065_2.png) [@Praful\_Rakhade](https://community.mosip.io/u/Praful_Rakhade)\ **Post date:** [May 19, 2025, 11:24am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/13 "2025-05-19T11:24:40Z")
Hi @Johnny, Please follow the official documentation here: 👉 [MOSIP On-Prem Installation Guide](https://github.com/mosip/k8s-infra/tree/main/mosip/on-prem) and try again. **Note:** Your `hosts.ini` file format appears to be incorrect — kindly review and update it as per the structure mentioned in the documentation. ---
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [June 9, 2025, 12:42pm UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/14 "2025-06-09T12:42:33Z")
Hi @Praful_Rakhade I am installing mosip services but getting this error: john@mosipwgbst1:~/mosip-infra/deployment/v3/mosip/all$ ./install-all.sh Installing MOSIP services Create conf-secrets namespace Error from server (AlreadyExists): namespaces “conf-secrets” already exists Istio label namespace/conf-secrets not labeled Hang tight while we grab the latest from your chart repositories… …Successfully got an update from the “ingress-nginx” chart repository …Successfully got an update from the “longhorn” chart repository …Successfully got an update from the “mosip” chart repository Update Complete. ⎈Happy Helming!⎈ Installing Secrets required by config-server NAME: conf-secrets LAST DEPLOYED: Mon Jun 9 12:39:27 2025 NAMESPACE: conf-secrets STATUS: deployed REVISION: 1 TEST SUITE: None Is conf-secrets module installed?(Y/n) Y Is values.yaml for config-server chart set correctly as part of Pre-requisites?(Y/n) Y Create config-server namespace Error from server (AlreadyExists): namespaces “config-server” already exists Istio label namespace/config-server not labeled Hang tight while we grab the latest from your chart repositories… …Successfully got an update from the “longhorn” chart repository …Successfully got an update from the “ingress-nginx” chart repository …Successfully got an update from the “mosip” chart repository Update Complete. ⎈Happy Helming!⎈ Copy configmaps Error from server (NotFound): configmaps “global” not found error: no objects passed to create Kindly assist ---
**Author:** ![Praful\_Rakhade](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/praful_rakhade/32/1065_2.png) [@Praful\_Rakhade](https://community.mosip.io/u/Praful_Rakhade)\ **Post date:** [June 10, 2025, 5:13am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/15 "2025-06-10T05:13:45Z")
Hello @Johnny, I hope you were able to successfully create the Rancher cluster. As per the logs you shared, it seems the `global` ConfigMap is missing. Could you please confirm if you followed the documentation provided above? The steps for creating the `global` ConfigMap are included there. Regards Team MOSIP ---
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [June 10, 2025, 11:08am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/16 "2025-06-10T11:08:16Z")
Hi @Praful_Rakhade , Trust your day is going well, appreciate your response. Yes, I followed the above documentation. ---
**Author:** ![Praful\_Rakhade](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/praful_rakhade/32/1065_2.png) [@Praful\_Rakhade](https://community.mosip.io/u/Praful_Rakhade)\ **Post date:** [June 10, 2025, 11:38am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/17 "2025-06-10T11:38:39Z")
Hi @Johnny, Thanks for confirming. Could you please check if the `global` ConfigMap is present in the **`default`** namespace? As per the error message you shared — `Error from server (NotFound): configmaps "global" not found` — it seems the ConfigMap might be missing or not created in the correct namespace. Regards Team MOSIP ---
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [June 10, 2025, 11:43am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/18 "2025-06-10T11:43:37Z")
Hello @Praful_Rakhade, I think I have solved the error but I am having issue installing keycloak: Installing Error: INSTALLATION FAILED: Get “[https://mosip.github.io/mosip-helm/keycloak-7.1.18.tgz](https://mosip.github.io/mosip-helm/keycloak-7.1.18.tgz)”: context deadline exceeded (Client.Timeout exceeded while awaiting headers) What can be done please. Thank you very much ---
**Author:** ![Johnny](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johnny](https://community.mosip.io/u/Johnny)\ **Post date:** [June 10, 2025, 9:55pm UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/19 "2025-06-10T21:55:39Z")
Hi @Praful_Rakhade, This is the new error I am getting: configmap/activemq-activemq-artemis-share created Error from server (NotFound): configmaps “s3” not found error: no objects passed to create And trying to setup s3, this was run ./cred.sh and from the option, if one select 2, it is asking for aws credential. kindly assist ---
**Author:** ![Praful\_Rakhade](https://dub1.discourse-cdn.com/flex017/user_avatar/community.mosip.io/praful_rakhade/32/1065_2.png) [@Praful\_Rakhade](https://community.mosip.io/u/Praful_Rakhade)\ **Post date:** [June 11, 2025, 6:48am UTC](https://community.mosip.io/t/cannot-download-repomd-xml/2020/20 "2025-06-11T06:48:15Z")
Hi @Johnny, The errors you are encountering across multiple services seem to be due to the installation sequence. According to the recommended process, you should install the **External modules** first, followed by the **MOSIP modules**. However, from your steps, it appears that you attempted to install the MOSIP modules before setting up the external modules, which is likely the cause of these errors. For your reference, here are the detailed installation guides: [External Modules Installation Guide](https://github.com/mosip/mosip-infra/tree/v1.2.0.2/deployment/v3/external) [Mosip Modules Installation Guide](https://github.com/mosip/mosip-infra/tree/v1.2.0.2/deployment/v3/mosip) Please follow the sequence as outlined in the documentation to resolve these issues. Please let us know if you require any additional assistance. Thanks, Team MOSIP [Next page](https://community.mosip.io/t/cannot-download-repomd-xml/2020.md?page=2)