I following the installation guidelines for deployment guide for V3, after the executing the cluster import command the cluster status in rancher shows pending and there is no progress.
When you go to the related resource tab of the cluster it says:
This resource is currently in a transitioning state, but there isn’t a detailed message available.
Clicking on the Mgmt Cluster there is a message Waiting for API to be available.
In the error log I can see the below for the pod
ERROR: .xyz Domain Names | Join Generation XYZ is not accessible (Could not resolve host: rancher.xyz.com).
But I don’t see any issues in resolution of the url. Please note I am using a selfsigned certificate for nginix generated through openssl.
Hi Arjun,
Thank you for reaching out regarding the deployment of V3 using the installation guidelines. We apologize for the inconvenience you are facing with the cluster import command.
Upon reviewing your message, it appears that the cluster status in Rancher is showing as “pending,” and there is no progress. Additionally, when accessing the related resource tab of the cluster, it indicates that the resource is in a transitioning state, but there is no detailed message available. Furthermore, clicking on the Mgmt Cluster shows a message stating “Waiting for API to be available.”
We aim to get back to you as soon as possible with a detailed and comprehensive answer to address your concerns. Your satisfaction is our priority, and we want to ensure that you have a smooth and successful experience with MOSIP.
Best Regards,
Team MOSIP
Ensure to resolve DNS rancher.xyz.com from your local as well as from the k8s cluster.
Setup DNS ( K8S cluster )
kubectl -n kube-system edit cm coredns
Update and add the below block In coredns cm.
hosts {
<INTERNAL_IP_OF_OBS_NGINX_NODE> rancher.xyz.net keycloak.xyz.net
fallthrough
}
Restart coredns pods
kubectl -n kube-system rollout restart deploy coredns coredns-autoscaler
Set DNS local
- Add the below values in the
/etc/hosts file.
<INTERNAL_IP_OF_OBS_NGINX_NODE> rancher.xyz.net keycloak.xyz.net
Hi
Thanks for getting back with the solution.
After the below steps should I remove the cluster showing pending state in the rancher console and re-register it?
Hi Syed,
The name resolution issue on the pod is resolved , but even after re-registering the cluster the status is pending and “Waiting for API to be available” on the rancher portal.
Hi Arun,
Thanks for the update as I can see you have followed up the steps shared by @syed.salman but still issue on the re-registration of cluster status is pending our team is looking into this and get back to you asap.
Best Regards,
Team MOSIP
@arjunsukumar can you please provide more information on this?
List the pods via
kubectl get pods -A
If any pod is crashing, please provide us the logs via the below command:
kubectl -n <NAMESPACE> logs <POD-NAME> --previous
Hi Syed,
Please find attached the requested logs for review and advise.
(Attachment Logs.rtf is missing)
Hello Syed,
Arjun not able to post. i’m sharing the logs
arjun@mass-master:~$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-system cattle-cluster-agent-547f7959b8-9z7s5 0/1 CrashLoopBackOff 239 (77s ago) 19h
httpbin httpbin-74fb669cc6-rmp56 2/2 Running 3 (17h ago) 41h
istio-operator istio-operator-f89b46bdf-s9x5w 1/1 Running 2 (17h ago) 2d21h
istio-system istio-ingressgateway-bbcd89c48-jhmzv 1/1 Running 2 (17h ago) 2d21h
istio-system istio-ingressgateway-internal-bcd5d5876-blrn8 1/1 Running 2 (17h ago) 2d21h
istio-system istiod-d97cbc5f-d4xls 1/1 Running 3 (17h ago) 2d21h
kube-system calico-kube-controllers-5685fbd9f7-cbjvh 1/1 Running 2 (17h ago) 3d16h
kube-system canal-2h9nq 2/2 Running 4 (17h ago) 3d16h
kube-system canal-78qjn 2/2 Running 4 (17h ago) 3d15h
kube-system canal-9k4b7 2/2 Running 5 (17h ago) 3d16h
kube-system canal-jwkhv 2/2 Running 4 (17h ago) 3d16h
kube-system canal-vv662 2/2 Running 4 (17h ago) 3d16h
kube-system coredns-6784669b74-qb75g 1/1 Running 1 (17h ago) 23h
kube-system coredns-6784669b74-x59gk 1/1 Running 1 (17h ago) 23h
kube-system coredns-autoscaler-7f74c8f58b-4sbr4 1/1 Running 1 (17h ago) 23h
kube-system metrics-server-6bc7854fb5-szwnw 1/1 Running 2 (17h ago) 3d16h
kube-system rke-coredns-addon-deploy-job--1-p9gkq 0/1 Completed 0 3d16h
kube-system rke-metrics-addon-deploy-job--1-dww4k 0/1 Completed 0 3d16h
kube-system rke-network-plugin-deploy-job--1-lqwhf 0/1 Completed 0 3d16h
longhorn-system csi-attacher-5f46994f7-b2dg4 1/1 Running 2 (17h ago) 2d21h
longhorn-system csi-attacher-5f46994f7-j7jjk 1/1 Running 3 (17h ago) 2d21h
longhorn-system csi-attacher-5f46994f7-v9hr4 1/1 Running 2 (17h ago) 2d21h
longhorn-system csi-provisioner-6ccbfbf86f-jrsb7 1/1 Running 2 (17h ago) 2d21h
longhorn-system csi-provisioner-6ccbfbf86f-ncd4s 1/1 Running 3 (17h ago) 2d21h
longhorn-system csi-provisioner-6ccbfbf86f-pt79b 1/1 Running 2 (17h ago) 2d21h
longhorn-system csi-resizer-6dd8bd4c97-5t22b 1/1 Running 2 (17h ago) 2d21h
longhorn-system csi-resizer-6dd8bd4c97-j8vw8 1/1 Running 2 (17h ago) 2d21h
longhorn-system csi-resizer-6dd8bd4c97-tkvg8 1/1 Running 3 (17h ago) 2d21h
longhorn-system csi-snapshotter-86f65d8bc-dpwdc 1/1 Running 2 (17h ago) 2d21h
longhorn-system csi-snapshotter-86f65d8bc-gf64f 1/1 Running 2 (17h ago) 2d21h
longhorn-system csi-snapshotter-86f65d8bc-vzs9z 1/1 Running 3 (17h ago) 2d21h
longhorn-system engine-image-ei-fa2dfbf0-42d94 1/1 Running 2 (17h ago) 2d21h
longhorn-system engine-image-ei-fa2dfbf0-9zt45 1/1 Running 2 (17h ago) 2d21h
longhorn-system engine-image-ei-fa2dfbf0-dcq6l 1/1 Running 2 (17h ago) 2d21h
longhorn-system engine-image-ei-fa2dfbf0-pshmv 1/1 Running 2 (17h ago) 2d21h
longhorn-system engine-image-ei-fa2dfbf0-ttg9c 1/1 Running 2 (17h ago) 2d21h
longhorn-system instance-manager-e-059e27d4 1/1 Running 0 17h
longhorn-system instance-manager-e-15adf935 1/1 Running 0 17h
longhorn-system instance-manager-e-2151ddc3 1/1 Running 0 17h
longhorn-system instance-manager-e-4241c261 1/1 Running 0 17h
longhorn-system instance-manager-e-cd1550f9 1/1 Running 0 17h
longhorn-system instance-manager-r-624b4869 1/1 Running 0 17h
longhorn-system instance-manager-r-9bb300a1 1/1 Running 0 17h
longhorn-system instance-manager-r-c3bf960e 1/1 Running 0 17h
longhorn-system instance-manager-r-cbab1bb8 1/1 Running 0 17h
longhorn-system instance-manager-r-dcbbb8a5 1/1 Running 0 17h
longhorn-system longhorn-csi-plugin-7f6nv 2/2 Running 5 (17h ago) 2d21h
longhorn-system longhorn-csi-plugin-csrlp 2/2 Running 6 (17h ago) 2d21h
longhorn-system longhorn-csi-plugin-ddn6b 2/2 Running 5 (17h ago) 2d21h
longhorn-system longhorn-csi-plugin-pfbcn 2/2 Running 7 (17h ago) 2d21h
longhorn-system longhorn-csi-plugin-rrthh 2/2 Running 6 (17h ago) 2d21h
longhorn-system longhorn-driver-deployer-6db849975f-2lc6t 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-iscsi-installation-5wxx7 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-iscsi-installation-6mzdn 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-iscsi-installation-9hwdx 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-iscsi-installation-jfpvw 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-iscsi-installation-v5fz5 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-manager-5fd85 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-manager-5sc8l 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-manager-bdtcd 1/1 Running 3 (17h ago) 2d21h
longhorn-system longhorn-manager-h52fj 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-manager-mgj5f 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-nfs-installation-7dwdf 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-nfs-installation-8rwz6 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-nfs-installation-bm27k 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-nfs-installation-f7zfm 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-nfs-installation-mkds2 1/1 Running 2 (17h ago) 2d21h
longhorn-system longhorn-ui-6f547c964-8gj5t 1/1 Running 4 (17h ago) 2d21h
arjun@mass-master:~$ kubectl -n cattle-system logs cattle-cluster-agent-547f7959b8-9z7s5 --previous
INFO: Environment: CATTLE_ADDRESS=10.42.3.26 CATTLE_CA_CHECKSUM= CATTLE_CLUSTER=true CATTLE_CLUSTER_AGENT_PORT=tcp://10.43.158.148:80 CATTLE_CLUSTER_AGENT_PORT_443_TCP=tcp://10.43.158.148:443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_ADDR=10.43.158.148 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PORT=443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_PORT_80_TCP=tcp://10.43.158.148:80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_ADDR=10.43.158.148 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PORT=80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_SERVICE_HOST=10.43.158.148 CATTLE_CLUSTER_AGENT_SERVICE_PORT=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTP=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTPS_INTERNAL=443 CATTLE_CLUSTER_REGISTRY= CATTLE_INGRESS_IP_DOMAIN=sslip.io CATTLE_INSTALL_UUID=7dfa2c7b-7ca4-4104-80ad-1dc41e457a2b CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=false CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-547f7959b8-9z7s5 CATTLE_RANCHER_WEBHOOK_MIN_VERSION= CATTLE_RANCHER_WEBHOOK_VERSION=2.0.5+up0.3.5 CATTLE_SERVER=https://rancher.kvaliteta.com CATTLE_SERVER_VERSION=v2.7.5
INFO: Using resolv.conf: nameserver 10.43.0.10 search cattle-system.svc.kvaliteta.com svc.kvaliteta.com kvaliteta.com options ndots:5
INFO: https://rancher.kvaliteta.com/ping is accessible
INFO: rancher.kvaliteta.com resolves to 192.168.1.46
time="2023-07-21T05:07:12Z" level=info msg="Listening on /tmp/log.sock"
time="2023-07-21T05:07:12Z" level=info msg="Rancher agent version v2.7.5 is starting"
time="2023-07-21T05:07:12Z" level=info msg="Certificate details from https://rancher.kvaliteta.com"
time="2023-07-21T05:07:12Z" level=info msg="Certificate #0 (https://rancher.kvaliteta.com)"
time="2023-07-21T05:07:12Z" level=info msg="Subject: CN=*.kvaliteta.com,O=Kvaliteta,L=Tvm,ST=Ker,C=IN"
time="2023-07-21T05:07:12Z" level=info msg="Issuer: CN=*.kvaliteta.com,O=Kvaliteta,L=Tvm,ST=Ker,C=IN"
time="2023-07-21T05:07:12Z" level=info msg="IsCA: true"
time="2023-07-21T05:07:12Z" level=info msg="DNS Names: <none>"
time="2023-07-21T05:07:12Z" level=info msg="IPAddresses: <none>"
time="2023-07-21T05:07:12Z" level=info msg="NotBefore: 2023-07-10 11:21:25 +0000 UTC"
time="2023-07-21T05:07:12Z" level=info msg="NotAfter: 2024-07-09 11:21:25 +0000 UTC"
time="2023-07-21T05:07:12Z" level=info msg="SignatureAlgorithm: SHA256-RSA"
time="2023-07-21T05:07:12Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2023-07-21T05:07:12Z" level=fatal msg="Get \https://rancher.kvaliteta.com\: x509: certificate relies on legacy Common Name field, use SANs instead"
arjun@mass-master:~$
Hi @ramkumar.kvaliteta
The import is getting failed due to a self-signed certificate, Use the curl --insecure option displayed on rancher UI to import the cluster
Hi Syed,
we are already using the insecure option
this warning is generated while registration
Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
Hey @ramkumar.kvaliteta
Thanks for letting us know where is the issue coming @syed.salman will look into this and help you out.
Best Regards,
MOSIP Team
Kindly note that the warning issued can be disregarded. However, it is imperative to confirm whether all the prerequisites outlined in the MOSIP documentation have been diligently followed.
Furthermore, please ensure that the Kubernetes cluster version precisely matches v1.22.9. To retrieve this information, kindly execute the provided command below.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready,SchedulingDisabled controlplane,etcd,worker 14d v1.22.9
node2 Ready,SchedulingDisabled controlplane,etcd,worker 14d v1.22.9
Hello Syed pls check the below```
arjun@mass-master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
mosip-app-host-01 Ready controlplane,etcd,worker 10d v1.22.9
mosip-app-host-02 Ready controlplane,etcd,worker 10d v1.22.9
mosip-app-host-03 Ready controlplane,etcd,worker 10d v1.22.9
mosip-app-host-04 Ready controlplane,worker 10d v1.22.9
mosip-app-host-05 Ready controlplane,worker 10d v1.22.9
1 Like
The version looks fine…
you can ignore the warning
1 Like
Hi @ramkumar.kvaliteta
Do let us know if you are facing any issues further !
Best Regards,
Team MOSIP
Hi Syed
So what is the next step, the cluster import is till in pending state.
Can you please check which pods are failing or in not ready state? And also provide us with the logs.
Hi Syed,
Please find below.
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-system cattle-cluster-agent-7b8fcf9545-qf5tv 0/1 CrashLoopBackOff 2459 (4m30s ago) 8d
kubectl -n cattle-system logs cattle-cluster-agent-7b8fcf9545-qf5tv --previous
INFO: Environment: CATTLE_ADDRESS=10.42.1.34 CATTLE_CA_CHECKSUM= CATTLE_CLUSTER=true CATTLE_CLUSTER_AGENT_PORT=tcp://10.43.158.148:80 CATTLE_CLUSTER_AGENT_PORT_443_TCP=tcp://10.43.158.148:443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_ADDR=10.43.158.148 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PORT=443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_PORT_80_TCP=tcp://10.43.158.148:80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_ADDR=10.43.158.148 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PORT=80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_SERVICE_HOST=10.43.158.148 CATTLE_CLUSTER_AGENT_SERVICE_PORT=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTP=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTPS_INTERNAL=443 CATTLE_CLUSTER_REGISTRY= CATTLE_INGRESS_IP_DOMAIN=sslip.io CATTLE_INSTALL_UUID=7dfa2c7b-7ca4-4104-80ad-1dc41e457a2b CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=false CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-7b8fcf9545-qf5tv CATTLE_RANCHER_WEBHOOK_MIN_VERSION= CATTLE_RANCHER_WEBHOOK_VERSION=2.0.5+up0.3.5 CATTLE_SERVER=https://rancher.kvaliteta.com CATTLE_SERVER_VERSION=v2.7.5
INFO: Using resolv.conf: nameserver 10.43.0.10 search cattle-system.svc.kvaliteta.com svc.kvaliteta.com kvaliteta.com options ndots:5
INFO: https://rancher.kvaliteta.com/ping is accessible
INFO: rancher.kvaliteta.com resolves to 192.168.1.46
time="2023-08-02T06:11:39Z" level=info msg="Listening on /tmp/log.sock"
time="2023-08-02T06:11:39Z" level=info msg="Rancher agent version v2.7.5 is starting"
time="2023-08-02T06:11:39Z" level=info msg="Certificate details from https://rancher.kvaliteta.com"
time="2023-08-02T06:11:39Z" level=info msg="Certificate #0 (https://rancher.kvaliteta.com)"
time="2023-08-02T06:11:39Z" level=info msg="Subject: CN=*.kvaliteta.com,O=Kvaliteta,L=Tvm,ST=Ker,C=IN"
time="2023-08-02T06:11:39Z" level=info msg="Issuer: CN=*.kvaliteta.com,O=Kvaliteta,L=Tvm,ST=Ker,C=IN"
time="2023-08-02T06:11:39Z" level=info msg="IsCA: true"
time="2023-08-02T06:11:39Z" level=info msg="DNS Names: <none>"
time="2023-08-02T06:11:39Z" level=info msg="IPAddresses: <none>"
time="2023-08-02T06:11:39Z" level=info msg="NotBefore: 2023-07-10 11:21:25 +0000 UTC"
time="2023-08-02T06:11:39Z" level=info msg="NotAfter: 2024-07-09 11:21:25 +0000 UTC"
time="2023-08-02T06:11:39Z" level=info msg="SignatureAlgorithm: SHA256-RSA"
time="2023-08-02T06:11:39Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2023-08-02T06:11:39Z" level=fatal msg="Get \"https://rancher.kvaliteta.com\": x509: certificate relies on legacy Common Name field, use SANs instead"
It appears to be a self-signed certificate problem. Could you kindly regenerate the self-signed certificates using Docker, following the instructions provided in this link and try to import again ?
Furthermore, I have noticed that the Kubernetes nodes are currently in the SchedulingDisabled state. Have you manually disabled them?
