MOSIP sandbox V2 Release 1.2.0 installation stops at TASK [helm : Install chart kernel]

First of all sorry for the long post. I tried this several times, re-installed kernel module, also re-installed MOSIP from scratch, I still end up with the same error. Can someone help me fix this error? Thanks!

I am installing MOSIP sandbox V2 Release 1.2.0 on prem with self-signed certificate. The installation was running smoothly until its stops at TASK [helm : Install chart kernel].

Below is the error, result of kc1 get pod, kc1 logs, kc1 describe pod

TASK [helm : Install chart kernel] *************************************************************************************************
fatal: [console.sb]: FAILED! => {“changed”: true, “cmd”: “/home/mosipuser/bin/helm --kubeconfig /home/mosipuser/.kube/mzcluster.config install kernel /home/mosipuser/mosip-infra/deployment/sandbox-v2/helm/charts/kernel -f /home/mosipuser/mosip-infra/deployment/sandbox-v2/helm/charts/kernel/values.yaml -n default --create-namespace --set-string dummy=value, --wait --timeout 20m0s --debug > /home/mosipuser/mosip-infra/deployment/sandbox-v2/tmp/yaml/kernel.yaml”, “delta”: “0:17:35.343644”, “end”: “2022-03-12 15:55:01.361541”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2022-03-12 15:37:26.017897”, “stderr”: “install.go:159: [debug] Original chart version: “”\ninstall.go:176: [debug] CHART PATH: /home/mosipuser/mosip-infra/deployment/sandbox-v2/helm/charts/kernel\n\nclient.go:108: [debug] creating 1 resource(s)\nclient.go:108: [debug] creating 1 resource(s)\nclient.go:467: [debug] Watching for changes to Job kernel-keys-generator with timeout of 20m0s\nclient.go:495: [debug] Add/Modify event for kernel-keys-generator: ADDED\nclient.go:534: [debug] kernel-keys-generator: Jobs active: 0, jobs failed: 0, jobs succeeded: 0\nclient.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED\nclient.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 0, jobs succeeded: 0\nclient.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED\nclient.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 1, jobs succeeded: 0\nclient.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED\nclient.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 2, jobs succeeded: 0\nclient.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED\nclient.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 3, jobs succeeded: 0\nclient.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED\nclient.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 4, jobs succeeded: 0\nclient.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED\nclient.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 5, jobs succeeded: 0\nclient.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED\nclient.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 6, jobs succeeded: 0\nclient.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED\nError: failed pre-install: job failed: BackoffLimitExceeded\nhelm.go:84: [debug] failed pre-install: job failed: BackoffLimitExceeded”, “stderr_lines”: [“install.go:159: [debug] Original chart version: “””, “install.go:176: [debug] CHART PATH: /home/mosipuser/mosip-infra/deployment/sandbox-v2/helm/charts/kernel”, “”, “client.go:108: [debug] creating 1 resource(s)”, “client.go:108: [debug] creating 1 resource(s)”, “client.go:467: [debug] Watching for changes to Job kernel-keys-generator with timeout of 20m0s”, “client.go:495: [debug] Add/Modify event for kernel-keys-generator: ADDED”, “client.go:534: [debug] kernel-keys-generator: Jobs active: 0, jobs failed: 0, jobs succeeded: 0”, “client.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED”, “client.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 0, jobs succeeded: 0”, “client.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED”, “client.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 1, jobs succeeded: 0”, “client.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED”, “client.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 2, jobs succeeded: 0”, “client.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED”, “client.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 3, jobs succeeded: 0”, “client.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED”, “client.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 4, jobs succeeded: 0”, “client.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED”, “client.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 5, jobs succeeded: 0”, “client.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED”, “client.go:534: [debug] kernel-keys-generator: Jobs active: 1, jobs failed: 6, jobs succeeded: 0”, “client.go:495: [debug] Add/Modify event for kernel-keys-generator: MODIFIED”, “Error: failed pre-install: job failed: BackoffLimitExceeded”, “helm.go:84: [debug] failed pre-install: job failed: BackoffLimitExceeded”], “stdout”: “”, “stdout_lines”: []}

kc1 get pods -A | grep kernel
default db-kernel-init-job-phbdz 0/1 Completed 0 87m
default kernel-keys-generator-5c29z 0/1 Error 0 30m
default kernel-keys-generator-bbn65 0/1 Error 0 40m
default kernel-keys-generator-c8r9s 0/1 Error 0 36m
default kernel-keys-generator-n48zr 0/1 Error 0 42m
default kernel-keys-generator-sdpwl 0/1 Error 0 41m
default kernel-keys-generator-vjcjd 0/1 Error 0 42m
default kernel-keys-generator-vqc7w 0/1 Error 0 39m
[mosipuser@console sandbox-v2]$

kc1 logs kernel-keys-generator-5c29z

Tomcat error not loading properly ?

{"@timestamp":“2022-03-12T15:49:10.331Z”,"@version":“1”,“message”:“Could not locate PropertySource: label not found”,“logger_name”:“org.springframework.cloud.config.client.ConfigServicePropertySourceLocator”,“thread_name”:“main”,“level”:“WARN”,“level_value”:30000,“appName”:“keys-generator”}

{"@timestamp":“2022-03-12T15:49:12.486Z”,"@version":“1”,“message”:“The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]”,“logger_name”:“org.apache.catalina.core.AprLifecycleListener”,“thread_name”:“localhost-startStop-1”,“level”:“INFO”,“level_value”:20000,“appName”:“keys-generator”}

{"@timestamp":“2022-03-12T15:49:13.259Z”,"@version":“1”,“message”:“Error starting Tomcat context. Exception: org.springframework.beans.factory.BeanCreationException. Message: Error creating bean with name ‘servletEndpointRegistrar’ defined in class path resource [org/springframework/boot/actuate/autoconfigure/endpoint/web/ServletEndpointManagementContextConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.endpoint.web.ServletEndpointRegistrar]: Factory method ‘servletEndpointRegistrar’ threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘healthEndpoint’ defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthEndpoint]: Factory method ‘healthEndpoint’ threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration’: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration$$EnhancerBySpringCGLIB$$f28ccab]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘keymanagerDataSource’ defined in class path resource [io/mosip/kernel/keymanagerservice/config/KeymanagerDaoConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method ‘keymanagerDataSource’ threw exception; nested exception is java.lang.NullPointerException”,“logger_name”:“org.springframework.boot.web.embedded.tomcat.TomcatStarter”,“thread_name”:“localhost-startStop-1”,“level”:“ERROR”,“level_value”:40000,“appName”:“keys-generator”}
{"@timestamp":“2022-03-12T15:49:13.278Z”,"@version":“1”,“message”:“Stopping service [Tomcat]”,“logger_name”:“org.apache.catalina.core.StandardService”,“thread_name”:“main”,“level”:“INFO”,“level_value”:20000,“appName”:“keys-generator”}
{"@timestamp":“2022-03-12T15:49:13.289Z”,"@version":“1”,“message”:“Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat”,“logger_name”:“org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext”,“thread_name”:“main”,“level”:“WARN”,“level_value”:30000,“appName”:“keys-generator”}
{"@timestamp":“2022-03-12T15:49:13.301Z”,"@version":“1”,“message”:"\n\nError starting ApplicationContext. To display the conditions report re-run your application with ‘debug’ enabled.",“logger_name”:“org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener”,“thread_name”:“main”,“level”:“INFO”,“level_value”:20000,“appName”:“keys-generator”}
{"@timestamp":“2022-03-12T15:49:13.305Z”,"@version":“1”,“message”:“Application run failed”,“logger_name”:“org.springframework.boot.SpringApplication”,“thread_name”:“main”,“level”:“ERROR”,“level_value”:40000,“stack_trace”:"org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:155)\n\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)\n\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)\n\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:327)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)\n\tat io.mosip.kernel.keygenerator.KeysGeneratorApplication.main(KeysGeneratorApplication.java:23)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:50)\n\tat org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:593)\nCaused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat\n\tat org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:126)\n\tat org.springframework.boot.web.embedded.tomcat.TomcatWebServer.(TomcatWebServer.java:86)\n\tat org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:409)\n\tat org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:174)\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:179)\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:152)\n\t… 16 common frames omitted\nCaused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘servletEndpointRegistrar’ defined in class path resource [org/springframework/boot/actuate/autoconfigure/endpoint/web/ServletEndpointManagementContextConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.endpoint.web.ServletEndpointRegistrar]: Factory method ‘servletEndpointRegistrar’ threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘healthEndpoint’ defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthEndpoint]: Factory method ‘healthEndpoint’ threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration’: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration$$EnhancerBySpringCGLIB$$f28ccab]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘keymanagerDataSource’ defined in class path resource [io/mosip/kernel/keymanagerservice/config/KeymanagerDaoConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method ‘keymanagerDataSource’ threw exception; nested exception is java.lang.NullPointerException\n\tat org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1254)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:541)\n\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)\n\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)\n\tat org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204)\n\tat org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:225)\n\tat org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:213)\n\tat org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:90)\n\tat org.springframework.boot.web.servlet.ServletContextInitializerBeans.(ServletContextInitializerBeans.java:79)\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:250)\n\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:237)\n\tat org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:54)\n\tat org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5245)\n\tat org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)\n\tat org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1421)\n\tat org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1411)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.endpoint.web.ServletEndpointRegistrar]: Factory method ‘servletEndpointRegistrar’ threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘healthEndpoint’ defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthEndpoint]: Factory method ‘healthEndpoint’ threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration’: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration$$EnhancerBySpringCGLIB$$f28ccab]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘keymanagerDataSource’ defined in class path resource [io/mosip/kernel/keymanagerservice/config/KeymanagerDaoConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method ‘keymanagerDataSource’ threw exception; nested exception is java.lang.NullPointerException\n\tat org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)\n\tat org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579)\n\t… 23 common frames omitted\nCaused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘healthEndpoint’ defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthEndpoint]: Factory method ‘healthEndpoint’ threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration’: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.autoconfigure.jdbc.DataSourceHealthIndicatorAutoConfiguration$$EnhancerBySpringCGLIB$$f28ccab]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘keymanagerDataSource’ defined in class path resource [io/mosip/kernel/keymanagerservice/config/KeymanagerDaoConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method ‘keymanagerDataSource’ threw exception; nested exception is java.lang.NullPointerException\n\tat org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)\n\tat

[mosipuser@console sandbox-v2]$ kc1 describe pod kernel
Name: kernel-keys-generator-5c29z
Namespace: default
Priority: 0
Node: mzworker8.sb/10.0.0.31
Start Time: Sat, 12 Mar 2022 15:49:01 +0000
Labels: app=kernel-keys-generator
component=kernel
controller-uid=59653766-a19d-49e8-abba-0b7728808cbd
job-name=kernel-keys-generator
Annotations:
Status: Failed
IP: 10.244.6.13
IPs:
IP: 10.244.6.13
Controlled By: Job/kernel-keys-generator
Containers:
kernel-keys-generator:
Container ID: docker://76b53ad07d6f2377b5130101f451d636ffdb53d1b05363bbde9b7170903a432a
Image: mosipid/keys-generator:1.2.0
Image ID: docker-pullable://mosipid/keys-generator@sha256:d1d0dddc8037551b68babc88abe954d3b101c4de306ef351b8b9333257c075d6
Port:
Host Port:
State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 12 Mar 2022 15:49:05 +0000
Finished: Sat, 12 Mar 2022 15:49:13 +0000
Ready: False
Restart Count: 0
Environment:
active_profile_env: default
spring_config_label_env: develop
spring_config_url_env: http://config-server.default:80/config
spring_config_name_env: kernel
artifactory_url_env: http://mz.ingress:30080/
is_glowroot_env: absent
PKCS11_PROXY_SOCKET: tcp://softhsm-keymanager:80
zip_file_path: artifactory/libs-release-local/hsm/client.zip
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lnfvr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-lnfvr:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message


Normal Scheduled 33m default-scheduler Successfully assigned default/kernel-keys-generator-5c29z to mzworker8.sb
Normal Pulling 33m kubelet Pulling image “mosipid/keys-generator:1.2.0”
Normal Pulled 33m kubelet Successfully pulled image “mosipid/keys-generator:1.2.0” in 3.125318947s
Normal Created 33m kubelet Created container kernel-keys-generator
Normal Started 33m kubelet Started container kernel-keys-generator

Name: kernel-keys-generator-bbn65
Namespace: default
Priority: 0
Node: mzworker8.sb/10.0.0.31
Start Time: Sat, 12 Mar 2022 15:39:01 +0000
Labels: app=kernel-keys-generator
component=kernel
controller-uid=59653766-a19d-49e8-abba-0b7728808cbd
job-name=kernel-keys-generator
Annotations:
Status: Failed
IP: 10.244.6.10
IPs:
IP: 10.244.6.10
Controlled By: Job/kernel-keys-generator
Containers:
kernel-keys-generator:
Container ID: docker://0fed5ffe8508a8d16ed608928b9a156b3029c5c02864943979183860806a0c27
Image: mosipid/keys-generator:1.2.0
Image ID: docker-pullable://mosipid/keys-generator@sha256:d1d0dddc8037551b68babc88abe954d3b101c4de306ef351b8b9333257c075d6
Port:
Host Port:
State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 12 Mar 2022 15:39:06 +0000
Finished: Sat, 12 Mar 2022 15:39:11 +0000
Ready: False
Restart Count: 0
Environment:
active_profile_env: default
spring_config_label_env: develop
spring_config_url_env: http://config-server.default:80/config
spring_config_name_env: kernel
artifactory_url_env: http://mz.ingress:30080/
is_glowroot_env: absent
PKCS11_PROXY_SOCKET: tcp://softhsm-keymanager:80
zip_file_path: artifactory/libs-release-local/hsm/client.zip
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gxqnw (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-gxqnw:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Can we get below details from cluster:

  1. description and status of config-server.
  2. description and status of artifactory service.
  3. description and status of all postgres-init jobs.

We suspect either one of the above pre-requisits for the key-generator job is not met causing the issue.

Thank you.

Here are the cluster details

mosipuser@console sandbox-v2]$ kc1 get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default artifactory-service-69f4644bff-nt4qv 1/1 Running 0 4h55m
default config-server-5cb566fd69-gf7p6 1/1 Running 0 4h49m
default db-audit-init-job-zk456 0/1 Completed 0 5h30m
default db-authdevice-init-job-rb2vm 0/1 Completed 0 5h30m
default db-credential-init-job-c2rj8 0/1 Completed 0 5h30m
default db-hotlist-init-job-lftjs 0/1 Completed 0 5h30m
default db-ida-init-job-kgkf5 0/1 Completed 0 5h30m
default db-idmap-init-job-kw6cc 0/1 Completed 0 5h30m
default db-idrepo-init-job-495g7 0/1 Completed 0 5h30m
default db-kernel-init-job-95ngk 0/1 Completed 0 5h30m
default db-keymgr-init-job-94rkr 0/1 Completed 0 5h30m
default db-master-init-job-54czw 0/1 Completed 0 5h30m
default db-pms-init-job-bsw62 0/1 Completed 0 5h30m
default db-prereg-init-job-xvsbp 0/1 Completed 0 5h30m
default db-regdevice-init-job-vj698 0/1 Completed 0 5h30m
default db-regprc-init-job-7wqhr 0/1 Completed 0 5h30m

[mosipuser@console sandbox-v2]$ kc1 get job
NAME COMPLETIONS DURATION AGE
db-audit-init-job 1/1 111s 5h25m
db-authdevice-init-job 1/1 3m12s 5h25m
db-credential-init-job 1/1 2m7s 5h25m
db-hotlist-init-job 1/1 114s 5h25m
db-ida-init-job 1/1 4m6s 5h25m
db-idmap-init-job 1/1 2m1s 5h25m
db-idrepo-init-job 1/1 3m8s 5h25m
db-kernel-init-job 1/1 2m16s 5h25m
db-keymgr-init-job 1/1 2m47s 5h25m
db-master-init-job 1/1 7m3s 5h25m
db-pms-init-job 1/1 5m26s 5h25m
db-prereg-init-job 1/1 3m10s 5h25m
db-regdevice-init-job 1/1 3m12s 5h25m
db-regprc-init-job 1/1 3m36s 5h25m
kernel-keys-generator 0/1 4h38m 4h38m

[mosipuser@console sandbox-v2]$ kc1 get pod -A | grep config-server
default config-server-5cb566fd69-gf7p6 1/1 Running 0 4h38m

[mosipuser@console sandbox-v2]$ kc1 describe pod config-server-5cb566fd69-gf7p6
Name: config-server-5cb566fd69-gf7p6
Namespace: default
Priority: 0
Node: mzworker1.sb/10.0.0.24
Start Time: Tue, 15 Mar 2022 09:10:33 +0000
Labels: app=config-server
exportLogs=enabled
pod-template-hash=5cb566fd69
Annotations:
Status: Running
IP: 10.244.1.5
IPs:
IP: 10.244.1.5
Controlled By: ReplicaSet/config-server-5cb566fd69
Containers:
config-server:
Container ID: docker://063a332d98e0c4678892ef7c891589f675e0ea8d4059a8bd932b08e053181402
Image: mosipid/sandbox-config-server:1.1.0
Image ID: docker-pullable://mosipid/sandbox-config-server@sha256:3f2a6e094ace6db61e717b23356b2181e2a58774b14896f713848bb339f7e523
Port: 51000/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 15 Mar 2022 09:10:38 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 300m
memory: 3Gi
Requests:
cpu: 200m
memory: 1Gi
Liveness: tcp-socket :51000 delay=0s timeout=5s period=10s #success=1 #failure=15
Readiness: tcp-socket :51000 delay=0s timeout=5s period=10s #success=1 #failure=15
Startup: tcp-socket :51000 delay=60s timeout=5s period=10s #success=1 #failure=15
Environment:
SPRING_CLOUD_CONFIG_SERVER_GIT_SEARCHPATHS: sandbox
ENCRYPT_KEY: xyz
SPRING_CLOUD_CONFIG_SERVER_OVERRIDES_MOSIPBOX_PUBLIC_URL: https://console.sb
SPRING_CLOUD_CONFIG_SERVER_GIT_URI: GitHub - mosip/mosip-config: This repository contains MOSIP configuration templates
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qq6gd (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-qq6gd:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

[mosipuser@console sandbox-v2]$ kc1 get pods -A | grep artifactory
default artifactory-service-69f4644bff-nt4qv 1/1 Running 0 4h24m

[mosipuser@console sandbox-v2]$ kc1 describe pod artifactory-service-69f4644bff-nt4qv
Name: artifactory-service-69f4644bff-nt4qv
Namespace: default
Priority: 0
Node: mzworker15.sb/10.0.0.46
Start Time: Tue, 15 Mar 2022 09:04:55 +0000
Labels: app=artifactory-service
exportLogs=enabled
pod-template-hash=69f4644bff
Annotations:
Status: Running
IP: 10.244.17.3
IPs:
IP: 10.244.17.3
Controlled By: ReplicaSet/artifactory-service-69f4644bff
Containers:
artifactory-service:
Container ID: docker://a003f7d7eb9a0efe167e0c9c041caadcc2c50b8eb3f8e06adf6020c263b6f828
Image: mosipid/artifactory-server:1.2.0
Image ID: docker-pullable://mosipid/artifactory-server@sha256:2616b8c3d7e84a4f0707a97279bc18ea99266cfa177e433f6857339f775d6a76
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 15 Mar 2022 09:05:43 +0000
Ready: True
Restart Count: 0
Liveness: tcp-socket :8080 delay=0s timeout=5s period=10s #success=1 #failure=30
Readiness: tcp-socket :8080 delay=0s timeout=5s period=10s #success=1 #failure=30
Startup: tcp-socket :8080 delay=60s timeout=5s period=10s #success=1 #failure=30
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bh8rn (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-bh8rn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

[mosipuser@console sandbox-v2]$ kc1 describe pod db-audit-init-job-zk456
Name: db-audit-init-job-zk456
Namespace: default
Priority: 0
Node: mzworker12.sb/10.0.0.43
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-audit-init-job
controller-uid=a76a1510-afa0-4136-ae82-f0d8950b38a9
job-name=db-audit-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.11.3
IPs:
IP: 10.244.11.3
Controlled By: Job/db-audit-init-job
Containers:
postgres-init:
Container ID: docker://87249182dd3808c244e896860ef1e09591df55761007bc8a35afb2ac4d3c9bb7
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:26 +0000
Finished: Tue, 15 Mar 2022 08:31:12 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-audit-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bgbwm (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-bgbwm:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

[mosipuser@console sandbox-v2]$ kc1 describe pod db-authdevice-init-job-rb2vm
Name: db-authdevice-init-job-rb2vm
Namespace: default
Priority: 0
Node: mzworker0.sb/10.0.0.23
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-authdevice-init-job
controller-uid=f0ff39e7-61aa-47f2-9de4-f0fe70cf29d1
job-name=db-authdevice-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.5.3
IPs:
IP: 10.244.5.3
Controlled By: Job/db-authdevice-init-job
Containers:
postgres-init:
Container ID: docker://eb1ffc27f125771451c2b03d13535aae8ef4427eaaa4a9b5091a6ad82f4896f3
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:26 +0000
Finished: Tue, 15 Mar 2022 08:32:32 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-authdevice-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vs44v (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-vs44v:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

[mosipuser@console sandbox-v2]$ kc1 describe pod db-credential-init-job-c2rj8
Name: db-credential-init-job-c2rj8
Namespace: default
Priority: 0
Node: mzworker7.sb/10.0.0.30
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-credential-init-job
controller-uid=951ba165-0845-4f47-a1e1-1154107b2010
job-name=db-credential-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.7.4
IPs:
IP: 10.244.7.4
Controlled By: Job/db-credential-init-job
Containers:
postgres-init:
Container ID: docker://548243ea1af22263f28f48b6ac34e1b9363cb24df278ee45430fbcfcd7f5854c
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:26 +0000
Finished: Tue, 15 Mar 2022 08:31:27 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-credential-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rgt42 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-rgt42:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

[mosipuser@console sandbox-v2]$ kc1 describe pod db-hotlist-init-job-lftjs
Name: db-hotlist-init-job-lftjs
Namespace: default
Priority: 0
Node: mzworker8.sb/10.0.0.31
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-hotlist-init-job
controller-uid=41bb8f5a-39a3-4bf3-9913-f36db056914a
job-name=db-hotlist-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.8.5
IPs:
IP: 10.244.8.5
Controlled By: Job/db-hotlist-init-job
Containers:
postgres-init:
Container ID: docker://ba6f6f27e74bfc0c1585b99fa2277c4e85c1ad7d7ac9ce470c4725e977c607e9
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:33 +0000
Finished: Tue, 15 Mar 2022 08:31:14 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-hotlist-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bbb85 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-bbb85:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

[mosipuser@console sandbox-v2]$ kc1 describe pod db-ida-init-job-kgkf5
Name: db-ida-init-job-kgkf5
Namespace: default
Priority: 0
Node: mzworker0.sb/10.0.0.23
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-ida-init-job
controller-uid=96ebfffb-78b9-49da-9127-d6f780e3c6cf
job-name=db-ida-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.5.5
IPs:
IP: 10.244.5.5
Controlled By: Job/db-ida-init-job
Containers:
postgres-init:
Container ID: docker://82556cde4d439d2508f249c5dd6bbe93f87187426e6ed1a5a8adc6a434d228f2
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:33 +0000
Finished: Tue, 15 Mar 2022 08:33:26 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-ida-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8hcpf (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-8hcpf:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

[mosipuser@console sandbox-v2]$ kc1 describe pod db-kernel-init-job-95ngk
Name: db-kernel-init-job-95ngk
Namespace: default
Priority: 0
Node: mzworker12.sb/10.0.0.43
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-kernel-init-job
controller-uid=a9411eb5-25f9-43b8-acd0-038f365ebde9
job-name=db-kernel-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.11.5
IPs:
IP: 10.244.11.5
Controlled By: Job/db-kernel-init-job
Containers:
postgres-init:
Container ID: docker://04732bf5f71103c1d97c2c1124f4c3efc8fd47802dcc1f24126c011dee8f90db
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:32 +0000
Finished: Tue, 15 Mar 2022 08:31:36 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-kernel-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m7q6b (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-m7q6b:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
[mosipuser@console sandbox-v2]$ kc1 describe pod db-keymgr-init-job-94rkr
Name: db-keymgr-init-job-94rkr
Namespace: default
Priority: 0
Node: mzworker9.sb/10.0.0.40
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-keymgr-init-job
controller-uid=d9b5fd05-00aa-4b07-8354-178e6065ecf2
job-name=db-keymgr-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.10.4
IPs:
IP: 10.244.10.4
Controlled By: Job/db-keymgr-init-job
Containers:
postgres-init:
Container ID: docker://5885596d865dc2a7098136aea1f984ed0955e54f9337c3c9776ffa42e8d9ff8a
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:26 +0000
Finished: Tue, 15 Mar 2022 08:32:07 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-keymgr-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nbllh (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-nbllh:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

[mosipuser@console sandbox-v2]$ kc1 describe pod db-master-init-job-54czw
Name: db-master-init-job-54czw
Namespace: default
Priority: 0
Node: mzworker8.sb/10.0.0.31
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-master-init-job
controller-uid=6519954a-d736-4e1c-9ece-16de55c0df3a
job-name=db-master-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.8.4
IPs:
IP: 10.244.8.4
Controlled By: Job/db-master-init-job
Containers:
postgres-init:
Container ID: docker://8fe73e6e4017a3fa607fb13a20c63e9417bf6b3f73931d310f2b5c85dccaf74d
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:30 +0000
Finished: Tue, 15 Mar 2022 08:36:24 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-master-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cj5xh (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-cj5xh:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
[mosipuser@console sandbox-v2]$ kc1 describe pod db-pms-init-job-bsw62
Name: db-pms-init-job-bsw62
Namespace: default
Priority: 0
Node: mzworker4.sb/10.0.0.27
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-pms-init-job
controller-uid=4b531741-25d7-4bff-877f-c379e42a7409
job-name=db-pms-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.2.4
IPs:
IP: 10.244.2.4
Controlled By: Job/db-pms-init-job
Containers:
postgres-init:
Container ID: docker://fbdb27f6534b9bcbfe062c8ee5144a92a47f8d368aa84082457a2cc4c470bc43
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:26 +0000
Finished: Tue, 15 Mar 2022 08:34:46 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-pms-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b9d48 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-b9d48:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

Here are the rest of the postgres-init

[mosipuser@console sandbox-v2]$ kc1 describe pod db-prereg-init-job-xvsbp
Name: db-prereg-init-job-xvsbp
Namespace: default
Priority: 0
Node: mzworker13.sb/10.0.0.44
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-prereg-init-job
controller-uid=249050ad-a881-4a24-b9aa-9bf5e7193b8c
job-name=db-prereg-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.14.3
IPs:
IP: 10.244.14.3
Controlled By: Job/db-prereg-init-job
Containers:
postgres-init:
Container ID: docker://c7ae61a224a0923a21d3676bd720be0deb2ddda911f0e5049bdedb679c86ac91
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:26 +0000
Finished: Tue, 15 Mar 2022 08:32:30 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-prereg-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fkq7j (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-fkq7j:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
[mosipuser@console sandbox-v2]$ kc1 describe pod db-regdevice-init-job-vj698
Name: db-regdevice-init-job-vj698
Namespace: default
Priority: 0
Node: mzworker17.sb/10.0.0.48
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-regdevice-init-job
controller-uid=9e047fd3-7e51-4586-8e93-5e5c890a1b74
job-name=db-regdevice-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.16.4
IPs:
IP: 10.244.16.4
Controlled By: Job/db-regdevice-init-job
Containers:
postgres-init:
Container ID: docker://ccccffb80ac61cbda3b4002556b3f64be71fc46e7663f866ad9a60cf7fc318bf
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:26 +0000
Finished: Tue, 15 Mar 2022 08:32:32 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-regdevice-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2n29t (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-2n29t:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

[mosipuser@console sandbox-v2]$ kc1 describe pod db-regprc-init-job-7wqhr
Name: db-regprc-init-job-7wqhr
Namespace: default
Priority: 0
Node: mzworker12.sb/10.0.0.43
Start Time: Tue, 15 Mar 2022 08:29:21 +0000
Labels: app.kubernetes.io/instance=postgres-init
app.kubernetes.io/name=db-regprc-init-job
controller-uid=19aced22-ccef-4cb0-8fec-bd6347f2971e
job-name=db-regprc-init-job
sidecar.istio.io/inject=false
Annotations:
Status: Succeeded
IP: 10.244.11.4
IPs:
IP: 10.244.11.4
Controlled By: Job/db-regprc-init-job
Containers:
postgres-init:
Container ID: docker://2a1cb91b1e625a5a5372e59fc6b7437c0637ff8ba77739c1fda1a8058c4adb84
Image: mosipdev/postgres-init:develop
Image ID: docker-pullable://mosipdev/postgres-init@sha256:1f489ee98f01ce7c95b8653997722873ca1430a580662ea2d7e423f93ca228a4
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 15 Mar 2022 08:29:29 +0000
Finished: Tue, 15 Mar 2022 08:32:56 +0000
Ready: False
Restart Count: 0
Environment Variables from:
db-regprc-init-env-config ConfigMap Optional: false
Environment:
SU_USER_PWD: <set to the key ‘postgresql-password’ in secret ‘postgres-postgresql’> Optional: false
DBUSER_PWD: <set to the key ‘db-dbuser-password’ in secret ‘db-common-secrets’> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8c5c2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-8c5c2:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

The issue seems to be with the posgres-init chart.
It was pointing to one of the older tag so DB was not correct for the same. we have republished the postgres-init-1.2.0 chart…
Can you perform below steps again…

  1.   helm1 delete postgres-init kernel
    
  2.   kc1 delete job <kernel-keys-gen job name>
    
  3.   an playbooks/postgres-init-db.yml
    
  4.   an playbooks/postgres-md-upload.yml
    
  5.   an playbooks/kernel.yml.

Thank you very much.

I will do these as soon as I get in to the office tomorrow. My Internet is acting up and I am intermittently losing my remote access connection to our machines.

I will post an update as soon as I can.

Best regards,
Ramon

Thank you very much for all your help.

Happy to say that there is no longer an error when installing postgres-init-db and postgres-md-upload.

However, the error in installing kernel module is still there.

Since, that is the case , at this time I am very sorry that I have decided to abandon this installation and reverted back to installing 1.1.5.4. I hope that I did not cause too much trouble and inconvenience while you are helping me with this installation.

I decided to revert back to 1.1.5.4 because I had an insight that I believe would solve the issues I encountered before. Glad to say that I was right. I managed to install MOSIP sandbox v2 1.1.5.4 and seems to be working fine except for the configuration of Prereg captcha and OTP. The trick seems to be is locking the specific Kubernetes version to 1.21.1. Using a different version creates undefined errors which I experienced that is why I decided installing MOSIP release-1.2.0.

Again, all your help and assistance are deeply appreciated.

Anyway, can/how do I edit this topic and label it closed?

All the best.

Thanks @rcsampang we will close this tpic