Hello and thank you for your help.
Just to clarify, this is a full V3 server-side deployment of MOSIP v1.2.0.1 on a dedicated Kubernetes cluster. It’s running on 2 nodes, below is more information about environment.
Could be the problem that config server seems to be having a problem loading its configuration from Git repository. This causes other services like admin-service and prereg-application to fail because they cannot get their configuration, which leads to the errors we see in the browser. The Git message appears in the log after a while. Not sure if this is it or if Mosip server is even supported on my system.
Every service is running but there is some miscommunication going on internally. I have tried to tinker with services and proxy without success. What would be the proper way to troubleshoot?
Going through admin-proxy http://10.4.0.219:30086/ I get these now:
127.0.0.6 - - [17/Jul/2025:11:38:04 +0000] "GET / HTTP/1.1" 426 16 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0" "-"
127.0.0.6 - - [17/Jul/2025:11:38:04 +0000] "GET /favicon.ico HTTP/1.1" 426 16 "http://10.4.0.219:30086/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0" "-"
Here’s just the message from config-server log:
2025-07-17 08:47:39,657 [http-nio-51000-exec-8] WARN [o.s.c.c.s.e.JGitEnvironmentRepository].warn.671 : Could not fetch remote for master remote: https://github.com/mosip/mosip-config
2025-07-17 08:47:39,666 [http-nio-51000-exec-8] INFO [o.s.c.s.AbstractApplicationContext].prepareRefresh.590 : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@185710ad: startup date [Thu Jul 17 08:47:39 UTC 2025]; root of context hierarchy
Below is server info and start of the config-server log.
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/rke2-coredns-rke2-coredns:udp-53/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
mosip-k8s Ready control-plane,etcd,master 17d v1.32.5+rke2r1 10.4.0.219 <none> Ubuntu 22.04.5 LTS 5.15.0-143-generic containerd://2.0.5-k3s1
mosip-k8s-2 Ready <none> 16d v1.32.6+rke2r1 10.4.0.221 <none> Ubuntu 22.04.5 LTS 5.15.0-143-generic containerd://2.0.5-k3s1
NAME STATUS AGE
abis Active 2d17h
activemq Active 16d
admin Active 2d17h
artifactory Active 16d
biosdk Active 16d
captcha Active 16d
clamav Active 16d
conf-secrets Active 10d
config-server Active 10d
datashare Active 16d
default Active 17d
ida Active 10d
idrepo Active 16d
ingress-nginx Active 17d
istio-system Active 17d
kafka Active 16d
kernel Active 2d18h
keycloak Active 16d
keymanager Active 16d
kube-node-lease Active 17d
kube-public Active 17d
kube-system Active 17d
landing-page Active 16d
local-path-storage Active 17d
longhorn-system Active 10d
masterdata-loader Active 16d
mimoto Active 6d22h
minio Active 16d
mock-smtp Active 16d
mosip-file-server Active 2d16h
msg-gateways Active 16d
onboarder Active 3d3h
packetmanager Active 16d
pms Active 16d
postgres Active 17d
prereg Active 2d18h
print Active 10d
regclient Active 2d5h
regproc Active 2d17h
resident Active 2d15h
s3 Active 16d
softhsm Active 16d
websub Active 16d
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
mosip-k8s 2893m 19% 43217Mi 79%
mosip-k8s-2 582m 3% 47923Mi 82%
mosipadmin@mosip-k8s:~/mosip-infra/deployment/v3/mosip/config-server$ kubectl get service --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
abis mock-abis ClusterIP 10.43.214.251 <none> 80/TCP 2d18h
abis mock-mv ClusterIP 10.43.80.88 <none> 80/TCP 2d18h
activemq activemq-activemq-artemis ClusterIP 10.43.79.46 <none> 8161/TCP,61616/TCP,5672/TCP 17d
activemq activemq-activemq-artemis-master ClusterIP 10.43.39.48 <none> 8161/TCP,61616/TCP,5672/TCP,9494/TCP 17d
activemq activemq-activemq-artemis-slave ClusterIP None <none> 8161/TCP,61616/TCP,5672/TCP,9494/TCP 17d
admin admin-hotlist ClusterIP 10.43.229.242 <none> 80/TCP 64m
admin admin-proxy NodePort 10.43.133.87 <none> 80:30086/TCP 56m
admin admin-service ClusterIP 10.43.1.227 <none> 80/TCP 64m
admin admin-ui ClusterIP 10.43.138.57 <none> 80/TCP 64m
artifactory artifactory ClusterIP 10.43.193.218 <none> 80/TCP 2d22h
biosdk biosdk-service ClusterIP 10.43.127.157 <none> 80/TCP 2d21h
clamav clamav ClusterIP 10.43.198.211 <none> 3310/TCP 17d
config-server config-server ClusterIP 10.43.89.207 <none> 80/TCP 2d22h
datashare datashare ClusterIP 10.43.38.182 <none> 80/TCP 2d21h
default auth-service-proxy ClusterIP 10.43.179.114 <none> 8091/TCP 3d6h
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 17d
default mailhog NodePort 10.43.215.214 <none> 1025:30025/TCP,8025:30825/TCP 17d
ida ida-auth ClusterIP 10.43.162.30 <none> 80/TCP 2d17h
ida ida-internal ClusterIP 10.43.116.42 <none> 80/TCP 2d17h
ida ida-otp ClusterIP 10.43.217.86 <none> 80/TCP 2d17h
idrepo credential ClusterIP 10.43.221.113 <none> 80/TCP 2d18h
idrepo credentialrequest ClusterIP 10.43.107.125 <none> 80/TCP 2d18h
idrepo identity ClusterIP 10.43.14.132 <none> 80/TCP 2d18h
idrepo vid ClusterIP 10.43.164.49 <none> 80/TCP 2d18h
ingress-nginx ingress-nginx-controller LoadBalancer 10.43.136.159 <pending> 80:32713/TCP,443:30331/TCP 17d
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.43.100.140 <none> 443/TCP 17d
istio-system istio-egressgateway ClusterIP 10.43.102.218 <none> 80/TCP,443/TCP 17d
istio-system istio-ingressgateway LoadBalancer 10.43.46.121 <pending> 15021:30821/TCP,80:32073/TCP,443:31423/TCP,31400:30220/TCP,15443:30443/TCP 17d
istio-system istio-ingressgateway-clusterip ClusterIP 10.43.121.123 <none> 443/TCP 8d
istio-system istiod ClusterIP 10.43.195.162 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 17d
kafka kafka ClusterIP 10.43.33.162 <none> 9092/TCP 17d
kafka kafka-headless ClusterIP None <none> 9092/TCP,9093/TCP 17d
kafka kafka-ui ClusterIP 10.43.150.150 <none> 80/TCP 17d
kafka kafka-zookeeper ClusterIP 10.43.220.62 <none> 2181/TCP,2888/TCP,3888/TCP 17d
kafka kafka-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 17d
kernel auditmanager ClusterIP 10.43.119.53 <none> 80/TCP 2d19h
kernel authmanager ClusterIP 10.43.66.203 <none> 80/TCP 2d19h
kernel idgenerator ClusterIP 10.43.221.196 <none> 80/TCP 2d19h
kernel masterdata ClusterIP 10.43.81.251 <none> 80/TCP 2d19h
kernel notifier ClusterIP 10.43.246.245 <none> 80/TCP 2d19h
kernel otpmanager ClusterIP 10.43.244.180 <none> 80/TCP 2d19h
kernel pridgenerator ClusterIP 10.43.13.96 <none> 80/TCP 2d19h
kernel ridgenerator ClusterIP 10.43.201.145 <none> 80/TCP 2d19h
kernel syncdata ClusterIP 10.43.155.169 <none> 80/TCP 2d19h
keycloak keycloak NodePort 10.43.44.87 <none> 80:30080/TCP,443:30955/TCP 17d
keycloak keycloak-headless ClusterIP None <none> 80/TCP 17d
keycloak keycloak-postgresql ClusterIP 10.43.63.182 <none> 5432/TCP 17d
keycloak keycloak-postgresql-hl ClusterIP None <none> 5432/TCP 17d
keymanager keymanager ClusterIP 10.43.255.145 <none> 80/TCP 2d22h
kube-system rke2-coredns-rke2-coredns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP 17d
kube-system rke2-metrics-server ClusterIP 10.43.69.156 <none> 443/TCP 17d
landing-page landing-page ClusterIP 10.43.105.107 <none> 80/TCP 16d
longhorn-system longhorn-admission-webhook ClusterIP 10.43.138.119 <none> 9502/TCP 10d
longhorn-system longhorn-backend ClusterIP 10.43.85.20 <none> 9500/TCP 10d
longhorn-system longhorn-conversion-webhook ClusterIP 10.43.112.89 <none> 9501/TCP 10d
longhorn-system longhorn-frontend ClusterIP 10.43.38.188 <none> 80/TCP 10d
longhorn-system longhorn-recovery-backend ClusterIP 10.43.44.251 <none> 9503/TCP 10d
longhorn-system pvc-54c7cc83-c0a6-435e-8c06-57785dda9717 ClusterIP 10.43.23.101 <none> 2049/TCP 2d18h
minio minio ClusterIP 10.43.208.119 <none> 9000/TCP,9001/TCP 17d
mock-smtp mock-smtp ClusterIP 10.43.42.15 <none> 8080/TCP,8081/TCP,8025/TCP 2d22h
mosip-file-server mosip-file-server ClusterIP 10.43.126.98 <none> 80/TCP 2d17h
packetmanager packetmanager ClusterIP 10.43.106.3 <none> 80/TCP 2d21h
pms pmp-ui ClusterIP 10.43.243.254 <none> 80/TCP 2d18h
pms pms-partner ClusterIP 10.43.1.164 <none> 80/TCP 2d18h
pms pms-policy ClusterIP 10.43.12.56 <none> 80/TCP 2d18h
postgres postgres-postgresql ClusterIP 10.43.164.189 <none> 5432/TCP 2d22h
postgres postgres-postgresql-hl ClusterIP None <none> 5432/TCP 2d22h
prereg prereg-application NodePort 10.43.179.158 <none> 80:30082/TCP 2d1h
prereg prereg-batchjob ClusterIP 10.43.97.183 <none> 80/TCP 2d19h
prereg prereg-booking ClusterIP 10.43.87.208 <none> 80/TCP 2d19h
prereg prereg-captcha ClusterIP 10.43.106.24 <none> 80/TCP 2d19h
prereg prereg-datasync ClusterIP 10.43.190.32 <none> 80/TCP 2d19h
prereg prereg-ui NodePort 10.43.124.146 <none> 80:30087/TCP 2d19h
print print-service ClusterIP 10.43.107.4 <none> 80/TCP 2d17h
regclient regclient NodePort 10.43.137.135 <none> 80:30083/TCP 2d5h
regproc regproc-camel ClusterIP 10.43.142.195 <none> 80/TCP 2d18h
regproc regproc-group1 ClusterIP 10.43.197.216 <none> 80/TCP 2d18h
regproc regproc-group2 ClusterIP 10.43.36.120 <none> 80/TCP,9072/TCP,8088/TCP 2d18h
regproc regproc-group3 ClusterIP 10.43.111.179 <none> 9096/TCP,9071/TCP,8084/TCP 2d18h
regproc regproc-group4 ClusterIP 10.43.136.22 <none> 8020/TCP,8091/TCP 2d18h
regproc regproc-group5 ClusterIP 10.43.211.133 <none> 8088/TCP,8089/TCP,8093/TCP,8094/TCP,8095/TCP 2d18h
regproc regproc-group6 ClusterIP 10.43.51.64 <none> 8087/TCP,8092/TCP 2d18h
regproc regproc-group7 ClusterIP 10.43.92.160 <none> 8099/TCP,8097/TCP 2d18h
regproc regproc-landingzone ClusterIP 10.43.243.190 <none> 80/TCP 2d18h
regproc regproc-notifier ClusterIP 10.43.231.80 <none> 80/TCP 2d18h
regproc regproc-pktserver ClusterIP 10.43.186.182 <none> 80/TCP 2d18h
regproc regproc-reprocess ClusterIP 10.43.158.176 <none> 80/TCP 2d18h
regproc regproc-status ClusterIP 10.43.192.211 <none> 80/TCP 2d18h
regproc regproc-trans ClusterIP 10.43.161.195 <none> 80/TCP 2d18h
regproc regproc-workflow ClusterIP 10.43.124.138 <none> 80/TCP,8023/TCP 2d18h
resident resident NodePort 10.43.32.237 <none> 80:30084/TCP 2d1h
resident resident-ui ClusterIP 10.43.24.89 <none> 8080/TCP 2d16h
softhsm softhsm-ida ClusterIP 10.43.77.98 <none> 80/TCP 17d
softhsm softhsm-kernel ClusterIP 10.43.82.30 <none> 80/TCP 17d
websub websub ClusterIP 10.43.186.47 <none> 80/TCP 2d22h
websub websub-consolidator ClusterIP 10.43.60.32 <none> 80/TCP 2d22h
mosipadmin@mosip-k8s:~/mosip-infra/deployment/v3/mosip/config-server$ kubectl logs -n config-server config-server-75fc94d9d9-9cccx
2025-07-17 07:49:18,368 [main] INFO [o.s.c.s.AbstractApplicationContext].prepareRefresh.590 : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@3d36e4cd: startup date [Thu Jul 17 07:49:18 UTC 2025]; root of context hierarchy
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (jar:file:/home/mosip/kernel-config-server.jar!/BOOT-INF/lib/spring-core-5.0.6.RELEASE.jar!/) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2025-07-17 07:49:23,072 [main] INFO [o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker].postProcessAfterInitialization.328 : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$127f577] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.0.2.RELEASE)
2025-07-17 07:49:27,270 [main] INFO [o.s.b.SpringApplication].logStartupProfileInfo.659 : No active profile set, falling back to default profiles: default
2025-07-17 07:49:27,669 [main] INFO [o.s.c.s.AbstractApplicationContext].prepareRefresh.590 : Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@236e3f4e: startup date [Thu Jul 17 07:49:27 UTC 2025]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@3d36e4cd
2025-07-17 07:49:39,071 [main] INFO [o.s.c.c.s.GenericScope].setSerializationId.294 : BeanFactory id=569534b7-b1f2-384d-bc50-ee91e861ca55
2025-07-17 07:49:40,171 [main] INFO [o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker].postProcessAfterInitialization.328 : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$127f577] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2025-07-17 07:49:43,575 [main] INFO [o.s.b.w.e.t.TomcatWebServer].initialize.91 : Tomcat initialized with port(s): 51000 (http)
2025-07-17 07:49:43,869 [main] INFO [o.a.j.l.DirectJDKLog].log.180 : Initializing ProtocolHandler ["http-nio-51000"]
2025-07-17 07:49:44,169 [main] INFO [o.a.j.l.DirectJDKLog].log.180 : Starting service [Tomcat]
2025-07-17 07:49:44,169 [main] INFO [o.a.j.l.DirectJDKLog].log.180 : Starting Servlet Engine: Apache Tomcat/8.5.37
2025-07-17 07:49:44,275 [localhost-startStop-1] INFO [o.a.j.l.DirectJDKLog].log.180 : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
2025-07-17 07:49:45,468 [localhost-startStop-1] INFO [o.a.j.l.DirectJDKLog].log.180 : Initializing Spring embedded WebApplicationContext
2025-07-17 07:49:45,469 [localhost-startStop-1] INFO [o.s.b.w.s.c.ServletWebServerApplicationContext].prepareWebApplicationContext.285 : Root WebApplicationContext: initialization completed in 17800 ms
2025-07-17 07:50:43,469 [localhost-startStop-1] INFO [o.s.b.w.s.ServletRegistrationBean].addRegistration.185 : Servlet dispatcherServlet mapped to [/config/*]
2025-07-17 07:50:43,472 [localhost-startStop-1] INFO [o.s.b.w.s.AbstractFilterRegistrationBean].configure.244 : Mapping filter: 'characterEncodingFilter' to: [/*]
2025-07-17 07:50:43,473 [localhost-startStop-1] INFO [o.s.b.w.s.AbstractFilterRegistrationBean].configure.244 : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2025-07-17 07:50:43,568 [localhost-startStop-1] INFO [o.s.b.w.s.AbstractFilterRegistrationBean].configure.244 : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2025-07-17 07:50:43,569 [localhost-startStop-1] INFO [o.s.b.w.s.AbstractFilterRegistrationBean].configure.244 : Mapping filter: 'requestContextFilter' to: [/*]
2025-07-17 07:50:43,569 [localhost-startStop-1] INFO [o.s.b.w.s.AbstractFilterRegistrationBean].configure.244 : Mapping filter: 'httpTraceFilter' to: [/*]
2025-07-17 07:50:43,569 [localhost-startStop-1] INFO [o.s.b.w.s.AbstractFilterRegistrationBean].configure.244 : Mapping filter: 'webMvcMetricsFilter' to: [/*]
2025-07-17 07:50:46,074 [main] INFO [o.s.w.s.h.AbstractUrlHandlerMapping].registerHandler.373 : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2025-07-17 07:50:50,970 [main] INFO [o.s.w.s.m.m.a.RequestMappingHandlerAdapter].initControllerAdviceCache.588 : Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@236e3f4e: startup date [Thu Jul 17 07:49:27 UTC 2025]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@3d36e4cd
2025-07-17 07:50:52,574 [main] INFO [o.s.w.s.h.AbstractHandlerMethodMapping$MappingRegistry].register.550 : Mapped "{[/refresh],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.String> io.mosip.kernel.config.server.controller.RefreshController.refreshContext(java.lang.String)
2025-07-17 07:50:52,576 [main] INFO [o.s.w.s.h.AbstractHandlerMethodMapping$MappingRegistry].register.550 : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2025-07-17 07:50:52,669 [main] INFO [o.s.w.s.h.AbstractHandlerMethodMapping$MappingRegistry].register.550 : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2025-07-17 07:50:52,971 [main] INFO [o.s.w.s.h.AbstractHandlerMethodMapping$MappingRegistry].register.550 : Mapped "{[/encrypt/status],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> org.springframework.cloud.config.server.encryption.EncryptionController.status()
2025-07-17 07:50:52,971 [main] INFO [o.s.w.s.h.AbstractHandlerMethodMapping$MappingRegistry].register.550 : Mapped "{[/key/{name}/{profiles}],methods=[GET]}" onto public java.lang.String org.springframework.cloud.config.server.encryption.EncryptionController.getPublicKey(java.lang.String,java.lang.String)
