UIN data not syncing with IDA, empty identity_cache and credential_event_store tables

Platform
1

Dear MOSIP Team,

We’ve done MOSIP on-prem setup for v1.2.0.1. When I process a packet, UIN gets generated but it doesn’t reflect in mosip_ida database as the identity_cache and credential_event_store tables remain empty. Same issue is there when I upload certificates, I can see certificates in mosip_keymgr db but they don’t reflect in other databases like mosip_ida or mosip_master.

This issue is in continuation with following issue:

we resolved the certificate signature not matching issue but the main root cause was that the data is not syncing properly. We suspect that there’s an issue with websub. Can anyone from Devops team help us on this?

Thanks,
Durgesh

2

Hi @Durgesh,

Thank you for reaching out to us.

Will connect you with devops team as you have requested to check meanwhile any logs you can share to debug to make sure whether the issue is from websub?

Also can you confirm the relevant event have been published from ID Repo to send data to IDA

Regards,
Varaniya S

3

Hi @Varaniya1,

Thank you for the reply.

Here are the logs:

Websub logs:

127.0.0.6 - - [26/Sep/2025:06:56:21 +0000] "GET /hub/actuator/health HTTP/1.1" 200 215 "-" "kube-probe/1.32"
127.0.0.6 - - [26/Sep/2025:06:56:29 +0000] "POST /hub/?hub.mode=publish&hub.topic=mpartner-default-auth/CREDENTIAL_ISSUED HTTP/1.1" 200 17 "-" "Apache-HttpClient/4.5.6 (Java/11.0.16)"
127.0.0.6 - - [26/Sep/2025:06:56:31 +0000] "GET /hub/actuator/health HTTP/1.1" 200 215 "-" "kube-probe/1.32"

IDA internal logs:

{"@timestamp":"2025-09-26T06:56:22.850Z","@version":"1","message":"sessionId -  - healthCheck - READ_KEY_SUCCESS, Key Algorithm: AES","logger_name":"io.mosip.kernel.keymanager.hsm.health.HSMHealthCheck","thread_name":"http-nio-8093-exec-7","level":"INFO","level_value":20000,"appName":"id-authentication,id-authentication-internal","traceId":"a1b1e04e4359a76e","spanExportable":"false","req.requestURI":"/idauthentication/v1/internal/actuator/health","X-Span-Export":"false","req.method":"GET","req.userAgent":"kube-probe/1.32","spanId":"a1b1e04e4359a76e","X-B3-SpanId":"a1b1e04e4359a76e","X-B3-TraceId":"a1b1e04e4359a76e","req.remoteHost":"127.0.0.6","req.requestURL":"http://192.168.154.136:8093/idauthentication/v1/internal/actuator/health"}
2025-09-26 06:56:22.850  INFO [id-authentication,id-authentication-internal,a1b1e04e4359a76e,a1b1e04e4359a76e,false] 64 --- [nio-8093-exec-7] i.m.k.k.hsm.health.HSMHealthCheck        : sessionId -  - healthCheck - READ_KEY_SUCCESS, Key Algorithm: AES
{"@timestamp":"2025-09-26T06:56:22.851Z","level":"ACCESS","level_value":70000,"traceId":"a1b1e04e4359a76e","statusCode":200,"req.requestURI":"/idauthentication/v1/internal/actuator/health","bytesSent":309,"timeTaken":0.049,"appName":"id-authentication,id-authentication-internal","req.userAgent":"kube-probe/1.32","req.xForwardedFor":"-","req.referer":"-","req.method":"GET","req.remoteHost":"127.0.0.6"}
{"@timestamp":"2025-09-26T06:56:32.606Z","@version":"1","message":"sessionId -  - healthCheck - READ_KEY_SUCCESS, Key Algorithm: AES","logger_name":"io.mosip.kernel.keymanager.hsm.health.HSMHealthCheck","thread_name":"http-nio-8093-exec-9","level":"INFO","level_value":20000,"appName":"id-authentication,id-authentication-internal","traceId":"ce8958b624c91a13","spanExportable":"false","req.requestURI":"/idauthentication/v1/internal/actuator/health","X-Span-Export":"false","req.method":"GET","req.userAgent":"kube-probe/1.32","spanId":"ce8958b624c91a13","X-B3-SpanId":"ce8958b624c91a13","X-B3-TraceId":"ce8958b624c91a13","req.remoteHost":"127.0.0.6","req.requestURL":"http://192.168.154.136:8093/idauthentication/v1/internal/actuator/health"}
2025-09-26 06:56:32.606  INFO [id-authentication,id-authentication-internal,ce8958b624c91a13,ce8958b624c91a13,false] 64 --- [nio-8093-exec-9] i.m.k.k.hsm.health.HSMHealthCheck        : sessionId -  - healthCheck - READ_KEY_SUCCESS, Key Algorithm: AES
{"@timestamp":"2025-09-26T06:56:32.607Z","level":"ACCESS","level_value":70000,"traceId":"ce8958b624c91a13","statusCode":200,"req.requestURI":"/idauthentication/v1/internal/actuator/health","bytesSent":309,"timeTaken":0.016,"appName":"id-authentication,id-authentication-internal","req.userAgent":"kube-probe/1.32","req.xForwardedFor":"-","req.referer":"-","req.method":"GET","req.remoteHost":"127.0.0.6"}
{"@timestamp":"2025-09-26T06:56:32.809Z","@version":"1","message":"sessionId -  - healthCheck - READ_KEY_SUCCESS, Key Algorithm: AES","logger_name":"io.mosip.kernel.keymanager.hsm.health.HSMHealthCheck","thread_name":"http-nio-8093-exec-10","level":"INFO","level_value":20000,"appName":"id-authentication,id-authentication-internal","traceId":"61e3476eb7fcd531","spanExportable":"false","req.requestURI":"/idauthentication/v1/internal/actuator/health","X-Span-Export":"false","req.method":"GET","req.userAgent":"kube-probe/1.32","spanId":"61e3476eb7fcd531","X-B3-SpanId":"61e3476eb7fcd531","X-B3-TraceId":"61e3476eb7fcd531","req.remoteHost":"127.0.0.6","req.requestURL":"http://192.168.154.136:8093/idauthentication/v1/internal/actuator/health"}
2025-09-26 06:56:32.809  INFO [id-authentication,id-authentication-internal,61e3476eb7fcd531,61e3476eb7fcd531,false] 64 --- [io-8093-exec-10] i.m.k.k.hsm.health.HSMHealthCheck        : sessionId -  - healthCheck - READ_KEY_SUCCESS, Key Algorithm: AES
{"@timestamp":"2025-09-26T06:56:32.810Z","level":"ACCESS","level_value":70000,"traceId":"61e3476eb7fcd531","statusCode":200,"req.requestURI":"/idauthentication/v1/internal/actuator/health","bytesSent":309,"timeTaken":0.009,"appName":"id-authentication,id-authentication-internal","req.userAgent":"kube-probe/1.32","req.xForwardedFor":"-","req.referer":"-","req.method":"GET","req.remoteHost":"127.0.0.6"}

I can only see health check logs in ida-internal, the event is not being consumed here.

Credential Service logs:

{"@timestamp":"2025-09-26T06:56:29.584Z","@version":"1","message":"service-account-mosip-crereq-client - REQUEST_ID - 18f0144c-8629-48cf-95b1-d837c4856c9c - Building Event JSON Completed.","logger_name":"io.mosip.credentialstore.service.impl.CredentialStoreServiceImpl","thread_name":"http-nio-8095-exec-7","level":"INFO","level_value":20000,"appName":"credential-service,id-repository,application","traceId":"41d495c5fe728d45","spanExportable":"false","req.requestURI":"/v1/credentialservice/issue","X-Span-Export":"false","req.method":"POST","req.userAgent":"Apache-HttpClient/4.5.6 (Java/11.0.16)","spanId":"41d495c5fe728d45","X-B3-SpanId":"41d495c5fe728d45","X-B3-TraceId":"41d495c5fe728d45","req.remoteHost":"127.0.0.6","req.requestURL":"http://credential.idrepo/v1/credentialservice/issue"}
{"@timestamp":"2025-09-26T06:56:29.631Z","@version":"1","message":"published topic mpartner-default-auth/CREDENTIAL_ISSUED update at hub","logger_name":"io.mosip.kernel.websub.api.client.PublisherClientImpl","thread_name":"http-nio-8095-exec-7","level":"INFO","level_value":20000,"appName":"credential-service,id-repository,application","traceId":"41d495c5fe728d45","spanExportable":"false","req.requestURI":"/v1/credentialservice/issue","X-Span-Export":"false","req.method":"POST","req.userAgent":"Apache-HttpClient/4.5.6 (Java/11.0.16)","spanId":"41d495c5fe728d45","X-B3-SpanId":"41d495c5fe728d45","X-B3-TraceId":"41d495c5fe728d45","req.remoteHost":"127.0.0.6","req.requestURL":"http://credential.idrepo/v1/credentialservice/issue"}
{"@timestamp":"2025-09-26T06:56:29.631Z","@version":"1","message":"service-account-mosip-crereq-client - REQUEST_ID - 18f0144c-8629-48cf-95b1-d837c4856c9c - Publish the update successfully","logger_name":"io.mosip.credentialstore.util.WebSubUtil","thread_name":"http-nio-8095-exec-7","level":"INFO","level_value":20000,"appName":"credential-service,id-repository,application","traceId":"41d495c5fe728d45","spanExportable":"false","req.requestURI":"/v1/credentialservice/issue","X-Span-Export":"false","req.method":"POST","req.userAgent":"Apache-HttpClient/4.5.6 (Java/11.0.16)","spanId":"41d495c5fe728d45","X-B3-SpanId":"41d495c5fe728d45","X-B3-TraceId":"41d495c5fe728d45","req.remoteHost":"127.0.0.6","req.requestURL":"http://credential.idrepo/v1/credentialservice/issue"}
{"@timestamp":"2025-09-26T06:56:29.663Z","level":"ACCESS","level_value":70000,"traceId":"41d495c5fe728d45","statusCode":200,"req.requestURI":"/v1/credentialservice/issue","bytesSent":2646,"timeTaken":3.080,"appName":"credential-service,id-repository,application","req.userAgent":"Apache-HttpClient/4.5.6 (Java/11.0.16)","req.xForwardedFor":"-","req.referer":"-","req.method":"POST","req.remoteHost":"127.0.0.6"}

I can see the topic mpartner-default-auth/CREDENTIAL_ISSUED being published from credential service.

4

Hi @Varaniya1,

Could you please connect me with devops team?

Thanks,
Durgesh

5

Hi Durgesh, please check in Kafka UI if the subscriber is empty. If the subscriber is inactive or unstable, events will not be delivered.

6

Hi @dhanendra06 ,

Thank you for the reply.

I can see that there are no consumers for the topic.

image
image1897×648 40.1 KB

I restarted the ida pods but in the logs of ida-internal I can see

Subscribed to topic: mpartner-default-auth/CREDENTIAL_ISSUED

But still the consumers list for the topic is empty on Kafka-UI

Here are the logs of ida-internal:

2025-10-08 04:32:55.180  INFO [id-authentication,id-authentication-internal,,,] 64 --- [ask-scheduler-6] i.m.a.c.s.w.i.IdChangeEventsInitializer  : sessionId - doSubscribe - IdChangeEventsInitializer - Initializing Internal Auth subscribptions..
{"@timestamp":"2025-10-08T04:32:55.209Z","@version":"1","message":"subscribing for topic mpartner-default-auth/CREDENTIAL_ISSUED at hub","logger_name":"io.mosip.kernel.websub.api.client.SubscriberClientImpl","thread_name":"task-scheduler-6","level":"INFO","level_value":20000,"appName":"id-authentication,id-authentication-internal"}
2025-10-08 04:32:55.209  INFO [id-authentication,id-authentication-internal,,,] 64 --- [ask-scheduler-6] i.m.k.w.api.client.SubscriberClientImpl  : subscribing for topic mpartner-default-auth/CREDENTIAL_ISSUED at hub
{"@timestamp":"2025-10-08T04:32:55.210Z","@version":"1","message":"sessionId - subscribeForCredentialIssueanceEvents -  - Subscribed to topic: mpartner-default-auth/CREDENTIAL_ISSUED","logger_name":"io.mosip.authentication.common.service.websub.impl.IdChangeEventsInitializer","thread_name":"task-scheduler-6","level":"INFO","level_value":20000,"appName":"id-authentication,id-authentication-internal"}
2025-10-08 04:32:55.210  INFO [id-authentication,id-authentication-internal,,,] 64 --- [ask-scheduler-6] i.m.a.c.s.w.i.IdChangeEventsInitializer  : sessionId - subscribeForCredentialIssueanceEvents -  - Subscribed to topic: mpartner-default-auth/CREDENTIAL_ISSUED
7

Hi @dhanendra06 ,

I even restarted websub, but still the issue persists, also I can see these consumers having state Empty.

image
image1870×859 113 KB

8

Hi @dhanendra06 ,

Could you please check this?

9

Hi @Durgesh, if you need to restart the websub services, please follow this sequence:

  1. consolidator-websub-service

  2. websub-service

Once the consolidator pod is up and running, then restart the websub service, followed by the dependent services in sequential order then check the mpartner-default-auth/CREDENTIAL_ISSUED consumer it must be stable after restarting all the services

10

Hi @dhanendra06 ,

Thank you for the response.

I restarted the services as per the sequence suggested by you:

Consolidator, Websub-service followed by Idrepo and IDA.

But still I cannot see any consumer called mpartner-default-auth/CREDENTIAL_ISSUED.

These are all the consumer groups for your reference:

image
image1876×875 98.5 KB

image
image1885×820 72.9 KB

There’s a topic mpartner_default_auth_CREDENTIAL_ISSUED but it has No active consumer groups.

image
image1889×585 31.6 KB

These are subscription logs for your reference:

Websub-service logs:

time = 2025-10-13T12:35:24.926Z level = INFO module = mosip/kafkaHub message = "Subscription request received" payload = {"hub":"http://0.0.0.0:9191/hub","hubMode":"subscribe","hubCallback":"http://ida-internal.ida/idauthentication/v1/internal/callback/idchange/credential_issued/mpartner-default-auth","hubTopic":"mpartner-default-auth/CREDENTIAL_ISSUED","hubLeaseSeconds":"864000","hubSecret":"doqeMxfDwC5njh/bGKk8mYHyMQG39V4KyKZPfy8IXQg=","hub.mode":"subscribe"}
127.0.0.6 - - [13/Oct/2025:12:35:24 +0000] "POST /hub/ HTTP/1.1" 202 0 "-" "Apache-HttpClient/4.5.6 (Java/11.0.16)"
time = 2025-10-13T12:35:24.928Z level = INFO module = mosip/kafkaHub message = "Validation done before sending intent verification" payload = {"hub":"http://0.0.0.0:9191/hub","hubMode":"subscribe","hubCallback":"http://ida-internal.ida/idauthentication/v1/internal/callback/idchange/credential_issued/mpartner-default-auth","hubTopic":"mpartner-default-auth/CREDENTIAL_ISSUED","hubLeaseSeconds":"864000","hubSecret":"doqeMxfDwC5njh/bGKk8mYHyMQG39V4KyKZPfy8IXQg=","hub.mode":"subscribe"}
time = 2025-10-13T12:35:24.970Z level = INFO module = mosip/kafkaHub message = "Subscription Intent verfication done and stored to kafka" payload = {"verificationSuccess":true,"hub":"http://0.0.0.0:9191/hub","hubMode":"subscribe","hubCallback":"http://ida-internal.ida/idauthentication/v1/internal/callback/idchange/credential_issued/mpartner-default-auth","hubTopic":"mpartner-default-auth/CREDENTIAL_ISSUED","hubLeaseSeconds":"864000","hubSecret":"cipher{TGaYf2NtR/qKphowZGx5sxNbaQXjV6UgJ/phiGIlCJSwbtwW7gR8Kb8q5x69i0t+}","consumerGroup":"mpartner_default_auth_CREDENTIAL_ISSUED___http___ida_internal_ida_idauthentication_v1_internal_callback_idchange_credential_issued_mpartner_default_auth___23533735_125648717d"}

Consolidator logs:

time = 2025-10-13T12:06:11.445Z level = INFO module = mosip/consolidatorService message = "websub event received in consolidator" payload = "{\"verificationSuccess\":true, \"hub\":\"http://0.0.0.0:9191/hub\", \"hubMode\":\"subscribe\", \"hubCallback\":\"http://ida-internal.ida/idauthentication/v1/internal/callback/idchange/credential_issued/mpartner-default-auth\", \"hubTopic\":\"mpartner-default-auth/CREDENTIAL_ISSUED\", \"hubLeaseSeconds\":\"864000\", \"hubSecret\":\"cipher{TzyKUkS219RiHVYZd56DiH8yinyXSmjarDJzGODRXxkY8nLaDLePa0tUiQxqdVhj}\", \"consumerGroup\":\"mpartner_default_auth_CREDENTIAL_ISSUED___http___ida_internal_ida_idauthentication_v1_internal_callback_idchange_credential_issued_mpartner_default_auth___23514258_089065336d\"}"
11

Hi @dhanendra06 ,

Could you please help me here?