CTK Signature/Trust Root Validation fails with 403 Forbidden when calling Keymanager (deviceInfo signed JWT)

Developer
1

I’m testing MOSIP SBI compliance using the Compliance Toolkit (CTK). My device responses (deviceInfo + digitalId) are signed JWS (RS256) and decode correctly on jwt.io, but CTK fails during signature / trust root validation.

CTK logs show that it attempts to call Keymanager for trust validation, and that call returns 403 Forbidden.

What I’m trying to do

  • I want to validate Signed deviceInfo responses for an L0 registered device.
  • deviceInfo is returned as a compact JWS: base64url(header).base64url(payload).base64url(signature)
  • Header uses alg=RS256, typ=JWT, and x5c includes the device certificate (leaf first).

Observed behavior

  • CTK JSON/schema validation passes.
  • CTK fails at signature / trust root validation:
    • In SignatureValidator - 403 Forbidden
    • Sometimes I also see ResourceCacheService 401 Unauthorized in logs.
  • CTK also logs Illegal base64 character 2e sometimes, but I believe CTK intentionally tries to base64-decode unsigned payloads in some paths (not sure).

My questions

  1. What are the most common reasons CTK would get 403 Forbidden from Keymanager during trust root validation?
  2. Does CTK require a specific service account / client role configuration in Keycloak so it can call Keymanager?
  3. If a Partner/Device certificate is uploaded correctly, could 403 still happen due to missing authorization scopes/roles between CTK ↔ Keymanager?
  4. Is there a recommended way to test the Keymanager trust-validation endpoint directly (e.g., partnercertmanager /verifyCertificateTrust) to confirm whether the issue is authorization vs certificate chain/trust setup?

Below is the log for reference.
CTK
Safari/537.36",“req.xForwardedFor”:“172.17.0.2,10.42.0.0”,“req.referer”:“https://compliance.sandbox.test.mosip.net/",“req.method”:“POST”,“req.remoteHost”:"10.42.1.11”}
2025-12-09 05:16:55,518 [http-nio-8099-exec-1] INFO [i.m.k.a.d.h.ValidateTokenHelper].getPublicKey.222 : offline verification for environment profile. UserName: test_1
2025-12-09 05:16:55,521 [http-nio-8099-exec-1] INFO [i.m.k.a.d.h.ValidateTokenHelper].buildMosipUser.299 : user (offline verification done): test_1
2025-12-09 05:16:55,533 [http-nio-8099-exec-1] INFO [i.m.k.l.l.i.Slf4jLoggerImpl].info.32 : sessionId - idType - id - JSON is as expected. All mandatory values are available and they all have valid expected values.
2025-12-09 05:16:55,535 [http-nio-8099-exec-1] ERROR [i.m.k.l.l.i.Slf4jLoggerImpl].error.27 : sessionId - idType - id - In SBIValidator - Illegal base64 character 2e
2025-12-09 05:16:55,564 [http-nio-8099-exec-1] ERROR [i.m.k.l.l.i.Slf4jLoggerImpl].error.27 : sessionId - idType - id - In SignatureValidator - 403 Forbidden
{“@timestamp”:“2025-12-09T05:16:55.567Z”,“level”:“ACCESS”,“level_value”:70000,“traceId”:“340977500688a19fc979516d24b6adb1”,“statusCode”:200,“req.requestURI”:“/v1/toolkit/validateResponse”,“bytesSent”:734,“timeTaken”:0.052,“appName”:“compliance-toolkit-service”,“req.userAgent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36”,“req.xForwardedFor”:“172.17.0.2,10.42.0.0”,“req.referer”:“https://compliance.sandbox.test.mosip.net/",“req.method”:“POST”,“req.remoteHost”:"10.42.1.11”}
2025-12-09 05:16:55,582 [http-nio-8099-exec-10] INFO [i.m.k.a.d.h.ValidateTokenHelper].getPublicKey.222 : offline verification for environment profile. UserName: test_1
2025-12-09 05:16:55,584 [http-nio-8099-exec-10] INFO [i.m.k.a.d.h.ValidateTokenHelper].buildMosipUser.299 : user (offline verification done): test_1

keymanager
{“@timestamp”:“2025-12-09T09:39:01.116Z”,“@version”:“1”,“message”:“Exception Root Cause: Access is denied “,“logger_name”:“io.mosip.kernel.core.exception.ExceptionUtils”,“thread_name”:“http-nio-8088-exec-5”,“level”:“ERROR”,“level_value”:40000,“appName”:“kernel-keymanager-service”,“traceId”:“4b296817ca215e15a60b4bfc7fc51216”,“spanExportable”:“false”,“req.requestURI”:”/v1/keymanager/verifyCertificateTrust”,“req.xForwardedFor”:“10.0.2.13,10.42.0.0”,“X-Span-Export”:“false”,“X-B3-ParentSpanId”:“35b4bf73a8e80d76”,“req.method”:“POST”,“parentId”:“35b4bf73a8e80d76”,“req.userAgent”:“Apache-HttpClient/4.5.6 (Java/11.0.16)”,“spanId”:“a038c0b663f7d578”,“X-B3-SpanId”:“a038c0b663f7d578”,“X-B3-TraceId”:“4b296817ca215e15a60b4bfc7fc51216”,“req.remoteHost”:“127.0.0.6”,“req.requestURL”:“http://api-internal.sandbox.test.mosip.net/v1/keymanager/verifyCertificateTrust”}

{“@timestamp”:“2025-12-09T09:39:01.117Z”,“level”:“ACCESS”,“level_value”:70000,“traceId”:“4b296817ca215e15a60b4bfc7fc51216”,“statusCode”:403,“req.requestURI”:“/v1/keymanager/verifyCertificateTrust”,“bytesSent”:165,“timeTaken”:0.011,“appName”:“kernel-keymanager-service”,“req.userAgent”:“Apache-HttpClient/4.5.6 (Java/11.0.16)”,“req.xForwardedFor”:“10.0.2.13,10.42.0.0”,“req.referer”:“-”,“req.method”:“POST”,“req.remoteHost”:“127.0.0.6”}
{“@timestamp”:“2025-12-09T09:39:08.837Z”,“level”:“ACCESS”,“level_value”:70000,“traceId”:“7f57632b40a78b75”,“statusCode”:200,“req.requestURI”:“/v1/keymanager/actuator/health”,“bytesSent”:302,“timeTaken”:0.005,“appName”:“kernel-keymanager-service”,“req.userAgent”:“kube-probe/1.22”,“req.xForwardedFor”:“-”,“req.referer”:“-”,“req.method”:“GET”,“req.remoteHost”:“127.0.0.6”}

help me please

2

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.