Kafka issue in sandbox default installation

hello, may I know what is the issue on below logs, I’m trying to build sandbox from the scratch and got this error in kafka deployment

[2023-06-08 20:38:44,344] INFO [MetadataLoader 0] initializeNewPublishers: the loader is still catching up because we still don’t know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)

[2023-06-08 20:38:44,358] WARN [RaftManager nodeId=0] Error connecting to node kafka-2.kafka-headless.default.svc.cluster.local:9093 (id: 2 rack: null) (org.apache.kafka.clients.NetworkClient)

java.net.UnknownHostException: kafka-2.kafka-headless.default.svc.cluster.local

at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)

at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)

at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)

at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)

at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)

at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)

at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:510)

at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)

at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)

at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:990)

at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)

at kafka.common.InterBrokerSendThread.$anonfun$sendRequests$1(InterBrokerSendThread.scala:103)

at kafka.common.InterBrokerSendThread.$anonfun$sendRequests$1$adapted(InterBrokerSendThread.scala:99)

at scala.collection.Iterator.foreach(Iterator.scala:943)

at scala.collection.Iterator.foreach$(Iterator.scala:943)

at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)

at scala.collection.IterableLike.foreach(IterableLike.scala:74)

at scala.collection.IterableLike.foreach$(IterableLike.scala:73)

at scala.collection.AbstractIterable.foreach(Iterable.scala:56)

at kafka.common.InterBrokerSendThread.sendRequests(InterBrokerSendThread.scala:99)

at kafka.common.InterBrokerSendThread.pollOnce(InterBrokerSendThread.scala:73)

at kafka.common.InterBrokerSendThread.doWork(InterBrokerSendThread.scala:94)

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

[2023-06-08 20:38:44,436] WARN [RaftManager nodeId=0] Graceful shutdown timed out after 5000ms (org.apache.kafka.raft.KafkaRaftClient)

[2023-06-08 20:38:44,437] ERROR [kafka-raft-io-thread]: Graceful shutdown of RaftClient failed (kafka.raft.KafkaRaftManager$RaftIoThread)

java.util.concurrent.TimeoutException: Timeout expired before graceful shutdown completed

at org.apache.kafka.raft.KafkaRaftClient$GracefulShutdown.failWithTimeout(KafkaRaftClient.java:2416)

at org.apache.kafka.raft.KafkaRaftClient.maybeCompleteShutdown(KafkaRaftClient.java:2163)

at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:2230)

at kafka.raft.KafkaRaftManager$RaftIoThread.doWork(RaftManager.scala:65)

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

[2023-06-08 20:38:44,437] INFO [kafka-raft-io-thread]: Stopped (kafka.raft.KafkaRaftManager$RaftIoThread)

[2023-06-08 20:38:44,437] INFO [kafka-raft-io-thread]: Shutdown completed (kafka.raft.KafkaRaftManager$RaftIoThread)

[2023-06-08 20:38:44,444] INFO [kafka-raft-outbound-request-thread]: Shutting down (kafka.raft.RaftSendThread)

[2023-06-08 20:38:44,444] INFO [kafka-raft-outbound-request-thread]: Stopped (kafka.raft.RaftSendThread)

[2023-06-08 20:38:44,444] INFO [kafka-raft-outbound-request-thread]: Shutdown completed (kafka.raft.RaftSendThread)

[2023-06-08 20:38:44,444] INFO [MetadataLoader 0] initializeNewPublishers: the loader is still catching up because we still don’t know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)

[2023-06-08 20:38:44,454] INFO [SocketServer listenerType=CONTROLLER, nodeId=0] Stopping socket server request processors (kafka.network.SocketServer)

[2023-06-08 20:38:44,458] INFO [SocketServer listenerType=CONTROLLER, nodeId=0] Stopped socket server request processors (kafka.network.SocketServer)

[2023-06-08 20:38:44,459] INFO [Controller 0] QuorumController#beginShutdown: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)

[2023-06-08 20:38:44,459] INFO [SocketServer listenerType=CONTROLLER, nodeId=0] Shutting down socket server (kafka.network.SocketServer)

[2023-06-08 20:38:44,465] INFO [SocketServer listenerType=CONTROLLER, nodeId=0] Shutdown completed (kafka.network.SocketServer)

[2023-06-08 20:38:44,465] INFO [data-plane Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)

[2023-06-08 20:38:44,466] INFO [data-plane Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)

[2023-06-08 20:38:44,466] INFO [ExpirationReaper-0-AlterAcls]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2023-06-08 20:38:44,466] INFO [ExpirationReaper-0-AlterAcls]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2023-06-08 20:38:44,467] INFO [ExpirationReaper-0-AlterAcls]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

[2023-06-08 20:38:44,467] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,467] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,467] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,467] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,467] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,467] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,467] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,467] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,467] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,467] INFO [ThrottledChannelReaper-ControllerMutation]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,468] INFO [ThrottledChannelReaper-ControllerMutation]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,468] INFO [ThrottledChannelReaper-ControllerMutation]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)

[2023-06-08 20:38:44,468] INFO [Controller 0] closed event queue. (org.apache.kafka.queue.KafkaEventQueue)

[2023-06-08 20:38:44,469] INFO [SharedServer id=0] Stopping SharedServer (kafka.server.SharedServer)

[2023-06-08 20:38:44,469] INFO [MetadataLoader 0] beginShutdown: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)

[2023-06-08 20:38:44,470] INFO [SnapshotGenerator 0] close: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)

[2023-06-08 20:38:44,470] INFO [SnapshotGenerator 0] closed event queue. (org.apache.kafka.queue.KafkaEventQueue)

[2023-06-08 20:38:44,470] INFO [MetadataLoader 0] closed event queue. (org.apache.kafka.queue.KafkaEventQueue)

[2023-06-08 20:38:44,470] INFO [SnapshotGenerator 0] closed event queue. (org.apache.kafka.queue.KafkaEventQueue)

[2023-06-08 20:38:44,472] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)

[2023-06-08 20:38:44,472] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)

[2023-06-08 20:38:44,472] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)

[2023-06-08 20:38:44,475] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)

[2023-06-08 20:38:44,476] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)

It looks like a DNS issue in your cluster, can you please restart the coredns pods in the kube-system namespace and then try?

kubectl -n kube-system rollout restart deploy coredns