DNS discovery mode not working any more after OpenShift Update from 4.7.9 to 4.7.11 #323
Description
We have a StatefulSet with pods running Java software that embeds OrientDB 3.1 in a cluster.
We configured hazelcast to use com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy with DNS discovery of the hazelcast service in OpenShift.
Everything worked fine in OpenShift 4.7.9 but after updating to OpenShift 4.7.11 the cluster nodes could not find each other any more. The logged error messages were all like DNS lookup for serviceDns 'service-name' failed: name not found.
After digging into the code, we could narrow the issue down to the method used for DNS resolving in class com.hazelcast.kubernetes.DnsEndpointResolver. We extracted this method (which uses JNDI and com.sun.jndi.dns.DnsContextFactory) into a sample program and got the Exception
Exception in thread "main" javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name 'service-name'
at jdk.naming.dns/com.sun.jndi.dns.DnsClient.checkResponseCode(Unknown Source)
at jdk.naming.dns/com.sun.jndi.dns.DnsClient.isMatchResponse(Unknown Source)
at jdk.naming.dns/com.sun.jndi.dns.DnsClient.doUdpQuery(Unknown Source)
at jdk.naming.dns/com.sun.jndi.dns.DnsClient.query(Unknown Source)
at jdk.naming.dns/com.sun.jndi.dns.Resolver.query(Unknown Source)
at jdk.naming.dns/com.sun.jndi.dns.DnsContext.c_getAttributes(Unknown Source)
at java.naming/com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(Unknown Source)
at java.naming/com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(Unknown Source)
at java.naming/com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(Unknown Source)
at java.naming/javax.naming.directory.InitialDirContext.getAttributes(Unknown Source)
at DnsTest.lookup(DnsTest.java:43)
at DnsTest.main(DnsTest.java:30)
whereas resolving the service DNS simply by calling
InetAddress.getAllByName(serviceDns);
always worked and returned the IP addresses of all pods.
Is there a reason, why DNS resolving is done with JNDI and com.sun.jndi.dns.DnsContextFactory and not with InetAddress.getAllByName()? or could this be changed?
The JRE we are using is
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)