[Bug] Nondeterministic Assertions in KafkaAbstractSinkTest #24876
Description
Search before reporting
- I searched in the issues and found nothing similar.
Read release policy
- I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
User environment
Broker version: 4.2.0-SNAPSHOT
Broker Operating system and hardware type: Linux x86_64 GNU/Linux
Broker Java version: 17.0.16
Issue Description
The original test performed six assertions using the expectThrows method but was based on two incorrect assumptions. First, it implicitly assumed that field validation within IOConfigUtils.java would occur in a consistent order. However, IOConfigUtils iterates over configuration fields using the loop (Field field : getAllFields(clazz)), and getAllFields() calls Class.getDeclaredFields, which returns an array of fields that are not sorted and not in any particular order. Second, the test initialized multiple required fields as null simultaneously, assuming the validation would fail in a specific order. In reality, the nondeterministic field iteration order can cause different missing fields to be detected first, leading to inconsistent exception messages and nondeterministic test failures due to different environments producing the contents in different orders despite the logical contents being the same.
- pulsar-io/kafka/src/test/java/org/apache/pulsar/io/kafka/sink/KafkaAbstractSinkTest.java
I discovered the problem with the NonDex tool. NonDex systematically detects incorrect tests that rely on non-deterministic behaviors in Java APIs—like assuming order of name/value pairs in json files — by exploring all specification-allowed outcomes. It does this by instrumenting undetermined APIs and randomizing the returned order/behavior within what the spec allows. Failures it exposes reliably indicate flawed assumptions in order that were never promised and should be fixed.
Error messages
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.pulsar.io.kafka.sink.KafkaAbstractSinkTest
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 2.947 s <<< FAILURE! -- in org.apache.pulsar.io.kafka.sink.KafkaAbstractSinkTest
[ERROR] org.apache.pulsar.io.kafka.sink.KafkaAbstractSinkTest.testInvalidConfigWillThrownException -- Time elapsed: 0.004 s <<< FAILURE!
java.lang.AssertionError: expected [topic cannot be null] but found [acks cannot be null]
at org.testng.Assert.fail(Assert.java:110)
at org.testng.Assert.failNotEquals(Assert.java:1577)
at org.testng.Assert.assertEqualsImpl(Assert.java:149)
at org.testng.Assert.assertEquals(Assert.java:131)
at org.testng.Assert.assertEquals(Assert.java:655)
at org.testng.Assert.assertEquals(Assert.java:665)
at org.apache.pulsar.io.kafka.sink.KafkaAbstractSinkTest.expectThrows(KafkaAbstractSinkTest.java:68)
at org.apache.pulsar.io.kafka.sink.KafkaAbstractSinkTest.testInvalidConfigWillThrownException(KafkaAbstractSinkTest.java:202)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] org.apache.pulsar.io.kafka.sink.KafkaAbstractSinkTest.testInvalidConfigWillThrownException
[INFO] Run 1: PASS
[ERROR] Run 2: KafkaAbstractSinkTest.testInvalidConfigWillThrownException:202->expectThrows:68 expected [topic cannot be null] but found [acks cannot be null]
[INFO]
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
Reproducing the issue
You can replicate the problem with the NonDex tool command:
mvn -pl pulsar-io/kafka -Dtest=org.apache.pulsar.io.kafka.sink.KafkaAbstractSinkTest#testInvalidConfigWillThrownException -DforkCount=1 -DnondexRuns=1 -DnondexSeed=974622 -DreuseForks=false edu.illinois:nondex-maven-plugin:2.2.1:nondex
Additional information
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!