HIVE-28603 Fixing Multiple Flaky tests in module serde which are due to non determinism

[nikunjagarwal321]

What changes were proposed in this pull request?

Sorting the fields in function getDeclaredNonStaticFields() in ObjectInspectorUtils.java class in order to make the function deterministic. The below flaky tests will be fixed by this.

Why are the changes needed?

Multiple flaky tests were detected in serde module while trying to run the tests using the nondex tool. NonDex is a tool for detecting and debugging wrong assumptions on under-determined Java APIs.

Steps to reproduce flakiness using nondex -

mvn install -pl serde -DskipTests

mvn -pl serde test -Dtest=org.apache.hadoop.hive.serde2.objectinspector.TestProtocolBuffersObjectInspectors#testProtocolBuffersObjectInspectors

mvn -pl serde edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=org.apache.hadoop.hive.serde2.objectinspector.TestProtocolBuffersObjectInspectors#testProtocolBuffersObjectInspectors

ERROR logs:

[ERROR] Failures: 
[ERROR]   TestProtocolBuffersObjectInspectors.testProtocolBuffersObjectInspectors:71 expected:<test> but was:<[one, two]>

Reason for Failure

The method getDeclaredNonStaticFields(Class<?> c) in class ObjectInspectorUtils.java uses class.getDeclaredFields() method which is nondeterministic as per java docs.

Field[] f = c.getDeclaredFields();

As per the test cases, it is assumed that the return order of the fields will always be the same, which may not be the case.

All the below test cases are impacted because of the same reason :

• org.apache.hadoop.hive.serde2.objectinspector.TestProtocolBuffersObjectInspectors.testProtocolBuffersObjectInspectors
• org.apache.hadoop.hive.serde2.objectinspector.TestThriftObjectInspectors.testThriftObjectInspectors
• org.apache.hadoop.hive.serde2.objectinspector.TestReflectionObjectInspectors.testReflectionObjectInspectors
• org.apache.hadoop.hive.serde2.columnar.TestLazyBinaryColumnarSerDe.testLazyBinaryColumnarSerDeWithEmptyBinary
• org.apache.hadoop.hive.serde2.columnar.TestLazyBinaryColumnarSerDe.testSerDeInnerNulls
• org.apache.hadoop.hive.serde2.lazybinary.TestLazyBinarySerDe.testLazyBinarySerDe
• org.apache.hadoop.hive.serde2.TestStatsSerde.testLazyBinarySerDe
• org.apache.hadoop.hive.serde2.lazy.TestLazySimpleSerDe#testSerDeParameters
• org.apache.hadoop.hive.serde2.binarysortable.TestBinarySortableSerDe#testBinarySortableSerDe

Does this PR introduce any user-facing change?

No

Is the change a dependency upgrade?

No

How was this patch tested?

The changes were tested by running the unit tests locally. The code is already covered in the above test cases and is actually fixing the flaky tests.

Please let me know if you have any questions or would like to discuss this further for any clarificaitions.

[difin]

LGTM, +1.

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[nikunjagarwal321]

Thanks @difin !

[kokila-19]

@nikunjagarwal321 Hi, It appears that this commit does not work with JDK 17 and has resulted in multiple test failures. Could you please check this issue?

You can check out the PR here: #5404

[nikunjagarwal321]

@nikunjagarwal321 Hi, It appears that this commit does not work with JDK 17 and has resulted in multiple test failures. Could you please check this issue?

You can check out the PR here: #5404

Hey,
Thanks for bringing this to my attention. From my understanding, the current master branch is still on Java 8, and the changes I made were based on that version. The modifications I implemented focused on resolving flaky tests in the existing main code, and all related unit tests passed successfully at the time.

I see that the transition to Java 17 is a new initiative you're working on. If the previous changes are not fully compatible with Java 17, we may need to explore alternative approaches. I’d be happy to provide more context on my changes or collaborate on finding a resolution.

Let’s continue the discussion in the new PR where the Java 17 upgrade is being worked on, as this change seems primarily related to the version upgrade and keeping all discussions in one place would be helpful.