[GPU][TESTS] Fix race condition in GpuCacheDirWithDotsParamTest#35698
Open
zhanmyz wants to merge 1 commit intoopenvinotoolkit:masterfrom
Open
[GPU][TESTS] Fix race condition in GpuCacheDirWithDotsParamTest#35698zhanmyz wants to merge 1 commit intoopenvinotoolkit:masterfrom
zhanmyz wants to merge 1 commit intoopenvinotoolkit:masterfrom
Conversation
### Details: - Replace deterministic cacheDir path with generateTestFilePrefix() to ensure each parallel process uses a unique cache directory. - The original hash-only prefix was identical across processes, causing removeDir() to race with concurrent write_cache_entry(). ### Tickets: - *CVS-182616* Signed-off-by: zhanmyz <yazhan.ma@intel.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes an intermittent CI failure in the Intel GPU functional test suite by eliminating a cross-process cache directory collision in GpuCacheDirWithDotsParamTest during parallel execution.
Changes:
- Replace a deterministic cache directory name (hash of test name) with
ov::test::utils::generateTestFilePrefix()to ensure per-invocation uniqueness. - Prevent concurrent gtest-parallel workers from deleting each other’s cache directories/files during
SetUp()/TearDown()cleanup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details:
GpuCacheDirWithDotsParamTestthat causes intermittent[GPU] Failed to write 4 bytes to stream! Wrote 0in CI parallel execution.Description of the issue (symptom, root-cause, how it was resolved)
Symptom:
CacheDirDotVariants/GpuCacheDirWithDotsParamTest.smoke_PopulateAndReuseCache/1fails intermittently in CI with:Root Cause:
cacheDirfromstd::hash<std::string>{}(test_name) + GetParam(), which is fully deterministic — every process resolves to the same on-disk path (7815f5b1e52eaecf/test_encoder/test_encoder.encrypted/).removeDir(cacheDir)inSetUp()/TearDown()deletes the.blobfile while another process is mid-write viawrite_cache_entry()→export_model()→sputn().sputn()to return 0.Resolution:
ov::test::utils::generateTestFilePrefix(), which incorporates thread ID and timestamp to guarantee a unique cache directory per process invocation.The code and line that caused this issue (if it is not changed directly)
openvino/src/plugins/intel_gpu/tests/functional/subgraph_tests/serialize.cpp
Lines 75 to 94 in ea2a3c9
Reproduction step and snapshot (if applicable)
Checklist
Tickets: