gen/c: resolve each file at most once in the native-input cache scan - #28940
Open
quaesitor-scientiam wants to merge 1 commit into
Open
quaesitor-scientiam wants to merge 1 commit into
quaesitor-scientiam wants to merge 1 commit into
Conversation
cache_external_input_snapshot_with_resolved_flags checks, for every file node it visits, whether the file is a program file under its written or its resolved path. It called os.real_path for each visit that missed the written path, so the same files were resolved again and again. On Windows each call opens the file and queries its final path (roughly 120-170 us). Reuse the per-path memo that FlatGen.file_is_cache_program_file already used, through a shared cache_program_file_matches helper. A serial Linux cmd/v build now makes 1988 os.real_path calls instead of 2278. Co-Authored-By: WOZCODE <contact@withwoz.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cache_external_input_snapshot_with_resolved_flagsdecides, for every file node it visits, whether the file is a program file under its written or its resolved path. It calledos.real_pathon each visit that missed the written path, with no memo, so the same files were resolved repeatedly.This PR reuses the per-path memo that
FlatGen.file_is_cache_program_filealready had, through one shared helper,cache_program_file_matches.Why
On Windows,
os.real_pathopens the file, queries its final path and closes it again, which costs roughly 120-170 us per call here (Ryzen 9 5900X, NTFS). A serial-profilerun of acmd/vbuild (on the head of #28925, whose changes do not touch these paths) spent 358 ms inos.real_pathon Windows against 22 ms on Linux, for 2614 calls on only 438 distinct paths. This scan was the largest single source of repeats.Measured
os.real_pathcalls in a serialcmd/vbuild (-d v3_no_parallel -gcompiler,-no-parallel -no-memory-limit, counted with a gdb breakpoint, Linux/WSL2):9bb5b5b31dIn a normal parallel
cmd/vbuild this scan runs on the native-inputs thread next to the checker, so the wall-clock gain there is small. Builds that run it on the main thread save the full time.Tests
vlib/v/gen/c/cache_program_file_test.v: a program file is still matched under a spelling that only matches after resolution, answers are memoized per written path, and a memoized answer is reused without resolving again. The memo assertions fail when the memo lookup is disabled.vlib/v/gen/c/andvlib/v/driver/on Windows: the only failures (cache_prune_test.v,c_compiler_flags_test.v,implicit_output_test.v) fail the same way on master../v -silent teston everyvlib/vtest file exceptvlib/v/compiler_tests/andvlib/v/types/checker_ownership_alias_test.v(V3: checker_ownership_alias_test.v never finishes compiling (-d ownership compiler spins at 100% CPU) #28923), Linux/WSL2: 2377 passed, 45 failed, 24 skipped of 2446. The same 45 files fail on master9bb5b5b31d.Not in this PR
The remaining repeats come from about ten other functions that each resolve the same source files once for their own purposes (import resolution, the macOS fallback report, the checker's shadow-file resolver, cgen). Removing those needs one shared resolution per build, which touches the driver, checker and cgen; that is a separate change.
🧙 Built with WOZCODE