Skip to content

gen/c: resolve each file at most once in the native-input cache scan - #28940

Open
quaesitor-scientiam wants to merge 1 commit into
vlang:masterfrom
quaesitor-scientiam:perf-v3-fewer-real-path-calls
Open

quaesitor-scientiam wants to merge 1 commit into
vlang:masterfrom
quaesitor-scientiam:perf-v3-fewer-real-path-calls

Conversation

@quaesitor-scientiam

Copy link
Copy Markdown
Contributor

cache_external_input_snapshot_with_resolved_flags decides, for every file node it visits, whether the file is a program file under its written or its resolved path. It called os.real_path on each visit that missed the written path, with no memo, so the same files were resolved repeatedly.

This PR reuses the per-path memo that FlatGen.file_is_cache_program_file already had, through one shared helper, cache_program_file_matches.

Why

On Windows, os.real_path opens the file, queries its final path and closes it again, which costs roughly 120-170 us per call here (Ryzen 9 5900X, NTFS). A serial -profile run of a cmd/v build (on the head of #28925, whose changes do not touch these paths) spent 358 ms in os.real_path on Windows against 22 ms on Linux, for 2614 calls on only 438 distinct paths. This scan was the largest single source of repeats.

Measured

os.real_path calls in a serial cmd/v build (-d v3_no_parallel -g compiler, -no-parallel -no-memory-limit, counted with a gdb breakpoint, Linux/WSL2):

calls
master 9bb5b5b31d 2278
this PR 1988

In a normal parallel cmd/v build this scan runs on the native-inputs thread next to the checker, so the wall-clock gain there is small. Builds that run it on the main thread save the full time.

Tests

  • New vlib/v/gen/c/cache_program_file_test.v: a program file is still matched under a spelling that only matches after resolution, answers are memoized per written path, and a memoized answer is reused without resolving again. The memo assertions fail when the memo lookup is disabled.
  • vlib/v/gen/c/ and vlib/v/driver/ on Windows: the only failures (cache_prune_test.v, c_compiler_flags_test.v, implicit_output_test.v) fail the same way on master.
  • ./v -silent test on every vlib/v test file except vlib/v/compiler_tests/ and vlib/v/types/checker_ownership_alias_test.v (V3: checker_ownership_alias_test.v never finishes compiling (-d ownership compiler spins at 100% CPU) #28923), Linux/WSL2: 2377 passed, 45 failed, 24 skipped of 2446. The same 45 files fail on master 9bb5b5b31d.

Not in this PR

The remaining repeats come from about ten other functions that each resolve the same source files once for their own purposes (import resolution, the macOS fallback report, the checker's shadow-file resolver, cgen). Removing those needs one shared resolution per build, which touches the driver, checker and cgen; that is a separate change.

🧙 Built with WOZCODE

cache_external_input_snapshot_with_resolved_flags checks, for every file
node it visits, whether the file is a program file under its written or its
resolved path. It called os.real_path for each visit that missed the written
path, so the same files were resolved again and again. On Windows each call
opens the file and queries its final path (roughly 120-170 us).

Reuse the per-path memo that FlatGen.file_is_cache_program_file already
used, through a shared cache_program_file_matches helper.

A serial Linux cmd/v build now makes 1988 os.real_path calls instead of
2278.

Co-Authored-By: WOZCODE <contact@withwoz.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants