Skip to content

fix(cli): resolve subpackage register paths before deduplication#2428

Open
Anai-Guo wants to merge 2 commits into
huggingface:mainfrom
Anai-Guo:fix/cli-symlink-dedupe
Open

fix(cli): resolve subpackage register paths before deduplication#2428
Anai-Guo wants to merge 2 commits into
huggingface:mainfrom
Anai-Guo:fix/cli-symlink-dedupe

Conversation

@Anai-Guo

Copy link
Copy Markdown

Problem

Closes #2417.

On RHEL / CentOS-derived distributions lib64 is a symlink to lib, so both site-packages directories end up in commands_register_spec.submodule_search_locations:

/home/user/.venv/lib64/python3.12/site-packages
/home/user/.venv/lib/python3.12/site-packages

register_optimum_cli_subcommands deduplicates that list with a plain set(...), which compares the raw path strings. Since the strings differ, the same register module is imported twice — once via lib, once via lib64. The duplicate REGISTER_COMMANDS entries then try to register identical subparsers and any optimum-cli invocation that pulls in a subpackage (e.g. optimum-cli export onnx --help) blows up with:

argparse.ArgumentError: argument {onnx}: conflicting subparser: onnx

Ubuntu / Debian only ship lib, so the bug never surfaced there. The reporter narrowed it down on CentOS Stream 10; it also reproduces on Fedora, RHEL 8/9 and any virtualenv created on those distros.

Fix

Resolve each candidate path with Path.resolve() before building the deduplication set. Symlinked siblings collapse to one canonical absolute path, so the registration loop runs exactly once per real directory.

register_paths = {
    Path(register_path).resolve()
    for register_path in commands_register_spec.submodule_search_locations
}
for register_path in register_paths:
    ...

The rest of the loop is unchanged — register_path was already being wrapped in Path(...) on the first line of the body, so the resolved Path object is a drop-in replacement.

Why this is safe

  • No behavior change on Ubuntu / Debian / Windows / macOSsubmodule_search_locations contains a single entry, Path.resolve() returns it unchanged.
  • No new dependencypathlib.Path is already imported at the top of the module.
  • No double-iteration — the comprehension materializes the set before the loop begins, matching the previous semantics.
  • commands_register_spec.submodule_search_locations is None short-circuit is preserved (lines above the change).

Reproduction (matches the issue)

# CentOS Stream 10 / Fedora / RHEL 8+
python3 -m venv .venv && source .venv/bin/activate
pip install optimum optimum-onnx
python -c "import sys; [print(p) for p in sys.path if 'site-packages' in p]"
# -> shows both lib and lib64 paths
optimum-cli export onnx --help
# Before: argparse.ArgumentError: conflicting subparser: onnx
# After:  prints the help text

🤖 Generated with Claude Code

Tai An added 2 commits April 25, 2026 00:16
On RHEL/CentOS-derived distros lib64 is a symlink to lib, so both paths appear in submodule_search_locations. The previous set() compared raw strings and treated the symlink siblings as distinct entries, causing each register module to be imported twice and raising 'argparse.ArgumentError: conflicting subparser' at optimum-cli startup.

Resolving each path with Path.resolve() before building the set lets symlinked locations collapse to a single canonical entry, so the registration loop runs once per real directory.

Fixes huggingface#2417
Covers the fix in this PR: a directory and a symlink resolving to the same
location in optimum.commands.register's submodule_search_locations must not
import the register module twice (which would raise argparse conflicting
subparser at CLI startup, as seen on RHEL/CentOS lib64 -> lib hosts).
@Anai-Guo

Anai-Guo commented Jun 7, 2026

Copy link
Copy Markdown
Author

Added a hermetic regression test (tests/cli/test_cli.py::test_symlinked_register_paths_are_deduplicated) that reproduces the double-import: it stubs submodule_search_locations with a directory plus a symlink resolving to the same path and asserts the register module is imported exactly once. It fails on the previous set(...) dedup and passes with the Path(...).resolve() fix. No production code changed.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cannot run sub-commands on RHL because optimum mishandles the lib64 symlink to lib

1 participant