multiaddr vs py-libp2p multihash Conflict

[acul71]

Verification: multiaddr vs py-libp2p multihash Conflict

Verification Date: 2025-11-08
Status: ✅ CONFIRMED - Issue is REAL

---

Executive Summary

The dependency conflict between multiaddr 0.1.0 and py-libp2p regarding multihash packages is confirmed to be a real issue. When multiaddr 0.1.0 is installed, it pulls in py-multihash 2.0.1, which conflicts with pymultihash 0.8.2 that py-libp2p requires. Both packages provide a multihash module, causing namespace collision and AttributeError when py-libp2p tries to use APIs that only exist in pymultihash.

---

Dependency Conflict Schema

Scenario 1: Current Working State (multiaddr 0.0.12)

┌────────────────────────────────────────────────────────────┐
│                    py-libp2p                               │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  Requires:                                          │   │
│  │  • multiaddr>=0.0.11                                │   │
│  │  • pymultihash>=0.8.2  ───────────────┐             │   │
│  └───────────────────────────────────────┼─────────────┘   │
│                                          │                 │
│  ┌───────────────────────────────────────▼─────────────┐   │
│  │  Installed:                                         │   │
│  │  • multiaddr 0.0.12  ──────┐                        │   │
│  │  • pymultihash 0.8.2  ─────┼───┐                    │   │
│  └────────────────────────────┼───┼────────────────────┘   │
│                               │   │                        │
│                               │   │                        │
│  ┌────────────────────────────▼───▼────────────────────┐   │
│  │  Python imports: import multihash                   │   │
│  │  ✅ Gets: pymultihash (only one installed)          │   │
│  │  ✅ Provides: Func, digest(), FuncReg, decode()     │   │
│  │  ✅ py-libp2p works correctly                       │   │
│  └─────────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────────┘

multiaddr 0.0.12 dependencies:
┌─────────────────┐
│ multiaddr 0.0.12│
│                 │
│ Does NOT depend │
│ on py-multihash │
└─────────────────┘

Scenario 2: Conflict State (multiaddr 0.1.0)

┌─────────────────────────────────────────────────────────────┐
│                    py-libp2p                                │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  Requires:                                           │  │
│  │  • multiaddr>=0.0.11                                 │  │
│  │  • pymultihash>=0.8.2  ───────────────┐              │  │
│  └───────────────────────────────────────┼──────────────┘  │
│                                          │                 │
│  ┌───────────────────────────────────────▼──────────────┐  │
│  │  Installed:                                          │  │
│  │  • multiaddr 0.1.0   ──────┐                         │  │
│  │  • pymultihash 0.8.2  ─────┼───┐                     │  │
│  │  • py-multihash 2.0.1  ────┼───┼───┐  ⚠️ CONFLICT!   │  │
│  └────────────────────────────┼───┼───┼─────────────────┘  │
│                               │   │   │                    │
│                               │   │   │                    │
│  ┌────────────────────────────▼───▼───▼────────────────┐  │
│  │  Python imports: import multihash                   │  │
│  │  ❌ Gets: py-multihash (imported first)             │  │
│  │  ❌ Missing: Func, digest(), FuncReg                │  │
│  │  ❌ py-libp2p fails with AttributeError             │  │
│  └─────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

multiaddr 0.1.0 dependencies:
┌─────────────────┐
│ multiaddr 0.1.0 │
│                 │
│  ┌──────────────▼──────────────┐
│  │  Depends on:                │
│  │  • py-multihash 2.0.1  ⚠️   │
│  └─────────────────────────────┘
└─────────────────┘

Dependency Tree Comparison

WORKING (multiaddr 0.0.12):              CONFLICT (multiaddr 0.1.0):
───────────────────────────              ───────────────────────────
py-libp2p                                 py-libp2p
├── multiaddr 0.0.12                      ├── multiaddr 0.1.0
│   └── (no multihash dependency)         │   └── py-multihash 2.0.1 ⚠️
└── pymultihash 0.8.2                     └── pymultihash 0.8.2
    └── multihash module ✅                   └── multihash module
                                                └── py-multihash 2.0.1
                                                    └── multihash module ⚠️
                                                    
Result: ✅ Only one multihash          Result: ❌ Two multihash modules
        module, no conflict                      conflict!

Namespace Collision Visualization

┌──────────────────────────────────────────────────────────────┐
│                    Python sys.path                           │
│                                                              │
│  When both packages are installed:                          │
│                                                              │
│  ┌──────────────────────┐    ┌──────────────────────┐      │
│  │  pymultihash 0.8.2   │    │  py-multihash 2.0.1  │      │
│  │                      │    │                      │      │
│  │  Provides module:    │    │  Provides module:    │      │
│  │  ┌────────────────┐  │    │  ┌────────────────┐  │      │
│  │  │  multihash     │  │    │  │  multihash     │  │      │
│  │  │                │  │    │  │                │  │      │
│  │  │  • Func        │  │    │  │  • encode()    │  │      │
│  │  │  • digest()    │  │    │  │  • decode()    │  │      │
│  │  │  • FuncReg     │  │    │  │  • constants   │  │      │
│  │  │  • decode()    │  │    │  │  • Multihash   │  │      │
│  │  └────────────────┘  │    │  └────────────────┘  │      │
│  └──────────────────────┘    └──────────────────────┘      │
│                                                              │
│  Python: import multihash                                   │
│  ──────────────────────────                                 │
│  ❌ Gets FIRST one found in sys.path                        │
│  ❌ Usually: py-multihash (wrong one!)                      │
│  ❌ py-libp2p expects: pymultihash                          │
│  ❌ Result: AttributeError                                  │
└──────────────────────────────────────────────────────────────┘

API Compatibility Matrix

┌─────────────────────┬──────────────────┬──────────────────┐
│      API            │  pymultihash     │  py-multihash    │
│                     │     0.8.2        │     2.0.1        │
├─────────────────────┼──────────────────┼──────────────────┤
│ multihash.Func      │       ✅         │       ❌         │
│ multihash.digest()  │       ✅         │       ❌         │
│ multihash.FuncReg   │       ✅         │       ❌         │
│ multihash.decode()  │       ✅         │       ✅         │
│ multihash.encode()  │       ❌         │       ✅         │
│ multihash.constants │       ❌         │       ✅         │
│ multihash.Multihash │       ❌         │       ✅         │
└─────────────────────┴──────────────────┴──────────────────┘

py-libp2p needs: ✅ Func, ✅ digest(), ✅ FuncReg, ✅ decode()
py-multihash provides: ❌ Func, ❌ digest(), ❌ FuncReg, ✅ decode()

Result: 3 out of 4 APIs missing → INCOMPATIBLE

---

Verification Results

✅ Confirmed Facts

1. multiaddr 0.1.0 exists and requires py-multihash 2.0.1

   • Verified via: pip install --dry-run multiaddr==0.1.0
   • Output confirms: Would install multiaddr-0.1.0 py-multihash-2.0.1
   • multiaddr 0.1.0 is the latest version on PyPI

2. py-libp2p requires pymultihash>=0.8.2 (NOT py-multihash)

   • Current pyproject.toml line 33: "pymultihash>=0.8.2"
   • py-libp2p uses these APIs from pymultihash:
     • multihash.Func.sha2_256 (used in libp2p/peer/id.py:87)
     • multihash.digest() (used in multiple files)
     • multihash.FuncReg (used in libp2p/peer/id.py:35)
     • multihash.decode() (used in libp2p/records/pubkey.py:39)

3. API Incompatibility Between Packages

   • pymultihash 0.8.2 provides:

     • multihash.Func (enum with hash function codes)
     • multihash.digest(data, code) (creates multihash from data)
     • multihash.FuncReg (registry for custom hash functions)
     • multihash.decode() (decodes multihash bytes)

   • py-multihash 2.0.1 provides:

     • multihash.encode(digest, code) (encodes digest as multihash)
     • multihash.decode() (decodes multihash bytes)
     • multihash.constants.HASH_CODES (dictionary of hash codes)
     • multihash.Multihash class
     • Does NOT provide: multihash.Func, multihash.digest(), multihash.FuncReg

4. Namespace Collision

   • Both packages provide a Python module named multihash
   • Python imports the first one found in sys.path
   • If py-multihash is imported first, py-libp2p will fail with:

     AttributeError: module 'multihash' has no attribute 'Func'
     AttributeError: module 'multihash' has no attribute 'digest'
     

5. Current State in py-libp2p

   • pyproject.toml line 28: "multiaddr>=0.0.11" (allows 0.1.0 to be installed)
   • Currently installed in dev environment: multiaddr 0.0.12
   • multiaddr 0.0.12 does NOT require py-multihash (no conflict)
   • Conflict only appears when multiaddr 0.1.0 is installed

📋 Files in py-libp2p Using multihash APIs

The following files use pymultihash-specific APIs that would break with py-multihash:

1. libp2p/peer/id.py

   • Line 35: multihash.FuncReg.register() - registers identity hash function
   • Line 87: multihash.Func.sha2_256 - gets SHA-256 hash function code
   • Line 90: multihash.digest(serialized_key, algo) - creates multihash from key

2. libp2p/kad_dht/routing_table.py

   • Line 50: multihash.digest(peer_id.to_bytes(), "sha2-256").digest - converts peer ID to DHT key

3. libp2p/kad_dht/utils.py

   • Line 104: multihash.digest(binary_data, "sha2-256").digest - creates DHT key
   • Line 160: multihash.digest(peer_id.to_bytes(), "sha2-256").digest - creates peer hash

4. libp2p/security/secio/transport.py

   • Line 211: multihash.digest(data, "sha2-256") - creates multihash for score

5. libp2p/records/pubkey.py

   • Line 39: multihash.decode(keyhash) - decodes multihash (this API exists in both packages)

---

Why It Doesn't Manifest in Development

The conflict does exist but doesn't manifest in py-libp2p's development environment because:

• Developers' virtual environments have multiaddr 0.0.12 installed (NOT 0.1.0)
• multiaddr 0.0.12 does NOT depend on py-multihash
• So only pymultihash is installed, no conflict occurs

The conflict only appears when:

• Installing fresh dependencies in a clean environment
• pip resolves to multiaddr 0.1.0 (the latest version)
• Which pulls in py-multihash 2.0.1 as a dependency

---

Test Results

Current Environment (multiaddr 0.0.12)

$ pip list | grep -E "multiaddr|multihash"
multiaddr                     0.0.12
pymultihash                   0.8.2

$ python3 -c "import multihash; print('Has Func:', hasattr(multihash, 'Func')); print('Has digest:', hasattr(multihash, 'digest'))"
Has Func: True
Has digest: True
✅ Works correctly

What Would Happen with multiaddr 0.1.0

$ pip install --dry-run multiaddr==0.1.0
Would install multiaddr-0.1.0 py-multihash-2.0.1
❌ Would install conflicting package

Verification of py-libp2p APIs

# Tested in current environment (pymultihash installed)
from libp2p.peer.id import ID
from libp2p.crypto.secp256k1 import create_new_key_pair
import multihash

key_pair = create_new_key_pair()
peer_id = ID.from_pubkey(key_pair.public_key)  # ✅ Works

# APIs used:
multihash.Func.sha2_256        # ✅ Available
multihash.digest()             # ✅ Available
multihash.FuncReg              # ✅ Available
multihash.decode()             # ✅ Available

---

Impact Assessment

Severity: HIGH

• Affects: All fresh installations of py-libp2p when multiaddr 0.1.0 is resolved
• Symptoms: AttributeError when importing or using py-libp2p core functionality
• Affected Operations:
  • Creating peer IDs from public keys
  • DHT routing table operations
  • SECIO security transport
  • Any code path that uses ID.from_pubkey()

Affected Scenarios

1. Fresh Docker builds - Will fail if multiaddr 0.1.0 is installed
2. CI/CD pipelines - Will fail on clean environments
3. New developer setups - Will fail when setting up from scratch
4. Production deployments - Will fail if dependencies are updated

---

Recommended Solutions

Option 1: Fix in multiaddr (Recommended - Long-term)

Problem: multiaddr 0.1.0 added py-multihash as a dependency, but this conflicts with pymultihash that py-libp2p needs.

Solution: multiaddr should either:

1. Use pymultihash instead of py-multihash (if multihash functionality is needed)
   • This would align with what py-libp2p already uses
   • Both packages would use the same multihash implementation
2. Remove the py-multihash dependency (if it doesn't actually need multihash functionality)
   • Check if multiaddr actually uses multihash APIs
   • If not, remove the dependency to avoid the conflict
3. Make the dependency optional or conditional
   • Only require py-multihash if actually needed
   • Or provide a way to use either package

Action Items:

• File an issue/PR with the multiaddr maintainers
• Investigate why py-multihash was added as a dependency in 0.1.0
• Determine if multiaddr actually needs multihash functionality
• If yes, switch to pymultihash to align with py-libp2p
• If no, remove the dependency

Repository: https://github.com/multiformats/py-multiaddr

Option 2: Fix in py-libp2p (Not Recommended)

Problem: py-libp2p uses multihash.Func and multihash.digest() APIs that only exist in pymultihash, not in py-multihash.

Solution: py-libp2p could update its code to use py-multihash's API instead:

• Replace multihash.Func.sha2_256 with multihash.constants.HASH_CODES["sha2-256"]
• Replace multihash.digest() with a function using multihash.encode() and hashlib
• Update all code that uses these APIs (5+ files)

Trade-offs:

• ✅ Would work with multiaddr 0.1.0's dependency
• ❌ Requires significant code changes in py-libp2p
• ❌ py-multihash might not have all the features pymultihash provides (e.g., FuncReg)
• ❌ Other projects might depend on py-libp2p using pymultihash

Why Not Recommended:

• pymultihash is already working well for py-libp2p
• Changing would require extensive refactoring
• The conflict is caused by multiaddr adding a new dependency, not by py-libp2p
• pymultihash provides FuncReg which is used for identity hash function registration

Option 3: Pin multiaddr Version in py-libp2p (Temporary Workaround)

Problem: py-libp2p specifies multiaddr>=0.0.11, which allows 0.1.0 to be installed.

Solution: Pin multiaddr to >=0.0.11,<0.1.0 in py-libp2p's pyproject.toml:

dependencies = [
    # ... other dependencies ...
    "multiaddr>=0.0.11,<0.1.0",  # Pin to avoid py-multihash conflict
    # ... other dependencies ...
]

Trade-offs:

• ✅ Quick fix, no code changes needed
• ✅ Prevents the conflict from occurring
• ❌ Blocks multiaddr updates (might miss bug fixes or features)
• ❌ Not a long-term solution
• ❌ Doesn't fix the root cause

Action: This is a temporary workaround, not a proper fix. Should be used only until multiaddr is fixed.

---

Current Workaround (If Needed)

Until the upstream fix is implemented, Docker builds or CI/CD can use this workaround:

RUN pip install --no-cache-dir -e . && \
    pip uninstall -y py-multihash && \
    pip install --no-cache-dir --force-reinstall pymultihash>=0.8.2

This ensures only pymultihash is installed, avoiding the conflict.

---

Verification Commands

To verify the fix works after multiaddr is fixed:

# After multiaddr is fixed
pip install multiaddr  # Should not install py-multihash
pip install py-libp2p  # Should work without conflicts
python -c "from libp2p.peer.id import ID; print('OK')"

# Check installed packages
pip list | grep multihash
# Should only show: pymultihash 0.8.2
# Should NOT show: py-multihash

---

Historical Context

Previous Attempts to Address multiaddr Issues

The repository has a history of dealing with multiaddr dependency issues:

1. PR Fix multiaddr dep to use specific commit hash to resolve install issue #928 (September 14, 2025) - "Fix multiaddr dep to use specific commit hash to resolve install issue"

   • Author: @acul71
   • Commit: 52625e0f
   • Action: Pinned multiaddr to a specific git commit hash (b186e2ccadc22545dec4069ff313787bf29265e0)
   • Reason: To resolve installation issues
   • News fragment: "Fix multiaddr dependency to use the last py-multiaddr commit hash to resolve installation issues"

2. PR Update multiaddr to version 0.0.11 #935 / Issue Update multiaddr to version 0.0.11 #934 (September 16, 2025) - "Update multiaddr to version 0.0.11"

   • Author: @acul71
   • Commit: 35a4bf2d
   • Action: Switched from git dependency to PyPI package version multiaddr>=0.0.11
   • Reason: Move from git-based dependency to stable PyPI release
   • News fragment: "Updated multiaddr dependency from git repository to pip package version 0.0.11"
   • Note: This change explicitly avoided multiaddr 0.1.0 by using >=0.0.11 (which allows 0.0.11, 0.0.12, etc., but not 0.1.0 if properly constrained)

3. Earlier commits (July 2025):

   • Multiple commits pinning multiaddr to specific git commits to avoid build issues
   • Commit 746d0a46: "fix: update multiaddr dependency to fix py-cid build issue"
   • Commit 21ee4177: "Pin py-multiaddr dependency to specific git commit"

Why Previous Fixes Don't Address 0.1.0 Conflict

The previous fixes addressed:

• Build/installation issues with git-based dependencies
• Dependency resolution problems with py-cid
• Moving to stable PyPI releases

However, they did NOT address the multiaddr 0.1.0 → py-multihash conflict because:

• At the time, multiaddr 0.1.0 may not have existed yet, or
• The conflict wasn't discovered until someone tried to install multiaddr 0.1.0
• The current constraint multiaddr>=0.0.11 technically allows 0.1.0 to be installed

Current State

• Current constraint: multiaddr>=0.0.11 (allows 0.1.0)
• Currently installed in dev: multiaddr 0.0.12 (safe, no conflict)
• Risk: Fresh installs may get multiaddr 0.1.0 → conflict occurs

Important Clarification: Not an API Regression in multiaddr

Important: The issue is NOT that multiaddr 0.1.0 changed its API or has fewer functions. The problem is:

1. multiaddr 0.0.12 does NOT depend on py-multihash ✅
2. multiaddr 0.1.0 ADDED py-multihash as a new dependency ❌
3. When py-multihash is installed (via multiaddr 0.1.0), it conflicts with pymultihash that py-libp2p needs
4. The two multihash packages (pymultihash vs py-multihash) have incompatible APIs and both provide a module named multihash

What changed in multiaddr 0.1.0:

• Added py-multihash 2.0.1 as a dependency (this is the new/regressive change)
• The multiaddr package itself may not have changed its API significantly
• The problem is the new transitive dependency that creates a namespace collision

The "fewer functions" issue:

• It's not that multiaddr has fewer functions
• It's that py-multihash (installed by multiaddr 0.1.0) has a different API than pymultihash (needed by py-libp2p)
• py-multihash doesn't provide multihash.Func, multihash.digest(), or multihash.FuncReg that py-libp2p requires
• When Python imports multihash, it gets py-multihash instead of pymultihash, causing AttributeError

Timeline

• July 2025: Multiple attempts to fix multiaddr dependency issues by pinning to git commits
• September 14, 2025: PR Fix multiaddr dep to use specific commit hash to resolve install issue #928 (@acul71) - Fixed multiaddr to use specific commit hash
• September 16, 2025: PR Update multiaddr to version 0.0.11 #935 (@acul71) - Updated to multiaddr>=0.0.11 from PyPI
• Current: Workaround in Docker build (if needed)
• Short-term: Pin multiaddr version in py-libp2p to <0.1.0 if needed
• Long-term: Fix in multiaddr to remove or change the dependency

---

References

• multiaddr repository: https://github.com/multiformats/py-multiaddr
• py-libp2p repository: https://github.com/libp2p/py-libp2p
• pymultihash repository: https://github.com/ivilata/pymultihash
• py-multihash repository: https://github.com/multiformats/py-multihash
• PyPI: multiaddr - https://pypi.org/project/multiaddr/
• PyPI: pymultihash - https://pypi.org/project/pymultihash/
• PyPI: py-multihash - https://pypi.org/project/py-multihash/

---

Summary

The conflict exists because multiaddr 0.1.0 added py-multihash as a dependency, which conflicts with pymultihash that py-libp2p needs. The fix should be in multiaddr:

• Either switch to pymultihash (if multihash is needed)
• Or remove the dependency (if multihash is not needed)

This is a real issue that manifests in fresh installs, even though it doesn't appear in existing development environments that have multiaddr 0.0.12 installed.

Next Steps:

1. File an issue with multiaddr maintainers
2. Consider pinning multiaddr<0.1.0 in py-libp2p as a temporary workaround
3. Monitor multiaddr repository for fixes

[sumanjeet0012]

@acul71 @seetadev

After analyzing both repositories, I believe we should consider migrating from pymultihash to py-multihash by implementing the missing functionality in py-multihash. Here's my reasoning:

Current Dependency Analysis

py-libp2p currently depends on three primary APIs from pymultihash:

• multihash.Func - Enum for hash function codes
• multihash.digest() - Creates multihash from data
• multihash.FuncReg - Registry for custom hash functions

These APIs are not present in py-multihash. However, if we can port these functions to py-multihash, we could eliminate the pymultihash dependency entirely.

Why This Migration is Critical

1. Maintenance Status: pymultihash was archived 2 years ago and is no longer actively maintained. This poses long-term sustainability risks for py-libp2p.

2. Ecosystem Alignment: py-multihash is the canonical multiformats implementation. Continuing with pymultihash creates friction with the broader multiformats ecosystem.

3. Feature Parity: Since pymultihash is unmaintained, we cannot update it to achieve feature parity with the Go implementation of multihash, limiting our ability to keep py-libp2p aligned with other libp2p implementations.

4. Future Compatibility: As other multiformats Python packages (e.g., py-multibase, py-multicodec) standardize on py-multihash, continuing with pymultihash will create additional dependency conflicts.

5. Blocking multiaddr Updates: The current situation prevents us from upgrading to multiaddr 0.1.0+, limiting access to bug fixes and new features.

Feasibility Assessment

Both libraries are relatively small:

• pymultihash: ~700-800 lines of code
• py-multihash: ~200-300 lines of code

The missing functionality (Func, digest(), FuncReg) could be ported to py-multihash with reasonable effort. This would:

• Provide backward-compatible APIs for existing py-libp2p code
• Future-proof py-libp2p against dependency conflicts
• Align with the canonical multiformats ecosystem
• Enable us to track upstream Go multihash features

Proposed Path Forward

1. Contribute to py-multihash: Add the missing APIs (Func, digest(), FuncReg) to maintain compatibility with py-libp2p's current usage patterns.

2. Update py-libp2p: Migrate from pymultihash to the enhanced py-multihash once the APIs are available.

3. Remove pymultihash dependency: Eliminate the archived dependency and fully adopt the maintained multiformats implementation.

This approach addresses both the immediate multiaddr 0.1.0 conflict and the long-term sustainability concerns with depending on an archived package.