Roadmap: Updating py-multihash, py-multibase, and py-multicodec to Full Parity with Go, Rust, and JS Implementations

[seetadev]

Now that the py-multiformats repos have been unarchived, and with active collaboration from the Multiformats team, this roadmap outlines the steps needed to bring Python parity with the current Go, Rust, and JavaScript stacks.

This roadmap includes functional parity, spec compliance, test coverage, and ecosystem readiness.

---

1. 🎯 Phase 1 — Baseline Assessment & Repository Cleanup

Goal: Establish an accurate understanding of gaps vs. go-multiformats, rust-multiformats, and js-multiformats.

Tasks

• Audit APIs and feature gaps for each Python repo:

  • py-multihash
  • py-multibase
  • py-multicodec

• Compare with:

  • go-multihash, go-multibase, go-multicodec
  • rust-multihash, rust-multibase, rust-multicodec
  • js-multiformats

• Identify missing codecs, bases, hashing algorithms, error behaviors, encoding rules.

• Clean up stale PRs and issues, document missing items.

• Modernize repo setup:

  • Python 3.10+ baseline
  • pyproject.toml migration
  • Ruff/Black formatting
  • pre-commit hooks
  • GitHub Actions CI

Deliverables:
📄 Gap analysis document
🚦 CI pipeline green
📦 Modernized Python project layout

---

2. 🎯 Phase 2 — Feature Parity: py-multibase

Goal: Match the full multibase table supported in Go, Rust, and JS.

Tasks

• Implement missing bases:

  • base16 upper/lower
  • base32 variants
  • base58btc (full compatibility test matrix)
  • base64, base64url, no-padding variants

• Ensure byte-level equivalence with Go/Rust/JS implementations.

• Add multibase-name → prefix mappings identical to reference tables.

• Implement round-trip property tests.

Deliverables:
✨ py-multibase v2.0 with full support & test parity

---

3. 🎯 Phase 3 — Feature Parity: py-multicodec

Goal: Support the complete codec table and wire representations.

Tasks

• Import latest multicodec-table from upstream.

• Implement automated code-gen for Python from codec table JSON (matching Go’s generator approach).

• Ensure compatibility with:

  • varint encoding rules
  • identity codecs
  • named aliases

• Add fuzzy tests comparing Python → Go → Rust → JS resolution.

Deliverables:
📦 py-multicodec v2.0 with up-to-date codec table
🛠 Codegen workflow for long-term maintainability

---

4. 🎯 Phase 4 — Feature Parity: py-multihash

Goal: Support all stable hashing algorithms and matching output encoding.

Tasks

• Complete algorithm support:

  • sha1, sha2 variants
  • sha3 / keccak
  • blake2b / blake2s
  • blake3 (align with Rust & Go)

• Match output formatting with:

  • go-multihash binary representation
  • rust-multihash strict length checks
  • js-multiformats flexible input APIs

• Implement strict & relaxed mode hashing (like Go version).

Deliverables:
🔐 py-multihash v2.0 with full algorithm coverage
🧪 Cross-language test suite

---

5. 🎯 Phase 5 — Unified Cross-Language Test Suite

Goal: Ensure bit-for-bit compatibility across all languages.

Tasks

• Implement docker-compose based test harness:

  • Python ↔ Go ↔ Rust ↔ JS round-trip tests

• Test categories:

  • Encode/decode consistency (multibase, multicodec, multihash)
  • Error mode consistency
  • Invalid input vectors
  • Fuzzing (with Python Hypothesis + cross-lang comparisons)

• Add snapshot-based tests synced with upstream reference tables.

Deliverables:
📊 Interop matrix (Python vs Go/Rust/JS)
🧪 CI running cross-language interop tests automatically

---

6. 🎯 Phase 6 — Documentation, Migration Guide & Release

Goal: Provide high-quality user documentation and release stable versions.

Tasks

• Write docs explaining:

  • Supported codecs/bases/hashes
  • API parity with Go/JS
  • Upgrade guide from older py-multiformats

• Add examples mirroring JS + Go examples (CID creation, encoding/decoding).

• Release:

  • py-multibase 2.0
  • py-multicodec 2.0
  • py-multihash 2.0

Deliverables:
📝 Complete docs on ReadTheDocs
🚀 Stable releases to PyPI

---

7. 🎯 Phase 7 — Long-Term Maintenance & Alignment with Multiformats Team

Goal: Ensure continued health and parity.

Tasks

• Set up monthly sync with multiformats maintainers.

• Subscribe Python repos to automated multicodec table update PRs.

• Add GitHub bots for:

  • stale issue cleanup
  • automated version bumping

• Establish roadmap for:

  • CID support in Python
  • multistream-select
  • varint improvements

---

✅ Expected Outcome

By following this roadmap, the Python multiformats libraries will achieve:

• Complete parity with Go, Rust, and JS
• Fully updated codec tables and hashing algorithms
• Identical behavior across all language ecosystems
• CI-supported interop test harness
• Long-term maintainability with automated updates

[seetadev]

@acul71, @pacrob, @sumanjeet0012, @lla-dane, @yashksaini-coder: Hi everyone, please find a brief roadmap for updating the newly un-archived py-multiformats packages (py-multihash, py-multibase, py-multicodec) shared above.

@acul71 : Wish if you could open up issues like the one you opened up for py-multiaddr: multiformats/multiaddr#181

@sumanjeet0012 : Wish to recommend you to start exploring the fix for #1030 with @acul71.

@lla-dane, @acul71 and @sumanjeet0012, @yashksaini-coder - As discussed, wish if we could open newer discussion pages, issues and PRs for improving each of these multiformat packages. Nice references from our collaborative efforts in improving py-multiaddr repository are:

1. Add SNI, NOISE, CERTHASH, WEBRTC, WEBRTC-DIRECT in py-multiaddr multiformats/py-multiaddr#97
2. py-multiaddr improvement #676
3. small cleanup multiformats/py-multiaddr#93
4. Add release commands multiformats/py-multiaddr#88
5. ipcidr Protocol Usage in libp2p #976
6. Fix Tag-Only Protocols validation error multiformats/py-multiaddr#99
7. Add ipcidr protocol in reference with go-multiaddr multiformats/py-multiaddr#95

Me and @pacrob will try setting up these packages at pypi and readthedocs.io/, and will join you in the above mentioned project tasks soon. We will also try and see if we could get the ownership transferred from the previous maintainers of these pages.

[acul71]

@seetadev
py-multihash Implementation Comparison: Gap Analysis with Go, Rust, and JavaScript #1061

The Python py-multihash implementation provides solid foundational functionality for encoding and decoding multihashes, but lacks several key features present in other implementations:

1. Hash computation - The most critical gap
2. Missing hash functions - 14 hash functions/codecs are missing from Python's constants table (MD5, SHA2-224, SHA2-384, SHA2-512/224, SHA2-512/256, Blake3, SHA2-256-trunc254-padded, X11, Poseidon-BLS12-381, RIPEMD variants, Strobe variants)
3. Streaming support - Important for large files
4. Extensibility - Needed for custom hash functions
5. Advanced features - Truncation, collections, serialization

[acul71]

@seetadev @pacrob
The first thing we should do is update

• py-multihash
• py-multibase
• py-multicodec

to match a similar py-libp2p setup
Python 3.10+ (Is this ok ? )
and updated lint, typecheck and CI/CD utilities

If relinquishing the repos will take a long time, we could fork the repos and work on the fork, until we got access.

[pacrob]

Python 3.10+ (Is this ok ? )
It is OK. That's where py-libp2p, py-multihash, and py-cid are set, and it gets us some nice typing things.

[seetadev]

@pacrob : Thank you Paul for sharing feedback. Appreciate it.

@acul71 : Thank you so much for sharing detailed analysis and insights on the step by step milestones needed in each of the multiformats packages. Appreciate it. Lets get started, and accomplish the goals in the coming weeks.

[acul71]

Lets get started,

So fork the repo ?
How long before we have access ?

[seetadev]

@acul71 : Wish to share that all the 3 repos have been un-archived and accessible to us. Please visit https://github.com/orgs/multiformats/repositories. We do have the full access.

You can get started on the key initiatives with @sumanjeet0012, @yashksaini-coder and @lla-dane.

Me and @pacrob will arrange for the pypi and readthedocs.io access soon.

[acul71]

@seetadev

py-multibase Gap Analysis #1062

py-multibase currently supports 58% of the standard multibase encodings (14 out of 24). The missing encodings primarily fall into three categories:

1. Padding variants (4 encodings) - High priority for RFC compliance
2. Uppercase variants (6 encodings) - Medium priority for case-sensitive systems
3. Specialized encoding (1 encoding: base256emoji) - Low priority

Additionally, the API lacks some advanced features present in other implementations, such as reusable encoder/decoder objects and type-safe base selection.

[acul71]

@seetadev

py-multicodec gap analysis #1063

py-multicodec provides the core functionality for multicodec operations but is missing:

• ~230+ codecs present in go-multicodec
• Advanced features like codec enumeration, reserved ranges, type safety
• Integration features like serialization, CLI support

The codebase is well-structured and can be easily updated. The main gap is the outdated codec table, which can be fixed by running the update script.