py-multicodec gap analysis

[acul71]

py-multicodec Comparison Analysis

This document provides a comprehensive comparison between py-multicodec and other multicodec implementations:

• go-multicodec (Go)
• rust-multicodec (Rust)
• js-multiformats (JavaScript/TypeScript - multicodec portion)

Executive Summary

py-multicodec is a basic implementation that provides core prefix operations but lacks:

1. Many newer codecs present in go-multicodec
2. Advanced features like code enumeration, reserved range support
3. Type-safe codec constants
4. Integration with serialization frameworks

Feature Comparison Table

Feature	py-multicodec	go-multicodec	rust-multicodec	js-multiformats
Core Operations
Add prefix	✅ add_prefix()	✅ Via constants	✅ add()	✅ Via codec interface
Remove prefix	✅ remove_prefix()	❌	✅ remove()	✅ Via codec interface
Get codec from bytes	✅ get_codec()	❌	✅ get()	✅ Via codec interface
Get prefix bytes	✅ get_prefix()	❌	❌	❌
Extract prefix integer	✅ extract_prefix()	❌	❌	❌
Check if codec exists	✅ is_codec()	✅ Via Set()	❌	❌
Codec Management
Codec constants	❌	✅ Type-safe constants	❌	❌
List all codecs	❌	✅ KnownCodes()	❌	❌
Codec name → code	✅ Via NAME_TABLE	✅ Via Set()	❌	❌
Code → codec name	✅ Via CODE_TABLE	✅ Via String()	❌	❌
Reserved range support	❌	✅ ReservedStart/End	❌	❌
Serialization
Encode objects	❌	❌	✅ JSON only	✅ Via codec interface
Decode objects	❌	❌	✅ JSON only	✅ Via codec interface
Type Safety
Type-safe codec types	❌	✅ Code type	✅ Enum	✅ TypeScript types
Flag.Value interface	❌	✅	❌	❌
Codec Count
Total codecs	~370	~600+	1 (JSON only)	2 (raw, json)
Auto-update mechanism	✅ update-table.py	✅ go generate	❌	❌

API Comparison

py-multicodec API

from multicodec import add_prefix, remove_prefix, get_codec, get_prefix, is_codec, extract_prefix

# Add prefix
prefixed = add_prefix('sha2-256', data)

# Remove prefix
data = remove_prefix(prefixed)

# Get codec name
codec = get_codec(prefixed)

# Get prefix bytes
prefix_bytes = get_prefix('sha2-256')

# Extract prefix integer
prefix_int = extract_prefix(prefixed)

# Check if codec exists
exists = is_codec('sha2-256')

go-multicodec API

import "github.com/multiformats/go-multicodec"

// Type-safe constants
code := multicodec.Sha2_256
name := code.String() // "sha2-256"

// Set from string
var code multicodec.Code
code.Set("sha2-256")

// Set from number
code := multicodec.Code(0x12)

// List all known codecs
allCodes := multicodec.KnownCodes()

// Reserved range
if code >= multicodec.ReservedStart && code <= multicodec.ReservedEnd {
    // Internal use code
}

rust-multicodec API

use rust_multicodec::{codec, encode, decode};

// Encode with JSON codec
let encoded = encode(codec::CodecType::JSON, &object)?;

// Decode
let result: DecodeResult<T> = decode(encoded.as_ref())?;

js-multiformats API

import * as json from 'multiformats/codecs/json'
import * as raw from 'multiformats/codecs/raw'

// Encode
const bytes = json.encode({ hello: 'world' })

// Decode
const obj = json.decode(bytes)

Missing Codecs in py-multicodec

Based on comparison with go-multicodec (which appears to be the most comprehensive), the following codecs are missing in py-multicodec:

CID & Versioning

• ✅ cidv1 (0x01) - PRESENT
• ✅ cidv2 (0x02) - PRESENT
• ✅ cidv3 (0x03) - PRESENT

Multiformat Extensions

• ❌ varsig (0x34) - Variable signature format
• ❌ dnslink (0xe8) - DNS link format

Hash Functions

• ❌ sha2-384 (0x20) - SHA-384
• ❌ sha2-224 (0x1013) - SHA-224
• ❌ sha2-512-224 (0x1014) - SHA-512/224
• ❌ sha2-512-256 (0x1015) - SHA-512/256
• ❌ murmur3-x64-128 (0x1022) - Murmur3 x64 128-bit
• ❌ murmur3-x64-64 (0x22) - Murmur3 x64 64-bit (Note: py has murmur3-128 and murmur3-32)

Encryption & Cipher

• ❌ aes-128 (0xa0)
• ❌ aes-192 (0xa1)
• ❌ aes-256 (0xa2)
• ❌ chacha-128 (0xa3)
• ❌ chacha-256 (0xa4)
• ❌ aes-gcm-256 (0x2000)
• ❌ chacha20-poly1305 (0xa000)

Network & Transport

• ❌ ipcidr (0x2b) - IP CIDR mask
• ❌ webrtc-direct (0x0118)
• ❌ webrtc (0x0119)
• ❌ sni (0x01c1) - Server Name Indication
• ❌ noise (0x01c6) - Noise protocol
• ❌ shs (0x01c8) - Secure Handshake
• ❌ quic-v1 (0x01cd)
• ❌ webtransport (0x01d1)
• ❌ certhash (0x01d2)
• ❌ http-path (0x01e1)
• ❌ memorytransport (0x0309)

Blockchain & Cryptocurrency

• ❌ caip-50 (0xca) - Chain Agnostic Improvement Proposal 50
• ❌ streamid (0xce) - Stream ID
• ❌ lbry (0x8c) - LBRY protocol
• ❌ eth-receipt-log-trie (0x99)
• ❌ eth-receipt-log (0x9a)
• ❌ beeson (0xfc)

Checksums

• ❌ crc32 (0x0132)
• ❌ crc64-ecma (0x0164)
• ❌ crc64-nvme (0x0165)

Serialization & Formats

• ❌ car (0x0202) - Content Addressed Archive
• ❌ car-index-sorted (0x0400)
• ❌ car-multihash-index-sorted (0x0401)
• ❌ x509-certificate (0x0210)

IPLD & Data Structures

• ❌ ipns-record (0x0300)
• ❌ libp2p-relay-rsvp (0x0302)
• ❌ blake3-hashseq (0x80)

Cryptographic Keys & Signatures

• ❌ sr25519-pub (0xef) - Sr25519 public key
• ❌ rsa-pub (0x1205) - RSA public key
• ❌ sm2-pub (0x1206) - SM2 public key
• ❌ vlad (0x1207)
• ❌ provenance-log (0x1208)
• ❌ provenance-log-entry (0x1209)
• ❌ provenance-log-script (0x120a)
• ❌ mlkem-512-pub (0x120b) - ML-KEM 512 public key
• ❌ mlkem-768-pub (0x120c) - ML-KEM 768 public key
• ❌ mlkem-1024-pub (0x120d) - ML-KEM 1024 public key
• ❌ multisig (0x1239)
• ❌ multikey (0x123a)
• ❌ nonce (0x123b)
• ❌ secp256k1-priv (0x1301)
• ❌ x25519-priv (0x1302)
• ❌ sr25519-priv (0x1303)
• ❌ rsa-priv (0x1305)
• ❌ p256-priv (0x1306)
• ❌ p384-priv (0x1307)
• ❌ p521-priv (0x1308)
• ❌ bls12_381-g1-priv (0x1309)
• ❌ bls12_381-g2-priv (0x130a)
• ❌ bls12_381-g1g2-priv (0x130b)
• ❌ bls12_381-g1-pub-share (0x130c)
• ❌ bls12_381-g2-pub-share (0x130d)
• ❌ bls12_381-g1-priv-share (0x130e)
• ❌ bls12_381-g2-priv-share (0x130f)
• ❌ sm2-priv (0x1310)
• ❌ ed448-priv (0x1311)
• ❌ x448-priv (0x1312)

Post-Quantum Cryptography

• ❌ lamport-sha3-512-pub (0x1a14)
• ❌ lamport-sha3-384-pub (0x1a15)
• ❌ lamport-sha3-256-pub (0x1a16)
• ❌ lamport-sha3-512-priv (0x1a24)
• ❌ lamport-sha3-384-priv (0x1a25)
• ❌ lamport-sha3-256-priv (0x1a26)
• ❌ lamport-sha3-512-priv-share (0x1a34)
• ❌ lamport-sha3-384-priv-share (0x1a35)
• ❌ lamport-sha3-256-priv-share (0x1a36)
• ❌ lamport-sha3-512-sig (0x1a44)
• ❌ lamport-sha3-384-sig (0x1a45)
• ❌ lamport-sha3-256-sig (0x1a46)
• ❌ lamport-sha3-512-sig-share (0x1a54)
• ❌ lamport-sha3-384-sig-share (0x1a55)
• ❌ lamport-sha3-256-sig-share (0x1a56)

Hash Functions (Extended)

• ❌ xxh-32 (0xb3e1) - XXH32 hash
• ❌ xxh-64 (0xb3e2) - XXH64 hash
• ❌ xxh3-64 (0xb3e3) - XXH3 64-bit
• ❌ xxh3-128 (0xb3e4) - XXH3 128-bit
• ❌ sha256a (0x7012) - SHA256a variant

Merkle Trees & Data Structures

• ❌ fr32-sha256-trunc254-padbintree (0x1011)
• ❌ rdfc-1 (0xb403)
• ❌ ssz (0xb501) - Simple Serialize
• ❌ ssz-sha2-256-bmt (0xb502)
• ❌ sha2-256-chunked (0xb510)

Formats & Protocols

• ❌ json-jcs (0xb601) - JSON Canonicalization Scheme
• ❌ bittorrent-pieces-root (0xb702)
• ❌ iscc (0xcc01) - International Standard Content Code
• ❌ swhid-1-snp (0x01f0) - Software Heritage ID
• ❌ multidid (0x0d1d) - Multi-DID

Transport Protocols

• ❌ transport-bitswap (0x0900)
• ❌ transport-graphsync-filecoinv1 (0x0910)
• ❌ transport-ipfs-gateway-http (0x0920)
• ❌ transport-filecoin-piece-http (0x0930)

Signatures & Authentication

• ❌ nonstandard-sig (0xd000)
• ❌ bcrypt-pbkdf (0xd00d)
• ❌ es256k (0xd0e7)
• ❌ bls12_381-g1-sig (0xd0ea)
• ❌ bls12_381-g2-sig (0xd0eb)
• ❌ eddsa (0xd0ed)
• ❌ eip-191 (0xd191)
• ❌ jwk_jcs-pub (0xeb71)
• ❌ ed2k (0xed00)

Filecoin

• ✅ fil-commitment-unsealed (0xf101) - PRESENT
• ✅ fil-commitment-sealed (0xf102) - PRESENT

Shelter Protocol

• ❌ shelter-contract-manifest (0x511020)
• ❌ shelter-contract-text (0x511021)
• ❌ shelter-contract-data (0x511022)
• ❌ shelter-file-manifest (0x511023)
• ❌ shelter-file-chunk (0x511024)

Other Protocols

• ❌ plaintextv2 (0x708021)
• ❌ arweave-ns (0xb28a10)
• ❌ subspace-ns (0xb2f590)
• ❌ kumandra-ns (0xb4b510)

JWT & Signatures

• ❌ es256 (0xd00000) - ECDSA P-256
• ❌ es384 (0xd00001) - ECDSA P-384
• ❌ es512 (0xd00002) - ECDSA P-521
• ❌ rs256 (0xd00005) - RSA SHA-256
• ❌ es256k-msig (0xd00080)
• ❌ bls12_381-g1-msig (0xd00081)
• ❌ bls12_381-g2-msig (0xd00082)
• ❌ eddsa-msig (0xd00083)
• ❌ bls12_381-g1-share-msig (0xd00084)
• ❌ bls12_381-g2-share-msig (0xd00085)
• ❌ lamport-msig (0xd00086)
• ❌ lamport-share-msig (0xd00087)
• ❌ es256-msig (0xd00088)
• ❌ es384-msig (0xd00089)
• ❌ es521-msig (0xd0008a)
• ❌ rs256-msig (0xd0008b)
• ❌ scion (0xd00e00)

ML-KEM Private Keys

• ❌ mlkem-512-priv (0x130d)
• ❌ mlkem-768-priv (0x130e)
• ❌ mlkem-1024-priv (0x130f)
• ❌ jwk_jcs-priv (0x130e)

Missing Features in py-multicodec

1. Codec Enumeration

• Missing: KnownCodes() equivalent to list all supported codecs
• Impact: Cannot programmatically discover available codecs
• Workaround: Access CODECS.keys() directly (but not part of public API)

2. Reserved Range Support

• Missing: Support for reserved code range (0x300000-0x3FFFFF)
• Impact: Cannot use internal/reserved codes
• Reference: go-multicodec has ReservedStart and ReservedEnd constants

3. Type-Safe Codec Constants

• Missing: Type-safe codec constants (like multicodec.Sha2_256)
• Impact: Prone to typos, no IDE autocomplete
• Current: String-based codec names only

4. Flag.Value Interface

• Missing: Integration with command-line flag parsing
• Impact: Cannot use multicodec directly in CLI tools
• Reference: go-multicodec implements flag.Value interface

5. Serialization Integration

• Missing: Built-in encode/decode for common formats
• Impact: Must manually serialize before adding prefix
• Reference: rust-multicodec and js-multiformats provide encode/decode

6. Codec Metadata

• Missing: Codec descriptions, tags, status (permanent/draft)
• Impact: Cannot determine codec properties programmatically
• Reference: go-multicodec includes codec metadata in generated code

7. Error Types

• Missing: Specific error types for different failure modes
• Impact: Generic ValueError for all errors
• Suggestion: Create UnknownCodecError, InvalidPrefixError, etc.

8. Validation

• Missing: Validation of codec names/prefixes
• Impact: Runtime errors instead of validation errors
• Suggestion: Add validate_codec() function

9. Codec Registry

• Missing: Dynamic codec registration
• Impact: Cannot add custom codecs at runtime
• Suggestion: Allow registering custom codecs

10. Performance Optimizations

• Missing: Cached lookups, optimized data structures
• Impact: Linear search in some operations
• Suggestion: Use dictionaries for O(1) lookups (partially implemented)

Codec Count Summary

Implementation	Codec Count	Notes
py-multicodec	~370	Based on constants.py
go-multicodec	~600+	Most comprehensive
rust-multicodec	1	JSON only (very limited)
js-multiformats	2	raw, json (base package)

Note: The actual count varies as the multicodec table is updated. Both py-multicodec and go-multicodec can be updated from the official table.

Recommendations

High Priority

1. Update codec table - Run update-table.py to sync with latest multicodec table
2. Add missing codecs - Focus on commonly used ones (sha2-384, aes variants, etc.)
3. Add KnownCodes() equivalent - Expose list of all codecs
4. Improve error handling - Add specific error types

Medium Priority

5. Add reserved range support - For internal/experimental codecs
6. Add codec metadata - Descriptions, tags, status
7. Type hints - Add comprehensive type annotations
8. Performance - Optimize lookup operations

Low Priority

9. Serialization helpers - Optional encode/decode utilities
10. CLI integration - Optional argparse integration
11. Codec registry - Dynamic codec registration

Update Process

To update py-multicodec with missing codecs:

# Update from official multicodec table
curl -X GET https://raw.githubusercontent.com/multiformats/multicodec/master/table.csv | python tools/update-table.py

This will regenerate multicodec/constants.py with the latest codecs from the official table.

Conclusion

py-multicodec provides the core functionality for multicodec operations but is missing:

• ~230+ codecs present in go-multicodec
• Advanced features like codec enumeration, reserved ranges, type safety
• Integration features like serialization, CLI support

The codebase is well-structured and can be easily updated. The main gap is the outdated codec table, which can be fixed by running the update script.

---

Last Updated: Based on codebase analysis as of current date
Comparison Sources:

• py-multicodec: /home/luca/Informatica/Learning/PNL_Launchpad_Curriculum/Libp2p/py-multicodec
• go-multicodec: /home/luca/PNL_Launchpad_Curriculum/Libp2p/go-multicodec
• rust-multicodec: /home/luca/PNL_Launchpad_Curriculum/Libp2p/rust-multicodec
• js-multiformats: /home/luca/PNL_Launchpad_Curriculum/Libp2p/js-multiformats