Skip to content

Bittorrent Integration [draft]#1137

Draft
marcinczenko wants to merge 73 commits intomasterfrom
feat/bittorrent-integration
Draft

Bittorrent Integration [draft]#1137
marcinczenko wants to merge 73 commits intomasterfrom
feat/bittorrent-integration

Conversation

@marcinczenko
Copy link
Copy Markdown
Contributor

@marcinczenko marcinczenko commented Feb 28, 2025 •
edited
Loading

Draft of the BitTorrent Codex integration.

Should be enough to see the overall idea in code.

What is there:

  • storeTorrent is equivalent of Codex store - this is the entry point when seeding BitTorrent content into Codex network
  • retrieveTorrent is equivalent of Codex retrieve - entry point when requesting BitTorrent content having only info hash as the input.

To be functional still needs:

  • b-encoding and the actual info-hash validation (now there is only a placeholder around the place where it most probably will happen)
  • streaming pieces and validating them
  • REST API (download)
  • REST API (upload)
  • REST API (fetch torrent manifest only for the given info hash)
  • Try to make the Cid for BitTorrent Manifest a valid Cid - investigation done - not technically feasible and does not add any more security.
  • update docs
  • a bit more testing

See also logos-storage/logos-storage-pm-archived#211.

@marcinczenko marcinczenko self-assigned this Feb 28, 2025
@marcinczenko marcinczenko marked this pull request as draft February 28, 2025 17:11
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 3 times, most recently from bf800ac to 8a608e8 Compare March 5, 2025 14:45
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch from 1b28c03 to b531696 Compare March 11, 2025 01:05
@marcinczenko marcinczenko mentioned this pull request Mar 13, 2025
Open
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 2 times, most recently from aa9989c to e19065b Compare March 23, 2025 00:44
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 7 times, most recently from 820709c to 4237e8b Compare March 31, 2025 18:34
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 3 times, most recently from 7a76387 to 30414e8 Compare April 23, 2025 16:14
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch from 7e073ad to 295fabf Compare April 26, 2025 23:49
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 2 times, most recently from 60f16d1 to 7128f62 Compare May 20, 2025 13:21
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch from 29c6d93 to 6b88273 Compare June 2, 2025 02:37
marcinczenko added 29 commits June 3, 2025 03:00
@marcinczenko
refactor piece validator tests
ea3cfe0
@marcinczenko
adds torrent downloader abstraction
50ca6da
@marcinczenko
adds testing setup for torrentdownloader
526e3f1
@marcinczenko
do not randomize piece fetching sequence for now
ee2f403
@marcinczenko
adds some good weather tests for torrentdownloader
c859250
@marcinczenko
adds more tests and ability to retrieve the downloaded blocks
2aa91e5
@marcinczenko
updates and integrates torrentdownloader into api and node
534cb71
@marcinczenko
updates copyright
414906a
@marcinczenko
checked exceptions in stores
5abce81
@marcinczenko
better exception handling
b0be205
@marcinczenko
better exception handling in node
edf4627
@marcinczenko
uses SafeAsyncIter in "listBlocks" and in "getBlockExpirations"
4d5dfb1
@marcinczenko
uses SafeAsyncIter to stream the blocks in api
6000ae5
@marcinczenko
fix failing test in torrentdownloader
e3fd2fa
@marcinczenko
gets rid of ugly casts
6c32f6b
@marcinczenko
adds more tests to torrentdownloader
f9d2b21
@marcinczenko
gets rid of future casting in torrent downloader
85e0e58
@marcinczenko
Uploading directories - first draft.
912479e
@marcinczenko
moves tarball-related stuff out of node + adds explicit padding argum…
6852c1f 
…ent to "store" + adds "onBatch" callback to "fetchDatasetAsync"
@marcinczenko
Cleans up directory manifest and tartballs
6aad595
@marcinczenko
adds directory downloader
2e6d7d0
@marcinczenko
adds API for retrieving directory
108370e
@marcinczenko
convenience feature - download content using magnet links
8ac8f94
@marcinczenko
Updates magnet link tests
03ff25b
@marcinczenko
adds torrent parser to support native torrent files when downloading
0f62624
@marcinczenko
adds magent link and torrent parser tests to bittorrent tests
f9ce889
@marcinczenko
adds integration tests for using magnet links and torrent file while …
29bb017 
…retrieving BitTorrent content from Codex Net
@marcinczenko
removes not needed err from "without" in repostore test
ff40917
@marcinczenko
checking if auto in the block expiration test has any impact on lib…
0935abb 
…pcre requirement
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch from 74f39bd to 0935abb Compare June 3, 2025 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@marcinczenko marcinczenko

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@marcinczenko