New Extractor Idea: Find/write a "`cad-dl`" to save 3d assets, gltf files, CAD files, shapefiles, STLs, VR views, etc.

[fire]

My thoughts is a combination of having archive box save the asset as a Blender file and as a gltf2.

However, there's layers of problems here.

Any suggestions are welcome.

I can provide technical support and man-months, but not sure where to start.

[pirate]

What do you mean by 3d assets? Can you provide some examples of URLs and formats that you'd expect it to be able to save?

[fire]

Thanks for your prompt reply.

1. (gltf without animations) https://3d.si.edu/object/3d/command-module-apollo-11:d8c63e8a-4ebc-11ea-b77f-2e728ce88125
2. (blend) https://cloud.blender.org/p/gallery/5e46a80442fa9613e1cd1fca
3. (blend) https://cloud.blender.org/p/gallery/60337d495677e942564cce76
4. (gltf with culturally significant animations) https://sketchfab.com/3d-models/fortnite-floss-emote-0a52f8e8eaf7441faffd8efc8d8a9e0e
5. (VRM based on gltf) https://booth.pm/en/items/1050142 (This is a stretch goal, but I would handle this by either ignoring the format, importing into blend or identify it as a gltf file)
6. (VRM based on gltf) https://github.com/Miraikomachi/MiraikomachiVRM/blob/master/Miraikomachi.vrm

The typical open formats are FBX (no good opensource reader), glTF2(open source), Blend (has an implementation in Blender but is complicated) and USDZ (Newer standard. Do not use due to complexity). I think there are a few cad formats, but they're either convertable to gltf2 or blender.

There are some other formats that aren't mentioned like alembic but the point is to support a well defined, narrow format that can be opened in the future for archival use.

[pirate]

I think the best way to go about this is to find an existing program (or snippet of puppeteer/playwright JS) that can look for these assets on a page (given some html or a url) and add it as an extractor module to archivebox.

I don't know the 3D space at all, so I'm probably not the right person for this, but I'm happy to review PRs or design proposals for such an extractor.

[fire]

If I can script Blender would that be acceptable?

https://github.com/donmccurdy/glTF-Transform is web native.

[fire]

@pirate Can you link me some guides for writing extractors. Also what is the format for design proposals?

I think finding the urls can be worked around by linking a direct url for now.

[pirate]

@fire The process for adding a new extractor is documented here:

• https://github.com/ArchiveBox/ArchiveBox#archivebox-development
• https://github.com/ArchiveBox/ArchiveBox#contributing-a-new-extractor

Note the main constraint for ArchiveBox right now is deployment complexity, so I'm putting a hold adding new binary dependencies at the moment. I don't think that will necessarily impair your ability to download 3d files as long as they don't need any further 3d processing after download. If you have a pure python package or npm library that can snapshot 3d assets from a URL with minimal packaging complexity and linux/macOS + x86/arm7/arm64 support then I'm down to consider it.

[fire]

The standard tool for gltf is https://gltf-transform.donmccurdy.com/ like ffmpeg in video importing.

I'll have to find a url extractor.

[pirate]

Maybe we can borrow code from this extension: https://github.com/stephancasas/thingiverse-stl-downloader

[pirate]

If anyone knows of any youtube-dl / yt-dlp equivalent program to find + download 3d assets from a URL that would be super useful here. Please comment with any suggestions :)

At the moment I'm still not willing to write custom logic to do this extraction, as it would be too much for me to maintain as a solo developer working on ArchiveBox in my spare time, but if we can find an external program/library that can do it then the task is much easier.

[fire]

Suggest an interface for me, and I might take a try at making one from scratch.

[pirate]

Same CLI as YouTube-dl/yt-dlp would be great. E.g.

shapefile-dl [--max-size=750m] https://example.com/some/page/containing/cad/files

It should output one or more files to the current directory the command is run in, and return 1 exit status + error text if it fails, or 2 exit status if no shape files are found.

Pure Python would be ideal, but js is also ok.

[fire]

Oh. so it needs to be python or javascript, but not like c++ or elixir binaries, hmmm. My plan was to either write one from scratch or use Godot Engine's code I know the details for.

Godot Engine has a wasm platform.