Mounting filesystems

[mstade]

It seems there's only an in-memory filesystem included with wasm-git (totally makes sense) but if I'm not completely mistaken it also looks like this can quite easily be replaced simply by mounting a filesystem at a given path? Digging through the FS object I see this:

Is there any documentation you can point to on how to implement this interface, and is this all that's needed to plug in a different kind of filesystem?

What I'd like to do is actually write to disk, not memory, when my application is running on a local machine (in Electron, hence #2, but I can do this in Node just as well) but when it's running in a browser I obviously would use an in-memory filesystem. I don't want to just dump one big blob, I'd actually like to write proper files that could be used with a regular git client outside of my app as well.

[petersalomonsen]

If you build it yourself you should be able to include support for the file-systems that come with emscripten:

https://emscripten.org/docs/api_reference/Filesystem-API.html#file-systems

In your case you probably want NODEFS

I haven't tested this, but will try as soon as I get time. Thanks for feedback :-)

[mstade]

Brilliant, I'll see if I can create a build for this, thanks for the tips!

[samdenty]

Doesn't work with NodeFS, breaks creating .git/config.lock files.

Works better with emscripten NODERAWFS=1, but still has some issues.

emscripten-core/emscripten#10162 fixes the first issue

[petersalomonsen]

Does it work with emscripten-core/emscripten#10162 ?

I did some testing myself and found the reason for the lock failing to be that after renaming a file, the file still exist with the old name:

emscripten-core/emscripten#10628

[samdenty]

@petersalomonsen Lock file renaming doesn't fail with NODERAWFS. But it fails with readdir 'Invalid Arguments', which #10162 fixes.

Then failed in libgit2/src/unix/map.c#58, with failed to mmap. Could not write data: No such device

[petersalomonsen]

So this should fix the locking issue for NODEFS:

emscripten-core/emscripten#10631

Now I get the same mmap error as you get. Will look into this soon, unless someone is quicker than me :-)

[petersalomonsen]

this fixes the mmap error:

emscripten-core/emscripten#10669

after that you'll get permission errors on library_fs.js open method and library_nodefs.js setattr. A dirty workaround to this is to make sure that the mknod call in the open method always have write permission ( node = FS.mknod(path, mode | 0o200, 0); ), and before the truncateSync in setattr also give write permission. I was able to clone in NODEFS this way, but for a PR to emscripten I'll need something better than changing permissions.

[petersalomonsen]

I've successfully cloned using NODEFS with the changes committed here: f9ed551

[nachoab]

Would it be possible to use the new WasmFS and mount the OPFS fs?

I'm including this flags in the build command, but it's creating a lg.js file that doesn't contain the opfs implementation

-s WASMFS -s FORCE_FILESYSTEM

[petersalomonsen]

Thanks, I wasn't aware of this, but it would be interesting to check if it works out of the box. Otherwise I would assume there are minor adjustments needed to support it. I'll have a look in the near future.

[petersalomonsen]

Now compiling with the latest emscripten here: #58

Had to increase the stack size to make it work.

Not sure if it actually use WASMFS. Do I have to add the -s WASMFS switch?

[nachoab]

Now compiling with the latest emscripten here: #58

Had to increase the stack size to make it work.

Not sure if it actually use WASMFS. Do I have to add the -s WASMFS switch?

Including -s WASMFS is supposed to include the WASMFS JS libraries, but I'm not seeing any of the methods of the opfs js library being included in the exported js unless explicitly added on EXPORTED_RUNTIME_METHODS. But even then, is not useful since FS doesn't include a way to load the OPFS backend, so I guess FS it's just running on memory (with different API so maybe that's making git calls fail)

I asked how to test this here and was suggested to take a look at the tests. But those are tests for the C implementation and don't show how to mount a FS

[petersalomonsen]

Maybe it is too early? According to the tracking page here, many of the essential parts are still in status TODO: WasmFS (view)

[nachoab]

Yeah your right, I'll give a try in the future

[MeydanOzeri]

any update on this topic ? im also trying to create a custom file system