dirent and directory operations support

[MitchBradley]

I have VFS directory operations working, including stdfs::directory_iterator and friends, so I can do pretty much everything through std::filesystem. I would like to submit a PR, but I need some guidance about how to structure it. The change to VFS.cpp is pretty straightforward, adding functions like opendir(), readdir(), mkdir(), so that part is easy.

The complicated part is that it needs dirent.h and statvfs.h, as well as fs_dir.cc, fs_ops.cc, and fs_path.cc from the c++ library, which should probably go in the toolchain package in some form. I don't understand the various trees well enough to know where the files should go. In my application I have those extra files in a local directory.

Any suggestions? Or is this something that is a non-starter for one reason or another?

[earlephilhower]

Neat, thanks for hacking something out!

The "right" way is to update Newlib and make a new tuple OS, which is where dirent.h/etc. are implemented. Once you have opendir and struct dirent defined, everything should "just work" and be clean. Things like std::filesystem would then come right from the GCC headers, no need to include your own copy. It's also a pretty big lift.

I'm traveling today so can't really verify, but right now is there a dirent.h header/etc. that you need to override? If Newlib doesn't even include that header when we build for -none then I think shipping everything inside the library src folder would work fine. If you're trying to replace headers that are included in the toolchain already, it gets a little uglier. I'm not sure how the GCC paths would work out if you tried to override with that new header version (nor how the Newlib includes would handle it).

The other thing to be aware of is that the Arduino Directory might have subtly different semantics than POSIX dir_t. I'd verify things like parsing subdirs/etc. still work as expected. I know File does things like, for example, making directories automatically as needed since the original Arduino API didn't include dirs.

[MitchBradley]

My simple dirent.h has a two-field dirent struct with char[256] name and unsigned char type, plus the function prototypes for open/close/read/rewind dir(). To use it I compile with -D_GLIBCXX_HAVE_DIRENT_H and provide an include path that overrides toolchain-rp2040-earlephilhower/arm-none-eabi/include/sys/dirent.h .

My current packaging has a "stdfs17" library with
library.json
filesystem/
dir-common.h
ops-common.h
include/
dirent.h. (custom)
sys/
statvfs.h (custom)
bits/
fs_dir.h
fs_fwd.h
fs_ops.h
fs_path.h
largefile-config.h
src/
c++17/
fs_dir.cc
fs_ops.cc
fs_path.cc

Except for the two (custom) files, everything was lifted verbatim from https://github.com/gcc-mirror/gcc/tree/master/libstdc%2B%2B-v3

Everything works using std::filesystem ops - std::path, directories, subdirs, iterators, space() - using VFS with LittleFS and SD mounts.

Re the Arduino File/Dir thing: I did find one discrepancy with posix. If you try to create the file /foo/bar, where /foo does not exist, instead of failing, it creates a file named /foo. I suppose it would be possible to check for that case. I think that is an existing problem, not something I created, since it goes through the unmodified _open() method. The mkdir() that I added does not have the issue.

I suppose that one way to break down this process would be to first patch the arduino framework, thus enabling posix-style directory ops, then later figure out how best to incorporate the C++ stdfs stuff.

[earlephilhower]

...you try to create the file /foo/bar, where /foo does not exist, instead of failing, it creates a file named /foo.

That would be a bug that got in there somewhere. It's supposed to silently make subdirectories for files in paths which don't exist (because the Arduino API doesn't/didn't have a mkdir method). So if auto f = LittleFS.open("/this/is/a/path.txt", "w") doesn't make /this and /this/is/ and /this/is/a/ and put path.txt in it, it should be fixed in the core. I think the logic was duplicated in SDFS and LittleFS so it might be one didn't get it right...

[MitchBradley]

My recollection is that the Arduino dir behavior was an artifact of SPIFFS's flat structure, which was worked around, albeit poorly due to total name length limitations, by allowing / in file names.

Regardless, what do you thing of the posix-first, then worry about std::filesystem later, approach?