Controlling Caching

[atzannes]

Let me start by saying that I am a big fan of fsspec and universal_pathlib. I had written a version of fsspec which I am trying to migrate off of in favor of fsspec. Especially when combined with UPath, it is very powerful.

I do keep running up against some issues with the caching, however, and I am wondering if I am missing something. Whenever I use UPath/fsspec in a process that writes data, things work well, but whenever I try to use it to read files that are being updated I run into issues with the listings cache: the default behavior is to enable the listings cache with no timeout. UPath reuses fsspec instances, so even if I create a new UPath instance to the same URI, it will find and reuse the instance of the filesystem (which holds the listings cache). My (not so elegant solution) is to pass use_listings_cache=False to UPath, but I am coming to the realization that implementations of AbstractFileSystem don't have a uniform API for forwarding kwargs to super(). For example, FTPFileSystem takes kwargs that it forwards to super(), but SFTPFileSystem takes ssh_kwargs which it both forwards to its super() and uses to create the client, which chokes on use_listings_cache=False. Perhaps, this is simply a bug and I'm happy to report it, but I am wondering if I'm not just using UPath incorrectly because I imagine that using it for reading externally changing filesystems is a common pattern, and there may be a different preferred way of doing that.