Skip any erroring entry in FilesystemStore::list

[tnull]

Closes #3795.

Previously, we would bubble up any error that we'd encouter during retrieving the metadata for all listed entries. Here we relax this somewhat to allow for minor inconsistencies between reading the directory entries and checking whether they are valid key entries.

[ldk-reviews-bot]

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[ldk-reviews-bot]

🔔 1st Reminder

Hey @TheBlueMatt! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

[ldk-reviews-bot]

🔔 2nd Reminder

Hey @TheBlueMatt! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

[TheBlueMatt]

What do we want to do about MigratableKVStore::list_all_keys?

[TheBlueMatt]

I also feel like we need to specify the expected semantics at the API-level here. This change can allow for an inconsistent list view - eg if someone wrote key A then deleted key B the list API can return a value without A and without B, which is weird in some cases.

[tnull]

What do we want to do about MigratableKVStore::list_all_keys?

I don't think we should change anything about that, as for migration we'd want be as strict as possible? For one, users really shouldn't run migration in-parallel to anything else (which is basically already documented on migrate_kv_store_data), and if we hit some inconsistencies the user probably needs to take a look rather than the migration just silently continuing with some items missing.

I also feel like we need to specify the expected semantics at the API-level here. This change can allow for an inconsistent list view - eg if someone wrote key A then deleted key B the list API can return a value without A and without B, which is weird in some cases.

Are we sure this would be the case? I'm not sure if it is honestly. Before we change the docs, we should probably first try to understand better what exactly happened in the user-reported instance.

[TheBlueMatt]

Are we sure this would be the case? I'm not sure if it is honestly. Before we change the docs, we should probably first try to understand better what exactly happened in the user-reported instance.

I think so. We'd first list the directory entries that doesn't have A or B, then the other thread can remove A and write B and the loop will go through, detect that A is missing and ignore it, but never look for B. Absent a full restart of the whole loop and giving up after a while I'm not sure how we'd fix it, though, honestly.

[tnull]

I think so. We'd first list the directory entries that doesn't have A or B, then the other thread can remove A and write B and the loop will go through, detect that A is missing and ignore it, but never look for B. Absent a full restart of the whole loop and giving up after a while I'm not sure how we'd fix it, though, honestly.

Issues like this reminds me that we still need to provide a better default KVStore implementation and discourage FilesystemStore for production use, IMO.