Skip to content
This repository was archived by the owner on Apr 20, 2020. It is now read-only.

BlockwiseFileset: Avoid creating empty files #168

Open
stuarteberg opened this issue Feb 18, 2015 · 0 comments
Open

BlockwiseFileset: Avoid creating empty files #168

stuarteberg opened this issue Feb 18, 2015 · 0 comments

Comments

@stuarteberg
Copy link
Member

The BlockwiseFileset class creates a lot of small or empty files that are used for bookkeeping purposes. On HPC storage, metadata performance on such files introduces a large overhead. Somehow these book-keeping items need to be kept somewhere else. Probably a simple database or sqlite store would be fine, provided it can handle a bit of concurrency.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant