Fast way to check if a variable is in a list of netCDF files? Or load only the files that has a list of variables.

[iuryt]

Let's say I have a long list of files all_fnames and I want to get only those that have a variable param

import xarray as xr
from tqdm import tqdm
fnames = [fname for fname in tqdm(all_fnames) if param in list(xr.open_dataset(fname))]

This is returning

100%|██████████| 1295/1295 [00:33<00:00, 38.56it/s]

Is there a faster way to do that? Any suggestions?

My idea is actually to use open_mfdataset with a path with a wildcard * and load only the files that have all from a list of variables.

[iuryt]

Using netCDF4 package is faster, but still, I would like to see if there is a better way doing it with xarray

from netCDF4 import Dataset
fnames = [fname for fname in tqdm(all_fnames) if param in Dataset(fname).variables]

which returns

100%|██████████| 1295/1295 [00:12<00:00, 101.96it/s]

[dcherian]

There isn't. IMO the real solution is your problem is intake-esm or intake and/or kerchunk

[kmuehlbauer]

Using netCDF4 package is faster, but still, I would like to see if there is a better way doing it with xarray

from netCDF4 import Dataset
fnames = [fname for fname in tqdm(all_fnames) if param in Dataset(fname).variables]

which returns

100%|██████████| 1295/1295 [00:12<00:00, 101.96it/s]

You might check with h5netcdf or even just h5py to retrieve dataset/variable names. But of course only for hdf5 based netcdf flavours.

Update:

This should work. Normally it should be in the same range as netCDF4, maybe a bit faster.

import h5py
fnames = [fname for fname in tqdm(all_fnames) if param in h5py.File(fname).keys()]

Not sure if and when the open file handles go out of scope and are closed finally.

[iuryt]

Sorry for taking so long to answer.
I tested your solution with mine.

fnames = [fname for fname in tqdm(all_fnames) if param in Dataset(fname).variables]

gives 113.47it/s,

and yours

fnames = [fname for fname in tqdm(all_fnames) if param in h5py.File(fname).keys()]

gives incredible 3073.24it/s!

Thus, I am marking this as the answer.

[iuryt]

@dcherian
Do you think something like this could be implemented to xarray?
Maybe when selecting the variables while using open_mfdataset. Not sure if data_vars would be the correct argument, but we could check and give a warning when there are files that don't have one of those variables.