Locally running weather-dl creates empty grib files

[aperaza-bsc]

To understand how to reproduce the public dataset arco-era5 as explained in its open-sourced repository, I am trying to use weather-dl --local-run using the following config file.

[parameters]
client=cds
dataset=reanalysis-era5-complete
target_path={date}_hres_sfc.grb2
partition_keys=
    date

[parameters.a]
api_key=YourCdsKey

[selection]
class=ea
stream=oper
expver=1
levtype=sfc
type=an

date=19590101
time=00/to/23

# z, msl, 10u, 10v
param=129.128/151.128/165.128/166.128

Running weather-dl creates a single CDS request that is completed successfully. Still, the corresponding file in my filesystem has a size of zero as discussed in #261. I have tried this download multiple times without success, the local file is always empty even if the CDS request went. Because of that, I understand that even if a retrial was implemented, it might never succeed.

Do you happen to know what could be causing this issue?

Kindly looking forward to your response,
Alejandro

[mahrsee1997]

Hi Alejandro,

We will look into the issue & revert back. Thanks.

[mahrsee1997]

Also what is the api_url, that you are specifying for this download ? Also mind sharing the entire command you used for downloading.
ref: https://github.com/google/weather-tools/blob/main/configs/era5_example_config.cfg#L23

[dabhicusp]

Hello @aperaza-bsc I tried with & without --local-run flag and it's working flawlessly in my case.

Also in your config I can't find api_url=https://cds.climate.copernicus.eu/api inside [parameters.a] section so inside this file .cdsapirc of your system this line(url=https://cds.climate.copernicus.eu/api) must be there. Can you check this once?

Command: python weather_dl/weather-dl weather-tools/demo_cnf.cfg

Config:

[parameters]
client=cds
dataset=reanalysis-era5-single-levels

target_path={date}_hres_sfc.grb2
partition_keys=
    date

[selection]
class=ea
stream=oper
expver=1
levtype=sfc
type=an

date=19590101
time=00/to/23

# z, msl, 10u, 10v
param=129.128/151.128/165.128/166.128

[parameters.a]
api_key=XXXXXXX

Let me know if you still facing the error i am here to help you otherwise you can close this issue 😄 .

[aperaza-bsc]

Hi @mahrsee1997 and @dabhicusp thanks a lot for the quick responses 😄

About the api_url it's not exactly that one as CDS now changed the API and I think that one should be deprecated and my .cdsapirc now contains this one https://cds-beta.climate.copernicus.eu/api. I hadn't included it in the config file since the requests were reaching CDS and were going fine. I have tried adding it today but the result was the same.

For reference this is how I exactly ran weather-dl

weather-dl ~/git/climate-emulators/arco-era5-cfgs/sfc_raw.cfg --local-run

Inside a conda environment with python 3.8.13 with google-weather-tools=0.3.2.

Today I quickly tried directly with python weather_dl/weather-dl inside the repository but I'm seeing errors like the following.

WARNING 24-11-13 09:40:24 apache_beam.utils.retry: Retry with exponential backoff: waiting for 204.3839138173621 seconds before retrying retrieve because we caught exception: AttributeError: 'LegacyApiClient' object has no attribute 'fetch'

For context, I am exploring the possibility of having the ERA5 dataset in Zarr available in the MareNostrum5 HPC. As the compute nodes do not have internet access, we need to have the data in our filesystem. For that, I am trying first to download a subset of the ar arco-era5 data as it is explained in its repository before attempting to recreate something similar from raw grib data. That is the reason why I am trying to get weather-dl working with the local-run option

[dabhicusp]

@aperaza-bsc

I think that one should be deprecated and my .cdsapirc now contains this one https://cds-beta.climate.copernicus.eu/api.

• actually cds-beta deprecated on 20th Oct. Reference: https://forum.ecmwf.int/t/goodbye-legacy-climate-data-store-hello-new-climate-data-store-cds/6380

Also you used very old version of google-weather-tools as 0.3.2 version is published on Jun 27,2022.

• Can you fork the repo of weather-tools and create an conda env using of this commands: https://github.com/google/weather-tools?tab=readme-ov-file#installing.

Please run weather-dl once again after doing the above stuff and let me know if you faced issue still.

[aperaza-bsc]

Thanks for the help @dabhicusp

My bad on cds-beta, didn't notice that it already left the beta stage. Thanks for pointing that out. Only changing that did not solve the issue.

Building the environment and using a newer version of the package solved it 😄

[dabhicusp]

Yes.
One of the reason of it is the released version of weather-tools(0.3.2.) is not compatible with the cds-api but the current main branch's code is compatible so you must need to built the environment using of the main branch's code.

[nmeijers31]

Hi @dabhicusp , I am actually facing the same issue:

• With version 0.3.2 and dataflowrunner, I was getting empty files
• With the latest version of weather-dl, installed as mentioned in the readme from the forked repo & compatible with the new cds, I am able to trigger api & downloads locally but not with dataflowrunner. With dataflowrunner, the job is launched but is stuck in the create-config/prep-partition step...