Big Data Transfer From S3 to Google Cloud Bucket #44
Description
Hi,
We are trying to transfer some of the 2025 building profiles from oedi-data-lake/nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/2025/comstock_amy2018_release_2/timeseries_individual_buildings to our cloud instance with Google to conduct an internal benchmarking analysis.
By design GCP requires us to provide credentials or to have an IAM role setup (reading permission for the bucket) even if that S3 bucket is public. Given that this represents terabytes of data, we'd like to be able to leverage Google's optimized transfer capabilities.
I am curious if you had similar requests in the past and how you ended up proceeding.
So far, our attempts at configuring rclone as an alternative have been unfruitful.
Many thanks in advance,
Anabel for the Audette team