Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster data transfers from MARS. #235

Merged
merged 10 commits into from
Jan 4, 2023
Merged

Faster data transfers from MARS. #235

merged 10 commits into from
Jan 4, 2023

Conversation

alxmrs
Copy link
Collaborator

@alxmrs alxmrs commented Sep 19, 2022

I'm taking a leaf from @mahrsee1997's PR #148 so that we can copy data from the MARS server faster (using a larger buffer size).

Thanks for the primary contribution here, Rahul.

@alxmrs alxmrs marked this pull request as ready for review September 19, 2022 23:33
@alxmrs alxmrs marked this pull request as draft September 20, 2022 03:49
@mahrsee1997
Copy link
Collaborator

mahrsee1997 commented Sep 21, 2022

@alxmrs
After thinking a bit on this/discussing with the team -- what if ECMWF server outbound bandwidth is itself capped at some constant rate (too low), in that case this approach would be of no help.

Maybe we can wait until they are back online and do a POC to check their server outbound bandwidth. WDYT ?

@alxmrs
Copy link
Collaborator Author

alxmrs commented Sep 21, 2022

That's sounds very reasonable to me. Good insight.

@alxmrs
Copy link
Collaborator Author

alxmrs commented Dec 2, 2022

@mahrsee1997 @ksic8 Let's revisit this CL now that ECMWF is back online.

@alxmrs alxmrs marked this pull request as ready for review December 2, 2022 02:26
@alxmrs
Copy link
Collaborator Author

alxmrs commented Jan 4, 2023

This LGTM -- I cannot approve this CL since I am also an author.

Copy link
Collaborator

@fredzyda fredzyda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can approve it if Alex can't.

@alxmrs
Copy link
Collaborator Author

alxmrs commented Jan 4, 2023

Thanks Fred!

@alxmrs alxmrs merged commit 7cae996 into main Jan 4, 2023
@alxmrs alxmrs deleted the dl-faster-copy branch January 4, 2023 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants