Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] enhancement of microbenchmark #7953

Open
FelixYBW opened this issue Nov 14, 2024 · 0 comments
Open

[VL] enhancement of microbenchmark #7953

FelixYBW opened this issue Nov 14, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@FelixYBW
Copy link
Contributor

FelixYBW commented Nov 14, 2024

Description

  1. Currently if a task failed, the reducer stopped to read data and output to parquet. So the reducer data isn't completed. We need a way to read full data once the partition is enabled.
  2. currently we filter sample by stage id and task id. we need to add filter by partition size as well. @marin-ma can we get the records number from driver before reducer read the data? I'd think so.
@FelixYBW FelixYBW added the enhancement New feature or request label Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant