Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More flexible PR experiment requests. #830

Open
1 task
DonggeLiu opened this issue Mar 2, 2025 · 4 comments
Open
1 task

More flexible PR experiment requests. #830

DonggeLiu opened this issue Mar 2, 2025 · 4 comments
Assignees

Comments

@DonggeLiu
Copy link
Collaborator

  1. Pass arbitrary flags to run_all_experiments.py (Support passing arbitrary flags to run_all_experiments.py via request_pr_exp.py.  #245).
  2. Customize more fields in [oss-fuzz-gen/ci/k8s/pr-exp.yaml](https://github.com/google/oss-fuzz-gen/blob/main/ci/k8s/pr-exp.yaml). E.g.:
@Snehil-Shah
Copy link

@DonggeLiu I would like to help with the first issue. Can I be assigned?

@DonggeLiu
Copy link
Collaborator Author

@DonggeLiu I would like to help with the first issue. Can I be assigned?

Sure, thanks!

@Snehil-Shah
Copy link

@DonggeLiu for some clarification, the goal with the second task is to dynamically set the VERTEX_AI_LOCATIONS variable in the yaml file based on the model, right? (which is currently set to all locations?).

@DonggeLiu
Copy link
Collaborator Author

DonggeLiu commented Mar 9, 2025

@DonggeLiu for some clarification, the goal with the second task is to dynamically set the VERTEX_AI_LOCATIONS variable in the yaml file based on the model, right? (which is currently set to all locations?).

Yes.
You can probably hardcode the model-locations pairs in ci/request_pr_exp.py to begin with, then also allow users to overwrite locations.
For example, we can support both:

# Use the default location set of model_1
/gcbrun exp ... --llm model_1

# Use the specified locations (l1, l2, l3)
/gcbrun exp ... --llm model_1 --llm-locations l1,l2,l3

Note the current location flag is for GKE, not models.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants