Add e2e test make sure resource quota error is surfaced #3087
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
han-steve
changed the title
|
|
|
Another question, since each integration test creates its own namespace, is there a way to run the integration tests in parallel? I see we do |
|
@rueian would you mind reviewing this PR? |
rueian
reviewed
Feb 22, 2025
Comment on lines
+143
to
+154
| g.Eventually(func() bool { | ||
| rc, err := RayCluster(test, namespace.Name, rayCluster.Name)() | ||
| if err != nil { | ||
| return false | ||
| } | ||
| for _, condition := range rc.Status.Conditions { | ||
| if condition.Type == "ReplicaFailure" && strings.Contains(condition.Message, "forbidden: exceeded quota") { | ||
| return true | ||
| } | ||
| } | ||
| return false | ||
| }, TestTimeoutShort).Should(BeTrue(), "Expected ReplicaFailure condition with message containing 'forbidden: exceeded quota'") |
There was a problem hiding this comment.
Hi @han-steve,
Thank you for the new test!
Suggested change
| g.Eventually(func() bool { | |
| rc, err := RayCluster(test, namespace.Name, rayCluster.Name)() | |
| if err != nil { | |
| return false | |
| } | |
| for _, condition := range rc.Status.Conditions { | |
| if condition.Type == "ReplicaFailure" && strings.Contains(condition.Message, "forbidden: exceeded quota") { | |
| return true | |
| } | |
| } | |
| return false | |
| }, TestTimeoutShort).Should(BeTrue(), "Expected ReplicaFailure condition with message containing 'forbidden: exceeded quota'") | |
| g.Eventually(RayCluster(test, namespace.Name, rayCluster.Name), TestTimeoutMedium). | |
| Should(WithTransform(StatusCondition(rayv1.RayClusterReplicaFailure), MatchCondition(metav1.ConditionTrue, ...))) |
Could you also help us extend the existing MatchCondition or add a new matcher that can match condition message?
I think it could be a good try but should be careful about resource control for the stability of the tests themselves. Log isolation for easy debugging might be another issue to be considered. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Thanks for adding the new conditions to the RayCluster!
Small PR to add a quick integration test to make sure that we surface the resource quota error for better observability.