Add Fallback option `behavior` for dynamic fallback calculation #6464

rickbrouwer · 2025-01-05T17:50:37Z

When you are running a large number of pods and something goes wrong, Keda uses a fallback (if configured). However, the difference between the fallback.replicas and the current number of replicas can be large.
"Something going wrong" can also be temporary, which means it can suddenly scale down considerably and then return to the correct number of replicas.

With this PR, I want to make a new option behavior for dynamic fallback calculation.

You can indicate that it should use the current number of replicas (CurrentReplicasIfHigher ) as fallback if current replicas is higher. If the given fallback.replicas is higher than the current replicas, then the fallback.replicas will be used.
Also you can indicate that it should use the current number of replicas (CurrentReplicasIfLower ) as fallback if current replicas is lower. If the given fallback.replicas is higher than the current replicas, then the current replicas will be used.

A new parameter has been added to the fallback struct (behavior) with three options Static (default), CurrentReplicasIfHigher and CurrentReplicasIfLower.
An adjustment has been made in the fallback.go to retrieve the current number of replicas
In the scale_scaledobjects.go there was code that determined what the current number of replicas was. This has been moved to the scale_resolvers.go
Fallback modified to determine whether to fall back to the fallback.replicas or the current number of replicas

If for any reason this option is not desired, please let me know.

Checklist

I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added
Changelog has been updated and is aligned with our changelog requirements
A PR is opened to update the documentation on (repo) (if applicable)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #6450
Docs: kedacore/keda-docs#1519
Relates to discussion: #6334

Signed-off-by: rickbrouwer <[email protected]>

Signed-off-by: Rick Brouwer <[email protected]>

pkg/scaling/executor/scale_scaledobjects_test.go

Signed-off-by: Rick Brouwer <[email protected]>

wozniakjan

lgtm, I like this a lot!

pkg/fallback/fallback.go

apis/keda/v1alpha1/scaledobject_types.go

Signed-off-by: rickbrouwer <[email protected]>

wozniakjan

a minor suggestion for simplification, I think semantically this should be equivalent, just less code to maintain :)

Let me know if you find any discrepancies

apis/keda/v1alpha1/scaledobject_types.go

Co-authored-by: Jan Wozniak <[email protected]> Signed-off-by: rickbrouwer <[email protected]>

Signed-off-by: Rick Brouwer <[email protected]>

JorTurFer · 2025-01-18T19:28:50Z

Sorry because I'm late, I've posted a comment in the issue about the approach, PTAL and share your thoughts

Signed-off-by: Rick Brouwer <[email protected]>

zroubalik

Great stuff!

Shouldn't we also add currentReplicas mode to always fallback to current replicas no matter if the count is lower or higher?

apis/keda/v1alpha1/scaledobject_types.go

pkg/fallback/fallback.go

pkg/scaling/resolver/scale_resolvers.go

Signed-off-by: Rick Brouwer <[email protected]>

zroubalik

I think that we should also think about e2e test for this feature

Signed-off-by: rickbrouwer <[email protected]>

rickbrouwer · 2025-02-07T14:43:55Z

I think that we should also think about e2e test for this feature

added an e2e test

apis/keda/v1alpha1/scaledobject_types.go

Signed-off-by: rickbrouwer <[email protected]>

wozniakjan · 2025-02-20T16:43:26Z

/run-e2e fallback
Update: You can check the progress here

tests/internals/fallback/fallback_test.go

Signed-off-by: rickbrouwer <[email protected]>

wozniakjan · 2025-02-21T09:03:45Z

/run-e2e fallback
Update: You can check the progress here

Signed-off-by: rickbrouwer <[email protected]>

wozniakjan · 2025-02-21T12:55:47Z

/run-e2e fallback
Update: You can check the progress here

JorTurFer

❤️

zroubalik

LGTM, great stuff!

rickbrouwer added 3 commits January 4, 2025 16:24

Add Fallback option UseCurrentReplicasAsMinimum

1b0d21b

Signed-off-by: rickbrouwer <[email protected]>

Update the tests

67d8be1

Signed-off-by: rickbrouwer <[email protected]>

Update changelog

13c95df

Signed-off-by: Rick Brouwer <[email protected]>

rickbrouwer requested a review from a team as a code owner January 5, 2025 17:50

rickbrouwer marked this pull request as draft January 5, 2025 17:50

semgrep-app bot reviewed Jan 5, 2025

View reviewed changes

pkg/scaling/executor/scale_scaledobjects_test.go Outdated Show resolved Hide resolved

semgrep-app bot reviewed Jan 5, 2025

View reviewed changes

pkg/scaling/executor/scale_scaledobjects_test.go Outdated Show resolved Hide resolved

semgrep-app bot reviewed Jan 5, 2025

View reviewed changes

pkg/scaling/executor/scale_scaledobjects_test.go Outdated Show resolved Hide resolved

Fix

7513c7f

Signed-off-by: Rick Brouwer <[email protected]>

rickbrouwer force-pushed the fallback branch from f6fbcf4 to 7513c7f Compare January 6, 2025 08:23

rickbrouwer added 2 commits January 6, 2025 09:41

Fix

05ccab8

Signed-off-by: Rick Brouwer <[email protected]>

Fix

2bc1117

Signed-off-by: Rick Brouwer <[email protected]>

rickbrouwer marked this pull request as ready for review January 6, 2025 09:17

rickbrouwer mentioned this pull request Jan 6, 2025

Add Fallback option behavior for dynamic fallback calculation kedacore/keda-docs#1519

Merged

1 task

wozniakjan approved these changes Jan 13, 2025

View reviewed changes

pkg/fallback/fallback.go Outdated Show resolved Hide resolved

apis/keda/v1alpha1/scaledobject_types.go Outdated Show resolved Hide resolved

rickbrouwer marked this pull request as draft January 15, 2025 16:31

rickbrouwer force-pushed the fallback branch 3 times, most recently from caa059c to 34abee5 Compare January 15, 2025 18:03

rickbrouwer marked this pull request as ready for review January 15, 2025 18:04

rickbrouwer changed the title ~~Use CurrentReplicas as a minimum for fallback replicas~~ Add Fallback option UseCurrentReplicasAsMinimum Jan 16, 2025

rickbrouwer force-pushed the fallback branch from 34abee5 to e6a0cd2 Compare January 16, 2025 12:52

Update after feedback

c0b1d55

Signed-off-by: rickbrouwer <[email protected]>

rickbrouwer force-pushed the fallback branch from e6a0cd2 to c0b1d55 Compare January 16, 2025 13:22

wozniakjan reviewed Jan 16, 2025

View reviewed changes

apis/keda/v1alpha1/scaledobject_types.go Outdated Show resolved Hide resolved

apis/keda/v1alpha1/scaledobject_types.go Outdated Show resolved Hide resolved

rickbrouwer and others added 3 commits January 17, 2025 15:47

Update apis/keda/v1alpha1/scaledobject_types.go

ea4698d

Co-authored-by: Jan Wozniak <[email protected]> Signed-off-by: rickbrouwer <[email protected]>

Update apis/keda/v1alpha1/scaledobject_types.go

93e29ce

Co-authored-by: Jan Wozniak <[email protected]> Signed-off-by: rickbrouwer <[email protected]>

Update after feedback

6fbcb56

Signed-off-by: Rick Brouwer <[email protected]>

rickbrouwer force-pushed the fallback branch from e8c12c5 to 6fbcb56 Compare January 17, 2025 17:38

rickbrouwer force-pushed the fallback branch from ea4f575 to c7eb211 Compare February 4, 2025 16:52

JorTurFer mentioned this pull request Feb 4, 2025

Release: 2.17.0 #6502

Open

25 tasks

Add comment

f6563ad

Signed-off-by: Rick Brouwer <[email protected]>

zroubalik reviewed Feb 6, 2025

View reviewed changes

apis/keda/v1alpha1/scaledobject_types.go Outdated Show resolved Hide resolved

pkg/fallback/fallback.go Outdated Show resolved Hide resolved

pkg/scaling/resolver/scale_resolvers.go Show resolved Hide resolved

rickbrouwer force-pushed the fallback branch 3 times, most recently from fe1d3a0 to 76d35eb Compare February 6, 2025 14:43

Updates from feedback

c472516

Signed-off-by: Rick Brouwer <[email protected]>

rickbrouwer force-pushed the fallback branch from 76d35eb to c472516 Compare February 6, 2025 14:57

zroubalik reviewed Feb 6, 2025

View reviewed changes

Add e2e test

6545ef7

Signed-off-by: rickbrouwer <[email protected]>

rickbrouwer force-pushed the fallback branch from c6b0d8b to 6545ef7 Compare February 7, 2025 14:30

JorTurFer reviewed Feb 8, 2025

View reviewed changes

apis/keda/v1alpha1/scaledobject_types.go Show resolved Hide resolved

rickbrouwer added 2 commits February 18, 2025 13:46

Merge branch 'main' into fallback

dfe21b0

Signed-off-by: rickbrouwer <[email protected]>

Remove unnecessary tests

5ce7599

Signed-off-by: rickbrouwer <[email protected]>

JorTurFer reviewed Feb 20, 2025

View reviewed changes

tests/internals/fallback/fallback_test.go Show resolved Hide resolved

Add more e2e tests

dd86f7b

Signed-off-by: rickbrouwer <[email protected]>

rickbrouwer force-pushed the fallback branch from 547091b to 3780b83 Compare February 21, 2025 09:03

Add static e2e

aef8b42

Signed-off-by: rickbrouwer <[email protected]>

rickbrouwer force-pushed the fallback branch from 3780b83 to aef8b42 Compare February 21, 2025 09:13

rickbrouwer added 2 commits February 21, 2025 10:25

Adjust failureThreshold

98227b2

Signed-off-by: rickbrouwer <[email protected]>

Change time settings

350af40

Signed-off-by: rickbrouwer <[email protected]>

JorTurFer approved these changes Feb 21, 2025

View reviewed changes

zroubalik approved these changes Feb 22, 2025

View reviewed changes

zroubalik merged commit 4cd84f3 into kedacore:main Feb 22, 2025
20 checks passed

rickbrouwer deleted the fallback branch February 22, 2025 11:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Fallback option `behavior` for dynamic fallback calculation #6464

Add Fallback option `behavior` for dynamic fallback calculation #6464

rickbrouwer commented Jan 5, 2025 •

edited

Loading

wozniakjan left a comment

wozniakjan left a comment

JorTurFer commented Jan 18, 2025

zroubalik left a comment

zroubalik left a comment

rickbrouwer commented Feb 7, 2025

wozniakjan commented Feb 20, 2025 •

edited by github-actions bot

Loading

wozniakjan commented Feb 21, 2025 •

edited by github-actions bot

Loading

wozniakjan commented Feb 21, 2025 •

edited by github-actions bot

Loading

JorTurFer left a comment

zroubalik left a comment

Add Fallback option behavior for dynamic fallback calculation #6464

Add Fallback option behavior for dynamic fallback calculation #6464

Conversation

rickbrouwer commented Jan 5, 2025 • edited Loading

Checklist

wozniakjan left a comment

Choose a reason for hiding this comment

wozniakjan left a comment

Choose a reason for hiding this comment

JorTurFer commented Jan 18, 2025

zroubalik left a comment

Choose a reason for hiding this comment

zroubalik left a comment

Choose a reason for hiding this comment

rickbrouwer commented Feb 7, 2025

wozniakjan commented Feb 20, 2025 • edited by github-actions bot Loading

wozniakjan commented Feb 21, 2025 • edited by github-actions bot Loading

wozniakjan commented Feb 21, 2025 • edited by github-actions bot Loading

JorTurFer left a comment

Choose a reason for hiding this comment

zroubalik left a comment

Choose a reason for hiding this comment

Add Fallback option `behavior` for dynamic fallback calculation #6464

Add Fallback option `behavior` for dynamic fallback calculation #6464

rickbrouwer commented Jan 5, 2025 •

edited

Loading

wozniakjan commented Feb 20, 2025 •

edited by github-actions bot

Loading

wozniakjan commented Feb 21, 2025 •

edited by github-actions bot

Loading

wozniakjan commented Feb 21, 2025 •

edited by github-actions bot

Loading