Skip to content

Should we use a fixed seed for random in tests? #684

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
5 tasks done
allrob23 opened this issue Apr 1, 2025 · 2 comments
Open
5 tasks done

Should we use a fixed seed for random in tests? #684

allrob23 opened this issue Apr 1, 2025 · 2 comments
Labels
bug This points to a verified bug in the code

Comments

@allrob23
Copy link

allrob23 commented Apr 1, 2025

Checklist

  • I have looked into the Readme and Examples, and have not found a suitable solution or answer.
  • I have looked into the API documentation and have not found a suitable solution or answer.
  • I have searched the issues and have not found a suitable solution or answer.
  • I have searched the Auth0 Community forums and have not found a suitable solution or answer.
  • I agree to the terms within the Auth0 Code of Conduct.

Description

I was checking out the code in auth0/rest.py and noticed that _calculate_wait uses randint to add jitter to retry delays

auth0-python/auth0/rest.py

Lines 243 to 251 in 10555cb

def _calculate_wait(self, attempt: int) -> int:
# Retry the request. Apply a exponential backoff for subsequent attempts, using this formula:
# max(MIN_REQUEST_RETRY_DELAY, min(MAX_REQUEST_RETRY_DELAY, (100ms * (2 ** attempt - 1)) + random_between(1, MAX_REQUEST_RETRY_JITTER)))
# Increases base delay by (100ms * (2 ** attempt - 1))
wait = 100 * 2 ** (attempt - 1)
# Introduces jitter to the base delay; increases delay between 1ms to MAX_REQUEST_RETRY_JITTER (100ms)
wait += randint(1, self.MAX_REQUEST_RETRY_JITTER())

This randomness could make tests flaky, especially if they depend on timing or retry behavior, since the wait time varies between runs.

Fixing a seed (e.g., random.seed(42) in a test fixture) would make test outcomes consistent and easier to debug. Without it, we risk intermittent failures in CI or locally, which could slow down development. (Im not saying this problems is happening, but it can happen). I didn't see any seeds being set into the test code...

What do you think about adding this to the test setup? I’d be happy to submit a PR if you’re open to it!

One quick question: I saw pytest-randomly in poetry.lock under test-randomorder—does that already handle seeds in some way I might have missed?

Thanks for your input!

Reproduction

.

Additional context

No response

auth0-python version

master branch

Python version

3.12.1

@allrob23 allrob23 added the bug This points to a verified bug in the code label Apr 1, 2025
@kishore7snehil
Copy link
Contributor

kishore7snehil commented May 8, 2025

@allrob23 Please feel free to open a PR. I can review it and will update you acordingly.

@allrob23 allrob23 mentioned this issue May 8, 2025
6 tasks
@allrob23
Copy link
Author

allrob23 commented May 8, 2025

@kishore7snehil I opened the PR, can you please review? #690

I created conftest.py for each test folder

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This points to a verified bug in the code
Projects
None yet
Development

No branches or pull requests

2 participants