Skip to content

Conversation

@vinkal-chudgar
Copy link

@vinkal-chudgar vinkal-chudgar commented Oct 16, 2025

Motivation

Fixes #24693.

ResourceGroupService unconditionally schedules two periodic tasks (aggregating local usage and calculating quotas) during broker initialization, executing them every resourceUsageTransportPublishIntervalInSecs (default 60 seconds) regardless of whether any tenants or namespaces are registered to resource groups. This wastes resources on brokers where the resource group feature is unused.

This PR implements explicit lifecycle management: periodic tasks now start only when the first tenant or namespace is registered to any resource group, and stop when the last registration is removed. This eliminates unnecessary periodic task execution and resource consumption on brokers where the resource group feature is idle.

Modifications

  • Lazy scheduling: Periodic tasks (aggregateResourceGroupLocalUsages, calculateQuotaForAllResourceGroups) are no longer unconditionally started in initialize(). They start only when tenant or namespace registrations are added (via registerTenant/registerNameSpace calls) and stop when the last registration is removed.
  • Automatic cleanup: Tasks stop automatically when the last attachment is removed, eliminating background work when resource groups become unused.
  • CAS-guarded concurrency: Task lifecycle is controlled via an AtomicBoolean with compare-and-set to avoid double start/stop and blocking.
  • Dynamic config: Existing dynamic rescheduling based on resourceUsageTransportPublishIntervalInSecs is preserved; guards prevent reschedule attempts when schedulers are stopped.
  • Docs: Updated the ServiceConfiguration field doc to reflect “run only when in use” and to clarify behavior when a ResourceUsageTransportManager is configured.

Modified files

  • ResourceGroupService.java: add maybeStartSchedulers(), maybeStopSchedulersIfIdle(), hasActiveAttachments(), plus small guard/logging improvements.
  • ServiceConfiguration.java: clarify resourceUsageTransportPublishIntervalInSecs doc.
  • ResourceGroupServiceTest.java: add new tests and adapt existing tests for lazy lifecycle.

Verifying this change

  • Make sure that the change passes the CI checks.

This change is verified by unit tests. Tests are sleep free, run with per test isolation, and use 60s timeouts.

New unit tests (ResourceGroupServiceTest.java)

  • testLazyStartStopAndReschedule() verifies lazy start, deterministic reschedule on interval change, and stop on last detach.
  • testNoStartOnRGCreateOnly() ensures no schedulers start when a resource group has no attachments and direct calls do not auto start.
  • testStartOnTenantAttachment() confirms that a tenant only attachment starts both schedulers and detaching the last tenant stops them.
  • testStopOnLastDetachWithMixedRefs() checks that schedulers keep running while any attachment exists and stop only after the last is removed.
  • testNoRescheduleWhenStopped() ensures interval changes do not schedule tasks when the service is idle.

Updated unit tests (ResourceGroupServiceTest.java)

  • testClose(): start schedulers via an attachment, call close(), then assert scheduled task references are set to null and isSchedulersRunning() is false.

  • testResourceGroupOps(): The test now calls calculateQuotaForAllResourceGroups() before unregistering all attachments, rather than after.

Test utilities

  • isSchedulersRunning() accessor annotated with com.google.common.annotations.VisibleForTesting.
  • createResourceGroupService() helper that returns a fresh ResourceGroupService(pulsar) per test.

Personal CI Results

Tested in Personal CI fork: vinkal-chudgar#1

Status:

  • 43 checks passed (build, tests, validation)
  • 3 coverage upload jobs failed due to missing Codecov token (expected in forks)
  • All core test suites passed successfully

Evidence: https://github.com/vinkal-chudgar/pulsar/pull/1/checks

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Configuration note: Only the documentation string for resourceUsageTransportPublishIntervalInSecs was updated. No default values were changed.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Note: Inline configuration documentation was updated in ServiceConfiguration.java. No website docs need changes

Matching PR in forked repository

PR in forked repository: vinkal-chudgar#1

…idle

ResourceGroupService previously executed periodic tasks unconditionally
every resourceUsageTransportPublishIntervalInSecs. This change starts both
tasks only when at least one tenant or namespace is attached to any
resource group, and stops them when no attachments remain. Each task
reschedules itself if the interval changes at runtime. close() now cancels
and clears both futures.

Fixes apache#24693

Signed-off-by: Vinkal Chudgar <[email protected]>
Signed-off-by: Vinkal Chudgar <[email protected]>
@github-actions
Copy link

@vinkal-chudgar Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

@github-actions github-actions bot added doc-not-needed Your PR changes do not impact docs and removed doc-label-missing labels Oct 16, 2025
@vinkal-chudgar vinkal-chudgar marked this pull request as ready for review October 16, 2025 14:30
@lhotari
Copy link
Member

lhotari commented Oct 16, 2025

Thanks for the great contribution! I didn't do a full review yet. A few minor comments.

@vinkal-chudgar
Copy link
Author

Thanks for the great contribution! I didn't do a full review yet. A few minor comments.

Thanks for the quick look! I'll address the comments when available.

@vinkal-chudgar
Copy link
Author

@lhotari - When convenient, could you please review this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] ResourceGroupService requests all topic stats every 60 seconds by default even when it shouldn't be enabled

2 participants