fix(state/in-memory): avoid deadlock between Close and cleanup goroutine

[karprabha]

Description

Close() calls wg.Wait() while holding the write lock. The cleanup goroutine
started by Init() needs that same lock in doCleanExpiredItems(). If the
cleanup ticker fires in the window between Close() acquiring the lock and the
goroutine reaching Lock(), the two wait on each other and Close() never
returns.

This is a permanent hang — no timeout, no error, no retry. Under Kubernetes it
surfaces as a sidecar stuck terminating until SIGKILL, with nothing in the logs.

Fix: wait for the cleanup goroutine to exit before acquiring the lock.
Once wg.Wait() returns, nothing can contend for it.

Introduced in d098e38 (#2474, Feb 2023).

Reproduction

The window is microseconds wide, so it reproduces under scheduling pressure
rather than precise timing: many stores closed concurrently, timed to coincide
with their cleanup tick. It is not single-core specific — measured on a 10-core
machine, 6 rounds per configuration:

GOMAXPROCS	stores	before	after
1	2000	6/6 deadlocked	0/6
2	2000	4/6	0/6
4	2000	5/6	0/6
10	2000	3/6	0/6
10	4	1/6	0/6

Goroutine dump at the deadlock:

goroutine [sync.Mutex.Lock]:
  ...doCleanExpiredItems()  in_memory.go:444   <- blocked on store.lock
goroutine [sync.WaitGroup.Wait]:
  ...Close()                in_memory.go:90    <- holds store.lock

Cleanup ticker now uses the injected clock

startCleanThread used time.After directly, so the cleanup tick ignored the
clock injected into the store. It now uses store.clock.After, with the interval
hoisted into a package constant shared with the test.

This means a fake clock now governs the sweeper as well - fakeClock.Step in
TestReadAndWrite drives a sweep, so tests get full control of the component's
time rather than a background goroutine ticking on real time underneath them.
Existing tests pass unchanged.

Issue reference

Closes #4495

Checklist

• Code compiles correctly
• Created/updated tests
• Extended the documentation
  • Created the dapr/docs PR: N/A

No documentation changes are required: this is an internal concurrency fix with
no change to component behavior, configuration metadata, or public API.

[mikeee]

Could you please consider including the proposed changes to startCleanThread in this PR to reduce the CI run load?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 32.14%. Comparing base (cc03682) to head (6c3d2f0).
⚠️ Report is 25 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4496      +/-   ##
==========================================
+ Coverage   31.94%   32.14%   +0.20%     
==========================================
  Files         353      353              
  Lines       47723    37916    -9807     
==========================================
- Hits        15243    12188    -3055     
+ Misses      31277    24524    -6753     
- Partials     1203     1204       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[karprabha]

Done, pushed. startCleanThread now uses store.clock.After with the interval as a package constant, so the sweeper follows the injected clock like the rest of the component. Tests get full control of the component's time now - fakeClock.Step in TestReadAndWrite drives a sweep. Existing tests pass unchanged; -race and make lint clean.

Separately, pubsub.mqtt3 certification failed. This only touches state/in-memory and state.in-memory conformance passed, so it looks unrelated.

[karprabha]

Hi @mikeee I've made the said changes Kindly review it again