fix: do not create many redis subscriptions per token - fast yielding background

[benedikt-bartscher]

No description provided.

[benedikt-bartscher]

@masenf any idea why fast yielding background tasks seem to lock up after some state updates?

[benedikt-bartscher]

@masenf i accidentally pushed unfinished changes and forgot to mention that the test needs to be run with redis

[benedikt-bartscher]

I pushed 389f4c7 which fixes the issue.
Another idea to improve this even further: one shared subscription per token and worker (if I am not mistaken, we basically need to lock a token to a worker on redis, further locking can be done with asyncio). Reuse the subscription and release/garbage collect it after a delay.

However, I would merge this as-is, because it works and does not change the behavior much. I would then try implementing the other idea in a followup.

[benedikt-bartscher]

This is probably wrong. Clients can reconnect. Need to check if there are any unexpired redis tokens

[masenf]

edit: oops i thought this was the StateManagerMemory.disconnect func; either way we need to be careful about reconnects, which can be triggered by a simple page refresh

i had a draft comment on my in progress review calling out this function, but i suppose your testing found it first. we need to keep the in memory states around. we do have an internal bug to add an expiry for memory state manager, but it just hasn't been high priority because we assume no one is using memory state manager in production

[benedikt-bartscher]

Maybe one asyncio Lock is enough, we won't need to garbage collect then

[benedikt-bartscher]

Note to myself: subscribe to redis expire events to clean up locks, may be easier with #4459

[benedikt-bartscher]

@masenf is it true that the current state locking does not guarantee fifo? Imo the longest waiting task should get the lock next

[benedikt-bartscher]

I have a local draft where only one pubsub instance is created per reflex worker. All lock releases are distributed to the waiting tasks via asyncio locks

[masenf]

Closing in favor of #5932, which was inspired by this PR and also incorporates this concept:

Another idea to improve this even further: one shared subscription per token and worker (if I am not mistaken, we basically need to lock a token to a worker on redis, further locking can be done with asyncio). Reuse the subscription and release/garbage collect it after a delay.

@benedikt-bartscher if you have some cycles to test on #5932 i'd appreciate any input you have. I'm working on some test cases for it, but since this isn't my main project at the moment, it might have to wait until next weekend.