Fix for deadlock issue #3682

[rajagopalanand]

What does this PR do?

This PR is a fix for prometheus#3682. In some instances, mem.Alerts.Subscribe() and store.gc() can get deadlocked

1. Lock acquisition in store.Alerts.gc():
   • The method store.Alerts.gc() acquires a lock on its internal mutex store.Alerts.mtx
   • While holding the lock, it then calls the callback function which will try to acquire a lock on mem.Alerts.mtx
2. Concurrent Execution:
   • mem.Alerts.Subscribe() acquires a lock on its internal mutex mem.Alerts.mtx and calls store.Alerts.List()
3. Deadlock situation:
   • Callback function tries to acquire a lock on mem.Alerts.mtx. However this lock is already being held by mem.Subscribe()
   • Similarly mem.Subscribe() cannot proceed because store.List() cannot acquire lock (store.Alerts.mtx) because it is being held by store.gc()

Another way of summarizing this is store.Alerts.gc() was holding the lock until callback function completed which in turn was waiting to acquire the lock. Callback function could not acquire the lock because Subscribe() was holding the lock. Subscribe() cannot progress because it calls store.Alerts.List() which was waiting for lock acquisition which was being held by store.Alerts.gc(). This fix releases the lock held by store.Alerts.gc() prior to calling the callback function

[emanlodovice]

lgtm