Race condition causes metric updates to be lost #132
Comments
pwinckles
added a commit
to pwinckles/parfait
that referenced
this issue
Nov 17, 2024
Fixes a race condition where metric updates can be lost if the happen while `PcpMmvWriter` is being reset. Previously, updates were dropped if the writer was not in a started state. If this happened while the the writer was executing `start()`, then it's possible the updated value would be lost if the old metric value had already been written. The issue is addressed here by blocking the metric update when the writer is in a `STARTING` state, until the writer has been fully started. If the writer is in a `STARTED` state, the update is applied immediately. If the writer is in a `STOPPED` state, the update is dropped immediately. Resolves performancecopilot#132
|
Unfortunately, my prior PR only partially addressed the issue. The race condition still exists if a monitorable is updated before the writer is started again but after the monitorable has been re-added to the writer. I hope to get a PR up to fix this case tomorrow. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If a metric is updated while
QuiescentRegistryListeneris restartingPcpMmvWriterdue to a monitorable being added, then it's possible that the metric update will be lost. This can happen because the metric update callsPcpMmvWriter.updateMetric(), which immediately returns ifstartedisfalse. WhenQuiescentRegistryListenerrestartsPcpMmvWriter, it first stops it, settingstartedtofalse, and then starts it again.PcpMmvWriter.start()iterates over all of the metric data, writing it again, prior to settingstartedtotrue. So, the race occurs if a metric is updated between the time the old value is rewritten inPcpMmvWriter.start()and the timestartedis set totrue.The text was updated successfully, but these errors were encountered: