proposal: Remote Write: Restart from segment-based savepoint by kgeckhart · Pull Request #72 · prometheus/proposals

kgeckhart · 2025-12-18T20:05:14Z

I am proposing a solution to allow remote write to restart from a savepoint (avoiding the use of checkpoint as there's already a checkpoint involved in remote write which does not help). This is a long standing issue that has seen a few attempts with none quite being accepted. It includes starting with a more basic solution which can be iterated to eventually get to at-least-once delivery.

Related to: prometheus/prometheus#8809

Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

…heckpoint Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

bwplotka

Nice!

Thanks, this is super needed. Added some questions, but we need to play with a different strategies here. WAL segments based is a cool idea!

I'd recommend we stop talking about exact code components, it might add too much detail to the discussion (subjective opinion).

bwplotka · 2026-02-16T18:25:48Z

+### Pitfalls of the current solution
+
+As mentioned in the why, this behavior is often confusing to users who know a WAL is in use but still finds they have missing data on restart.


Suggested change

### Pitfalls of the current solution

As mentioned in the why, this behavior is often confusing to users who know a WAL is in use but still finds they have missing data on restart.

Probably dup?

bwplotka · 2026-02-16T18:26:44Z

+1. Support resuming from a savepoint for each configured `remote_write` destination.
+2. Taking a savepoint for a remote_write destination should not incur significant overhead.
+3. Changing the `queue_configuration` for a `remote_write` destination should not result in a new savepoint entry.
+   * The `queue_configuration` includes fields like min/max shards and other performance tuning parameter.s


Suggested change

* The `queue_configuration` includes fields like min/max shards and other performance tuning parameter.s

* The `queue_configuration` includes fields like min/max shards and other performance tuning parameters.

bwplotka · 2026-02-16T18:27:57Z

+
+## Goals
+
+1. Support resuming from a savepoint for each configured `remote_write` destination.


Some questions:

Do you propose it to be by default or optional?

What to do for users who prefer fresh data over persisting old data?

What to do for users who want persistence, but WAL grown so much they will 100% fail to catch up, so it's better to drop?

Do you propose it to be by default or optional?

Yes at some point in the future I think this should become the default option but I think we'll need to evolve some default config options (and potentially introduce new ones) to make it a safe transition.

What to do for users who prefer fresh data over persisting old data?

What to do for users who want persistence, but WAL grown so much they will 100% fail to catch up, so it's better to drop?

Thinking about this in terms of what users can do today to prevent these things. A user can use remote_write.queue_config.sample_age_limit to try to keep 2 in check, I think it's the same answer for 3. Today you cannot modify this setting "online", changing it causes us to dump the WAL so I think the answer mostly ends up being "restart it".

After we have a segment based savepoint, I think this setting is still the most valuable tool to prevent unsustainable WAL growth and manage the tradeoff between freshness and completeness of data. I suggested that enabling savepoint should come with adding a default value of 2 hours. Couple this with the adjustment to ensure queue config won't trigger a replay a user can drop this value lower to catch-up and put it back. WDYT?

bwplotka · 2026-02-16T18:28:38Z

+
+1. Support resuming from a savepoint for each configured `remote_write` destination.
+2. Taking a savepoint for a remote_write destination should not incur significant overhead.
+3. Changing the `queue_configuration` for a `remote_write` destination should not result in a new savepoint entry.


Do you mean?

Suggested change

3. Changing the `queue_configuration` for a `remote_write` destination should not result in a new savepoint entry.

3. Changing the `queue_configuration` for a `remote_write` destination should not result in losing a savepoint entry.

bwplotka · 2026-02-16T20:56:39Z

+
+## Alternatives
+
+1. `remote.QueueManager` should own syncing its own savepoint (most early implementations took this approach).


I am lost here, should we avoid diving into too much of code architecture in this proposal?

e.g remote.WriteStorage sounds fancy, but it's a literally trivial HTTP client for remote write AFAIK. If we do save point there or on caller side.. it's fine to decide during implementation.

Eventually if feels this style might hard to review by maintainers who didn't look on current Prometheus RW sending code recently. Brining it to high level design (shard, queues, WAL watching) might give more chances someone else will help in review (:

bwplotka · 2026-02-16T20:57:21Z

+1. `remote.QueueManager` should own syncing its own savepoint (most early implementations took this approach).
+   * `remote.QueueManager` already has a lot of responsibilities and will take on more for at-least-once.
+   * `remote.WriteStorage` has reasonable hook points to run this logic without adding a lot more  complexity.
+2. The savepoint should be synchronously updated when segments change.


Suggested change

2. The savepoint should be synchronously updated when segments change.

2. The savepoint should be synchronously updated when segments change during WAL watching.

bwplotka · 2026-02-16T21:00:41Z

+   * `remote.QueueManager` already has a lot of responsibilities and will take on more for at-least-once.
+   * `remote.WriteStorage` has reasonable hook points to run this logic without adding a lot more  complexity.
+2. The savepoint should be synchronously updated when segments change.
+   * Introducing a bit of time between knowing that a segment changed to persisting it gives us more time to fully deliver the batch before we persist the change.


Can we split into Pros and Cons?

The pro is simpler implementation. Plus given queues have limit (they can be full) WAL watching tracking could be good enough persistence. Not saying this is the best option - but something to consider.

bwplotka · 2026-02-16T21:03:34Z

+
+### Code flow
+
+1. Adding another configurable timer to [`remote.WriteStorage.run()`](https://github.com/prometheus/prometheus/blob/f50ff0a40ad4ef24d9bb8e81a6546c8c994a924a/storage/remote/write.go#L114-L125) periodically persisting the current segments for each queue.


Hmmmm, if we do it periodically anyway then relying only on WAL watching tracking might be good enough? 🤔

Also... why periodic, isn't segment rarely rotated/finished?

Hmmmm, if we do it periodically anyway then relying only on WAL watching tracking might be good enough? 🤔

That was my hypothesis when looking at alternatives which included a synchronous commit,

If we assume a 15 second queue delay then syncing the savepoint every 30 seconds gives a lot of room for the segment to be fully processed before being committed.

…tion proposal, expand more on at-least-once, alternatives pros/cons Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

bwplotka

LGTM, one question, but otherwise I would love to unblock this work - I hope it brings refresh to our codebase on the way for queue manager.

bwplotka · 2026-06-25T15:51:44Z

+If a config is reloaded or prometheus/agent is restarted before flushing pending samples we will skip those samples.
+Given we have a persistent WAL this behavior is unexpected by users and can cause a lot of confusion.
+
+## Goals


Is the goal to make sure we also always support best-effort mode for lightweight setup that prefers lower latency and HW use?

Ahh good point, I had actually assumed we would always want this behavior at some point in the future. As we introduce it, it will absolutely need to be opt-in but I imagine if we could make the breaking change we would want it on by default with an opt-out? Is that your perspective as well?

Remote Write: Restart from checkpoint

beb076e

Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

kgeckhart marked this pull request as ready for review December 18, 2025 20:14

kgeckhart changed the title ~~Remote Write: Restart from checkpoint~~ proposal: Remote Write: Restart from checkpoint Dec 18, 2025

kgeckhart added 2 commits December 18, 2025 15:16

Add pr id to the proposal file

d4945e1

Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

Remove duplicated block

df97ae1

Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

kgeckhart changed the title ~~proposal: Remote Write: Restart from checkpoint~~ proposal: Remote Write: Restart from savepoint Jan 29, 2026

Rename checkpoint -> savepoint to eliminate confusion with existing c…

cdc82e6

…heckpoint Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

kgeckhart force-pushed the kgeckhart/remote-write-restart-from-checkpoint branch from 96370f0 to cdc82e6 Compare January 29, 2026 19:11

kgeckhart mentioned this pull request Feb 9, 2026

tsdb/agent: Checkpoint based on Series in Memory prometheus/prometheus#17617

Closed

bwplotka reviewed Feb 16, 2026

View reviewed changes

bboreham mentioned this pull request Feb 24, 2026

prometheus.remote_write silently drops any data older than the age of the alloy process itself grafana/alloy#5320

Open

beorn7 mentioned this pull request Feb 24, 2026

tsdb/agent: Prevent unread segments from being truncated prometheus/prometheus#17616

Open

kgeckhart added 2 commits February 24, 2026 16:51

Remove feedback: typos and other small fixes, higher level implementa…

eb32233

…tion proposal, expand more on at-least-once, alternatives pros/cons Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

fmt

6f5e380

Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>

kgeckhart changed the title ~~proposal: Remote Write: Restart from savepoint~~ proposal: Remote Write: Restart from segment-based savepoint Mar 17, 2026

kgeckhart mentioned this pull request Mar 17, 2026

Implement replay grafana/alloy#5808

Open

x1unix mentioned this pull request Apr 8, 2026

remote_write: implement restart from segment-based savepoint prometheus/prometheus#18485

Open

kgeckhart mentioned this pull request Apr 30, 2026

Alloy doesn't replay wal for recoverable error grafana/alloy#4469

Open

bwplotka approved these changes Jun 25, 2026

View reviewed changes

		### Pitfalls of the current solution

		As mentioned in the why, this behavior is often confusing to users who know a WAL is in use but still finds they have missing data on restart.

	* The `queue_configuration` includes fields like min/max shards and other performance tuning parameter.s
	* The `queue_configuration` includes fields like min/max shards and other performance tuning parameters.


		## Goals

		1. Support resuming from a savepoint for each configured `remote_write` destination.

	3. Changing the `queue_configuration` for a `remote_write` destination should not result in a new savepoint entry.
	3. Changing the `queue_configuration` for a `remote_write` destination should not result in losing a savepoint entry.


		## Alternatives

		1. `remote.QueueManager` should own syncing its own savepoint (most early implementations took this approach).

	2. The savepoint should be synchronously updated when segments change.
	2. The savepoint should be synchronously updated when segments change during WAL watching.


		### Code flow

		1. Adding another configurable timer to [`remote.WriteStorage.run()`](https://github.com/prometheus/prometheus/blob/f50ff0a40ad4ef24d9bb8e81a6546c8c994a924a/storage/remote/write.go#L114-L125) periodically persisting the current segments for each queue.

Uh oh!

Conversation

kgeckhart commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kgeckhart Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kgeckhart commented Dec 18, 2025 •

edited

Loading

kgeckhart Feb 18, 2026 •

edited

Loading