You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: proposals/0072-remote-write-restart-from-savepoint.md
+15-15Lines changed: 15 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
## Remote Write: Restart from checkpoint
1
+
## Remote Write: Restart from savepoint
2
2
3
3
***Owners:**
4
4
*[@kgeckhart](https://github.com/kgeckhart)
@@ -18,7 +18,7 @@ I believe the issue for [tsdb/agent: Prevent unread segments from being truncate
18
18
19
19
***Other docs or links:**
20
20
21
-
> This effort aims to have an agreed upon design with requirements for completing the work to allow remote write to restart data delivery from a checkpoint and not from `time.Now()`
21
+
> This effort aims to have an agreed upon design with requirements for completing the work to allow remote write to restart data delivery from a savepoint and not from `time.Now()`
22
22
23
23
## Why
24
24
@@ -32,9 +32,9 @@ As mentioned in the why, this behavior is often confusing to users who know a WA
32
32
33
33
## Goals
34
34
35
-
1. Support resuming from a checkpoint for each configured `remote_write` destination.
36
-
2. Taking a checkpoint for a remote_write destination should not incur significant overhead.
37
-
3. Changing the `queue_configuration` for a `remote_write` destination should not result in a new checkpoint entry.
35
+
1. Support resuming from a savepoint for each configured `remote_write` destination.
36
+
2. Taking a savepoint for a remote_write destination should not incur significant overhead.
37
+
3. Changing the `queue_configuration` for a `remote_write` destination should not result in a new savepoint entry.
38
38
* The `queue_configuration` includes fields like min/max shards and other performance tuning parameter.s
39
39
* These can be expected to change under normal circumstances and should not trigger a data loss scenario.
40
40
4. Guards need to be in place to protect against infinite WAL growth.
@@ -83,17 +83,17 @@ A basic replay to accomplishing all non-stretch goals would be as follows
83
83
6. Update `wlog.NewWatcher.Run()` to start sending samples if a starting segment is configured
84
84
7. Walk through the `remote.QueueManager` send code to ensure duplicate data errors will not cause slow downs in data delivery (we have a high probability of sending duplicate data).
85
85
86
-
This flow should be enough to to accomplish Goal 1: Support resuming from a checkpoint for each configured `remote_write` destination.
86
+
This flow should be enough to to accomplish Goal 1: Support resuming from a savepoint for each configured `remote_write` destination.
87
87
88
-
The act of taking a checkpoint will require a lock to be held but given we do it on a schedule this will be infrequent enough that the implementation should safely accomplish Goal 2: Taking a checkpoint for a remote_write destination should not incur significant overhead (see testing for further info).
88
+
The act of taking a savepoint will require a lock to be held but given we do it on a schedule this will be infrequent enough that the implementation should safely accomplish Goal 2: Taking a savepoint for a remote_write destination should not incur significant overhead (see testing for further info).
89
89
90
-
### Checkpoint file format/location
90
+
### Savepoint file format/location
91
91
92
-
The segment checkpoint would be stored in the `remote.WriteStorage.dir` which would be next to the `/wal` directory.
92
+
The savepoint would be stored in the `remote.WriteStorage.dir` which would be next to the `/wal` directory.
93
93
94
94
We only care about the queue hash and the current segment so a json encoded file seems reasonable for this. A key value format should make it easier to evolve over time vs a more basic delimited file.
95
95
96
-
Solving for, Goal 3: Changing the `queue_configuration` for a `remote_write` destination should not result in a new checkpoint entry.
96
+
Solving for, Goal 3: Changing the `queue_configuration` for a `remote_write` destination should not result in a new savepoint entry.
97
97
98
98
This will be done via adding a specific toHash function for RemoteWriteConfig which zeros the QueueConfig before taking the hash. RemoteWriteConfig is managed as a pointer so we'll need to keep the value before, set to empty, and put the original value back but all is reasonably managed. We could look at identifying other "operational" fields which could be excluded from hashing for the same reasons.
99
99
@@ -103,11 +103,11 @@ This will change existing queue hashes but I don't believe that to be a big prob
103
103
104
104
Goal 4: Guards need to be in place to protect against infinite WAL growth is capable of being accomplished through adjusting config defaults when replaying is enabled. We would require `remote_write.queue_config.sample_age_limit` be non-zero and would have a default of `2h`.
105
105
106
-
I believe prombench is sufficient to prove Goal 2: Taking a checkpoint for a remote_write destination should not incur significant overhead. Open to further benchmarking ideas but given the components + time necessary for a proper test ensuring prombench is capable of covering this would be the most ideal.
106
+
I believe prombench is sufficient to prove Goal 2: Taking a savepoint for a remote_write destination should not incur significant overhead. Open to further benchmarking ideas but given the components + time necessary for a proper test ensuring prombench is capable of covering this would be the most ideal.
107
107
108
108
### Further reducing duplicated data sent
109
109
110
-
Replaying a whole segment can still result in a fair amount of duplicated data on startup. If we added tracking the lowest timestamp delivered via remote write to in the checkpoint it could reduce this number (lowest timestamp is required because the WAL supports out of order writes). At startup the tracked lowest timestamp would be used as marker for where to start writing data from within the checkpointed segment ideally reducing the amount of duplicated data replayed. At worst it would start from the beginning of the segment.
110
+
Replaying a whole segment can still result in a fair amount of duplicated data on startup. If we added tracking the lowest timestamp delivered via remote write to in the savepoint it could reduce this number (lowest timestamp is required because the WAL supports out of order writes). At startup the tracked lowest timestamp would be used as marker for where to start writing data, reducing the amount of duplicated data replayed. At worst it would start from the beginning of the segment.
111
111
112
112
### Goal 5: Stretch: Remote write supports at-least-once delivery of samples in the WAL.
113
113
@@ -144,13 +144,13 @@ I believe this bypasses resharding complexity, as a reshard triggers a purging o
144
144
145
145
## Alternatives
146
146
147
-
1.`remote.QueueManager` should own syncing its own checkpoint (most early implementations took this approach).
147
+
1.`remote.QueueManager` should own syncing its own savepoint (most early implementations took this approach).
148
148
*`remote.QueueManager` already has a lot of responsibilities and will take on more for at-least-once.
149
149
*`remote.WriteStorage` has reasonable hook points to run this logic without adding a lot more complexity.
150
-
2. The checkpoint should be synchronously updated when segments change.
150
+
2. The savepoint should be synchronously updated when segments change.
151
151
* Introducing a bit of time between knowing that a segment changed to persisting it gives us more time to fully deliver the batch before we persist the change.
152
152
* Synchronously committing it makes the potential gap larger.
153
-
* If we assume a 15 second queue delay then syncing the checkpoint every 30 seconds gives a lot of room for the segment to be fully processed before being committed.
153
+
* If we assume a 15 second queue delay then syncing the savepoint every 30 seconds gives a lot of room for the segment to be fully processed before being committed.
154
154
* The trade-off being more unnecessary data being replayed on startup.
155
155
* After implementing a solution for at-least-once we can reassess how often we commit/if we should make it synchronous.
0 commit comments