Skip to content

nomad: clarify lost_after default and fix conflicting statements #1460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mismithhisler wants to merge 3 commits into
base: nomad/1.11.2
Choose a base branch
from
+16 −18
Open

nomad: clarify lost_after default and fix conflicting statements #1460

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d26d249
nomad: updates disconnect block docs to clarify default lost_after an…
mismithhisler Dec 4, 2025
79fbbbc
more disconnect block updates and remove confusing links
mismithhisler Dec 8, 2025
39ac539
Update content/nomad/v1.11.x/content/docs/job-specification/disconnec…
aimeeu Dec 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 14 additions & 16 deletions content/nomad/v1.11.x/content/docs/job-specification/disconnect.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,15 @@ job "docs" {
group "example" {
disconnect {
lost_after = "6h"
replace = false
replace = true
reconcile = "keep_original"
}
}

group "example2" {
disconnect {
stop_on_client_after = "12h"
replace = false
replace = true
reconcile = "keep_original"
}
}
Expand All @@ -48,20 +48,21 @@ same `disconnect` block.

- `lost_after` `(string: "")` - Specifies a duration during which a Nomad client
will attempt to reconnect allocations after it fails to heartbeat in the
[`heartbeat_grace`][] window. It defaults to "", which is equivalent to
having the disconnect block be nil.

You cannot use `lost_after` and `stop_on_client_after` in the same
`disconnect` block.
[`heartbeat_grace`][] window. Allocations with a `lost_after` > 0s are put into
an `unknown` state until they reconnect or the `lost_after` expires.
It defaults to "", which is equivalent to a `lost_after` of `"0s"` where
allocations will immediately be marked lost.

Refer to [the Lost After section][lost-after] for more details.

- `replace` `(bool: true)` - Specifies if Nomad should replace the disconnected
allocation with a new one rescheduled on a different node. Nomad considers the
replacement allocation a reschedule and obeys the job's [`reschedule`][]
block. If false and the node the allocation is running on disconnects
or goes down, Nomad does not replace this allocation and reports `unknown`
until the node reconnects, or until you manually stop the allocation.
(i.e. `unknown`) allocation with a new one rescheduled on a different node.
Nomad considers the replacement allocation a reschedule and obeys the job's
[`reschedule`][] block. If false, Nomad does not replace the allocation
when the node it's running on disconnects or goes down.

Note: The `replace` field is only relevant for behavior when `lost_after`
is used and the allocation is in the `unknown` state.

```plaintext
`nomad alloc stop <alloc ID>`
Expand All @@ -71,10 +72,7 @@ same `disconnect` block.
disconnected.

- `stop_on_client_after` `(string: "")` - Specifies a duration after which a
disconnected Nomad client will stop its allocations. Setting
Copy link
Member Author

@mismithhisler mismithhisler Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this because it implies these two settings can be set at the same time, however they cannot, and that is mentioned in several places in the docs.

`stop_on_client_after` shorter than `lost_after` and `replace = false` at the
same time is not permitted and will cause a validation error, because this
would lead to a state where no allocations can be scheduled.
disconnected Nomad client will stop its allocations.

The Nomad client process must be running for this to occur.

Expand Down
2 changes: 1 addition & 1 deletion content/nomad/v1.11.x/content/docs/job-specification/migrate.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,6 @@ on node draining.
[count]: /nomad/docs/job-specification/group#count
[drain]: /nomad/commands/node/drain
[deadline]: /nomad/commands/node/drain#deadline
[replaces]: /nomad/docs/job-specification/disconnect#replace
[replaces]: /nomad/docs/job-specification/disconnect
Copy link
Member Author

@mismithhisler mismithhisler Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Linking to the replace parameter was misleading, so instead, just link to the disconnect overview.

[`restart`]: /nomad/docs/job-specification/restart
[reschedules]: /nomad/docs/job-specification/reschedule
2 changes: 1 addition & 1 deletion content/nomad/v1.11.x/content/docs/job-specification/reschedule.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,5 +139,5 @@ job "docs" {
[`progress_deadline`]: /nomad/docs/job-specification/update#progress_deadline
[`restart`]: /nomad/docs/job-specification/restart
[migrates]: /nomad/docs/job-specification/migrate
[replaces]: /nomad/docs/job-specification/disconnect#replace
[replaces]: /nomad/docs/job-specification/disconnect
[reschedules]: /nomad/docs/job-specification/reschedule
Loading