[WIP] Fix stale node monitor stats after wall-clock rollback

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>Node monitor stats can remain stale after wall-clock rollback because refresh checks use System.currentTimeMillis()</issue_title>
<issue_description>### Elasticsearch Version
7.17.x (based on the affected code path in the 7.17-derived branch we investigated; please adjust if maintainers need the exact released tag)

Installed Plugins

[]

Java Version

Bundled JDK / affected by JVM wall-clock drift on the node

OS Version

Linux

Problem Description

We observed that _nodes/<node>/stats/fs can keep reporting stale filesystem usage after the node clock jumps forward and is later moved back to the correct time.

Expected behavior:

• Node monitor stats should continue refreshing according to monitor.*.refresh_interval even if the wall clock is corrected.

Actual behavior:

• If a refresh happens while the wall clock is ahead, and the clock is then rolled back, the refresh condition can remain false for a long time.
• As a result, _nodes/stats may return stale data until the wall clock catches up again or Elasticsearch is restarted.

In our case:

• _nodes/<node>/stats/fs reported about 73.4% disk usage.
• The actual host filesystem usage was about 88%.
• Downstream disk-watermark-based logic did not trigger as expected because Elasticsearch kept exposing stale fs stats.
• Restarting Elasticsearch immediately made the reported stats become correct again.

From code inspection, this looks like a wall-clock-based cache expiry problem rather than an fs math problem:

• SingleObjectCache uses System.currentTimeMillis() both to set lastRefreshTimestamp and to decide whether the cache needs refresh:
  • server/src/main/java/org/elasticsearch/common/util/SingleObjectCache.java
• FsService uses SingleObjectCache for filesystem stats:
  • server/src/main/java/org/elasticsearch/monitor/fs/FsService.java
• OsService and ProcessService also use SingleObjectCache:
  • server/src/main/java/org/elasticsearch/monitor/os/OsService.java
  • server/src/main/java/org/elasticsearch/monitor/process/ProcessService.java
• JvmService has the same wall-clock refresh pattern inline:
  • server/src/main/java/org/elasticsearch/monitor/jvm/JvmService.java
• FsProbe itself reads FileStore values directly on refresh, so the stale result seems to come from refresh suppression, not from wrong free/available calculation:
  • server/src/main/java/org/elasticsearch/monitor/fs/FsProbe.java

This means the impact likely is not limited to fs stats. jvm, os, and process stats may have the same stale-refresh problem after clock rollback.

Steps to Reproduce

1. Start a single-node Elasticsearch instance with default monitor refresh intervals.
2. Call GET /_nodes/stats/fs,jvm,os,process at least once.
3. Move the host clock significantly forward (for example, +1 day).
4. Call GET /_nodes/stats/fs,jvm,os,process again so the monitor cache refreshes while the clock is in the future.
5. Move the host clock back to the correct current time.
6. Wait longer than monitor.fs.refresh_interval and change actual disk usage on the host.
7. Call GET /_nodes/stats/fs,jvm,os,process repeatedly.

Observed result:

• fs/jvm/os/process stats can remain stale for a long time.
• fs values do not reflect the current host filesystem state.
• Restarting Elasticsearch makes the stats immediately correct again.

Expected result:

• Refresh scheduling/expiry should use a monotonic time source for cache invalidation.
• Wall-clock corrections should not freeze monitor stats refresh.

Logs (if relevant)

There are no obvious error logs.
The problem is silent and is visible by comparing:

• _nodes/<node>/stats/fs
• host-side df -h
• behavior before and after Elasticsearch restart

Additional Notes

The field-level symptom can look like free_in_bytes / available_in_bytes or usage percentage being inconsistent with the actual filesystem state, but the deeper issue appears to be that node monitor stats are not refreshed after wall-clock rollback.
</issue_description>

Comments on the Issue (you are @copilot in this section)

@elasticsearchmachine Pinging @elastic/core-docs (:Docs/General) @DaveCTurner I'm going to call this a docs issue. We rely on the user organising their node clocks to be reasonably well-synced in various places for ~correct~ efficient operation, but AFAICT we don't say so explicitly in the docs today. I think we should mention it in the [Important system configuration](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/important-system-configuration) section.

For the avoidance of doubt we don't require synced clocks for any safety (i.e. correctness) properties, but we do not expect the system to behave optimally if clocks deviate significantly from reality.</comment_new>
<comment_new>@elasticsearchmachine
Pinging @elastic/admin-docs (:Docs/Admin)</comment_new>
<comment_new>@DaveCTurner

The stronger problem is that after a wall-clock jump forward and a later rollback

This is only a problem if you believe this is a reasonable thing for a wall-clock time to do, and we're saying that it isn't. We have to have some expectations about the surrounding environment behaving sensibly, and this is one of them. The problems with wall-clock time having large negative discontinuities are broader than just this cache-refresh behaviour, and distribute across nodes in ways that cannot be solved simply by switching to a monotonic time source in all cases.

Looking at the broader picture we must assume that users are configuring their systems not to have large clock skews or the kinds of massive clock discontinuities that you're describing. We're only missing an explicit mention of this assumption in the manual.</comment_new>

[elasticsearchmachine]

@copilot-swe-agent please enable the option "Allow edits and access to secrets by maintainers" on your PR. For more information, see the documentation.