Jepsen: define consistent outcome and Harness failure semantics

[Phoenix500526]

Background

An unsuccessful operation is not necessarily an error in a Jepsen test.
Timeouts, connection failures, version conflicts, and skipped fault injection
may be expected and necessary observations under faults. Calling all of them
"errors" obscures the boundary between valid test evidence and a broken test
harness.

We have already crossed this boundary in several different ways:

• refactor PR: test(jepsen): add error-aware CAS workload #1855 #1861 found that a raw InterruptedException could be converted into a
  generic :client-error outcome.
• Follow-ups for the Jepsen pause nemesis #1934 found that pause recovery history did not accurately describe what the
  Nemesis had done, and that teardown failure could prevent analysis.
• Jepsen partition teardown can prevent history analysis #1965 allowed analysis to continue after partition teardown cleanup failed.
• jepsen: an interrupted teardown heal is swallowed, and the two teardown regression scenarios share one deftest #1967 then found that the broad cleanup handler swallowed additional forms of
  interruption.
• The current Client still has a catch-all :client-error fallback for unknown
  exceptions, while some teardown paths treat every non-interruption exception
  as a cleanup warning.

These fixes address individual symptoms, but they do not establish one rule for
classifying future failures. A later change can therefore solve the same class
of problem differently and move the harness away from a consistent model.

Why Does It Matter?

Producing the appropriate unsuccessful outcome is part of validating OpenRaft.
For example, a timed-out write may or may not have taken effect. It should enter
the history as an indeterminate operation so the linearizability checker can
interpret it together with later reads and writes.

Conversely, an unknown exception caused by a Client or Nemesis implementation
bug is not an SUT observation. Converting it into an ordinary operation outcome
allows a broken harness to continue and may produce an untrustworthy verdict.

The opposite mistake is also harmful: treating an expected network or SUT
condition as a harness failure terminates a valid experiment and discards useful
evidence.

We should therefore use the neutral term outcome for expected observations
and reserve Harness failure for failures of the test implementation or its
environment.

Proposed Semantic Model

Category	Meaning	Primary evidence
Workload outcome	What a Client operation observed from OpenRaft	history.edn
Nemesis outcome	Whether a requested fault was installed, skipped, failed, or became indeterminate	history.edn
Checker verdict	A decision about a property, coverage, or final recovery	results.edn
Harness failure	A failure of test code, configuration, dependencies, setup, teardown, storage, or checker execution	run-level state and jepsen.log
Interruption	A lifecycle cancellation signal	propagated to the Jepsen coordinator

There is no separate catch-all "Client error" or "Nemesis error" category. The
classification depends on what the event means, not on whether it happened
inside a Client or Nemesis call.

Expected Behavior

1. A Client converts only recognized transport, HTTP, and OpenRaft API
   conditions into :ok, :fail, or :info outcomes.
2. A Nemesis converts only explicitly modeled operational conditions into
   structured outcomes.
3. An unknown Client or Nemesis implementation exception is a Harness failure,
   not a generic operation outcome.
4. The first Harness failure triggers controlled fail-fast:
   • retain the first exception and its source as run-level state;
   • stop scheduling new Client and Nemesis runtime operations;
   • retain completions from operations already in flight;
   • attempt final best-effort fault cleanup;
   • preserve available artifacts and run applicable checkers;
   • exit nonzero.
5. Interruption is never converted into :fail, :info, or a cleanup warning.
   Client and Nemesis code restores the interrupt flag and rethrows it.
6. Teardown does not manufacture successful cleanup outcomes. An expected
   best-effort cleanup failure may be logged as a warning; an unexpected
   teardown implementation exception remains a Harness failure.

Jepsen normally finishes by exhausting the generator, draining outstanding
operations, and stopping its workers. An abnormal interpreter exit may then
interrupt workers as a secondary cleanup action. The harness should therefore
propagate interruption without trying to infer its origin. Controlled fail-fast
needs run-level first-failure state, not cancellation-origin fields in
history.edn.

Verdict Semantics

A Harness failure makes the whole run unacceptable, but it must not erase
evidence already established by a nested checker.

• A successful nested property verdict is inconclusive because the Harness may
  not have completed the intended experiment.
• A failed linearizability verdict retains a concrete counterexample and must
  not be discarded.
• The counterexample establishes an OpenRaft failure only after confirming that
  the Harness failure did not compromise the history, model, or checker that
  produced it.

The top-level :valid? therefore answers only whether the whole run can be
accepted. It does not independently describe OpenRaft correctness or Harness
health.

Questions for Discussion

1. Do we agree with using outcomes to distinguish expected observations from
   Harness failures?
2. Do we agree that a Harness failure should stop new runtime work while still
   preserving cleanup, artifacts, and applicable analysis?
3. Do we agree to preserve established counterexamples while requiring an
   evidence-integrity check before attributing them to OpenRaft?

Non-Goals

• Defining a generic classifier that converts arbitrary SSH or control-layer
  exceptions into expected Nemesis outcomes.
• Adding cancellation-origin fields to operation history.
• Changing OpenRaft's safety or liveness guarantees.

This issue first seeks agreement on the semantic boundary. The implementation
can then be split into small, independently reviewable changes.

[github-actions]

👋 Thanks for opening this issue!

Get help or engage by:

• /help : to print help messages.
• /assignme : to assign this issue to you.

[Phoenix500526]

@drmingdrmer