Bytes test expansion, safer diagnostics, and doc anchors — with realistic JaCoCo gates

NEW_TESTS.md

@@ -0,0 +1,51 @@
+# Chronicle Bytes – In-Flight Test Additions
+
+Short log of the extra tests added while working through `TESTS_TODO.md`. Keep
+this in sync with `TESTS_TODO.md` so the next session can immediately see what
+landed vs. what is pending.
+
+## Implemented This Pass
+
+1. **`MappedFileTest.insideHonoursSafeLimitWhenPageSizeDiffers`**
+   * Exercises a chunk acquired with a page size larger than the OS default.
+   * Verifies that `MappedBytesStore#inside` respects `safeLimit()` so elastic
+     mappings do not overrun chunk boundaries.
+2. **`MappedBytesTest.zeroOutRespectsCustomPageSize`**
+   * Confirms `MappedBytes.zeroOut` clears all bytes when the mapping page size
+     is tuned (captures previous regressions when the OS default was assumed).
+3. **`TempDirectoryIntegrationTest`**
+   * Uses `IOTools.createTempDirectory` end-to-end and asserts the artefacts are
+     rooted under `OS.getTarget()` and removed afterwards.
+4. **`ReferenceTracingLeakTest`**
+   * Leaks a direct `Bytes` on purpose and ensures
+     `AbstractReferenceCounted.assertReferencesReleased()` raises the expected
+     diagnostic with `createdHere()`.
+5. **`DecoratedBufferOverflowExceptionTest`**
+   * Documents the null-cause semantics: passing `null` mirrors the
+     single-argument constructor, while a real `Throwable` is preserved as the
+     cause.
+6. **`NativeBytesOverflowTest#overflowWithoutTracingKeepsCauseNull`**
+   * Forces a capacity overflow with reference tracing disabled to ensure the
+     `DecoratedBufferOverflowException` thrown by `NativeBytes.newDBOE(...)`
+     retains a `null` cause instead of throwing.
+
+All of the above pass under `mvn -q -Dtest=... test` (see latest run in shell
+history) and under the module-wide `mvn -q verify`.
+
+## Still Outstanding
+
+* Safe-page defaults for builder APIs (Windows vs. other OSes) – see
+  `TESTS_TODO.md` item “Safe-Page Block Size for Builders”.
+* Additional `OS.pageAlign` coverage for chunked mappings – see
+  `TESTS_TODO.md` item “OS.pageAlign Consumers”.
+
+Once those tests are implemented, rerun:
+
+```bash
+mvn -q -Dtest=MappedFileTest,MappedBytesTest,TempDirectoryIntegrationTest,ReferenceTracingLeakTest,DecoratedBufferOverflowExceptionTest,NativeBytesOverflowTest test
+mvn -q verify
+```
+
+Remember to update the Matching AsciiDoc (especially `memory-management.adoc`)
+whenever diagnostics or behavioural guarantees change, and keep the AGENTS
+guidelines (British English + ISO-8859-1) in mind.

TESTS_TODO.md

@@ -0,0 +1,69 @@
+# Chronicle Bytes – Test Coverage TODO
+
+Keeps track of Byte-specific test work (including batch additions beyond the module hand-offs). Follow AGENTS rules (British English, ISO-8859-1) and run `mvn -q verify` before publishing.
+
+## Repo & Commands
+
+* Repo root: `/home/peter/Build-All/Chronicle-Bytes`
+* Iteration command (targeted):
+  ```bash
+  mvn -q -Dtest=BytesLifecycleTest,BytesCopyMatrixTest,UncheckedBytesBehaviourTest test
+  ```
+* Full suite: `mvn -q verify`
+
+## Completed This Session
+
+1. Added `BytesLifecycleTest` (slice ref-count lifecycle, elastic-growth monotonicity, `copyTo(BytesStore)` correctness).
+2. Added `BytesCopyMatrixTest` (heap→native copy and direct→`OutputStream` copy coverage).
+3. Extended `UncheckedBytesBehaviourTest` with `uncheckedModeAllowsWritePastLimit`.
+4. Targeted tests run via `mvn -q -Dtest=BytesLifecycleTest,BytesCopyMatrixTest,UncheckedBytesBehaviourTest test` – presently passing.
+5. Added `MappedFileTest.insideHonoursSafeLimitWhenPageSizeDiffers` to exercise `MappedBytesStore#inside` across custom page sizes.
+6. Added `MappedBytesTest.zeroOutRespectsCustomPageSize` to cover single mapping zero-out behaviour when the mapping page size is larger than the OS default.
+7. Added `TempDirectoryIntegrationTest` to verify `IOTools.createTempDirectory` locates paths beneath `OS.getTarget()` and that `deleteDirWithFiles` removes them.
+8. Added `ReferenceTracingLeakTest` to confirm `AbstractReferenceCounted.assertReferencesReleased()` surfaces leaks with the recorded `createdHere` trace.
+9. Added `DecoratedBufferOverflowExceptionTest` to spell out the null-cause semantics (`DecoratedBufferOverflowException` mirrors the single-argument constructor when tracing is disabled).
+10. Extended `NativeBytesOverflowTest` with `overflowWithoutTracingKeepsCauseNull` so we catch regressions where `createdHere()` returns `null` (ties back to <<CB-NF-O-006>> diagnostics in `memory-management.adoc`).
+
+## Outstanding Batches
+
+The following scenarios (from the user brief) remain **unimplemented** and should be prioritised next:
+
+1. **Safe-Page Block Size for Builders** – test builder defaults so Windows uses `OS.SAFE_PAGE_SIZE` while other OSes use `OS.pageSize()`. Add a new `MappedBytesQueueBuilderTest` or equivalent.
+2. **OS.pageAlign Consumers** – assert `MappedBytesStore.map()` (or helpers) align offsets using `OS.pageAlign`, especially for large offsets/Windows safe pages; extend `MappedBytesTest` with a synthetic offset scenario.
+
+## Additional Proposed Tests
+
+6. **`BytesComparisonMatrixTest`**
+   * Exercise `BytesStore.compareAndSwapInt/Long` across heap, direct, and mapped stores to ensure mixed-type comparisons behave consistently and honour alignment rules. Include negative tests where the address straddles the capacity boundary to confirm deterministic exceptions.
+
+7. **`ElasticByteBufferReuseTest`**
+   * Reproduce the pattern from Chronicle Wire: allocate an elastic direct buffer, write past the original capacity with `writePosition` growth, then hand the buffer to `BinaryWire`. Assert that ref-counts stay positive, and once `releaseLast()` is invoked the backing `ByteBuffer` becomes inaccessible (guards against use-after-free regressions).
+
+8. **`TempFileCleanupTest`**
+   * Create temporary `Bytes` backed by `IOTools.createTempFile()` and verify that `Bytes.deleteFile()` removes both the data file and the `.tmp` companion. Mimic busy Windows behaviour by holding a channel open and confirming the new retry loop logs the appropriate warning.
+
+9. **`BytesHexDumpDeterminismTest`**
+   * Feed `HexDumpBytes` with sequences containing control characters and high-bit values, ensuring `toHexString()` and `parseHexString()` round-trip identically (protects tooling that relies on deterministic dumps when debugging network captures).
+
+10. **`UnsafeDirectStoreAlignmentTest`**
+    * Parameterise over odd/even alignment sizes to assert that `UnsafeDirectStore.of()` respects the requested alignment and that `addressForRead`/`addressForWrite` never exceed `realCapacity()`. This mirrors the Chronicle Queue use-case where misalignment breaks memory-mapped header validation.
+
+## Next Steps
+
+1. Design tests for items 1–5 above; consider using `OS.memory()`/wrapper to mock page sizes where needed.
+2. After implementing each chunk, re-run targeted tests plus `mvn -q verify`.
+3. Update this file (and per-module `NEW_TESTS.md` if you add repo-specific notes) once the outstanding scenarios are covered.
+
+## Additional Ideas (Queue-derived backlog)
+| Batch | Scope | Goal | Notes |
+|------|-------|------|-------|
+| B-07 | Zero-length frame handling | Add Bytes-level tests that write/read empty payloads using `Bytes` and ensure downstream consumers observe zero-length content without blocking the next message. | Mirrors `ReaderResizesFileTest.testZeroLengthDocumentDoesNotBlockTailer`; ensures Bytes APIs surface the same invariants. |
+| B-08 | Partial frame mutation guards | Extend Bytes fuzz tests to mutate SPB headers mid-read (length mismatch, corrupted prefix) and assert defensive checks fire before data becomes visible. | Aligns with queue tests for partial frames; keeps raw Bytes primitives honest. |
+
+## Fresh Candidates (add to queue soon)
+
+1. **PinnedMappedBytesShutdownTest** – simulate long-lived `MappedBytes` held open during JVM shutdown, assert `MappedFile.release()` frees native handles and log noise stays below the warning threshold (covers Windows busy-file reports).
+2. **BytesUTF8SurrogateMatrixTest** – feed `AppendableUtil` with mixed valid and broken surrogate pairs, verifying encoder paths either emit replacement chars or throw per `StopCharTesters` expectations; ensures UTF-8 helpers keep Chronicle Wire interop safe.
+3. **NativeBytesBoundsRaceTest** – run two threads performing `writePosition` + `readPosition` mutations against the same `NativeBytes` instance under a `Pauser` to confirm CAS guards prevent negative capacities and throw `BufferOverflowException` deterministically.
+4. **ReadonlyMappedBytesRelaxedViewTest** – open the same mapped file as read-only and read-write Bytes, flip the writable view via `MappedBytes.writingDocument()`, and assert the read-only view never observes partial header writes (protects replication recovery scans).
+5. **BytesCompressionRoundTripTest** – integrate `LZ4Compressor`/`SnappyCompressor` adapters to compress into a `Bytes` sink, then decompress via the matching codec to prove direct/native stores do not require intermediary arrays and capacity grows elastically without leaks.

src/main/docs/architecture-overview.adoc

@@ -212,6 +212,7 @@ This architecture directly supports and enables the fulfilment of the functional
 
 * Core API exposure for various memory types and operations (CB-FN-001 .. CB-FN-008, CB-FN-012, CB-FN-014).
 * Performance targets for serialization, access speed, and atomic operations (e.g., <<CB-NF-P-001>>, <<CB-NF-P-003>>, <<CB-NF-P-004>>).
+* Multi-JDK build coverage ensures both baseline Java 8 users and teams piloting the latest releases stay unblocked (<<CB-NF-MP-006>>).
 * Operability hooks like JMX metrics for `BytesMetrics` (CB-NF-O-002) allow integration into standard monitoring dashboards.
 * Security considerations like robust bounds checking (CB-NF-S-001) are integral to the `Bytes` API design.
 

src/main/docs/memory-management.adoc

@@ -50,6 +50,18 @@ resources.
 Enable `Jvm.isResourceTracing()` during testing to record stack traces for
 allocations. This helps locate leaks if `releaseLast()` is not called.
 
+`AbstractReferenceCounted` implements `ReferenceCountedTracer`, so classes such
+as `NativeBytes` can call `createdHere()` to recover the stack trace of the
+allocation site. Chronicle Bytes feeds that trace into diagnostics like
+`DecoratedBufferOverflowException` in `NativeBytes.newDBOE(...)`, which means a
+resizing failure reports both the write that triggered it and where the buffer
+originated. This is invaluable when multiple components reuse elastic buffers
+and you need to pinpoint which call chain produced a leaked or overgrown
+instance. The richer exception context fulfils the diagnostic requirement in
+<<CB-NF-O-006>>. When tracing is disabled `createdHere()` returns `null`, so
+Chronicle Bytes automatically falls back to the standard constructor to avoid
+raising a misleading `NullPointerException`.
+
 == 6  Elastic Buffers
 
 Elastic `Bytes` can grow when more space is required. For native buffers a new