fix: AwkwardForth VM and libawkward correctness issues

[henryiii]

🤖 AI text below 🤖

This PR fixes a batch of correctness and memory-safety issues in the C++ libawkward sources (AwkwardForth VM, the JSON-schema reader, the layout builders, and a couple of small utilities), found by an automated multi-agent code review (Claude Code).

All fixes are in awkward-cpp/src/libawkward/ (plus the ForthMachine.h member declarations):

1. io/json.cpp nulls_for_optiontype — the switch fell through from case FillIndexedOptionArray: into case KeyTableHeader:, double-executing push_stack and leaving the instruction stack unbalanced for nullable records. This produced a spurious JSON schema mismatch when a record typed ["object","null"] had a missing option-type key, while the equivalent non-nullable record null-filled it correctly. Added break; so each case pushes once, matching the single pop_stack. Nullable records now null-fill missing keys.

2. forth/ForthMachine.cpp (Nbit read, ~3068) — uint64_t mask = (1 << bit_width) - 1; shifted an int literal; is_nbit allows widths up to 64, so this was UB / wrong for N >= 31. Now (bit_width >= 64) ? ~0ULL : ((uint64_t)1 << bit_width) - 1.

3. forth/ForthMachine.cpp constructor / ForthMachine.h — the seven scratch arrays were raw new[], and a user-reachable std::invalid_argument from tokenize/compile on bad AwkwardForth source leaked them (~50 KB per failed compile). Converted to std::unique_ptr<T[]>; destructor defaulted.

4. forth/ForthInputBuffer.cpp read/peek_byte — read sizes derive from stack-supplied item counts and could overflow negative. Now reject num_bytes < 0 || next < pos_ || next > length_ (and the analogous check in peek_byte).

5. forth/ForthInputBuffer.cpp read_textfloat — the integral-mantissa loop had no digit cap, causing signed-int64 overflow UB on long literals. Now accumulates in int64_t for the first 18 digits, then continues in double.

6. forth/ForthMachine.cpp is_integer — used std::stoul (caught only invalid_argument, accepted trailing junk like 123abc, handled negatives only by wraparound). Now uses std::stoll, also catches out_of_range, and requires full-string consumption via the pos out-parameter.

7. forth/ForthMachine.cpp (enum/enumonly error path) — could read tokenized[pos + 1] with pos + 1 == stop (out-of-bounds vector read) when no s" strings followed. Now reads the already-validated keyword token at pos - 1 with a bounds guard.

8. forth/ForthMachine.cpp stack_at — off-by-one: stack_at(0) read one past the top. Fixed to stack_depth_ - 1 - from_top. (Part of the public C++ API; no in-repo callers.)

9. forth/ForthOutputBuffer.cpp maybe_resize — reservation = ceil(reservation * resize_) never terminated when constructed with output_initial_size=0 or resize_ <= 1.0 (both reachable from Python). Now guarantees strict growth each iteration.

10. builder/RecordBuilder.cpp clear — cleared keys_/pointers_ and set length_ = -1 while keeping contents_, so subsequent form()/to_buffers() read keys_[i] out of bounds and __len__ raised ValueError. Now keeps the record structure consistent and resets length_ to 0, mirroring TupleBuilder::clear(). Reachable via _ext.ArrayBuilder.clear().

11. util.cpp dtype_to_format — #if defined _MSC_VER || defined INTPTR_MAX == INT32_MAX parsed as (defined INTPTR_MAX) == INT32_MAX (always false) for the uint32/uint64 cases, unlike the correct int32/int64 lines. Dropped the stray defined.

Tests

New regression test file covering the behaviorally-verifiable fixes: from_json with a nullable-record schema + missing option-type keys (#1), ArrayBuilder clear-then-reuse (#10), ForthMachine64(output_initial_size=0) (#9), and Nbit reads with N >= 32 (#2). The full existing suite passes.

Notes

• No package versions bumped here. Since this changes C++, the awkward-cpp version bump (in awkward-cpp/pyproject.toml and the pin in pyproject.toml) and an awkward-cpp release happen at release time, before the next awkward release.
• AI assistance: these fixes were found and implemented with Claude Code as part of an automated multi-agent review, then verified locally.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.55%. Comparing base (f3255a3) to head (bbfbe8f).
✅ All tests successful. No failed tests found.

Additional details and impacted files

[github-actions]

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR4105