SHA-3/SHAKE/EVP_ zeroed ctx guard

[manastasova]

Context and motivation

KECCAK1600_STATE_ABSORB flag was set to 0, so an all-zero KECCAK1600_CTX — never initialised, or cleansed after finalisation — was indistinguishable from one legitimately mid-absorb. The phase checks were deny-lists (state == SQUEEZE || state == FINAL), so such a context reached the primitives with block_size == 0: Keccak1600_Absorb spins forever on while (len >= r) with r == 0, and KeccakSponge_AbsorbFinal indexes ctx->buf[block_size - 1].

The same context is reachable through EVP: EVP_DigestFinal_ex cleanses md_data but leaves ctx->digest set, so a second call re-enters final. That slot returns void, so SHA3_Final refusing the call could not be reported and surfaced as an abort.

Description of changes

Adds KECCAK1600_STATE_UNINIT = 0 and renumbers the rest (ABSORB = 1, SQUEEZE = 2, FINAL = 3), giving a zeroed context a distinct, invalid phase. The checks become allow-lists (state != KECCAK1600_STATE_ABSORB), rejecting uninitialised and already-finalised contexts alike. Keccak1600_Squeeze's padded parameter becomes state.

At the EVP layer, adds EVP_MD_CTX_FINALISED, set by EVP_DigestFinal_ex and cleared by EVP_DigestInit_ex, so a second finalisation returns 0 instead of re-entering the digest. Also fixes used_for_hmac to mask EVP_MD_CTX_HMAC rather than compare the whole flags word.

Testing

New sha3_test.cc cases drive a zeroed context through SHA3_Update/SHA3_Final and SHAKE_Absorb/SHAKE_Final/SHAKE_Squeeze, confirm the existing phase transitions are unchanged, and pin Keccak1600_Squeeze's phase semantics against SHAKE128. keccak256_test.cc adds streaming-misuse and EVP double-final coverage. Without the guards these tests hang or abort rather than fail.

Review considerations

Public behaviour change: a second EVP_DigestFinal_ex without re-initialising now returns 0 for every digest, where some previously aborted and others silently returned success. This diverges from OpenSSL/BoringSSL; include/openssl/digest.h documents it.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[github-actions]

🔒 Security Review — View Report

Please review before merging.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.27%. Comparing base (51cbecc) to head (6b8fb57).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3454      +/-   ##
==========================================
+ Coverage   78.06%   78.27%   +0.21%     
==========================================
  Files         700      700              
  Lines      124705   124726      +21     
  Branches    17326    17332       +6     
==========================================
+ Hits        97352    97634     +282     
+ Misses      26484    26224     -260     
+ Partials      869      868       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.