Switch to cargo-fuzz

[uncomputable]

Adapt the fuzzing infrastructure to use cargo-fuzz. I keep the existing shell scripts because they turned out to be useful.

Everything should work as before, except that the fuzzing flag is producing warnings when the tests are run. Let's discuss solutions here.

[apoelstra]

Can you update the lockfiles as suggested by CI?

Everything should work as before, except that the fuzzing flag is producing warnings when the tests are run. Let's discuss solutions here.

What kind of warnings?

[uncomputable]

warning: unexpected `cfg` condition name: `fuzzing`
 --> fuzz/fuzz_targets/parse_human.rs:3:13
  |
3 | #![cfg_attr(fuzzing, no_main)]
  |             ^^^^^^^
  |
  = help: expected names are: `clippy`, `debug_assertions`, `doc`, `docsrs`, `doctest`, `feature`, `fmt_debug`, `miri`, `overflow_checks`, `panic`, `proc_macro`, `relocation_model`, `rustfmt`, `sanitize`, `sanitizer_cfi_generalize_pointers`, `sanitizer_cfi_normalize_integers`, `target_abi`, `target_arch`, `target_endian`, `target_env`, `target_family`, `target_feature`, `target_has_atomic`, `target_has_atomic_equal_alignment`, `target_has_atomic_load_store`, `target_os`, `target_pointer_width`, `target_thread_local`, `target_vendor`, `test`, `ub_checks`, `unix`, and `windows`
  = help: consider using a Cargo feature instead
  = help: or consider adding in `Cargo.toml` the `check-cfg` lint config for the lint:
           [lints.rust]
           unexpected_cfgs = { level = "warn", check-cfg = ['cfg(fuzzing)'] }
  = help: or consider adding `println!("cargo::rustc-check-cfg=cfg(fuzzing)");` to the top of the `build.rs`
  = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration
  = note: `#[warn(unexpected_cfgs)]` on by defaul

[uncomputable]

I fixed the normal CI, but the fuzz test CI seems to be disabled. Investigating.

[uncomputable]

Fuzzing CI works now :) I had to update the upload-artifact and download-artifact actions because of deprecation. I also updated the cache action because it will be deprecated soon.

[apoelstra]

Gonna try to run this in my local CI. It may fail because the first two commits need to be squashed (OTOH it might not because I might not be checking that generate-files.sh is correct here..).

I hope that the 1.84 MSRV-aware resolver works! Will be nice to be able to cronjob that in rust-bitcoin and elsewhere, so that our "recent" lockfile starts staying reasonably up to date.

Would be nice to add some comments saying that set -o errexit and set -o xtrace mean the same thing as set -e and set -x since the latter are familiar to everybody and the former I've never heard of. But that's just a nit.

utACK 4837317

[apoelstra]

To deal with the cfg(fuzzing) error add a line to Cargo.toml like

[lints.rust]
unexpected_cfgs = { level = "deny", check-cfg = ['cfg(fuzzing)'] }

[uncomputable]

I fixed all warnings and lints. We can run clippy over the entire workspace without issues: cargo clippy --all-targets --workspace -- --deny warnings.

[apoelstra]

It'll be tough for me to compile this. It puts libfuzzer-sys 0.4.8 into the lockfiles but this doesn't build with any stable Rust compiler, and my local CI wants to be able to build everything in the lockfile.

[apoelstra]

Oh, maybe it's just a MSRV thing, not a "needs nightly" thing, and maaybe it applies only to the fuzz workspace. Will need to experiment more.

[uncomputable]

For what it's worth, I pinned version 2024-07-01 of the nightly compiler. You can use that version in your local CI.

[apoelstra]

It's not worth a lot -- having things in the lockfile that stable Rust can't build is a problem.

But having said that, somehow cargo build works in the fuzz/ directory even though the fuzz crate has a direct depenedency on libfuzzer-sys. So I think this may be some quirk of crate2nix. But I need to spend some time digging into it to move forward.

[apoelstra]

Needs rebase. (I'm not blocked on it, but hopefully I can resolve the nightly-rustc thing soon and then I will be).

Heads up that I plan to work on this today and on the weekend -- now that I've got the libfuzzer tests sorted out in Elements I really want to be able to use them here.

[apoelstra]

Ok, it's not about nightly (phew) but just that libfuzzer doesn't support 1.63. It needs 1.70.

We have two choices, basically:

1. Provide a separate lockfile for the fuzz tests (by putting it in the exclude list in the main Cargo.toml) which is designed to only work with nightly.
2. Bump our MSRV.

Personally I am leaning toward the latter. This is a fast moving project and we have no particular reason to have a conservative MSRV. The MSRV of lwk is 1.78.0. Let's just copy them.

[uncomputable]

I rebased onto master and bumped the MSRV.

[apoelstra]

Thanks! The tip commit passes -- I am just working on getting the other ones to go (having some trouble with honggfuzz as the new MSRV, which ultimately really doesn't matter, but I'd like to get it working if I can). If I don't succeed today I'll just merge with only the tip tested.

[apoelstra]

ACK ae12708

[apoelstra]

I haven't updated my local CI to actually run these fuzztests, but it does compile-test the fuzz target so if we break them it should detect it. And I at some point when my server has some free cycles I'll start running a long-term test.