build: de-waf the post-link pipeline (phase 0 + 1) - #1886
Open
jplexer wants to merge 11 commits into
Open
Conversation
Remove three custom waf tools that nothing uses: - binary_header.py: loaded at configure time but no task generator in the tree uses the binary_header feature. It also references the re and os modules without importing them, so parts of it would raise NameError if ever exercised. - c_inject_include_files.py: never loaded by any wscript, and it references waflib.Node.Nod3 (typo), so loading it would fail. - compress.py: no importers anywhere in the tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
Two latent defects in the clar test harness: - clar() declared mutable default arguments (test_libs=[], override_includes=[]) and add_clar_test() appended pthread/m to test_libs and libutil/libbtutil/duma to the caller's use list in place. The shared defaults accumulated duplicates across all call sites, and multi-platform tests grew their use list once per platform. Copy the lists before modifying them. - clar() derived test_name in a loop over test_sources but built the test binaries in a separate loop afterwards, using the loop-leaked last test_source with the first source's name. A glob matching more than one file would silently drop all but the last. Nest the platform loop so every matched source gets its own test. No behavior change for existing call sites: every current test glob matches exactly one file, and the duplicate libs were harmless. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
PebbleFirmwareBinaryInfo and the FirmwareDescription struct insertion existed in two diverged copies: tools/fw_binary_info.py + tools/insert_firmware_descr.py (manual CLIs) and a vendored copy in pebble-commander's imaging command, which is the live path for boards without pblboot (asterix gets the struct prepended at PULSE flash time). Make pebble.commander.util.fw_binary_info the canonical implementation, next to the vendored stm32_crc it depends on, so the package stays self-contained. The tools/ scripts become thin shims: fw_binary_info.py subclasses to keep its .elf input support (which needs the repo-local binutils helper), and insert_firmware_descr.py keeps its CLI. Output is byte-identical to the previous implementations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
Move the git revision logic from tools/waf/gitinfo.py into tools/gitinfo.py with a plain subprocess implementation and a small CLI (--json), so post-link and bundle tooling can use it without waf. The waf module becomes a shim keeping the ctx-based signature for the remaining wscript callers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
Replace the hardcoded per-SoC size tables in wscript and pbl with two derived Kconfig symbols: - FW_MAX_SIZE: defaults to FW_FLASH_SIZE, which already encodes the per-board/per-variant bank sizes, with an nRF52 exception keeping 32 KiB reserved for the bootloader in the non-recovery bank. - SYSTEM_RESOURCES_MAX_SIZE: per-SoC pbpack cap. Verified against the old tables for every board across normal/prf/prf+mfg variants; all values are identical. One edge case improves: PRF built with RECOVERY_FW_AS_FW now gets the normal-area cap on SF32LB52 instead of the 576 KiB recovery-bank cap, since it is linked into the normal firmware area. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
Write a neutral JSON description of the configured build (board, platform, variant, slot, runners, artifact paths, and every CONFIG_* symbol) to build/build-info.json. This is the interface the standalone build tooling reads instead of parsing waf's c4che, and it is producer-agnostic: any build system that writes the same file can drive the same tools. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
Replace the five per-step waf rules in _link_firmware (objcopy hex, objcopy bin, two pblboot header inserts, loghash extraction) plus the merge_loghash_dicts post-build function with a single standalone CLI driven by build-info.json. waf now invokes it as one rule with the ELF as input and the hex/bin/loghash dictionaries as outputs. Along the way: - tools/waf/pblboot.py moves to tools/pblboot.py, dropping its two waf task-rule wrappers; the argparse CLI and the header/priority functions are unchanged. - tools/waf/generate_log_strings_json.py is removed; the extraction and format-specifier check now live in fw_image.py, writing both pebbleos_loghash_dict.json and src/fw/loghash_dict.json (the merge step had exactly one input dict, so both files were always identical). - bld.LOGHASH_DICTS and env.PBLBOOT_PRIORITY/FIRMWARE_OFFSET plumbing goes away. Verified: objcopy outputs and the loghash dictionary are byte- identical to what the replaced waf rules produce for the same ELF, and the pblboot header CRC/offset/content check out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
Move the .pbz bundling glue (_make_bundle/_get_version_info) from the root wscript into a standalone CLI driven by build-info.json: version info via tools/gitinfo, the CONFIG_FW_MAX_SIZE gate, resource pack / loghash / layouts attachment and the output naming convention. The waf bundle command is gone: ./pbl bundle invokes the tool without spawning a waf process, and CI uses it directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
Move the inline qemu_image_micro/qemu_image_spi logic from the root wscript into a standalone CLI driven by build-info.json. The waf qemu_image_* commands are gone: ./pbl qemu builds the images directly, a new ./pbl qemu_images command covers the CI usage, and pbl's last waf-forwarding helper (_run_waf) goes away. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
Load build state from the neutral build/build-info.json written at configure time rather than importing waf's c4che/_cache.py. The Env shim keeps waf's ConfigSet semantics (missing keys read as []) so the runner helpers are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
All remaining ./waf invocations in the workflows become ./pbl (configure/build/test forward through pbl's passthrough; bundle and qemu_images are pbl commands now). The workflow path filters gain 'pbl' so CLI changes retrigger the builds, and the docs drop stale ./waf and binary_header references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Joshua Jun <lets@throw.rocks>
jplexer
force-pushed
the
build/unwaf-phase0
branch
from
August 18, 2026 21:49
993815f to
ed76351
Compare
gmarull
reviewed
Aug 24, 2026
Comment on lines
+42
to
+45
| def _objcopy(elf, out, fmt): | ||
| subprocess.check_call( | ||
| ["arm-none-eabi-objcopy"] + OBJCOPY_STRIP_ARGS + ["-O", fmt, elf, out] | ||
| ) |
Member
There was a problem hiding this comment.
note: on a build system based e.g. on CMake, having this outside of the build system will be non-idiomatic, and probably cause problems like dependencies not being tracked properly (to determine when something has to be re-run). Also, we may not always want to invoke such tool, what if we support one day native builds, llvm, etc.?
gmarull
reviewed
Aug 24, 2026
Comment on lines
+37
to
+55
| config FW_MAX_SIZE | ||
| hex | ||
| default 0xf8000 if SOC_NRF52 && (!RECOVERY_FW || MFG) | ||
| default FW_FLASH_SIZE | ||
| help | ||
| Maximum size of the firmware image binary, enforced when | ||
| bundling and flashing. Defaults to the firmware flash bank | ||
| size; the nRF52 normal-firmware bank keeps 32 KiB reserved for | ||
| the bootloader. | ||
|
|
||
| config SYSTEM_RESOURCES_MAX_SIZE | ||
| hex | ||
| default 0x100000 if SOC_NRF52 | ||
| default 0x200000 if SOC_SF32LB52 | ||
| default 0x200000 if SOC_QEMU | ||
| default 0x40000 | ||
| help | ||
| Maximum size of the system resources pbpack, enforced after the | ||
| resource build. |
Member
There was a problem hiding this comment.
we should define some sort of partition map at board level for that I think
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First steps toward making waf a thin orchestrator over standalone tooling to later switch it out with another build system. No behavior changes intended; artifacts were verified byte-identical where deterministic.
Cleanup
binary_header,c_inject_include_files,compress). They were unused, and two had latent import bugs.pebble_test.pydefects: mutable default args accumulating libs across all 285clar()call sites, and a loop-leakedtest_sourcethat would drop files if a test glob ever matched more than one.pebble.commander.util.fw_binary_info(the live flash-time path for non-pblboot boards, i.e. asterix); thetools/scripts are thin shims keeping their.elfsupport and CLIs.build-info.json
configurenow writesbuild/build-info.json: board, platform, variant, slot, runners, artifact paths, and allCONFIG_*symbols. This is the neutral interface the extracted tools read: any future build system that writes the same file drives the same tools unchanged.Extracted tools (waf calls them; nothing about them is waf-specific)
tools/fw_image.py: post-link pipeline: objcopy hex/bin, pblboot header, loghash extraction/check. Replaces five waf rules + a post-build merge.tools/make_fw_bundle.py: .pbz bundlingtools/make_qemu_images.py: QEMU micro/SPI flash imagestools/gitinfo.py: standalone git version info with--json.FW_MAX_SIZE,SYSTEM_RESOURCES_MAX_SIZE); verified identical for every board × variant combination../pblno longer parses waf's c4che or spawns waf for bundle/qemu images: it reads build-info.json.Verification