fix: compiler outputs might exceed the max buffer size

[galargh]

• Because this PR includes a bug fix, relevant tests have been included.
• Because this PR includes a new feature, the change was previously discussed on an Issue or with someone from the team.
• I didn't do anything of this.

---

Resolves: #6336
Resolves: #6328

In this PR, I start using streams when handling the solc compiler outputs to support the compilation of very large codebases where the compilation outputs might exceed the maximum buffer size/string length.

Changes to the build system and cache

I added two new file system access functions:

• readJsonFileAsStream: opens the file as a stream, passes it to the streaming JSON parser, picks the full parsed object from the JSON values stream
• writeJsonFileAsStream: parses the JSON object as a stream of tokens, writes it to the file

These functions allow us to avoid storing the full compiler output as a string, which can exceed the maximum string length at times.

To implement these functions, I added two new dependencies:

• @streamparser/json-node: this is a completely new dependency
• json-stream-stringify: this dependency was present in Hardhat 2

These functions are used when interacting with compiler output objects, in the cache and when emitting the build info output.

Changes to the compiler

Previously, the compiler would execute the underlying solc compiler using the execFile function. With that function, we would end up waiting for the entire output to be returned to us as a string. Unfortunately, for very large compiler outputs, the resulting output would exceed the max buffer size and if we increased the buffer size further, we would eventually hit the maximum string length limit.

I removed the usage of execFile from the compiler and replaced it with spawn, where we pipe the output to a temporary file which we later read using the newly added readJsonFileAsStream function. This allows us to avoid creating a string of the compiler output.

Follow-ups

Deduplicate compiler output file creation

At the moment, we create a file with compiler output three times, in the Compiler, when saving the output to the cache, and when emitting the build info output. We should consider creating it only once and then copying/moving it when needed. This will require changing the build info output format which currently has some extra information apart from the compiler output.

Testing

I rerun the solidity testing testing suite using this branch which cleared all the 143s and max buffer size exceeded errors. My hypothesis on why the 143 were cleared as well is that in these cases, we weren't exceeding the max buffer size, but we were running very near it, which caused the memory pressure.

https://github.com/galargh/solidity-testing-testing/actions/runs/13542801435

[changeset-bot]

🦋 Changeset detected

Latest commit: 27764db

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@nomicfoundation/hardhat-utils	Patch
hardhat	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
hardhat	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 10, 2025 4:05pm

[github-actions]

hardhat

Total size of the bundle: 216M
Total number of dependencies (including transitive): 57

List of dependencies (sorted by size)

211M	total
29M	@ignored/edr-optimism-linux-x64-musl
29M	@ignored/edr-optimism-linux-x64-gnu
26M	@ignored/edr-optimism-linux-arm64-musl
26M	@ignored/edr-optimism-linux-arm64-gnu
22M	@ignored/edr-optimism-win32-x64-msvc
20M	esbuild
20M	@ignored/edr-optimism-darwin-x64
19M	@ignored/edr-optimism-darwin-arm64
2.8M	@sentry/tracing
2.5M	micro-eth-signer
1.9M	@noble/curves
1.7M	undici
1.2M	@sentry/types
1.2M	@noble/hashes
932K	@sentry/node
920K	@sentry/utils
864K	@streamparser/json
860K	@nomicfoundation/hardhat-utils
856K	zod
624K	micro-packed
576K	tsx
548K	@sentry/core
544K	fast-equals
492K	@scure/bip39
408K	json-stream-stringify
368K	ethereum-cryptography
356K	@nomicfoundation/hardhat-errors
344K	@sentry/hub
336K	@ignored/edr
332K	@streamparser/json-node
320K	enquirer
288K	semver
280K	@ignored/edr-optimism
196K	ws
168K	@scure/base
136K	get-tsconfig
136K	adm-zip
96K	@scure/bip32
92K	chalk
88K	tslib
88K	@sentry/minimal
76K	agent-base
72K	@nomicfoundation/solidity-analyzer
68K	debug
64K	lru_map
64K	https-proxy-agent
64K	@nomicfoundation/hardhat-zod-utils
56K	rfdc
48K	ansi-colors
44K	resolve.exports
40K	resolve-pkg-maps
36K	p-map
32K	cookie
24K	strip-ansi
24K	env-paths
24K	ansi-regex
20K	ms