Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# What is JetStream?

JetStream 3 is a JavaScript and WebAssembly benchmark suite.
For more information see the index and in-depth pages of the deployed benchmark.

A preview of the current main branch is available at [https://webkit-jetstream-preview.netlify.app/](https://webkit-jetstream-preview.netlify.app/).

<img src="./resources/screenshot.png">

## Open Governance

See [Governance.md](Governance.md) for more information.

## Getting Started, Setup Instructions

- Install Node.js and (optionally) [jsvu](https://github.com/GoogleChromeLabs/jsvu) for conveniently getting recent builds of engine shells.
- `npm install` the necessary dependencies.
- `npm run server` for starting a local development server, then browse to http://localhost:8010.
- `npm run test:shell` for running the benchmark in engine shells, or alternatively running directly, e.g., via `jsc cli.js`.

See [package.json](package.json) and [.github/workflows/test.yml](.github/workflows/test.yml) for more details and available commands.

### Shell Runner

For the shell runner, see the available options by passing `--help` to `cli.js`. (Note that this requires `--` for JavaScriptCore and V8 to separate VM arguments from script arguments.):

```
$ v8 cli.js -- --help
JetStream Driver Help

Options:
--help Print this help message.
--iteration-count Set the default iteration count.
...

Available tags:
all
...

Available tests:
8bitbench-wasm
...
```

### Browser Runner

The browser version also supports passing parameters as URL query parameters, e.g., pass the `test` parameter (aliases are `tests` or `testList`) with a comma-separated list to run only specific workloads: [https://webkit-jetstream-preview.netlify.app/?test=8bitbench-wasm,web-ssr](https://webkit-jetstream-preview.netlify.app/?test=8bitbench-wasm,web-ssr).
See [utils/params.js](utils/params.js) and [JetStreamDriver.js](JetStreamDriver.js) for more details.

## Technical Details

The main file of the benchmark harness is `JetStreamDriver.js`, which lists the individual workloads and their parameters, implements measurement and scoring, etc.
The individual workloads are in subdirectories.

### Preloading and Compression

The JetStream driver (both in the browser and shell runners) _preloads_ some large assets and source files.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit unclear what preloading means in this context. Might be worth splitting this into two parts e.g.

#### Network prefetching

In order to avoid the CPU frequency spinning down between tests we prefetch all assets before any of the tests start in the browser (In the CLI we assume all assets are on disk). Assets are saved in a blob URL so they can be cached on disk. This lowers the peak memory footprint of the benchmark to a sustainable level.

#### Large Asset preloading
The JetStream driver (both in the browser and shell runners) _preloads_ some large assets and source files. This avoids extensive disk I/O from the memory.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good points, I'll use that tomorrow in a follow-up (or feel free to send a quick PR, whatever works).

That is, it reads those files from disk or fetches them from the network before running the workloads.
This is in order to exclude network latency and disk I/O from the benchmark measurements and reduce variance. Otherwise, OS scheduling or CPU frequency scaling may affect the measurement.

Some workloads also utilize large assets (e.g., ML models, heavy JavaScript bundles in the order of 10s of MBs).
In order to limit the repository size and network transfers, such large assets are stored as compressed .z files.
Preloading handles the decompression of these assets (using `DecompressionStream` or a Wasm-based zlib polyfill) upfront so that decompression overhead does not affect the benchmark score.

Both preloading and compression can be disabled, e.g., to inspect raw files or because it sometimes helps with debugging (e.g., proper URLs instead of Blobs for resources).

- Compression: Run `npm run decompress` to decompress all .z files before running the benchmark.
- No prefetching for shells: Pass the `--no-prefetch` flag, e.g., `jsc cli.js -- --no-prefetch`.
- No prefetching in browsers: Append the query parameter `?prefetchResources=false` to the URL.

See `JetStreamDriver.js` and `utils/compress.mjs` for more details.

### Score Calculation
Comment thread
danleh marked this conversation as resolved.

Scores in JetStream are dimensionless floats, where a higher score is better.
When scores are aggregated (e.g., multiple sub-scores for each workload, or to determine the total score of the full benchmark suite), we use the [geometric mean](https://en.wikipedia.org/wiki/Geometric_mean).
This ensures that an x% improvement of any individual score has the same effect on the aggregated score, regardless of the absolute value of the individual score.

See the [in-depth.html](https://webkit-jetstream-preview.netlify.app/in-depth.html) and `JetStreamDriver.js` for more details.
2 changes: 1 addition & 1 deletion in-depth.html
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@

<title>JetStream 3 In-Depth Analysis</title>

<link rel="stylesheet" href="ressources/JetStream.css">
<link rel="stylesheet" href="resources/JetStream.css">

</head>
<body class="overflow-scroll">
Expand Down
2 changes: 1 addition & 1 deletion index.html
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ <h1 class="logo">
</h1>

<main>
<p class="summary">JetStream 3 is a JavaScript and WebAssembly benchmark suite focused on the most advanced web applications. It rewards browsers that start up quickly, execute code quickly, and run smoothly. For more information, read the <a href="resources/in-depth.html">in-depth analysis</a>. Bigger scores are better.</p>
<p class="summary">JetStream 3 is a JavaScript and WebAssembly benchmark suite focused on the most advanced web applications. It rewards browsers that start up quickly, execute code quickly, and run smoothly. For more information, read the <a href="/in-depth.html">in-depth analysis</a>. Bigger scores are better.</p>
<div id="non-default-params">
<h2>Non-standard Parameters</h2>
<p>
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
"prepare": "node utils/version-check.mjs",
"server": "npm run prepare --silent && node tests/server.mjs",
"compress": "npm run prepare --silent && node utils/compress.mjs",
"decompress": "npm run prepare --silent && node utils/compress.mjs --decompress --keep ",
"decompress": "npm run prepare --silent && node utils/compress.mjs --decompress --keep",
"lint:check": "npm run prepare --silent && eslint **/*.{js,mjs,jsx,ts,tsx}",
"pretty:check": "npm run prepare --silent && prettier --check ./",
"pretty:format": "npm run prepare --silent && prettier --write ./",
Expand Down
Binary file added resources/screenshot.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading