Skip to content

cts is not executed by ci workflow, cts_runner seems broken #6838

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
turbocrime opened this issue Dec 30, 2024 · 4 comments · Fixed by #7675
Closed

cts is not executed by ci workflow, cts_runner seems broken #6838

turbocrime opened this issue Dec 30, 2024 · 4 comments · Fixed by #7675
Labels
area: infrastructure Testing, building, coordinating issues type: enhancement New feature or request

Comments

@turbocrime
Copy link
Contributor

Description

it seems like the cts workflow was removed, and thus this repository's ci has no way to use cts_runner since 34e947d

34e947d#diff-6867f1034f74cf18cb5e7345b15c2b5ae21eb47bb0a53d2b0c3bb4947753f4b6

cts_runner execution has since broken (rust panic aborts execution instead of simply failing), and before that, some cts pass/fail status regressed.

Expected vs observed behavior

cts_runner should not panic.

ci workflow should at least optionally execute deno cts. ideally, success of each individual test could be recorded periodically, or for each set of changes.

if cts_runner is executed and tracked by some other system, this should be documented and public. if cts_runner is no longer used, it should be removed.

Extra materials

cts_runner panics for everything in test.lst including unittests on 0f5f058 (tip of trunk)

cts_runner passes two sets of tests on 5543961 (firefox dependency)

passing:

unittests:*
webgpu:api,operation,command_buffer,basic:*

panic and abort:

webgpu:api,operation,compute,basic:*

=== Running webgpu:api,operation,compute,basic:* ===
    Finished dev [unoptimized + debuginfo] target(s) in 0.13s
     Running `/Users/yet/Developer/github.com/gfx-rs/wgpu/target/debug/cts_runner ./tools/run_deno --verbose 'webgpu:api,operation,compute,basic:*'`
thread 'main' panicked at /Users/yet/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.272.0/gotham_state.rs:82:3:
required type wgpu_core::global::Global is not present in GothamState container
stack backtrace:
   0: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: deno_core::gotham_state::missing
             at /Users/yet/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.272.0/gotham_state.rs:82:3
   3: deno_core::gotham_state::GothamState::borrow::{{closure}}
             at /Users/yet/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.272.0/gotham_state.rs:42:41
   4: core::option::Option<T>::unwrap_or_else
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/option.rs:976:21
   5: deno_core::gotham_state::GothamState::borrow
             at /Users/yet/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.272.0/gotham_state.rs:42:5
   6: deno_webgpu::compute_pass::op_webgpu_compute_pass_set_pipeline::call
             at /Users/yet/Developer/github.com/gfx-rs/wgpu/deno_webgpu/compute_pass.rs:34:5
   7: deno_webgpu::compute_pass::op_webgpu_compute_pass_set_pipeline::slow_function_impl
             at /Users/yet/Developer/github.com/gfx-rs/wgpu/deno_webgpu/compute_pass.rs:20:1
   8: deno_webgpu::compute_pass::op_webgpu_compute_pass_set_pipeline::v8_fn_ptr
             at /Users/yet/Developer/github.com/gfx-rs/wgpu/deno_webgpu/compute_pass.rs:20:1
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5

exception and failure:

webgpu:api,operation,compute,basic:*
webgpu:api,operation,rendering,basic:clear:*
webgpu:api,operation,rendering,basic:fullscreen_quad:*

webgpu:api,operation,rendering,blending:*
webgpu:api,operation,rendering,blending:GPUBlendComponent:*
webgpu:api,operation,rendering,depth:*
webgpu:api,operation,rendering,draw:*

all seem to fail the same way

=== Running webgpu:api,operation,rendering,basic:clear:* ===
    Finished dev [unoptimized + debuginfo] target(s) in 0.15s
     Running `/Users/yet/Developer/github.com/gfx-rs/wgpu/target/debug/cts_runner ./tools/run_deno --verbose 'webgpu:api,operation,rendering,basic:clear:*'`
[fail] webgpu:api,operation,rendering,basic:clear: (20ms). Log:
  - EXCEPTION: Bad resource ID
    BadResource: Bad resource ID
        at GPURenderPassEncoder.end (ext:deno_webgpu/01_webgpu.js:3833:21)
        at RunCaseSpecific.fn (file:///Users/yet/Developer/github.com/gfx-rs/wgpu/cts/out/webgpu/api/operation/rendering/basic.spec.js:36:8)

** Failures **
[fail] webgpu:api,operation,rendering,basic:clear: (20ms). Log:
  - EXCEPTION: Bad resource ID
    BadResource: Bad resource ID
        at GPURenderPassEncoder.end (ext:deno_webgpu/01_webgpu.js:3833:21)
        at RunCaseSpecific.fn (file:///Users/yet/Developer/github.com/gfx-rs/wgpu/cts/out/webgpu/api/operation/rendering/basic.spec.js:36:8)

** Summary **
Passed  w/o warnings = 0 / 1 =   0.00%
Passed with warnings = 0 / 1 =   0.00%
Skipped              = 0 / 1 =   0.00%
Failed               = 1 / 1 = 100.00%
=== Running webgpu:api,operation,rendering,basic:fullscreen_quad:* ===
    Finished dev [unoptimized + debuginfo] target(s) in 0.12s
     Running `/Users/yet/Developer/github.com/gfx-rs/wgpu/target/debug/cts_runner ./tools/run_deno --verbose 'webgpu:api,operation,rendering,basic:fullscreen_quad:*'`
[fail] webgpu:api,operation,rendering,basic:fullscreen_quad: (24ms). Log:
  - EXCEPTION: serde_v8 error: invalid type; expected: string, got: Number
    TypeError: serde_v8 error: invalid type; expected: string, got: Number
        at GPUDevice.createRenderPipeline (ext:deno_webgpu/01_webgpu.js:1441:26)
        at RunCaseSpecific.fn (file:///Users/yet/Developer/github.com/gfx-rs/wgpu/cts/out/webgpu/api/operation/rendering/basic.spec.js:60:29)

** Failures **
[fail] webgpu:api,operation,rendering,basic:fullscreen_quad: (24ms). Log:
  - EXCEPTION: serde_v8 error: invalid type; expected: string, got: Number
    TypeError: serde_v8 error: invalid type; expected: string, got: Number
        at GPUDevice.createRenderPipeline (ext:deno_webgpu/01_webgpu.js:1441:26)
        at RunCaseSpecific.fn (file:///Users/yet/Developer/github.com/gfx-rs/wgpu/cts/out/webgpu/api/operation/rendering/basic.spec.js:60:29)

** Summary **
Passed  w/o warnings = 0 / 1 =   0.00%
Passed with warnings = 0 / 1 =   0.00%
Skipped              = 0 / 1 =   0.00%
Failed               = 1 / 1 = 100.00%

Platform

i'm running cts on macos sonoma and an m3 pro.

i understand macos was removed from CI test coverage some time ago, but all cargo xtask test (except stack depth) is still successful on 5543961 and tip of trunk.
@turbocrime
Copy link
Contributor Author

pr to repair cts_runner in #6840 but i will leave ci and workflow to maintainers

@cwfitzgerald
Copy link
Member

We haven't really maintained the cts_runner of late, as we have two downstream projects running the cts (firefox and servo) - that said we'd definitely like this to keep working.

@cwfitzgerald
Copy link
Member

i understand macos was removed from CI test coverage some time ago

It was added back shortly after, it was an administrative thing

I'll see if I can tackle putting the cts job back

@cwfitzgerald cwfitzgerald added type: enhancement New feature or request area: infrastructure Testing, building, coordinating issues labels Jan 11, 2025
@andyleiserson
Copy link
Contributor

It looks like there are some additional problems with the CTS after #7113. I had to update some deno crate versions, and I had to add a call to webgpu.initGPU in bootstrap.js. But there are some additional problems that I was not immediately able to resolve:

  • Installing a device lost callback is no longer working, instead it throws TypeError "Cannot read properties of undefined (reading 'then')".
  • Before the object wrap change, onSubmittedWorkDone returned a promise that immediately resolved. Now, onSubmittedWorkDone raises an error that it is not implemented.

andyleiserson added a commit to andyleiserson/wgpu that referenced this issue May 7, 2025
@andyleiserson
Revive the CTS job
764afe0 
Fixes gfx-rs#6838
@andyleiserson andyleiserson mentioned this issue May 7, 2025
Merged
6 tasks
andyleiserson added a commit to andyleiserson/wgpu that referenced this issue May 7, 2025
@andyleiserson
Revive the CTS job
04c96eb 
Fixes gfx-rs#6838
andyleiserson added a commit to andyleiserson/wgpu that referenced this issue May 7, 2025
@andyleiserson
Revive the CTS job
27be211 
Fixes gfx-rs#6838
andyleiserson added a commit to andyleiserson/wgpu that referenced this issue May 7, 2025
@andyleiserson
Revive the CTS job
2f220ec 
Fixes gfx-rs#6838
andyleiserson added a commit to andyleiserson/wgpu that referenced this issue May 7, 2025
@andyleiserson
Revive the CTS job
e508f7c 
Fixes gfx-rs#6838
@andyleiserson andyleiserson mentioned this issue May 14, 2025
Merged
6 tasks
andyleiserson added a commit to andyleiserson/wgpu that referenced this issue May 20, 2025
@andyleiserson
Revive the CTS job
4c59919 
Includes the following minor functional fixes to deno_webgpu:
* Don't throw an error immediately when `create_buffer` is called
  with invalid usage flags.
* Implement `on_submitted_work_done`.
* Correct validation of GPUExtent3D element count.
* Run without tracing (instead of panic) if the DENO_WEBGPU_TRACE env
  var is not set.

Fixes gfx-rs#6838
@github-project-automation github-project-automation bot moved this from Todo to Done in WebGPU for Firefox May 21, 2025
@github-project-automation github-project-automation bot moved this from Todo to Done in WebGPU for Firefox May 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: infrastructure Testing, building, coordinating issues type: enhancement New feature or request
Projects
WebGPU for Firefox
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revive the CTS job andyleiserson/wgpu
3 participants
@cwfitzgerald @andyleiserson @turbocrime