Support wasm #91

danielgranhao · 2025-02-26T13:47:15Z

Resolves #90.

This PR:

Changes the http client from ureq to async reqwest
Switches to a different secp256k1-zkp fork that circumvents wasm compatibility issues
Changes the websocket crate from tungstenite to tokio-tungstenite-wasm - as the name indicates, this crate uses tokio-tungstenite for native builds
Adds esplora as a supported chain data backend. Electrum is still supported. Both can be enabled using features, allowing clients to switch backends at runtime (which may be useful for redundancy). Only esplora is set as a default so that wasm is supported by default.
Allows reusing esplora and electrum clients instead of having to create new ones on every method.

~~There are still some TODOs, but I would appreciate some initial feedback.~~

TODO:

Testing (adding new tests for esplora + run ignored tests)
- I've made the testnet and mainnet integration tests run against regtest and added regtest as a submodule. These tests now run in CI both on native and wasm builds.
Try to upstream changes to secp256k1-zkp
- I've rebased sanket's PR (on which we depended already) on top of the secp256k1-zkp main branch (to pull in a WASM fix) and added an additional workaround/fix on top. I won't be trying to upstream the changes because his PR has been dropped in favor of adding musig2 support on the base secp256k1 crate (WIP here). Hopefully soon a fork won't be needed any longer.

roeierez · 2025-02-26T17:39:36Z

src/swaps/liquid.rs

-        let electrum_client = network_config.clone().build_client()?;
+    pub async fn fetch_utxo<LC: LiquidClient, N: LiquidNetworkConfig<LC>>(
+        &self,
+        network_config: &N,


Passing a config forces this function to call build_client (and create a new client) which prevents from the caller to reuse a client implementation. In electrum we have seen it has a big performance penalty.
I know this was the implementation before this PR but perhaps if we are going for such a big change we can improve it?
I commented on this method as an example but this pattern repeats itself.

Thanks @roeierez. I agree. I've made the methods take a client instead of the config in a6d9d1c

roeierez

Looks good! Added two NITs.

src/network/electrum.rs

Co-authored-by: michael1011 <[email protected]>

tests/regtest/mod.rs

danielgranhao · 2025-03-10T17:33:37Z

There is an issue with the websocket client. tokio-tungstenite-wasm doesn't expose websocket pings because they are not available in the javascript API. IIUC some browsers handle it automatically. This wrapper crate tries to keep the API consistent between native and WASM, so it also hides the ping-pong API for native builds.

I see two main solution paths:

Fork the tokio-tungstenite-wasm in order to expose the ping API on native (eventually try to upstream it). Not sure how viable this is. Also, I'm not sure how reliable the auto-keep-alive implementation of the browsers is.
Implement a keep-alive ping-pong on the app level. This could be as simple as the text "ping" and "pong" but it requires adapting the boltz backend accordingly cc @michael1011. This is the safest option IMO and the overhead should be insignificant.

michael1011 · 2025-03-10T18:26:02Z

There is an issue with the websocket client. tokio-tungstenite-wasm doesn't expose websocket pings because they are not available in the javascript API. IIUC some browsers handle it automatically. This wrapper crate tries to keep the API consistent between native and WASM, so it also hides the ping-pong API for native builds.

I see two main solution paths:

Fork the tokio-tungstenite-wasm in order to expose the ping API on native (eventually try to upstream it). Not sure how viable this is. Also, I'm not sure how reliable the auto-keep-alive implementation of the browsers is.

Implement a keep-alive ping-pong on the app level. This could be as simple as the text "ping" and "pong" but it requires adapting the boltz backend accordingly cc @michael1011. This is the safest option IMO and the overhead should be insignificant.

I'd prefer a ping/pong JSON message in the application layer over having to fork

danielgranhao · 2025-03-11T13:33:47Z

I opened a PR on boltz-backend to add these ping/pong messages.

danielgranhao · 2025-03-12T01:35:05Z

In f48f3d5 I've adjusted the ws message structs to align with https://github.com/BoltzExchange/boltz-backend, including the newly added ping/pong messages. Added also regtest test for the pinging.

This is ready for another review round. cc @michael1011 @i5hi @roeierez

Cargo.toml

roeierez

LGTM

i5hi · 2025-03-12T14:06:46Z

Sorry about the delay in reviewing; closing up another release now and will take a look at this and test it tomorrow morning.

i5hi · 2025-03-14T04:38:52Z

Getting this error when I run the tests:

---- btc_submarine_refund stdout ----

thread 'btc_submarine_refund' panicked at tests/test_framework/mod.rs:46:14:
called `Result::unwrap()` on an `Err` value: JsonRpc(Transport(SocketError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" })))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- btc_reverse_claim_relative_fee stdout ----

Similar error for :
test btc_submarine_refund ... FAILED
test btc_reverse_claim_relative_fee ... FAILED
test btc_reverse_claim_size ... FAILED
test btc_reverse_claim ... FAILED
test btc_submarine_refund_size ... FAILED
test btc_submarine_refund_relative_fee ... FAILED

danielgranhao · 2025-03-14T11:05:44Z

Getting this error when I run the tests:
---- btc_submarine_refund stdout ----

thread 'btc_submarine_refund' panicked at tests/test_framework/mod.rs:46:14:
called `Result::unwrap()` on an `Err` value: JsonRpc(Transport(SocketError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" })))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- btc_reverse_claim_relative_fee stdout ----
Similar error for : test btc_submarine_refund ... FAILED test btc_reverse_claim_relative_fee ... FAILED test btc_reverse_claim_size ... FAILED test btc_reverse_claim ... FAILED test btc_submarine_refund_size ... FAILED test btc_submarine_refund_relative_fee ... FAILED

@i5hi Could you please try to run the same tests on trunk (cargo test)? Those tests also fail on my machine (I have MacOS, not sure if you do too), but run fine in CI. Because for me, they already fail on trunk, I intended to leave out of this PR any potential fix. My assumption is that there is some issue with the bitcoind crate on MacOS.

I've opened an issue #93 for this.

i5hi · 2025-03-14T16:23:41Z

Getting this error when I run the tests:
---- btc_submarine_refund stdout ----

thread 'btc_submarine_refund' panicked at tests/test_framework/mod.rs:46:14:
called `Result::unwrap()` on an `Err` value: JsonRpc(Transport(SocketError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" })))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- btc_reverse_claim_relative_fee stdout ----
Similar error for : test btc_submarine_refund ... FAILED test btc_reverse_claim_relative_fee ... FAILED test btc_reverse_claim_size ... FAILED test btc_reverse_claim ... FAILED test btc_submarine_refund_size ... FAILED test btc_submarine_refund_relative_fee ... FAILED
@i5hi Could you please try to run the same tests on trunk (cargo test)? Those tests also fail on my machine (I have MacOS, not sure if you do too), but run fine in CI. Because for me, they already fail on trunk, I intended to leave out of this PR any potential fix. My assumption is that there is some issue with the bitcoind crate on MacOS.

I've opened an issue #93 for this.

Yes, I had tested it on a mac. Tested on Linux and its all good.

i5hi · 2025-03-14T16:27:48Z

LGTM.

Thanks @danielgranhao !

roeierez reviewed Feb 26, 2025

View reviewed changes

danielgranhao added 12 commits February 27, 2025 15:14

Fix secp256k1-zkp WASM compatibility

ef22334

Enable js feature in getrandom

e5c4f25

Use async reqwest

0ebb55b

Use tokio-tungstenite-wasm for websocket stream

85d56dc

Add esplora support

c3af0f6

Adjust GH actions

89da463

Update README.md

9a353e7

Fix esplora timeout const name

04e2408

Esplora fixes

2ff289d

Use test macros where missing

3598ac2

Add missing esplora conditionals

2ea8ad7

Fix get swap

d7395ba

danielgranhao force-pushed the support-wasm branch from d4aee44 to d7395ba Compare February 27, 2025 15:28

Fix fmt lint

840404a

danielgranhao force-pushed the support-wasm branch 11 times, most recently from 0366f4c to 8ce332e Compare March 6, 2025 17:10

Run tests on regtest

8aab73a

danielgranhao force-pushed the support-wasm branch from 8ce332e to 8aab73a Compare March 6, 2025 18:02

danielgranhao added 2 commits March 7, 2025 12:21

Update secp256k1-zkp patch

e2b7034

Allow chain client reuse

a6d9d1c

michael1011 requested review from i5hi and roeierez and removed request for roeierez March 10, 2025 12:12

roeierez reviewed Mar 10, 2025

View reviewed changes

src/network/electrum.rs Outdated Show resolved Hide resolved

src/network/electrum.rs Outdated Show resolved Hide resolved

danielgranhao and others added 6 commits March 10, 2025 13:00

Add missing new line

c1caf6c

Co-authored-by: michael1011 <[email protected]>

Switch to vendored native tls on tokio-tungetenite

3cebee2

Expose WebSocketStream

3f7ee4c

Optimize electrum's fetch_utxos_core

2fb92c9

Rename fetch_utxo_core to extract_address_utxos

58f93e5

Split configs and chain into Bitcoin and Liquid counterparts

bd904fc

danielgranhao requested review from roeierez and michael1011 March 10, 2025 14:35

Expose entire tokio-tungstenite-wasm

d43ffc9

michael1011 reviewed Mar 10, 2025

View reviewed changes

tests/regtest/mod.rs Show resolved Hide resolved

Drop client configs

73a41a3

danielgranhao mentioned this pull request Mar 10, 2025

WASM: Update boltz client dependency and use included wasm tokio-tungstenite breez/breez-sdk-liquid#769

Merged

Refactor ws message structs and add ping/pong messages

f48f3d5

dangeross reviewed Mar 12, 2025

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

Bump crate version

662ab3b

roeierez approved these changes Mar 12, 2025

View reviewed changes

i5hi merged commit f78e159 into SatoshiPortal:trunk Mar 14, 2025
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support wasm #91

Support wasm #91

danielgranhao commented Feb 26, 2025 •

edited

Loading

roeierez Feb 26, 2025

danielgranhao Mar 7, 2025 •

edited

Loading

roeierez left a comment

danielgranhao commented Mar 10, 2025

michael1011 commented Mar 10, 2025

danielgranhao commented Mar 11, 2025

danielgranhao commented Mar 12, 2025

roeierez left a comment

i5hi commented Mar 12, 2025

i5hi commented Mar 14, 2025

danielgranhao commented Mar 14, 2025

i5hi commented Mar 14, 2025

i5hi commented Mar 14, 2025

Support wasm #91

Support wasm #91

Conversation

danielgranhao commented Feb 26, 2025 • edited Loading

roeierez Feb 26, 2025

Choose a reason for hiding this comment

danielgranhao Mar 7, 2025 • edited Loading

Choose a reason for hiding this comment

roeierez left a comment

Choose a reason for hiding this comment

danielgranhao commented Mar 10, 2025

michael1011 commented Mar 10, 2025

danielgranhao commented Mar 11, 2025

danielgranhao commented Mar 12, 2025

roeierez left a comment

Choose a reason for hiding this comment

i5hi commented Mar 12, 2025

i5hi commented Mar 14, 2025

danielgranhao commented Mar 14, 2025

i5hi commented Mar 14, 2025

i5hi commented Mar 14, 2025

danielgranhao commented Feb 26, 2025 •

edited

Loading

danielgranhao Mar 7, 2025 •

edited

Loading