Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic: Response JSON should have an id #9314

Open
emersonian opened this issue Mar 1, 2025 · 4 comments · May be fixed by #9341
Open

Panic: Response JSON should have an id #9314

emersonian opened this issue Mar 1, 2025 · 4 comments · May be fixed by #9341

Comments

@emersonian
Copy link

emersonian commented Mar 1, 2025 •
edited
Loading

One of our Zebra v2.2.0 mainnet servers is panicking in production. Lightwalletd is its only client.

The crash only occurs if lightwalletd is running before Zebra starts, pointed at Zebra's RPC. If lightwalletd is started after Zebra is running the panic does not occur.

It is crashing over and over again with the following logs:

2025-03-01T19:50:03.121072Z  INFO {zebrad="6d01f05" net="Main"}: zebrad::commands::start: spawned initial Zebra tasks
2025-03-01T19:50:03.121086Z  INFO {zebrad="6d01f05" net="Main"}: zebrad::components::mempool::gossip: initializing transaction gossip task
2025-03-01T19:50:03.121080Z  INFO {zebrad="6d01f05" net="Main"}: zebrad::components::sync::gossip: initializing block gossip task
2025-03-01T19:50:03.121129Z  INFO {zebrad="6d01f05" net="Main"}: zebrad::components::sync::end_of_support: Starting end of support task
2025-03-01T19:50:03.121151Z  INFO {zebrad="6d01f05" net="Main"}: zebrad::components::mempool::crawler: initializing mempool crawler task
2025-03-01T19:50:03.121287Z  INFO {zebrad="6d01f05" net="Main"}: zebra_state::config: checking for old database versions db_kind="state"
2025-03-01T19:50:03.121409Z  INFO {zebrad="6d01f05" net="Main"}: zebra_state::config: finished old database version cleanup task
2025-03-01T19:50:03.121923Z  INFO {zebrad="6d01f05" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2838566))
2025-03-01T19:50:03.556200Z  INFO {zebrad="6d01f05" net="Main"}:sync:try_to_sync:obtain_tips: zebra_consensus::primitives::groth16::params: checking and loading Zcash Sapling and Sprout parameters
2025-03-01T19:50:03.985873Z  INFO {zebrad="6d01f05" net="Main"}:sync:try_to_sync:obtain_tips: zebra_consensus::primitives::groth16::params: Zcash Sapling and Sprout parameters loaded and verified
2025-03-01T19:50:04.207445Z  INFO {zebrad="6d01f05" net="Main"}:init{config=Config { checkpoint_sync: true } network=Mainnet}: zebra_consensus::router: finished state checkpoint validation
The application panicked (crashed).
Message:  response json should have an id

Error

response json should have an id

Metadata

key value
version 2.2.0
Zcash network Mainnet
running state version 26.0.0
initial disk state version 26.0.0
features default,default_release_binaries,getblocktemplate_rpcs,howudoin,indicatif,progress_bar,release_max_level_info,sentry
git commit 6d01f05
target triple x86_64-unknown-linux-gnu
rust compiler 1.82.0
rust release date 2024-10-15
optimization level 3
debug checks false
location zebra-rpc/src/server/rpc_call_compatibility.rs:56:26

Backtrace

Backtrace:
   0: color_eyre::config::PanicHook::panic_report
   1: <zebrad::application::ZebradApp as abscissa_core::application::Application>::register_components::{{closure}}
   2: std::panicking::rust_panic_with_hook
   3: std::panicking::begin_panic_handler::{{closure}}
   4: std::sys::backtrace::__rust_end_short_backtrace
   5: rust_begin_unwind
   6: core::panicking::panic_fmt
   7: core::option::expect_failed
   8: <zebra_rpc::server::rpc_call_compatibility::FixRpcResponseMiddleware as jsonrpsee_server::middleware::rpc::RpcServiceT>::call::{{closure}}
   9: <jsonrpsee_server::server::TowerServiceNoHttp<RpcMiddleware> as tower_service::Service<http::request::Request<Body>>>::call::{{closure}}
  10: <zebra_rpc::server::http_request_compatibility::HttpRequestMiddleware<S> as tower_service::Service<http::request::Request<jsonrpsee_core::http_helpers::Body>>>::call::{{closure}}
  11: <jsonrpsee_server::utils::TowerToHyperServiceFuture<S,R> as core::future::future::Future>::poll
  12: <hyper_util::server::conn::auto::UpgradeableConnection<I,S,E> as core::future::future::Future>::poll
  13: jsonrpsee_server::server::process_connection::{{closure}}
  14: tokio::runtime::task::core::Core<T,S>::poll
  15: tokio::runtime::task::harness::Harness<T,S>::poll
  16: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  17: tokio::runtime::scheduler::multi_thread::worker::Context::run
  18: tokio::runtime::context::scoped::Scoped<T>::set
  19: tokio::runtime::context::runtime::enter_runtime
  20: tokio::runtime::scheduler::multi_thread::worker::run
  21: tokio::runtime::task::core::Core<T,S>::poll
  22: tokio::runtime::task::harness::Harness<T,S>::poll
  23: tokio::runtime::blocking::pool::Inner::run
  24: std::sys::backtrace::__rust_begin_short_backtrace
  25: core::ops::function::FnOnce::call_once{{vtable.shim}}
  26: std::sys::pal::unix::thread::Thread::new::thread_start
  27: <unknown>
  28: <unknown>
@github-project-automation github-project-automation bot moved this to New in Zebra Mar 1, 2025
@mpguerra
Copy link
Contributor

mpguerra commented Mar 4, 2025

Do we know what lightwalletd sent to zebra to cause the panic?

@emersonian
Copy link
Author

Not yet, I can try to add some logging and track it down but it will be in around two weeks out once my schedule opens up a bit.

@conradoplg
Copy link
Collaborator

I can't seem to reproduce this. Do you have the lightwalletd version and logs?

@str4d
Copy link
Contributor

str4d commented Mar 21, 2025

I have reproduced this in Zallet's JSON-RPC (which was based on zebra-rpc). What appears to be happening is that in some cases, JSON-RPC IDs are numbers instead of strings, but zebra-rpc assumes they are always strings:

zebra/zebra-rpc/src/server/rpc_call_compatibility.rs

Lines 54 to 62 in f0140a4

let id = json["id"]
.as_str()
.expect("response json should have an id")
.to_string();
return MethodResponse::error(
jsonrpsee_types::Id::Str(id.into()),
ErrorObject::borrowed(new_error_code, "Invalid params", None),
);

str4d added a commit to zcash/wallet that referenced this issue Mar 21, 2025
@str4d
rpc: Permit JSON-RPC IDs to be non-strings
e174e6f 
This is ZcashFoundation/zebra#9314 (the code in question was reused in
Zallet).
@str4d str4d mentioned this issue Mar 21, 2025
Merged
str4d added a commit to str4d/zebra that referenced this issue Mar 21, 2025
@str4d
rpc: Permit JSON-RPC IDs to be non-strings
7c84ec6 
Fixes ZcashFoundation#9314.
@str4d str4d linked a pull request Mar 21, 2025 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Zebra
Status: New
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rpc: Permit JSON-RPC IDs to be non-strings str4d/zebra
4 participants
@conradoplg @mpguerra @str4d @emersonian