ERR re-enqueueing failed block: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg

[RaulBernal]

Bug description

I get this error in loop:

11:07AM DBG processing block height=11134297
11:07AM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=11134297
11:07AM DBG processing block height=11134297
11:07AM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=12324351

As consequence, the MainNET explorer is 10 hours behind the current synced block.

I have another instance (a backup server) running the same software and is not stucked in the same ERR

Steps to reproduce

BDJuno version is (both servers):

bdjuno version 
version: 4.0.0-51-g340f59b4
commit: 340f59b4d0b3faeb00aa99bf69f17a5b95339dfb
go: go1.21.6 linux/amd64

Expected behavior

Expected a bypass if it can't parse the message not a stuck :)

[RaulBernal]

Maybe this issue could be related with this other
cosmos/relayer#1359 (comment)
and this race condition:
cosmos/relayer#1363
@riccardom

[RaulBernal]

I've asked related validators that please turn off the buggy relayer until I check if is related

[RaulBernal]

Maybe this issue could be related with this other cosmos/relayer#1359 (comment) and this race condition: cosmos/relayer#1363 @riccardom

Not related, is about Client Updates, not about IBC acks (so forget that comment)

[MonikaCat]

Hey @RaulBernal can I have a link to your bdjuno implementation? I think you can first try replacing juno version to the latest one v5.2.1-0.20240201075935-851426ddd905 and pulling any changes from BDJuno branch cosmos/v0.47.x Also you should add new unbonding_tokens column inside staking_pool table to avoid the staking pool error

[RaulBernal]

Hey @RaulBernal can I have a link to your bdjuno implementation? I think you can first try replacing juno version to the latest one v5.2.1-0.20240201075935-851426ddd905 and pulling any changes from BDJuno branch cosmos/v0.47.x Also you should add new unbonding_tokens column inside staking_pool table to avoid the staking pool error

Upgraded and running, but error still there:
10:01AM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=12401971

How can I link you? (maybe pass me your SSH public key by DM using email/telegram/discord ? )

[RaulBernal]

For give you more context:

• Last week we upgraded SDK from v0.46 to v0.47
• We upgraded the bdjuno from to v0.47 branch

[RaulBernal]

So maybe I need to exec: bdjuno migrate v5 @MonikaCat

I have two identical servers; If I try to make the migration in one of them I get:
bdjuno migrate v5 ERROR: error while migrating database: pq: relation "message_type" does not exist

[RaulBernal]

BTW added two collumns to staking_pool to fix one of the errors:

• unbonding_tokens
• staked_not_bonded_tokens

[rider-cantuaria-eng]

3:25PM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=14582098

Any idea guys?

running this config

on callisto branch cosmos/v0.47.x branch

[RaulBernal]

Hey @RaulBernal can I have a link to your bdjuno implementation? I think you can first try replacing juno version to the latest one v5.2.1-0.20240201075935-851426ddd905 and pulling any changes from BDJuno branch cosmos/v0.47.x Also you should add new unbonding_tokens column inside staking_pool table to avoid the staking pool error

Certainly, error has disappears adding:

• unbonding_tokens
• staked_not_bonded_tokens

[RaulBernal]

Hi @MonikaCat I'm using BDjuno branch cosmos/v0.47.x with no changes.
I started a new fresh installation using this release of juno v5.2.1-0.20240201075935-851426ddd905.

Everything started smooth but again:

10:31AM DBG processing block height=14729369
10:31AM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=14729369
10:31AM DBG processing block height=14729369
10:31AM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=14729369
10:31AM DBG processing block height=14729369
10:31AM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=14729369
10:31AM DBG processing block height=14729369
10:31AM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=14729369
10:31AM DBG processing block height=14729369
10:31AM DBG processing block height=14729369
10:31AM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=14729369
10:31AM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=14729369
10:31AM DBG processing block height=14729369
10:31AM DBG processing block height=14729574
10:31AM ERR re-enqueueing failed block err="failed to get transactions for block: error while unpacking message: no concrete type registered for type URL /ibc.core.client.v1.MsgUpdateClient against interface *types.Msg" height=14729574

It stucks the live sync (https://explorer2.bitcanna.io)

When I check in other explorer the block 14729369: (https://www.mintscan.io/bitcanna/block/14729369?chainId=bitcanna-1)
I get some IBC client updates & IBC ACKs