[Bug]: Configuration doesn't fully restore

[overlord23]

Category

Other

Hardware

Heltec V3

Firmware Version

2.5.11.8e2a3e5

Description

The configuration doesn't fully restore on Heltec V3 and Heltec Wireless Stick Lite.
Here the applied steps to reproduce the problem

• issued nodedb reset and factory reset
• configured node and exported config with Meshtastic CLI
• issued nodedb reset and factory reset again
• exported config from Meshtastic CLI and compared with the previously exported yml

The following parameters included in the configuration file didn't restore:

• autoScreenCarouselSecs:
• txPower
• adcMultiplierOverride
• root (lora region)

Here the serial and debug log
debug_aftrestore.txt
serial_log_cfgrestore.txt

This bug has been discussed in #5331 for reference.

Relevant log output

No response

[ianmcorvidae]

@garthvh I think this is, in fact, a firmware bug. Python is just starting a settings transaction, sending the settings, and then committing; it seems like firmware is what isn't saving these values appropriately. There's been sporadic reports of this, particularly since 2.5. (noting, also, that some of these reports are from Android, which also uses the transaction mechanism)

@overlord23 if possible, could you also share the yaml file you're using to import, and the full output of meshtastic --debug --configure <file.yaml> (replacing <file.yaml> with your exported configuration, of course). I'd like to confirm that python is in fact sending the expected values, which don't all appear in the serial log.

[lucaskatayama]

I get something like this...

Export config

1. Change mqtt.enable from true to false
2. Apply with --configure <YAML> --port /dev/<PORT>
3. Doesn't change anything...

BUT

Using bluetooth (--ble <name>), it works...
It only happens with serial port ....

[fliphess]

unstale

[overlord23]

@garthvh I think this is, in fact, a firmware bug. Python is just starting a settings transaction, sending the settings, and then committing; it seems like firmware is what isn't saving these values appropriately. There's been sporadic reports of this, particularly since 2.5. (noting, also, that some of these reports are from Android, which also uses the transaction mechanism)

@overlord23 if possible, could you also share the yaml file you're using to import, and the full output of meshtastic --debug --configure <file.yaml> (replacing <file.yaml> with your exported configuration, of course). I'd like to confirm that python is in fact sending the expected values, which don't all appear in the serial log.

Here is the debug log.
Just for reference it has been collected using meshtastic cli 2.5.12 running firmware 2.5.20.

debug.log

[pdxlocations]

Adding some logs for another failed attempt to restore. I just randomly set some configs and channels to test. Some settings do stick but you can see I'm missing at least Channels and Display:

afterconfig.txt
config-command.txt
config-log.txt
config.yaml.txt

[syssi]

I can confirm the issue. A simple

meshtastic --port /dev/ttyACM0 --export-config > config.yaml
meshtastic --port /dev/ttyACM0 --configure config.yaml

isn't able to restore the configuration. If I reduce the exported configuration to a single paragraph:

config:
  security:
    privateKey: base64:...
    publicKey: base64:...
    serialEnabled: true

The setting doesn't get applied. Verified by

meshtastic --port /dev/ttyACM0 --export-config

[bodhar]

meshtastic-cli: 2.6.2
Device: Heltec V3
Device meshtastic version: 2.6.7

---

I can confirm that, while exporting works, using --configure to restore in the backed up configuration to a freshly wiped device doesn't restore the settings.

The output from meshtastic --configure ... does print out the correct details when running through the file and does successfully connect to the Heltec V3 I'm testing with, successfully sending a reboot command, but on reboot I need to configure everything from scratch.

Happy to help debug if anything is required in addition to the information provided by others reporting issues.

[interfect]

I'm also seeing this issue, with both a T3S3 an a LoRa32 v2.1_1.6. The transaction that is supposedly started does not actually apply, and some sent settings are updated but not others. If I cut down my YAML I am trying to apply to small pieces and do those one at a time, I can eventually get them all applied.

Adding some logs for another failed attempt to restore. I just randomly set some configs and channels to test. Some settings do stick but you can see I'm missing at least Channels and Display:

afterconfig.txt config-command.txt config-log.txt config.yaml.txt

@pdxlocations I compared your config-command.txt to your config-log.txt, and it looks like while the config command thinks it is starting a transaction (INFO file:node.py beginSettingsTransaction line:580 Telling open a transaction to edit settings), the node never actually processes a start-transaction packet. We see on line 432 INFO | ??:??:?? 38 [Router] Handle admin payload 65 (which is the payload for committing a transaction), but we never see a payload 64 (which is the one to open a transaction).

Before it processes any of the packets, the radio log has a bunch of lines about PACKET FROM PHONE, so it looks like the config changes are all being sent before any of them are actually handled. And it looks like the later config update packets are kicking the earlier ones out of the buffer somehow:

DEBUG | ??:??:?? 35 [ServerAPI] PACKET FROM PHONE (id=0x48a9d0a9 fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)
DEBUG | ??:??:?? 35 [ServerAPI] Enqueued local (id=0x48a9d0a9 fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)
DEBUG | ??:??:?? 36 [ServerAPI] PACKET FROM PHONE (id=0x84e13caa fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)
DEBUG | ??:??:?? 36 [ServerAPI] Enqueued local (id=0x84e13caa fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)
DEBUG | ??:??:?? 36 [ServerAPI] PACKET FROM PHONE (id=0x2bc9b4ab fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)
DEBUG | ??:??:?? 36 [ServerAPI] Enqueued local (id=0x2bc9b4ab fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)
DEBUG | ??:??:?? 36 [ServerAPI] PACKET FROM PHONE (id=0x203f08ac fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)
DEBUG | ??:??:?? 36 [ServerAPI] Enqueued local (id=0x203f08ac fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)
DEBUG | ??:??:?? 36 [ServerAPI] PACKET FROM PHONE (id=0x1fc15cad fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)
DEBUG | ??:??:?? 36 [ServerAPI] Enqueued local (id=0x1fc15cad fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)
DEBUG | ??:??:?? 36 [ServerAPI] fromRadioQ full, drop oldest! (id=0x48a9d0a9 fr=0x00000000 to=0xfa9d1adc, WantAck=0, HopLim=3 Ch=0x0 Portnum=6 WANTRESP PKI priority=70)

See the fromRadioQ full, drop oldest!? It gets about 4 packets, and then the 5th one comes in and it starts dropping old packets to make room for new ones, including the initial packet that was meant to start the transaction. Then we have 4 repetitions of Handle admin payload, with the last one being payload 65 to close out the transaction that was never opened.

So the meshtastic CLI is sending the config packets too fast, or not actually waiting for the response from one packet being actually handled before sending the next, and it's filling up the 4-packet-big queue and dropping all but the last few packets. The transaction never opens, and whetever's in the last few packets gets applied as soon as it comes in, without any transaction protection.

So the solution is probably to send the packets slower.