fork observer deployment issues

[pinheadmz]

1. there's a race condition where FO won't see all the tanks when it starts up, it should probably wait until all tanks are alive and running before we deploy.
2. it should be namespace restricted to default so it doesnt pick up armada tanks
3. it should be set to restart on failure so it can be manually restarted with a kill

[willcl-ark]

How did 1) manifest?

I thought fork observer was quite resilient to configured nodes not being available and would continually try to (re)connect…

Ping @0xB10C

[0xB10C]

Correct, it will try to reconnect to nodes, but not try to get the node version again if it failed at the beginning (this is done as the RPC getnetworkinfo is an optional RPC for fork-observer, and not a strict requirement). See https://github.com/0xB10C/fork-observer/blob/6f6d33f0f8cbdd201f9f062e0638a47750871ff3/src/main.rs#L777-L813

It retries for 5 times and waits 10s after each failed attempt. Since you need the versions for the warnet game, might want to start fork-observer after the nodes are RPC reachable

[0xB10C]

If you have fork-observer logs, it should also tell you why it didn't get the versions for the nodes