Multiple NATS instances / same storage

[ealeykin]

Hi,

I got faulty setup and observed two NATS replicas running against same file storage.

Also i have a DLQ workqueue stream with advisory max deliveries subject.

So because of some health check issues I got two NATS running in parallel, one of those was constantly restarting next to the first one.

My scenario was to test DLQ, so target stream consumer was emulating an error - and I found that number of events in DLQ was constantly increasing, also number of events in target stream was staying at 3.

Also when looking into the DLQ info - I'm getting different number of events in DLQ stream, looks like because NLB load balancer is routing requests between two NATS instances and they return different results.

Although NATS setup in this case is faulty, supposed to be a simple single instance - why do I see above behaviour ?

Thanks
Eugene

[ripienaar]

Each instance requires an isolated storage.

[ealeykin]

@ripienaar Yes, that's clear. I'm mostly about rolling update scenario when I need to update NATS and in order to avoid downtime - I can start another NATS instance in parallel and then shutdown the old one.

So during some short period of time I will have two NATS instances running against same storage.

What is the recommended approach in this case (basically to avoid downtime) ?

[neilalexander]

You MUST NOT point two NATS instances to the same storage directory if there's even the remote possibility that they could run at the same time. If you need HA, you should use a cluster and replication.

[ripienaar]

With a cluster of 3 you can take one out and replace it without causing downtime. So that’s the solution.