Channel fails to close

[mgdotson]

Using the document example running against a local docker container: docker run -p5672:5672 -p 15672:15672 -p5671:5671 -p15692:15692 rabbitmq:3.9.17-management, putting the server into a blocking situation causes the channel.Close() to hang and not finish.

Steps to reproduce:

1. Start the container with docker run above
2. Start the server code and watch a couple of message confirmations
3. In the docker image, run: rabbitmqctl set_vm_memory_high_watermark 0.00000001
4. Watch the server code display "Push didn't confirm. Retrying..." (let 4 or so pass)
5. In the docker image, run: rabbitmqctl set_vm_memory_high_watermark 0.4
6. Notice Push confirmations happen
7. Context will time out and run the queue.Close()

At this point, the code will close the done channel but will hang on the queue.channel.Close() command.

Even if we wrap this in a context timeout to allow the calling code finish, if this is a long running process, this could cause leaks over time, especially if there are multiple channels that end up in this situation due to a blocked server.

This is also not the only scenario that can case a channel.Close() call to hang.

Best practices? TCP settings?

[Zerpet]

I can reproduce this problem. The root cause seems to be a confirm notification blocked on sending to a channel. Channel close tries to nack and close all outstanding confirm notifications (since they haven't been delivered to the user), and acquires a mutex lock on the Confirm struct for this. Confirming one message with Confirm.Once() also acquires a mutex lock on the Confirm struct. Channel close function is blocked waiting for Confirm.Once() to finish, which won't finish. I'm going to look into this.

[Zerpet]

Good call. I created #122 for this. It's a problem with the example, the client library is ok and behaves as it describes in Channel.Close. The problem is that the example "gives up" on some confirmations and moves sends a new message. However, the library still keeps track of those "given up" confirmations, which are expected to be consumed.

[mgdotson]

Having this documented will be incredibly useful. I had questions on the "consuming until closed piece" as well. And, most client code is going to have some sort of timeout with any of the mq calls which could easily lead to deadlocks or resource leaks.

Having the library take contexts for all the calls will help eliminate/reduce these types of issues.

Again, thank you for looking into this!

[lukebakken]

@mgdotson we would welcome a PR that adds contexts where appropriate in this library. That would make a great addition to a version 2.0 release.

[mgdotson]

@lukebakken Will do! But in short, I've found almost all of them need contexts. For example, in blocking mode, Exchange/Queue Creation/Binding work until a publish/notify is done, then even Exchange and Queue commands "hang" waiting on blocking to be resolved.

This means wrapping every function in a go func timer wrapper in order to meet SLA's to clients while ensuring that the message is actually enqueued.

I'm in the process of re-writing our internal package wrapper so I'll make notes and look into PR's if I can - if nothing else, I'll create issues requests documenting reproducible test cases.

[lukebakken]

if nothing else, I'll create issues requests documenting reproducible test cases

That would be super useful in itself!

[thuan1412]

Hi, I still have this problem.
My environment is use

• amqp091-go: 1.8.1
• rabbitmq: docker bitnami/rabbitmq:3.7.18-debian-9-r1

The code is taken from https://github.com/rabbitmq/rabbitmq-tutorials/blob/main/go/send.go

[thuan1412]

I also tried code from https://github.com/rabbitmq/amqp091-go/blob/main/_examples/producer/producer.go and still got hang infinitely when closing the channel

2023/09/01 22:44:45 [INFO] producer: dialing amqp://admin:[email protected]:5672
2023/09/01 22:44:45 [INFO] confirm handler: sent OK to publish
2023/09/01 22:44:45 [INFO] producer: got Connection, getting Channel
2023/09/01 22:44:45 [INFO] producer: declaring exchange
2023/09/01 22:44:45 [INFO] producer: declaring queue 'test-queue'
2023/09/01 22:44:45 [INFO] producer: declared queue ("test-queue" 0 messages, 0 consumers), binding to Exchange (key "test-key")
2023/09/01 22:44:45 [INFO] producer: declaring binding
2023/09/01 22:44:45 [INFO] producer: enabling publisher confirms.
2023/09/01 22:44:45 [INFO] producer: waiting on the OK to publish...
2023/09/01 22:44:45 [INFO] producer: got the OK to publish
2023/09/01 22:44:45 [INFO] producer: publishing 6B body ("foobar")
2023/09/01 22:44:45 [INFO] producer: delivered deferred confirm to handler
2023/09/01 22:44:45 [INFO] confirm handler: checking 1 outstanding confirmations
2023/09/01 22:44:45 [INFO] confirm handler: sent OK to publish
2023/09/01 22:44:45 [INFO] producer: initiating stop
2023/09/01 22:44:45 [INFO] confirm handler: exit requested
2023/09/01 22:44:45 [INFO] confirm handler: waiting on 1 outstanding confirmations
2023/09/01 22:44:45 [INFO] confirm handler: checking 1 outstanding confirmations
2023/09/01 22:44:46 [WARNING] confirm handler: did not receive confirmation for tag 1
2023/09/01 22:44:46 [ERROR] confirm handler: exiting with 1 outstanding confirmations
2023/09/01 22:44:46 [INFO] confirm handler: exiting
2023/09/01 22:44:46 [INFO] producer: stopping, all confirms seen
^C2023/09/01 22:44:51 [INFO] close handler: Ctrl+C pressed in Terminal
panic: close of closed channel

goroutine 19 [running]:
main.setupCloseHandler.func1()
	/tmp/test/main.go:55 +0x51
created by main.setupCloseHandler
	/tmp/test/main.go:52 +0xd9
exit status 2

[thuan1412]

I've tried with image rabbitmq:3.12-management and it worked, seems like the problem comes from the docker image, not the library.

But I think the library should return an error or set a timeout in order to improve the DX. It took me hours to resolve this problem.

[lukebakken]

RabbitMQ 3.7 is completely out of support. Since 3.12 works, we won't investigate further unless you can reproduce this issue with that version. Thanks.

[lukebakken]

@thuan1412 just for my information, did you make ANY changes to the example code?

[thuan1412]

RabbitMQ 3.7 is completely out of support. Since 3.12 works, we won't investigate further unless you can reproduce this issue with that version. Thanks.

Yeah, I've changed to use 3.12.

@thuan1412 just for my information, did you make ANY changes to the example code?

No, I didn't.