Segmentation fault/Double-linked list corruption in rxm_close_conn due to rxm_flush_msg_cq

[dsciebu]

Environment

• Libfabric Version: v2.4.0
• Provider: verbs;rxm
• Endpoint Type: FI_EP_RDM
• OS: Ubuntu 22.04 / 24.04
• Architecture: Reproduced on both aarch64 and x86_64

Description

I am encountering intermittent crashes during the teardown phase of a library built on Libfabric. The error typically manifests as a segmentation fault or a corrupted double-linked list error.

Reproducer Scenario

The test involves $N$ nodes communicating over RDM verbs endpoints:

1. Node A (Inactive): Endpoint is opened but performs no traffic.
2. Node B (Sender): Performs fi_send to all other nodes but never posts receives (fi_recv).
3. Remaining Nodes: Perform standard bidirectional communication.

As I implemented timeout for communication, after the specified time it stops retrying and starts "closing" the communication. It calls fi_shutdown, then destroys memory region and deletes the data buffer. After that, the endpoint gets closed with fi_close.
When closing the endpoints, the application intermittently crashes.

Analysis & Debugging

I have traced the issue to the function rxm_flush_msg_cq, which is called internally by rxm_close_conn.

• Observation: During the flush, it appears that a fi_recv is being posted to the underlying MSG provider.
• The Conflict: If the application has already started cleaning up or deallocating data buffers (which does take place in my case) at the time of the fi_close call, this internal receive triggers memory corruption or a crash.
• Workaround: Commenting out the call to rxm_flush_msg_cq in rxm_conn.c (or relevant source) resolves all crashes in my test suite.

Questions for Maintainers

1. Purpose of the Flush: Is rxm_flush_msg_cq strictly intended to clean the CQ to allow for object reuse, or does it handle critical resource deallocation in the verbs layer?
2. Side Effects: What are the consequences of skipping this flush? Are there potential resource leaks or "dirty" states that would affect subsequent connections in a long-running process?
3. Buffer Ownership: Why is a receive being posted during the connection close if the user has not provided a buffer? Is there a recommended synchronization pattern to ensure rxm is finished with all internal operations before fi_close is called?

[j-xiong]

rxm_flush_msg_cq clears out the CQ so that any pending completions are reported. Not doing this may cause applications to miss completions. The fi_recv observed is reposting of receive buffer which is essential for core providers like verbs where preposted receives are expected in order to work properly. That is part of general workflow for handling completions. However, in the case of flushing, we may be able to handle it differently. That would need more investigation and some code change.

[dsciebu]

Thank you for the answer @j-xiong. Could you please clarify, what functionality I am loosing by removing the call of rxm_flush_msg_cq from rxm_close_conn? I understand, you need some time for investigation, implementation and finally releasing the change, so I wonder what would be the cost of removing it directly in my fork for now.

[j-xiong]

If there are pending completions in the lower layer (the verbs provider layer here), removing the call would cause those completions to be dropped silently instead of reporting to the application (at the rxm layer). If the application is waiting for such completions, it may wait forever for completions that are never going to come or interpret future completions as the missing one(s).