-
Notifications
You must be signed in to change notification settings - Fork 820
Update GRPC #6808
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Update GRPC #6808
Conversation
265068a
to
ec132e6
Compare
Signed-off-by: alanprot <[email protected]>
…arshalling Signed-off-by: alanprot <[email protected]>
Signed-off-by: alanprot <[email protected]>
Signed-off-by: alanprot <[email protected]>
Signed-off-by: alanprot <[email protected]>
Signed-off-by: alanprot <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
LGTM |
Signed-off-by: alanprot <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Great work
What this PR does:
This PR attempts to update the Go-GRPC library, which has been pinned for a while. The newer version of Go-GRPC reuses unmarshalling wire buffers internally, which can potentially race with our own buffer reuse logic, leading to message corruption.
A test was added to demonstrate this issue in the Push and QueryStream methods.
Problem background:
In Cortex, we use yolostring when unmarshalling bytes from the wire into proto structs (see: Cortex yolostring). Yolostring creates a string from the wire buffer’s bytes without copying them, effectively reusing the original bytes to reduce memory allocations.
Prior to Go-GRPC v1.65.0, wire buffers were created and used within a single request, so reusing those bytes via yolostring was safe.
However, starting with Go-GRPC v1.66.0, the wire buffers are now pooled and reused across multiple requests. The bytes are returned to the pool immediately after the Unmarshal method is called (reference and grpc/grpc-go#6619). This means that the bytes might be overwritten before the Cortex code finishes using them, leading to potential message corruption.
In this PR, we are creating a new Proto Codec that is not freeing the buffer used to unmarshall the message back to the pool: See this in comparison to the go-grpc code here - in practical terms, we are by default falling back to the behavior before v1.66.0. However, we can also now change the proto message to implement the
ReleasableMessage
interface and, if done, the message will now have an extraFree()
that can be called to release the buffer back to the pool when is safe to do so -> so far, only theWriteRequest
message is implementing this as this is the one that has a huge TPS with a very short lived lifecycle. In the future, we can add this capabilities to other proto messages.Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]