Remove Queue.js #233
Open
Remove Queue.js #233
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
he-is-harry
added a commit
to he-is-harry/node-hdb
that referenced
this pull request
Feb 28, 2025
Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors Implementation Details To implement the "blocked" mode, the Queue is modified to a data structure which supports 3 operations 1. Push a task (preserving the order in which tasks were pushed) 2. Pop a task (remove the task in the order they were pushed) 3. Selective pop (remove a task in the order they were pushed given that the type matches a given variety) For optimization purposes, the tasks that can block are only READ_LOB requests. As such, we only require another readLobQueue which stores READ_LOB tasks to allow those to skip the queue. - An invariant of the Queue is that the queue must store the same READ_LOB tasks as the readLobQueue and possibly more READ_LOB tasks that are already run. If in the future, it is found that other tasks can block, it is possible to maintain a map<message type, list of queue tasks> and still have these 3 operations run in constant time, because the number of message types is finite (and small).
he-is-harry
added a commit
to he-is-harry/node-hdb
that referenced
this pull request
Feb 28, 2025
Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors
he-is-harry
added a commit
to he-is-harry/node-hdb
that referenced
this pull request
Mar 6, 2025
Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have a scenario in
@cap-js/hanathat a blob column is streamed out of the database and is being inserted into a different table. When doing this theQueue.jsimplementation creates a dead lock. As the lob read request cannot be send while actively processing the insert statement.This PR solves this limitation by removing the
Queue.jsfrom the connection. Instead using thepacketCountto identify what callback to use when a response is received from the the HANA system.There is still a pseudo queue in the current implementation as simply sending the request to the system without waiting for an acknowledgement. Results in sporadic connection terminations by the HANA system. With the reason
invalid packet lengthwhile sending the exact same packets.