Skip to content

Add execute functions #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
May 7, 2025
Merged

Conversation

edmundmiller
Copy link

@edmundmiller edmundmiller commented May 1, 2025

This is a follow-up to #27

ResultSet was expected but not generated from query: INSERT INTO ap_demo.nextflowtest.table1

The problem is in the QueryHandler class. When executing a DDL statement like CREATE TABLE, the code is using executeQuery(), which expects a ResultSet to be returned. However, DDL statements don't return ResultSets; they should use executeUpdate() instead.

A couple of updates

  • Added a new sqlExecute function to execute SQL statements that do not return result sets, such as DDL (e.g., CREATE, DROP) and DML (e.g., INSERT, UPDATE, DELETE) operations. It returns the number of affected rows for DML statements and null for DDL statements.
  • Enhanced error handling for unsupported database operations (e.g., setAutoCommit, setObject, commit, executeBatch) to provide debug logs instead of failing outright. This is particularly useful for databases like Databricks that may not support certain JDBC features.
  • Added functional tests in the same style as the nf-schema plugin to quickly capture tests at a Nextflow level.
  • Added a version variable in the Makefile to dynamically fetch the plugin version from the MANIFEST.MF file and install locally.

…SQL execution logic

- Removed the executeUpdate parameter from QueryHandler and ChannelSqlExtension.
- Simplified SQL execution by directly executing queries without checking for DDL or UPDATE statements.

Signed-off-by: Edmund Miller <[email protected]>
- Introduced `execute` and `executeUpdate` methods for executing SQL statements without returning a result set.
- Added parameter validation and error handling for database connections.
- Implemented SQL statement normalization to ensure proper execution.

Signed-off-by: Edmund Miller <[email protected]>
- Added tests for DDL and DML operations, including table creation, updates, and deletions.
- Implemented error handling tests for invalid SQL statements and database configurations.
- Ensured proper handling of statement normalization.

Signed-off-by: Edmund Miller <[email protected]>
- Changed `execute` and `executeUpdate` methods from static to instance methods, allowing for better integration with session management.
- Updated test cases to reflect the new method calls and ensure proper session initialization.

Signed-off-by: Edmund Miller <[email protected]>
- Changed the database configuration from in-memory to file-based for persistence.
- Updated SQL statements to use uppercase for table and column names for consistency.
- Enhanced logging to include results of table creation and updates.

Signed-off-by: Edmund Miller <[email protected]>
…ions

- Added try-catch blocks around `setAutoCommit` and `commit` methods to handle exceptions for databases that do not support these operations.
- Improved logging to provide feedback when default behaviors are used due to unsupported operations.

Signed-off-by: Edmund Miller <[email protected]>
Copy link
Member

@pditommaso pditommaso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done. Made a few comments

- Added a new `.envrc` file to manage environment variables for database connections.

Signed-off-by: Edmund Miller <[email protected]>
@edmundmiller edmundmiller marked this pull request as draft May 2, 2025 20:42
edmundmiller and others added 4 commits May 2, 2025 15:56
- Renamed the `execute` function to `sqlExecute` for clarity and consistency.
- Updated all references in the README, example scripts, and tests to reflect the new function name.
- Enhanced documentation to provide clearer usage examples for SQL execution functions.

Signed-off-by: Edmund Miller <[email protected]>

Co-authored-by: Edmund Miller <[email protected]>
Co-authored-by: Paolo Di Tommaso <[email protected]>
- Enhanced logging in examples to provide feedback on the number of affected rows and results of DDL operations.

Signed-off-by: Edmund Miller <[email protected]>

Co-authored-by: Paolo Di Tommaso <[email protected]>
- Modified the `sqlExecute` method to return a map containing success status, affected rows, and error messages for both DDL and DML operations.
- Updated documentation to reflect the new return structure.
- Adjusted unit tests to validate the new response format and ensure proper handling of success and error cases.

Signed-off-by: Edmund Miller <[email protected]>
Copy link
Member

@pditommaso pditommaso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems good, however, could you please provide a short summary of PR in the description to simplify the review? Also you may want to mark as "Read for review"

@edmundmiller edmundmiller marked this pull request as ready for review May 5, 2025 20:58
@edmundmiller
Copy link
Author

edmundmiller commented May 5, 2025

Also you may want to mark as "Read for review"

I wanted to sleep on it 😆

edmundmiller and others added 3 commits May 7, 2025 10:40
@edmundmiller edmundmiller merged commit e0337fe into nextflow-io:databricks May 7, 2025
3 checks passed
@edmundmiller edmundmiller deleted the execute branch May 12, 2025 16:47
@edmundmiller edmundmiller restored the execute branch May 27, 2025 14:37
@edmundmiller edmundmiller deleted the execute branch May 27, 2025 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants