Skip to content

feat: new queryWithTimeout method for customer-side wait #3995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Nov 12, 2025
Merged

feat: new queryWithTimeout method for customer-side wait #3995

merged 7 commits into from
Nov 12, 2025
+137 −46

Conversation

@logachev
Copy link
Contributor

@logachev logachev commented Nov 5, 2025 •
edited
Loading

Existing BigQuery.query() method always returns TableResult for both long-running a jobless queries.
One of the issues with this approach is that for long-running operations, consumer app has no way to track the Job because JobId is unknown until after query is complete. It prevents consumer apps from cancelling queries if they're running too long or if customer wants to cancel jobs due various reasons.

This PR introduces new queryWithTimeout method that returns either TableResult object for short queries or Job object for long-running query.
It will allow to manipulate Job before query is finished (can be relevant for queries that take more than a few minutes to complete).

It also refactors query methods to leverage new queryNoWait method to avoid code duplication which fixed an issue that query() version with JobId was never leveraging jobless query feature.

Added integration & unit test for new method covering timeout feature.

@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/java-bigquery API. labels Nov 5, 2025
@logachev logachev changed the title feat: new queryNoWait method for customer-side wait feat: new queryWithTimeout method for customer-side wait Nov 6, 2025
@logachev logachev marked this pull request as ready for review November 7, 2025 05:19
@logachev logachev requested a review from a team as a code owner November 7, 2025 05:19
@logachev logachev requested a review from GaoleMeng November 7, 2025 05:19
@tswast
Copy link
Contributor

tswast commented Nov 7, 2025

Samples failures:

Step #3: [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 s - in com.example.bigquery.AuthDriveScopeIT
Step #3: [INFO] 
Step #3: [INFO] Results:
Step #3: [INFO] 
Step #3: [ERROR] Errors: 
Step #3: [ERROR]   QueryExternalGcsTempIT.testQueryExternalGcsTemp:66 » IllegalArgument Provided dataset is null or empty
Step #3: [INFO] 
Step #3: [ERROR] Tests run: 137, Failures: 0, Errors: 1, Skipped: 19
Step #3: [INFO] 
Step #3: [INFO] 
Step #3: [INFO] --- failsafe:3.0.0:verify (default) @ bigquery-google-cloud-samples ---
Step #3: [INFO] 
Step #3: [INFO] -----------< com.google.cloud:google-cloud-bigquery-samples >-----------
Step #3: [INFO] Building Google BigQuery Samples Parent 0.0.1-SNAPSHOT             [4/4]
Step #3: [INFO]   from pom.xml
Step #3: [INFO] --------------------------------[ pom ]---------------------------------
Step #3: [INFO] 
Step #3: [INFO] --- jacoco:0.8.12:prepare-agent (default) @ google-cloud-bigquery-samples ---
Step #3: [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.12/org.jacoco.agent-0.8.12-runtime.jar=destfile=/workspace/samples/target/jacoco.exec
Step #3: [INFO] 
Step #3: [INFO] --- jacoco:0.8.12:report (report) @ google-cloud-bigquery-samples ---
Step #3: [INFO] Skipping JaCoCo execution due to missing execution data file.
Step #3: [INFO] 
Step #3: [INFO] --- failsafe:3.0.0:integration-test (default) @ google-cloud-bigquery-samples ---
Step #3: [INFO] No tests to run.
Step #3: [INFO] 
Step #3: [INFO] --- failsafe:3.0.0:verify (default) @ google-cloud-bigquery-samples ---
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [INFO] Reactor Summary:
Step #3: [INFO] 
Step #3: [INFO] Google BigQuery Install Without Bom 1.2.2 .......... FAILURE [04:36 min]
Step #3: [INFO] Google BigQuery Snapshot Samples 1.2.2 ............. FAILURE [04:10 min]
Step #3: [INFO] bigquery-google-cloud-samples 1.2.2 ................ FAILURE [04:12 min]
Step #3: [INFO] Google BigQuery Samples Parent 0.0.1-SNAPSHOT ...... SUCCESS [  0.569 s]
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [INFO] BUILD FAILURE
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [INFO] Total time:  13:12 min
Step #3: [INFO] Finished at: 2025-11-07T02:47:23Z
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.0.0:verify (default) on project bigquery-install-without-bom: 
Step #3: [ERROR] 
Step #3: [ERROR] Please refer to /workspace/samples/install-without-bom/target/failsafe-reports for the individual test results.
Step #3: [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Step #3: [ERROR] -> [Help 1]
Step #3: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.0.0:verify (default) on project bigquery-snapshot: 
Step #3: [ERROR] 
Step #3: [ERROR] Please refer to /workspace/samples/snapshot/target/failsafe-reports for the individual test results.
Step #3: [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Step #3: [ERROR] -> [Help 1]
Step #3: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.0.0:verify (default) on project bigquery-google-cloud-samples: 
Step #3: [ERROR] 
Step #3: [ERROR] Please refer to /workspace/samples/snippets/target/failsafe-reports for the individual test results.
Step #3: [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Step #3: [ERROR] -> [Help 1]
Step #3: [ERROR] 
Step #3: [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
Step #3: [ERROR] Re-run Maven using the -X switch to enable full debug logging.
Step #3: [ERROR] 
Step #3: [ERROR] For more information about the errors and possible solutions, please read the following articles:
Step #3: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Step #3: /workspace
Step #3: basename: missing operand
Step #3: Try 'basename --help' for more information.
Step #3: exiting with 1

Seems like the Step #3: [ERROR] QueryExternalGcsTempIT.testQueryExternalGcsTemp:66 » IllegalArgument Provided dataset is null or empty failure might be related to this PR?

@tswast tswast requested a review from lqiu96 November 7, 2025 18:04
@logachev
Copy link
Contributor Author

logachev commented Nov 7, 2025

Samples failures:

Step #3: [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 s - in com.example.bigquery.AuthDriveScopeIT
Step #3: [INFO] 
Step #3: [INFO] Results:
Step #3: [INFO] 
Step #3: [ERROR] Errors: 
Step #3: [ERROR]   QueryExternalGcsTempIT.testQueryExternalGcsTemp:66 » IllegalArgument Provided dataset is null or empty
Step #3: [INFO] 
Step #3: [ERROR] Tests run: 137, Failures: 0, Errors: 1, Skipped: 19
Step #3: [INFO] 
Step #3: [INFO] 
Step #3: [INFO] --- failsafe:3.0.0:verify (default) @ bigquery-google-cloud-samples ---
Step #3: [INFO] 
Step #3: [INFO] -----------< com.google.cloud:google-cloud-bigquery-samples >-----------
Step #3: [INFO] Building Google BigQuery Samples Parent 0.0.1-SNAPSHOT             [4/4]
Step #3: [INFO]   from pom.xml
Step #3: [INFO] --------------------------------[ pom ]---------------------------------
Step #3: [INFO] 
Step #3: [INFO] --- jacoco:0.8.12:prepare-agent (default) @ google-cloud-bigquery-samples ---
Step #3: [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.12/org.jacoco.agent-0.8.12-runtime.jar=destfile=/workspace/samples/target/jacoco.exec
Step #3: [INFO] 
Step #3: [INFO] --- jacoco:0.8.12:report (report) @ google-cloud-bigquery-samples ---
Step #3: [INFO] Skipping JaCoCo execution due to missing execution data file.
Step #3: [INFO] 
Step #3: [INFO] --- failsafe:3.0.0:integration-test (default) @ google-cloud-bigquery-samples ---
Step #3: [INFO] No tests to run.
Step #3: [INFO] 
Step #3: [INFO] --- failsafe:3.0.0:verify (default) @ google-cloud-bigquery-samples ---
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [INFO] Reactor Summary:
Step #3: [INFO] 
Step #3: [INFO] Google BigQuery Install Without Bom 1.2.2 .......... FAILURE [04:36 min]
Step #3: [INFO] Google BigQuery Snapshot Samples 1.2.2 ............. FAILURE [04:10 min]
Step #3: [INFO] bigquery-google-cloud-samples 1.2.2 ................ FAILURE [04:12 min]
Step #3: [INFO] Google BigQuery Samples Parent 0.0.1-SNAPSHOT ...... SUCCESS [  0.569 s]
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [INFO] BUILD FAILURE
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [INFO] Total time:  13:12 min
Step #3: [INFO] Finished at: 2025-11-07T02:47:23Z
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.0.0:verify (default) on project bigquery-install-without-bom: 
Step #3: [ERROR] 
Step #3: [ERROR] Please refer to /workspace/samples/install-without-bom/target/failsafe-reports for the individual test results.
Step #3: [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Step #3: [ERROR] -> [Help 1]
Step #3: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.0.0:verify (default) on project bigquery-snapshot: 
Step #3: [ERROR] 
Step #3: [ERROR] Please refer to /workspace/samples/snapshot/target/failsafe-reports for the individual test results.
Step #3: [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Step #3: [ERROR] -> [Help 1]
Step #3: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.0.0:verify (default) on project bigquery-google-cloud-samples: 
Step #3: [ERROR] 
Step #3: [ERROR] Please refer to /workspace/samples/snippets/target/failsafe-reports for the individual test results.
Step #3: [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Step #3: [ERROR] -> [Help 1]
Step #3: [ERROR] 
Step #3: [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
Step #3: [ERROR] Re-run Maven using the -X switch to enable full debug logging.
Step #3: [ERROR] 
Step #3: [ERROR] For more information about the errors and possible solutions, please read the following articles:
Step #3: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Step #3: /workspace
Step #3: basename: missing operand
Step #3: Try 'basename --help' for more information.
Step #3: exiting with 1

Seems like the Step #3: [ERROR] QueryExternalGcsTempIT.testQueryExternalGcsTemp:66 » IllegalArgument Provided dataset is null or empty failure might be related to this PR?

This seems to be unrelated, I see the same failure in another PR: #3982

@lqiu96 lqiu96 requested a review from jinseopkim0 November 7, 2025 19:41
jinseopkim0
jinseopkim0 requested changes Nov 11, 2025
View reviewed changes
Copy link
Contributor

@jinseopkim0 jinseopkim0 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this PR. The clirr CI is failing with the following error:

Error:  7012: com.google.cloud.bigquery.BigQuery: Method 'public java.lang.Object queryWithTimeout(com.google.cloud.bigquery.QueryJobConfiguration, com.google.cloud.bigquery.JobId, java.lang.Long, com.google.cloud.bigquery.BigQuery$JobOption[])' has been added to an interface

Could you please add a default implementation to the queryWithTimeout method? An empty implementation throwing an exception would suffice.

Also, would it make sense to add @InternalApi annotation to this method?

@logachev logachev requested a review from a team as a code owner November 12, 2025 00:38
@jinseopkim0 jinseopkim0 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 12, 2025
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 12, 2025
@jinseopkim0 jinseopkim0 added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Nov 12, 2025
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Nov 12, 2025
@jinseopkim0 jinseopkim0 merged commit 9c0df54 into main Nov 12, 2025
23 of 24 checks passed
@jinseopkim0 jinseopkim0 deleted the kirl/query_rpc branch November 12, 2025 19:48
@release-please release-please bot mentioned this pull request Nov 12, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jinseopkim0 jinseopkim0 jinseopkim0 approved these changes

@GaoleMeng GaoleMeng Awaiting requested review from GaoleMeng GaoleMeng is a code owner automatically assigned from googleapis/api-bigquery

@lqiu96 lqiu96 Awaiting requested review from lqiu96

Assignees

No one assigned

Labels

api: bigquery Issues related to the googleapis/java-bigquery API. size: m Pull request size is medium.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@logachev @tswast @jinseopkim0 @yoshi-kokoro