Skip to content

fix: include location in Session-based temporary storage manager DDL queries #1780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

tswast
Copy link
Collaborator

@tswast tswast commented May 29, 2025

…queries

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@tswast tswast requested review from a team as code owners May 29, 2025 18:40
@tswast tswast requested a review from GarrettWu May 29, 2025 18:40
@product-auto-label product-auto-label bot added size: xs Pull request size is extra small. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels May 29, 2025
@tswast
Copy link
Collaborator Author

tswast commented May 29, 2025

Tested manually locally and so far it seems to fix the issue on HEAD.

(venv) ➜  python-bigquery-dataframes git:(tswast-session-location) pytest tests/system/large/test_location.py -v
================================================================================================================================= test session starts ==================================================================================================================================
platform linux -- Python 3.12.10, pytest-8.3.5, pluggy-1.6.0 -- /home/swast/src/github.com/googleapis/python-bigquery-dataframes/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/swast/src/github.com/googleapis/python-bigquery-dataframes
configfile: pytest.ini
plugins: snapshot-0.9.0
collected 159 items                                                                                                                                                                                                                                                                    

tests/system/large/test_location.py::test_bq_location_default PASSED                                                                                                                                                                                              
tests/system/large/test_location.py::test_bq_location[EU] PASSED                                                                                                                                                                                                  
tests/system/large/test_location.py::test_bq_location[US] PASSED                                                                                                                                                                                                  
tests/system/large/test_location.py::test_bq_location[africa-south1] PASSED                                                                                                                                                                                       
tests/system/large/test_location.py::test_bq_location[asia-east1] PASSED                      
tests/system/large/test_location.py::test_bq_location[asia-northeast1] PASSED                                                                                                                                                                                     
tests/system/large/test_location.py::test_bq_location[asia-northeast2] PASSED                                                                                                                                                                                     
tests/system/large/test_location.py::test_bq_location[asia-northeast3] PASSED                                                                                                                                                                                     
tests/system/large/test_location.py::test_bq_location[asia-south1] PASSED       
...
tests/system/large/test_location.py::test_bq_location_non_canonical[Eu-EU] PASSED                                                                                                                                                                                 
tests/system/large/test_location.py::test_bq_location_non_canonical[Us-US] PASSED                                                                                                                                                                                 
tests/system/large/test_location.py::test_bq_location_non_canonical[Africa-south1-africa-south1] PASSED                                                                                                                                                           
tests/system/large/test_location.py::test_bq_location_non_canonical[Asia-east1-asia-east1] PASSED                                                                                                                                                                 
tests/system/large/test_location.py::test_bq_location_non_canonical[Asia-east2-asia-east2] PASSED
...

Genesis929
Genesis929 previously approved these changes May 29, 2025
@tswast tswast enabled auto-merge (squash) May 29, 2025 18:51
@tswast tswast disabled auto-merge May 29, 2025 19:05
@product-auto-label product-auto-label bot added size: s Pull request size is small. and removed size: xs Pull request size is extra small. labels May 29, 2025
@@ -63,28 +62,6 @@ def _assert_bq_execution_location(
expected_result, result.to_pandas(), check_dtype=False, check_index_type=False
)

# Ensure BQ Storage Read client operation succceeds
table = result.query_job.destination
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This table and query_job might not actually exist / refer to an older object since to_pandas() will likely end up in the "jobless" API path.

Copy link
Contributor

@shobsi shobsi May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this code is testing that the result of a BigFrames operation are stored in that location itself, which has value in asserting the location behavior, especially with regional endpoints. Rather than deleting, can we not do one of the following:

  1. Force a query job via allow_large_results=True in the Session creation in all tests
  2. Move this test code to a special test which enforces allow_large_results=True

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this would verify the location though. You'd have to check that the dataset is in the correct location. Otherwise, this would read from whereever the table is.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have forced a job and validate the location of the stored results in 15fa728.

@Genesis929 @shobsi Please review.

@tswast tswast requested a review from Genesis929 May 29, 2025 20:16
@tswast tswast changed the title fix: include location in Session-based temporary storage manager DDL … fix: include location in Session-based temporary storage manager DDL queries May 29, 2025
@tswast
Copy link
Collaborator Author

tswast commented May 29, 2025

New e2e failure is FAILED tests/system/small/ml/test_multimodal_llm.py::test_multimodal_embedding_generator_predict_default_params_success which is unrelated.

New notebook failures is notebooks/generative_ai/bq_dataframes_llm_kmeans.ipynb, which is unrelated.

@Genesis929 please review again. I think the setting to dismiss stale reviews has been turned on for security reasons.

Genesis929
Genesis929 previously approved these changes May 29, 2025
@tswast tswast enabled auto-merge (squash) May 29, 2025 21:29
@product-auto-label product-auto-label bot added size: m Pull request size is medium. and removed size: s Pull request size is small. labels May 30, 2025
@tswast tswast requested review from shobsi and Genesis929 and removed request for GarrettWu May 30, 2025 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. size: m Pull request size is medium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants