Skip to content

🐛 [fs-bq-import-collection] Multi-Threaded import Script does not Support Cross-Project loading #2298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
justin-plancraft opened this issue Feb 20, 2025 · 4 comments · May be fixed by #2299
Open

🐛 [fs-bq-import-collection] Multi-Threaded import Script does not Support Cross-Project loading #2298

justin-plancraft opened this issue Feb 20, 2025 · 4 comments · May be fixed by #2299
Assignees
@CorieW
Labels
type: bug Something isn't working

Comments

@justin-plancraft
Copy link

  • Extension name: fs-bq-import-collection
  • Extension version: 0.1.23

Steps to reproduce:

When trying to Backfill via the Import Script in a Cross-Project setup (reading from project-dev-01 and writing to bigquery in project-prod-01 for example) - and running the import script multi-threaded, the worker.ts does not pass the neccessary bqProjectId to the BigQuery Writer, therefore always failing because the project-dev-01 does not contain the BigQuery Dataset and Table (and should not contain it).

Expected result

Should work

Actual result

Fails because view not found
@justin-plancraft justin-plancraft added the type: bug Something isn't working label Feb 20, 2025
justin-plancraft added a commit to justin-plancraft/extensions that referenced this issue Feb 20, 2025
@justin-plancraft
fix: Pass through BigQuery Config in multithreaded worker (closes fir…
f4dcff1 
…ebase#2298)
@justin-plancraft justin-plancraft linked a pull request Feb 20, 2025 that will close this issue
Open
@CorieW
Copy link
Contributor

CorieW commented Feb 20, 2025

Hi @justin-plancraft

Thanks for reporting this issue! We’ve received it and are reviewing it. We’ll provide updates as soon as possible.

@justin-plancraft
Copy link
Author

Awesome, thanks so much for the prompt response :) Hope my fix is helpful, I got it working locally by using these changes directly and editing inline.

@cabljac cabljac assigned CorieW and cabljac and unassigned CorieW Mar 17, 2025
@cabljac cabljac moved this from Needs Acknowledgment to Ready for Development in [Cloud] Extensions + Functions May 6, 2025
@cabljac cabljac moved this from Ready for Development to In Progress in [Cloud] Extensions + Functions May 6, 2025
@cabljac cabljac removed their assignment May 6, 2025
@cabljac cabljac moved this from In Progress to In Review in [Cloud] Extensions + Functions May 6, 2025
@cabljac
Copy link
Contributor

cabljac commented May 6, 2025

Hi @justin-plancraft We've been testing this issue and the PR, and we haven't been able to reproduce the original issue at all so far.

Can you confirm for us whether this is still occurring on the latest version of the extension? If it is then we'll dig into it further and figure out the root issue.

@justin-plancraft
Copy link
Author

I only had the problem back then and was not able to get it to run without patching the package - currently do not have a setup to reproduce with feasible effort, would have to touch quite a few things since right now all the extensions are in productive use 😬
Sorry - however also happy if it does not reproduce and it might have been just my idiosyncratic setup - feel free to close if you do not see value in it for the general repo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CorieW CorieW

Labels
type: bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Pass through BigQuery Config in multithreaded worker (closes #2298) justin-plancraft/extensions
3 participants
@cabljac @CorieW @justin-plancraft