[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

馃悰 [Stream Firestore to BigQuery] Update cloud function extension to gen 2 #2072

Open
leslie-zi-pan opened this issue Apr 22, 2024 · 4 comments
Open

馃悰 [Stream Firestore to BigQuery] Update cloud function extension to gen 2 #2072

leslie-zi-pan opened this issue Apr 22, 2024 · 4 comments
Labels
extension: firestore-bigquery-export Related to firestore-bigquery-export extension type: feature request New feature or request

Comments

@leslie-zi-pan
Copy link

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository.
If you have a general question, need help debugging, or fall into some
other category use one of these other channels:

  • For general technical questions, post a question on StackOverflow
    with the firebase tag.
  • For general Firebase discussion, use the firebase-talk
    google group.
  • To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to
    Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

  • Extension name: Stream Firestore to BigQuery
  • Extension version: 0.1.49

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

Failed extension deployment - specifically the firestore component of this extension. The extension allows database targetting (other than the default DB). However, deployment of this extension is still in 1st Gen Cloud function which limits the trigger to default database. This is contradictory to what the extension feature allows as there is a field to specify database ID. Please update to Gen 2. Thank you.
@leslie-zi-pan leslie-zi-pan added the type: bug Something isn't working label Apr 22, 2024
@mikeymace
Copy link

Hi!

Do we have a time frame for when the extensions will be upgraded to use the 2nd Gen Cloud Functions?

We are currently in the process of regionalising our Firestore databases (i.e. eur, us, shared) and we currently rely on the 'Stream Firestore to BigQuery' extension to get our data into BigQuery. Ideally, each database would be mapped to an equivalent (localised) dataset in BigQuery.

I have found the following behaviour for firebase/firestore-bigquery-export@0.1.49:

  • If I don't set the DATABASE_ID .env variable then the Cloud Function automatically defaults to 1st gen and DATABASE_ID = (default) see screenshot below.
  • If I do try and set DATABASE_ID = <location> in the .env then I get the following error message during deployment: ResourceErrorCode":"400","ResourceErrorMessage":"Cannot create Firestore trigger for .../databases/us-west2. Triggers must be created with a \"(default)\" Firestore Native database. See https://firebase.google.com/docs/firestore/extend-with-functions#limitations.

We also rely on npx firebaseextensions/fs-bq-import-collection for manual backfilling of existing Firestore data. Can you also confirm that this will be updated so that we can pass the database_id as an argument?

This is urgent from our side as our database migration is planned to go ahead early June and therefore any information on expected timeframes for this to be upgraded would be greatly appreciated otherwise unfortunately we will be forced to look for an alternative.

Thank you!

image

@mikeymace mikeymace mentioned this issue May 10, 2024
Merged
@IchordeDionysos
Copy link
Contributor

@mikeymace I'd suggest looking for an alternative, e.g. copy the code and deploy it directly (aka not as Firebase Extension).
This will take some time until it's fixed ...

@pr-Mais pr-Mais added type: feature request New feature or request extension: firestore-bigquery-export Related to firestore-bigquery-export extension and removed type: bug Something isn't working labels May 10, 2024
@pr-Mais pr-Mais mentioned this issue May 10, 2024
Closed
@pr-Mais
Copy link
Member
pr-Mais commented May 10, 2024

Hello everyone, tagging this as a feature request.

Unfortunately, it's not possible now to upgrade any extension to v2 Firestore triggers since extensions don't yet support them, once it does we will update the triggers.

For the contradiction between the params and actual functionality, I'm tracking this as an issue here #2090.

@basvandorst
Copy link

+1

general note: would be great if all (official) Firebase extensions also moving towards gen 2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
extension: firestore-bigquery-export Related to firestore-bigquery-export extension type: feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@basvandorst @IchordeDionysos @pr-Mais @leslie-zi-pan @mikeymace