[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logging: prioritize GOOGLE_CLOUD_PROJECT env variable over project id from metadata server for GKE resource detection #10326

Open
GoosvandenBekerom opened this issue Jun 5, 2024 · 4 comments
Assignees
Labels
api: logging Issues related to the Cloud Logging API. triage me I really want to be triaged.

Comments

@GoosvandenBekerom
Copy link
GoosvandenBekerom commented Jun 5, 2024

Is your feature request related to a problem? Please describe.
In our company, we have a central project in which GKE is managed, but then every application has its own project which gets it's own namespace in that central GKE setup. Now we are trying to integrate with the central logging architecture provided by our central TPS team.

Everything works, but there is a slight problem for us in the autodetection of the MonitoredResource. It detects the central project instead of the project that owns the workload. Because this central team wants to forward logging bills to the projects that are actually doing the request, they enforce us to provide the project id in the resource.project_id field.

I've tried a couple things, without success:

  • Use the logging.DetectProjectID setup
  • Only setting the project_id field and hope the auto-detection will set the rest (this was a desperate attempt but I wanted to make sure it didn't work before filing this issue)

Describe the solution you'd like
The Java library used by other teams in our company seems to first check for the "GOOGLE_CLOUD_PROJECT" environment variable before going to the metadata server. If we also do this here (at least for the GKE detection) this would solve our issue.

Describe alternatives you've considered
I want a way to override the detected resource.project_id field for log entries.
We could implement a new LoggerOption like logging.OverrideResourceProjectId(projectID string). which would then set the "project_id" label on l.commonResource to the given projectID.

The only other alternative I see for us is to write our own auto-detection setup, which I really dislike, as this feels like something that will change often.

Additional context
I would be happy to implement this feature if maintainers agree that it would fit.

@GoosvandenBekerom GoosvandenBekerom added the triage me I really want to be triaged. label Jun 5, 2024
@product-auto-label product-auto-label bot added the api: logging Issues related to the Cloud Logging API. label Jun 5, 2024
@GoosvandenBekerom GoosvandenBekerom changed the title logging: LoggingOption to override detected project_id for MonitoredResource logging: LoggerOption to override detected project_id for MonitoredResource Jun 5, 2024
GoosvandenBekerom pushed a commit to GoosvandenBekerom/google-cloud-go that referenced this issue Jun 6, 2024
…ect id from metadata server for GKE resource detection

related issue googleapis#10326
GoosvandenBekerom pushed a commit to GoosvandenBekerom/google-cloud-go that referenced this issue Jun 6, 2024
…ect id from metadata server for GKE resource detection (googleapis#10326)
GoosvandenBekerom added a commit to GoosvandenBekerom/google-cloud-go that referenced this issue Jun 6, 2024
…ect id from metadata server for GKE resource detection (googleapis#10326)
GoosvandenBekerom added a commit to GoosvandenBekerom/google-cloud-go that referenced this issue Jun 10, 2024
…ect id from metadata server for GKE resource detection (googleapis#10326)
@GoosvandenBekerom GoosvandenBekerom changed the title logging: LoggerOption to override detected project_id for MonitoredResource logging: prioritize GOOGLE_CLOUD_PROJECT env variable over project id from metadata server for GKE resource detection Jun 11, 2024
GoosvandenBekerom added a commit to GoosvandenBekerom/google-cloud-go that referenced this issue Jun 11, 2024
…ect id from metadata server for GKE resource detection (googleapis#10326)
GoosvandenBekerom added a commit to GoosvandenBekerom/google-cloud-go that referenced this issue Jun 12, 2024
…ect id from metadata server for GKE resource detection (googleapis#10326)
@gkevinzheng
Copy link
Contributor

@GoosvandenBekerom Is the Java library that you mention the Java Cloud Logging client library or some other proprietary library that interfaces with Cloud Logging?

@GoosvandenBekerom
Copy link
Author
GoosvandenBekerom commented Jun 14, 2024

From what I understood from our jvm team, it uses this method from the java google cloud client https://googleapis.dev/java/google-cloud-clients/0.98.0-alpha/com/google/cloud/ServiceOptions.html#getDefaultProjectId--

They use com.google.cloud.logging.Logging which allows them to set the projectId as an option, they use the above mentioned api to get the project ID, which seems to check several places before falling back to the metadata server.

For now our logging team is solving this for our Go apps by overwriting the detected project ID with the project ID in the logName. But that isn't ideal of course.

@gkevinzheng
Copy link
Contributor

Is this overwriting something that is happening to the MonitoredResource object? In the Go logging library the detected resource isn't exposed as an object.

@GoosvandenBekerom
Copy link
Author

Yes that is the problem, the monitored resource can either be detected (default) or fully overwritten using this option

For us it seems to work fine on other compute (cloud run) but for that centralized GKE setup it doesn't work as required, I figured this can easily be solved by simply checking the GOOGLE_CLOUD_PROJECT before going to the metadata server. That variable is set correctly for all our projects and judging by other detection methods (i.e. App Engine) I see that this variable is also used, so I assume this is a standard.

I figured by first checking that variable our issue would be solved, and it would not harm users that do not have that variable set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: logging Issues related to the Cloud Logging API. triage me I really want to be triaged.
Projects
None yet
Development

No branches or pull requests

2 participants