Client

cloudtasks

Environment

Image built with alpine running on Cloud Run

Go Environment

$ gcloud version

Google Cloud SDK 478.0.0
beta 2024.05.24
bq 2.1.5
core 2024.05.24
gcloud-crc32c 1.0.0
gsutil 5.29

Code

Note: Anything within <> is a replacement of the actual values.

Updating queue to have queue-level routing

gcloud tasks queues update <queue-name> \
--http-uri-override="mode:ALWAYS,host:<cloud-run-worker-service-host>,path:<worker-path>,port:443" --location="<...>" --project="<...>" \
--http-oidc-service-account-email-override="<service-acount-email>" \
--http-oidc-token-audience-override="https://<cloud-run-worker-service-host>"

Task creation code:

// buildTaskRequest build task request from event and publisher config
func (p *PublisherImpl) buildTaskRequest(env Envelope) (*taskspb.CreateTaskRequest, error) {
	...
	headers := map[string]string{
		"Content-Type": "application/json",
	}

	req := &taskspb.CreateTaskRequest{
		Parent: queuePath,
		Task: &taskspb.Task{
			// https://godoc.org/google.golang.org/genproto/googleapis/cloud/tasks/v2#HttpRequest
			MessageType: &taskspb.Task_HttpRequest{
				HttpRequest: &taskspb.HttpRequest{
					Url:        <irrelevantWorkerUrl>,// this is a different endpoint than what is being defined in the queue-level routing
					HttpMethod: taskspb.HttpMethod_POST,
					Headers:    headers,
					Body:       payload,
					AuthorizationHeader: &taskspb.HttpRequest_OidcToken{
						OidcToken: &taskspb.OidcToken{
							ServiceAccountEmail: "<service-acount-email>",// this is the same as what was defined on the queue-level routing
						},
					},
				},
			},
		},
	}

	return req, nil
}

// publish event to Cloud Tasks
func (p *PublisherImpl) publish(ctx context.Context, ee Envelope) error {
	req, err := p.buildTaskRequest(ee)
	if err != nil {
		return err
	}

	createdTask, err := p.cloudTaskClient.CreateTask(context.Background(), req)
	if err != nil {
		return err
	}

	...

	return nil
}

Expected behavior

We expect the irrelevantWorkerUrl being set during task creation to not matter and have our tasks routed and authenticated using the values set on the queue.

Actual behavior

Everything set as part of URI Override works perfectly.

However, the JWT being created for the service account has an audience that matches irrelevantWorkerUrl.

So our worker service refuses to accept the request because the audience on the token is irrelevantWorkerUrl instead of the https://<cloud-run-worker-service-host> that we set on the queue.

We can get the code working if we pass cloud-run-worker-service-host as the audience or as the worker url, but that defeats our goal of using queue-level routing.

When checking the queue, everything looks correct.

gcloud tasks queues describe

httpTarget:
  oidcToken:
    audience: https://<cloud-run-worker-service-host>
    serviceAccountEmail: <service-acount-email>
  uriOverride:
    host: <cloud-run-worker-service-host>
    pathOverride:
      path: <worker-path>
    port: '443'
    queryOverride: {}
    scheme: HTTPS
    uriOverrideEnforceMode: ALWAYS

Additional context

The whole reason we want to use queue-level routing is so we don't have to pass the worker service's host to all the places where we create tasks.

I am not sure if this feature is out of beta but I see the options documented here without any indication that it is a beta feature.

https://cloud.google.com/sdk/gcloud/reference/tasks/queues/create
https://cloud.google.com/sdk/gcloud/reference/tasks/queues/update
https://cloud.google.com/tasks/docs/configuring-queues#gcloud

For this test, we are using existing golang code for creating tasks but we generated a new queue with queue-level routing.

We also tried creating tasks using gcloud:

gcloud tasks create-http-task --queue=<queue-name> --url=<irrelevantWorkerUrl> --oidc-service-account-email="<service-acount-email>"

Same result where the audience is irrelevantWorkerUrl

We also tried creating tasks using golang code and gcloud where we don't pass a service-account. In those cases, a token does not get generated at all.

Maybe we are misunderstanding what --http-oidc-token-audience-override is supposed to do, but so far, it doesn't seem like it does anything.