Cloud telemetry export fails with invalid_scope when ADC is a service-account key file

[DmGranit]

Summary

With get_fast_api_app(..., otel_to_cloud=True) and Application Default Credentials supplied as a service-account key file, every Cloud Trace and Cloud Monitoring export fails at the token grant, before any HTTP request is made:

google.auth.exceptions.RefreshError: ('invalid_scope: Invalid OAuth scope or ID token audience provided.', ...)
  ... opentelemetry/exporter/otlp/proto/http/{trace,metric}_exporter/__init__.py
  ... google/auth/transport/requests.py  request
  ... google/oauth2/service_account.py   _perform_refresh_token
  ... google/oauth2/_client.py           jwt_grant

The application itself is unaffected — only the exporters fail — so the symptom is a continuous stream of tracebacks in the container log (observed at roughly nine per minute) with no telemetry arriving and no other outward sign.

Environment

• google-adk 2.8.0, google-auth 2.57.0, Python 3.12
• Host not on GCP (self-hosted container), ADC via GOOGLE_APPLICATION_CREDENTIALS=/path/key.json
• GOOGLE_CLOUD_PROJECT set

Cause

ADC is resolved without scopes and the result is handed to the OTLP exporters' AuthorizedSession. A service-account key credential has requires_scopes = True and no scopes, so the JWT grant is rejected.

Two call sites resolve it that way:

• google/adk/cli/api_server.py:700 — _setup_gcp_telemetry: credentials, project_id = google.auth.default(), then passed as google_auth= to get_gcp_exporters. This is the one that fires on the get_fast_api_app path, because it supplies google_auth= and the fallback below is never reached.
• google/adk/telemetry/google_cloud.py:96 and :201 — the google_auth is None fallback inside get_gcp_exporters / the metrics helper.

It does not reproduce on GCE, Cloud Run, GKE or Agent Engine: the metadata server issues a usable token regardless of the scopes request, so the omission is invisible there. A key file is what exposes it.

Minimal reproduction

import google.auth
from google.auth.transport.requests import Request

# GOOGLE_APPLICATION_CREDENTIALS -> a service-account key file
creds, project = google.auth.default()
creds.requires_scopes            # True
creds.refresh(Request())         # RefreshError: invalid_scope

creds, project = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
creds.refresh(Request())         # OK

Suggested fix

Apply scopes where they are needed, leaving credentials that do not need them untouched:

from google.auth.credentials import with_scopes_if_required

credentials = with_scopes_if_required(
    credentials, ["https://www.googleapis.com/auth/cloud-platform"]
)

or resolve with google.auth.default(scopes=[...]), which routes through the same helper.

Worth noting for whichever is chosen: with_scopes_if_required rewrites any credential whose requires_scopes is true, and that includes google.auth.compute_engine.Credentials — measured requires_scopes is True on 2.57.0. On GCE the metadata server ignores the scopes parameter, but on Cloud Run / App Engine Flex it honours it, so this is not a no-op everywhere in-cloud even though it is harmless.

Workaround

Build the exporters in the application instead of letting otel_to_cloud=True resolve ADC, and pass otel_to_cloud=False so a second, unscoped set is not installed:

creds, project = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
hooks = get_gcp_exporters(
    enable_cloud_tracing=True, enable_cloud_metrics=True, enable_cloud_logging=True,
    google_auth=(creds, project),
)
maybe_set_otel_providers([hooks], otel_resource=get_gcp_resource(project))

Two things have to be reproduced by hand when doing this, and both are easy to miss:

• _setup_instrumentation_lib_if_installed — without GoogleGenAiSdkInstrumentor the export carries agent spans and no model calls (plus the HTTPX/gRPC instrumentors on Agent Engine).
• maybe_install_request_metrics_middleware, which get_fast_api_app skips under otel_to_cloud=False. On Agent Engine that is not just a flush-timing loss: _RequestDrivenMetricReader has no background ticker, note_request_start is the only writer of its in-flight counter, and the generate_content path refuses to collect while that counter is zero — so metrics stop entirely.

Possibly related: discussion #1009 reports the same error with a key file.

Related, same off-GCP class (observed after applying the workaround above)

Once the token grant succeeds, metric export from a non-GCP host is still rejected by telemetry.googleapis.com/v1/metrics with 400 INVALID_ARGUMENT: prometheus_target resource type must have an instance specified. Off Agent Engine, get_gcp_resource() builds a bare Resource({"gcp.project_id": ...}); the Telemetry API's Prometheus mapping requires instance (from service.instance.id) and location on the resource, and the GCP resource detector supplies neither outside Google Cloud. Setting OTEL_RESOURCE_ATTRIBUTES=service.instance.id=<id>,location=<region> makes the same export return 200, so it is only the resource that is missing. Traces are unaffected. Mentioned here because it hits the same deployment shape (key file, self-hosted) right after the scope problem is fixed; happy to open it separately if preferred.

[chelsealong]

Picking this one up now — opening a PR shortly. Flagging it here so nobody duplicates the work; if someone is already on it, say so and I will drop mine.