Data Connect: bursty load at half the documented 24k req/min limit fails with opaque "SQL execution failed"

[federicobaiocco]

[REQUIRED] Environment info

firebase-tools: 15.22.1

Platform: Cloud Run (Node 22, firebase-admin 13.10.0), Data Connect service in europe-west1 backed by Cloud SQL for PostgreSQL 17 (db-custom-8-16384)

We could not reproduce the issue on db-custom-1-3840

[REQUIRED] Test case

When Dataconnect is under pressure, we start getting

SQL execution failed
    at DataConnectApiClient.makeGqlRequest (/usr/src/app/typescript/node_modules/firebase-admin/lib/data-connect/data-connect-api-client-internal.js:270:19)

on random queries.

The error does not include any details about quota limits, etc. It just says "SQL execution failed".

In order to get more info about the error, we tried the following:

Standard Admin SDK usage through a generated connector SDK — one shared DataConnect instance for the whole process:

import { DataConnect, getDataConnect } from 'firebase-admin/data-connect'
import { connectorConfig } from '@firebasegen/backend-connector'

const dataConnect: DataConnect = getDataConnect({
  connector: connectorConfig.connector,
  serviceId: connectorConfig.serviceId,
  location: connectorConfig.location,
})
// generated SDK operations, e.g.:
// await sdk.touchContent(dataConnect, { contentId })

The mutation itself is trivial (single-row _update):

mutation TouchContent($contentId: String!) {
  content_update(id: $contentId, data: { updated_expr: "request.time" })
}

Minimal load generator (plain REST against the connector, which is how we captured the full error extensions that the Admin SDK drops):

import { GoogleAuth } from 'google-auth-library'

const URL = 'https://firebasedataconnect.googleapis.com/v1/projects/<project>/locations/europe-west1/services/<service>/connectors/<connector>:impersonateMutation'
const auth = new GoogleAuth({ scopes: ['https://www.googleapis.com/auth/cloud-platform'] })
const client = await auth.getClient()

const CONCURRENCY = 200

// each worker keeps exactly one request in flight, so total in-flight = CONCURRENCY
async function worker() {
  while (true) {
    const { token } = await client.getAccessToken()
    const res = await fetch(URL, {
      method: 'POST',
      headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
      body: JSON.stringify({ operationName: 'TouchContent', variables: { contentId: '<some-id>' } }),
    })
    const body = await res.json()
    if (body.errors?.length) console.error(JSON.stringify(body.errors))
  }
}
await Promise.all(Array.from({ length: CONCURRENCY }, () => worker()))

[REQUIRED] Steps to reproduce

1. Run the load generator above at concurrency 200. Each mutation takes ~1s server-side, so throughput settles around 8,000–12,000 operations/min — half of the documented limit of 24,000 GraphQL requests per minute per service (docs).
2. Let the concurrency ramp up quickly from idle
3. Within the first seconds/minutes of the burst, operations fail. The same failure signature occurs in production every time bursty load hits, while the Cloud SQL instance is demonstrably healthy: 17% CPU, 43% memory, ~140 write IOPS vs ~1,100 provisioned, zero errors in postgres.log.

[REQUIRED] Expected behavior

• Staying under the only documented limit (24,000 GraphQL requests/min) means operations don't fail due to internal capacity limits.
• If Data Connect's own infrastructure is throttled, the error is surfaced with a retryable code (UNAVAILABLE / RESOURCE_EXHAUSTED), not INTERNAL.
• firebase-admin surfaces the GraphQL error extensions so the cause is diagnosable. Today DataConnectApiClient.makeGqlRequest throws only the generic message and discards extensions.debugDetails — the only part of the response that explains what happened.

[REQUIRED] Actual behavior

The only signal we get is:

SQL execution failed Error: SQL execution failed
    at DataConnectApiClient.makeGqlRequest (.../firebase-admin/lib/data-connect/data-connect-api-client-internal.js:270:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async DataConnectApiClient.executeOperationHelper (.../firebase-admin/lib/data-connect/data-connect-api-client-internal.js:178:26)

No quota information. Calling the connector REST endpoint directly reveals the real cause in extensions.debugDetails:

{"message":"SQL execution failed","path":["content_update"],"extensions":{"code":"INTERNAL","debugDetails":"SQL runtime error: MyStubby connect error to instance \"<project>:europe-west1:<instance>\": cannot receive first message: ZERO_APP::1: stream aborted: server handler returned error: ERROR_NOT_AUTHORIZED: GetConnectSettings request failed on instance <project>/<instance> with error: boss::260: Quota exceeded for quota metric 'Connect Queries' and limit 'Connect Queries per minute per user per region' of service 'sqladmin.googleapis.com' for consumer 'project_number:<number>'. ..."}}

Note the consumer: project_number:<number> is not our project (neither our dev nor prod project number) — it appears to be a Google-managed tenant project where Data Connect's connection brokering runs. So this quota is invisible in our console/monitoring and cannot be raised by us.

What Cloud Monitoring on our side shows (metric cloudsql.googleapis.com/database/postgresql/new_connection_count, 1-min sum):

• Baseline: 8–20 new connections/min.
• During every failure burst the metric jumps to a plateau of ~2.000 new connections/min — exactly the shape of a 2,000/min quota ceiling. This correlates 1:1 with the error bursts in both our dev repro and production.

So Data Connect appears to open and close Cloud SQL connections at a very high rate under bursty load, each new connection paying GetConnectSettings/ephemeral-cert calls against the Cloud SQL Admin API Connect Queries quota (default 2,000/min per user per region) — and that internal quota, not anything documented or visible to us, is what fails our operations.

Please note that the issue did not reproduce when postgres db tier was db-custom-1-3840 and it only reproduced when upgrading to db-custom-8-16384

Asks:

1. Fix the connection churn (pool/reuse connections across bursts) or size the internal quota for the service's own behavior — or document the real limit ("max new connections per minute per service") so it can be designed around.
2. Return a retryable error code instead of INTERNAL / generic SQL execution failed for this failure mode.
3. Expose GraphQL error extensions on the thrown error

[mtr002]

Hey @federicobaiocco,

Thank you for the thorough issue report and for sharing your reproduction details. It helped us pinpoint the root cause quickly!

• We resolved a backend connection pooling bottleneck that occurred under bursty load. The fix is currently rolling out and should be available in the next 1–2 weeks.
• To ensure opaque errors like "SQL execution failed" don't mask the underlying cause in the future, we also opened a PR in the Node Admin SDK to bubble up debugDetails. This will go out with the next Admin SDK release.

Thanks again for your patience and for helping us improve Data Connect. I will keep this issue open. Feel free to follow up here if you notice any further issues once the rollout lands!