Let a person connect Google Drive and sync the folders they choose

[muemich]

Closes #1223.

Why

Most teams keep their documents in Google Drive. Today the only way into Utopia is to export them by hand or to copy them into a bucket, which loses each file's identity and its modification time. Drive passes the four tests of 0013. But no administrator key covers a person's own files, so this connector is the first that a person connects with their own account. The reasoning, including what was rejected (a service account, the Google Picker, the changes feed), is in ADR 0068.

What changes

• New source kind google_drive. An editor creates the source, clicks Connect Google Drive (an OAuth popup asking for drive.readonly plus openid email), then picks folders in a folder browser that covers My Drive, Shared with me and every shared drive. Subfolders are included.
• The refresh token is a sealed source secret. refresh_token is added to SOURCE_SECRET_KEYS. The browser never holds a Google token; the folder browser is served by the server.
• The connection flow (google_drive_flows, migration 0112) uses PKCE and a single-use state. The callback must arrive in the same session that started the flow, and the user must still be an editor. Without that check, someone could have another person click through consent and attach that person's Drive to their own source.
• Sync walks the selected folders. Docs, Sheets and Slides are exported to DOCX, XLSX and PPTX for the existing parsers; other files are downloaded as they are. Forms, shortcuts and files over 32 MB are skipped. doc_time is modifiedTime.
• Change detection uses Drive's own marker (google_drive_files): md5Checksum for binary files, modifiedTime for Google formats. An export of an unchanged Doc is a new zip every time, so comparing content hashes would re-extract every Doc on every run.
• "Not in source" is marked only after a complete walk. A selected folder that can no longer be opened fails the run by name and marks nothing.
• New endpoints: GET /google-drive/status, GET /google-drive/callback, and under /kbs/{id}/sources/{source_id}/google-drive/: POST connect, DELETE connection (also revokes the token at Google), GET folders, PUT folders.
• Configuration: UTOPIA_GOOGLE_DRIVE_CLIENT_ID, _CLIENT_SECRET and _REDIRECT_URI (documented in .env.example). When they are not set, the kind cannot be created.

Size. About 1,100 lines of Rust (roughly half of it comments, in the house style), 460 lines of tests, and 380 lines of web. I kept it as one PR because the backend can't be exercised without the connect-and-pick flow, and the UI is useless without the backend. If you'd rather review it in two parts (ADR, migration, connector and routes first, then the web panel), say so and I'll split it.

How it was checked

• Machine: macOS (arm64), rustc 1.99, node 20. The branch is based on dev at c278cb8.
• cargo fmt --all --check and cargo clippy --workspace --all-targets -- -D warnings are clean. cargo test --workspace passes, including 12 new tests in google_drive_tests.rs that use wiremock. They cover:
  • walking subfolders, de-duplicating nested selections, skipping unreadable formats
  • following page tokens
  • export versus download
  • keeping Google's exportSizeLimitExceeded reason in the error
  • unreachable and trashed folders
  • the folder browser, including a query-injection attempt
  • the code exchange (no refresh token, Drive scope unchecked on the consent screen)
  • a revoked grant
  • checking the ID token's issuer and audience
• pnpm build (which includes the design lint) and pnpm test (268 tests) pass.
• Against a real Google Drive, by hand. I built the image from this branch and ran it on GKE in a throwaway namespace with its own Postgres 16 and pgvector. All migrations, including 0112, applied. I then connected a Google account, chose folders and synced. It worked.
• Not covered by automated tests:
  • The wiremock fixtures are written from the Drive v3 and Google OAuth documentation, not recorded from real responses (0013 decision 4). The ADR's status says so. it_reads_from_a_real_drive runs only when given a client and a refresh token.
  • The database-backed SQL (the flow table, markers and reconciliation) was exercised only by the manual run above. Locally those tests skipped because there was no Postgres.

Before review

• Every commit is signed off (git commit -s)
• cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass
• SQL under crates/utopia-store/ was tested with UTOPIA_DATABASE_URL set (n/a: the new SQL lives in utopia-server; exercised by the manual run above)
• For changes under web/: pnpm build and pnpm test pass
• A new migration takes the next free number on dev, and CURRENT_SCHEMA_VERSION in crates/utopia-cli/src/main.rs equals the number of files in migrations/
• UI strings are in both web/src/i18n/en.ts and zh.ts
• A change to the data model, the ontology contract or a public API has its ADR in docs/decisions/

[WaylandYang]

Thank you for this, and for opening the issue with the use case spelled out. Yes, we want a Google Drive source, and the design reads well: read-only scope, the token never in the browser, PKCE with a single-use state bound to the session that started it, and Drive's own markers for change detection.

Two things before we review it in detail:

• Please split it the way you offered. First the ADR, the migration, the connector and the routes; then the web panel. Each part is then small enough to review properly, and the second can follow quickly.
• The SQL needs tests that run in CI. None of the current tests touch the database, so the flow table, the markers and the reconciliation are covered only by your manual run. Most of all, the callback's checks deserve database-backed tests: a state used twice, a callback from another session, a person who is no longer an editor, and an expired flow. CI runs every test with a migrated PostgreSQL (UTOPIA_TEST_REQUIRE_DB=1); crates/utopia-server/src/source_checkpoint_tests.rs shows the fixture.

Because this adds an OAuth entry point, we will do a security-focused pass on the first part once it is up.

Thanks again; this is a careful piece of work, and a connector people have asked for.

[WaylandYang]

Thank you again. I read the connector, the routes and the sync with an eye on the new OAuth entry point, so these can go into the first part when you split. The flow itself is right: PKCE, a single-use state deleted before use, the callback bound to the session that started it, the editor check repeated at the callback, and folder ids checked against a strict character set before they reach a Drive query.

Two things need changing, both about whose Drive is read:

1. Only the person who connected should choose what is read from their Drive. Today any editor of the knowledge base can browse the connected account's folders (GET …/folders?parent=…) and select any of them (PUT …/folders), so a colleague can pull folders the connecting person never chose into a base everyone in it can read. The same grant keeps syncing after that person leaves the base. Please record who connected (connected_by), allow only them to browse and select, and have a sync stop with a "reconnect" error once they are no longer an editor of the base.
2. A walk that stops at the cap never reaches the rest. The walk stops at 2,000 listed files, in the same order every time, and the run still finishes as ok, so a folder tree larger than that never brings in its tail (the same kind of gap as #1212). Unchanged files cost nothing thanks to the markers, so the cap is better placed on downloads than on the listing, and a run that stops early should say so rather than report success.

Smaller ones:

• One editor can fill the 1,000 pending flows and block every connection on the server for ten minutes; a cap per user avoids that.
• callback, select and disconnect read config, change it and write it back, so two of them at once can drop the other's change (a folder selection can overwrite a token just saved). Updating only the keys involved in one statement, or locking the row, avoids it.

The database-backed tests from my earlier note are the place to pin these down: the second state use, another session, an editor removed between start and callback, and a non-connecting editor trying to browse.

Thanks again; with these the first part should be close.