Skip to content

Store personal access tokens only as digests (D2 of 3: rerun the backfill, stop the plaintext) - #3202

Draft
jeremy wants to merge 8 commits into
mainfrom
oauth-token-digests-d2
Draft

jeremy wants to merge 8 commits into
mainfrom
oauth-token-digests-d2

Conversation

@jeremy

@jeremy jeremy commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Caution

Explicitly gated. Don't merge this until production is running D1 (#3192), on every production container, and the backfill rerun in this PR (20261017120000) has run against fizzy_production, digesting every personal access token production main wrote in plaintext meanwhile. This code stops falling back to plaintext: a token production main wrote without a digest, and never used since, stops authenticating the moment this code serves it, unless the rerun has digested it first.

Warning

This PR contains migrations. Don't merge it directly. It has to go through the beta integration branch (oauth-integration-2026-10-15) and be migrated by hand on beta before it lands. beta1 runs against fizzy_production (saas/config/database.yml ends beta: *production), so migrating on beta1 is migrating production. The rows to add to that branch's migration inventory are below; this PR doesn't push to the integration branch.

Stacked on #3192 (train: #2296 → #3080 → #3081 → #3082 → #3160 → #3161 → #3159 → #3163 → #3169 → #3192 → this → #3203).

Implements step D2 of hashing tokens at rest (rubric TK-2, bearer artifacts stored only as digests):

Step What it does Status
D1 (#3192) Add digest columns, backfill, read by digest with a plaintext fallback for personal access tokens, keep writing their plaintext for production main after the OAuth stack
D2 (this PR) Rerun the backfill. Stop writing plaintext and stop the fallback. Ignore token after D1 is deployed to production
D3 (#3203) Drop identity_access_tokens.token after D2 is deployed everywhere

What D2 does

Unchanged: the token format (fizzy_pat_ + base58 body), the create JSON fizzy-cli reads (id, token, description, permission, created_at), the revocation endpoint (it already goes through find_by_raw_token), and every OAuth path, which D1 already made digest-only.

Tests

  • ApiTest: the two scheme pins (bearer matched case-insensitively, Basic Bearer … not bearer) present the fixture's raw_token. (Store tokens and client secrets as digests (D1 of 3: read by digest, plaintext fallback) #3192 now sends grants' raw tokens in its three challenge pins itself.)
  • Fixtures store only digests; their raw values live in AccessTokenTestHelper::RAW_FIXTURE_TOKENS, and identity_access_tokens(...) hands back records carrying raw_token, as an issued token does. Call sites moved from .token to .raw_token.
  • Identity::AccessTokenTest: personal tokens store only their digest (the column is NULL), a token stored only in plaintext doesn't authenticate and issues no plaintext query, and the model ignores the column. D1's fallback tests are gone with the fallback.
  • BackfillRemainingTokenDigestsTest: digests the leftovers, keeps personal plaintext and clears a grant's, leaves updated_at alone, is idempotent, the result authenticates, refuses duplicates (including one already digested) before any write without naming the token, keeps the plaintext of the personal access client's tokens, and is a no-op once the column is gone. D1's BackfillTokenDigestsTest now writes and reads plaintext through the migration's own model.

Migration inventory rows for the integration branch

For docs/oauth/integration-migrations.md on oauth-integration-2026-10-15. This PR doesn't change that branch. Labeled D2-1 rather than numbered, so a sibling PR claiming the rows after #3192's 8–10 doesn't collide:

# Version / file PR What it does Production table MySQL 8.4 DDL
D2-1 20261017120000_backfill_remaining_token_digests.rb this Data only, no DDL. Refuses duplicate tokens first. Then one UPDATE … WHERE id = ? AND token_digest IS NULL per row with a token and no digest: token_digest, and token NULLed for OAuth rows (a personal access client's rows, once #3205 adds it, keep it). updated_at doesn't move. disable_ddl_transaction!, batches of 500. down is a no-op identity_access_tokens: ~3,500 rows No ALTER, so no exclusive metadata lock: only the shared MDL every read takes, and row locks on the primary key. token has no index, so the duplicate check and the batch selection each scan ~3,500 rows, milliseconds. Expect a handful of updates: only tokens production main wrote before D1 reached it and nobody has used since

Notes for the runbook:

  • Order. (1) D1 deployed to every production container; no main process older than D1 left running. (2) Run D2-1 on beta1. (3) Only then let D2 code serve, on beta or anywhere: beta's web role doesn't migrate on boot (see the integration doc), so either migrate with the new image first (its option B) or accept that until D2-1 runs, beta refuses the leftover plaintext-only tokens (production, still on D1, keeps accepting and healing them).
  • If anything wrote a plaintext-only token after D2-1 ran (a pre-D1 process that outlived the deploy), rerun it: bin/rails db:migrate:redo:primary VERSION=20261017120000. A quick check before landing: SELECT COUNT(*) FROM identity_access_tokens WHERE token_digest IS NULL should be 0.
  • Undo by version: bin/rails db:migrate:down:primary VERSION=20261017120000 is a no-op (the digests stay, and D1 reads them). Undoing the code is a plain revert: D1 still reads digests first and falls back to the plaintext that's still there. Never db:rollback (see the integration doc).
  • Self-hosted installs migrate on boot before serving (db:prepare), so one that upgrades straight past D1 runs D1's backfill and D2-1 before this code answers a request. With Kamal, the old container keeps serving until the new one passes its health check, so a personal access token created on a pre-D1 container in those seconds, after D2-1 has scanned, has no digest and never authenticates. Its holder sees a 401 on first use and creates another. Closing that would mean keeping the plaintext fallback D2 exists to remove, so it isn't done; D1 and D2 reach main separately, so an install that follows main passes through D1 first.

Verified

On thelio at the dcb914ceb head, whose diff differs from this head's only in three api_test.rb lines that #3192 now carries itself, gated on exit status (CI's commands: bin/rails db:setup then bin/rails test, against a dedicated mysql:8.4 container):

Suite Result
SaaS (MySQL 8.4) 2298 tests, 0 failures, 0 errors
OSS (SQLite) 2130 tests, 0 failures, 0 errors
OSS (MySQL 8.4, DATABASE_ADAPTER=mysql) 2130 tests, 0 failures, 0 errors

Migration round trip, SQLite and a dedicated MySQL 8.4 container: loaded #3192's schema and seeded a personal access token written by pre-D1 code (plaintext only, never used), one written by D1 (plaintext and digest) and an OAuth grant (digest only), all with updated_at five days back. Then up to 20261017120000: the pre-D1 token got its digest, everything kept its digest, personal plaintext stayed, no updated_at moved, and the dump matches this branch's db/schema.rb / db/schema_sqlite.rb exactly. redo of the rerun: same. (D3 continues the round trip.)

#3191 shipped-client pins, at 8340a8d (this head before a rebase that touched only files this PR doesn't) merged with test/pin-shipped-client-auth at 98572aa, locally, not pushed. Unchanged, ShippedClientAuthenticationTest errors (OSS: 8 tests, 6 errors), because its bearer helper reads the fixture's plaintext as .token, which no longer exists, and so does one SaaS audits test. With those two lines reading .raw_token, they pass: OSS (SQLite) 8 tests, SaaS (MySQL 8.4) 17 with the auditor pins, 0 failures. Whichever of #3191 and this lands second carries that two-line change. The token string and the create JSON shape are unchanged.

Not done here, on purpose

  • The plaintext column stays until D3, for the rollback above.
  • No change to oauth_*: D1 already made OAuth credentials digest-only.
  • No new third party or personal-data flow.

@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T18:55:45.036480Z 457a45e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c8786a9dcc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread app/models/identity/access_token.rb

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A human must confirm D1 is deployed everywhere and the production backfill has completed before digest-only authentication serves traffic.

No open issues 🎉

What changed in this PR

Implements D2 of the token-digest rollout, moving personal access token reads and writes to digests while preparing for D3’s plaintext-column removal.

Changes:

  • Adds a rerunnable backfill for remaining plaintext-only tokens.
  • Removes plaintext authentication fallback and writes; ignores the retained column.
  • Updates fixtures, regression tests, and conformance documentation.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File Description
test/​test_helpers/​access_token_test_helper.rb Supplies raw fixture tokens in memory.
test/​test_helper.rb Includes the token fixture helper.
test/​models/​identity/​access_token_test.rb Tests digest-only storage and authentication.
test/​migrations/​backfill_token_digests_test.rb Accesses plaintext through the migration model.
test/​migrations/​backfill_remaining_token_digests_test.rb Tests backfill safety and rerunnability.
test/​integration/​oauth_revocation_test.rb Creates legacy tokens through raw_token.
test/​integration/​oauth_grant_reach_test.rb Uses raw fixture credentials.
test/​integration/​oauth_flow_test.rb Preserves legacy-token authentication coverage.
test/​integration/​oauth_availability_test.rb Uses raw credentials with OAuth disabled.
test/​integration/​active_storage_authorization_test.rb Updates bearer credentials for download tests.
test/​fixtures/​identity/​access_tokens.yml Stores only token digests.
test/​controllers/​users_controller_test.rb Updates bearer authentication tests.
test/​controllers/​search_rate_limiting_test.rb Updates bearer credential helper.
test/​controllers/​my/​access_tokens_controller_test.rb Updates token-management authentication tests.
test/​controllers/​api_test.rb Uses raw fixture bearer tokens.
test/​controllers/​active_storage/​direct_uploads_controller_test.rb Updates upload authentication tests.
test/​controllers/​accounts/​exports_controller_test.rb Updates export authentication test.
docs/​oauth/​spec_conformance.md Documents D2’s digest-only behavior.
db/​schema.rb Advances MySQL schema version.
db/​schema_sqlite.rb Advances SQLite schema version.
db/​migrate/​20261016120000_backfill_remaining_token_digests.rb Backfills remaining digests without changing timestamps.
app/​models/​identity/​access_token.rb Stops plaintext reads and writes; ignores the column.

Review effort: Balanced

@jeremy
jeremy force-pushed the oauth-token-digests branch from 5cf35e0 to 1ef16f3 Compare October 17, 2026 07:33
@jeremy
jeremy force-pushed the oauth-token-digests-d2 branch from c8786a9 to 135b057 Compare October 17, 2026 07:43
@jeremy
jeremy requested a balanced review from Copilot October 17, 2026 08:16
@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 135b057970

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A human must confirm D1 is deployed everywhere and the production backfill has completed before digest-only authentication serves traffic.

No open issues 🎉

Review effort: Balanced

@jeremy
jeremy force-pushed the oauth-token-digests branch from 1ef16f3 to 2bf7e87 Compare October 17, 2026 08:22
@jeremy
jeremy force-pushed the oauth-token-digests-d2 branch from 135b057 to b6675c2 Compare October 17, 2026 08:31
@jeremy
jeremy requested a balanced review from Copilot October 17, 2026 08:44
@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 7a68c24560

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human confirmation is required that D1 runs everywhere and the production backfill has completed before digest-only authentication serves traffic.

No open issues 🎉

Review effort: Balanced

@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

Review threads: 1 resolved (0 fixed, 1 declined with the reasoning in the thread).
Declined: Codex, keep the plaintext fallback through a self-hosted rolling cutover: a seconds-wide Kamal cutover window, a loud and self-correcting 401, and the only fix is the fallback this step removes; recorded in the body's runbook notes.
Open for a decision: the merge gate in the PR body (D1 on every production container, then 20261017120000 run against fizzy_production), which Copilot also flags; Jeremy's call when D1 is deployed.

@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

@codex review

@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 3ac03cc45d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A maintainer must confirm D1 is deployed everywhere in production and the production backfill has completed before digest-only authentication serves traffic.

No open issues 🎉

Review effort: Balanced

Step D2 of hashing tokens at rest. Personal access tokens are no longer
written in plaintext, and a token without a digest no longer falls back to
a plaintext lookup. Identity::AccessToken ignores the token column, so
every process running this code leaves it out of its INSERTs and SELECTs,
ready for the column to be dropped.

BackfillRemainingTokenDigests reruns the backfill first: production code
that stores only plaintext kept writing personal access tokens until
step D1 reached it, and one never used since has no digest.

Fixture tokens now carry their raw value in memory, as raw_token, the
way an issued token does.
… renumber it

A personal access token may be issued by the personal access client
rather than carrying no client, so the rerun keeps the plaintext of
either, as BackfillTokenDigests does once that client exists.

Renumbered to 20261017120000: 20261016120000 is taken by
CreatePersonalAccessClient on a sibling branch.
@jeremy
jeremy force-pushed the oauth-token-digests-d2 branch from 3ac03cc to 457a45e Compare October 18, 2026 05:56
@jeremy
jeremy requested a balanced review from Copilot October 18, 2026 06:00
@jeremy

jeremy commented Oct 18, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human confirmation is required that D1 runs everywhere and the production backfill completed before digest-only authentication serves traffic.

No open issues 🎉

Review effort: Balanced

@jeremy

jeremy commented Oct 18, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@jeremy

jeremy commented Oct 18, 2026

Copy link
Copy Markdown
Member Author

Restacked onto #3192 617edb87b ("Keep identity_access_tokens.refresh_token until it can go with token"), now at 457a45e30. The rebase resolved one conflict: ignored_columns now lists both plaintext columns, token and refresh_token. On thelio: rubocop clean; SaaS (MySQL), OSS (SQLite) and OSS (MySQL) suites pass, 0 failures, 0 errors. CI is green and Copilot reviewed this head with no open issues. Codex didn't review it: two requests returned "You have reached your Codex usage limits for code reviews". Re-request it once the limit resets.

@jeremy

jeremy commented Oct 19, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jeremy

jeremy commented Oct 19, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 457a45e309

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ping partway

* origin/oauth-token-digests: (54 commits)
  Make this branch's migrations safe to run again after stopping partway
  Make this branch's migrations safe to run again after stopping partway
  Make the OAuth migrations safe to run again after stopping partway
  Redact retired refresh tokens from logged SQL
  Filter the code parameter by its whole name, so code_challenge stays readable
  Bring main's tests up to this branch: Ruby 4, webhook attribution, token digests
  Pin the fizzy_mcp accessory to fizzy-mcp-server v0.1.0 by digest
  Scope webhook dispatch by account as well as board
  Test the reviewer audit against the structured log line production writes
  Refuse an OAuth parameter that arrives as a list where one value is meant
  Revoke on the email alone, and record each use in the request's structured log
  Remove the reviewer's access tokens and passkeys on sign-out, so a code holder can't keep a way back in
  Wire the reviewer sign-in into production from Deploy/Fizzy
  Close the reviewer's live sockets on sign-out, and keep the identity id out of Sentry
  End the reviewer's sessions on rotation, so the old code's sessions don't outlive it
  Refuse a code that normalizes to nothing, not just a blank one
  Let the directory reviewer sign in with a fixed code, SaaS only and off unless configured
  Refuse docker-dev's shared default port in a git worktree
  Ruby 4.0.9 (#3234)
  Retry the offline clear only in an active worker
  ...
* origin/oauth-token-digests:
  Answer 404 from a dark endpoint however many times it's asked
Base automatically changed from oauth-token-digests to main October 27, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants