feat(gcs): Add resumable uploads by lcian · Pull Request #609 · getsentry/objectstore

lcian · 2026-09-01T14:49:54Z

Implements the Resumable Upload API on the GCS backend by forwarding to the upstream API.

2 notable things:

Resumable session tokens will now be encrypted with AES-256-GCM when Objecstore is configured with compatible symmetric keys. This is necessary for GCS, where passing through the session token verbatim would allow direct operations on the bucket.
In the previous PR/design, we didn't consider what happens when the user tries to upload a chunk at an offset lower than the persisted one.
GCS supports that, and just ignores the already persisted prefix. Objectstore could have a stricter contract where we enforce the client-provided offset to match what the backend has, but that would require a preflight request to query the offset on every chunk upload.
Instead, we document that Objecstore may or may not accept such requests, and in the case of GCS we just accept them, which practically means that we always pass through the offset and body without any pre-check.

Support resumable GCS sessions with optional service-level token encryption.

Validate session origins, preserve protocol error mapping, and report committed uploads.

Generate an ephemeral process-local key when no persistent keyring is configured, so backend capability tokens are never exposed to clients. Make service construction fallible when secure randomness is unavailable.

Remove the unintended TieredStorage forwarding implementation and restore its unsupported resumable-upload documentation.

lcian · 2026-09-02T08:58:23Z

bugbot run
@sentry review

Define completion relative to the backend handling the upload, so composed backends must finish publication before reporting terminal success. Clarify status-query recovery after a lost final response. BREAKING CHANGE: Rename Committed to Complete and CommitResponse to CompleteUploadResponse.

Return the authoritative terminal status when an offset-gap recovery query finds that the upload already completed. Preserve offset mismatch and query-failure behavior for the other outcomes.

Reintroduce the offset-query Content-Length and query precedence tests. Use a protected fixture token so both requests reach the unsupported filesystem backend after session-token encryption became mandatory.

Encode the validated active-key invariant in the keyring structure so token encryption no longer needs a map lookup or expect.

Keep LessSafeKey behind a token-specific wrapper that always generates sealing nonces internally and centralizes authenticated decryption.

Flatten the active key into the keyring and keep one shared random generator. Name the private AEAD wrapper and its operations after their protocol-level roles.

Use the GCS status codes, session version, and u64 offset type directly where their meaning is already clear from context.

Keep the backend session payload and protected service envelope minimal until format versioning is needed. Allow extra backend JSON fields for forward compatibility.

Call the canonical backend object identity storage_path to match ObjectId::as_storage_path and distinguish it from a logical key.

Parse and validate GCS session locations once, retain them as Url values, and reject malformed serialized sessions during deserialization.

Store the canonical object path and backend-owned token string in the encrypted service envelope. Backends now receive their original token string and remain responsible for parsing their own typed session state.

Encode the declared length and typed session URI directly in the backend token instead of serializing the GCS session as JSON.

lcian · 2026-09-03T16:08:16Z

bugbot run
@sentry review

Move response interpretation out of GcsBackend into range_response_to_upload_progress, which returns the progress GCS reported plus the completed object rather than writing to the change stream itself. Parsing a response now has no side effects and does not need a backend to exercise. Reporting moves to the two call sites, which invoke report_upload_completion after the parse. Calling it from both is preferred over folding the change stream back into the parser. Rename parse_resumable_range to range_response_to_offset so both helpers read as conversions from the same response.

lcian · 2026-09-04T11:16:16Z

bugbot run
@sentry review

lcian · 2026-09-04T11:35:39Z

+            .await
+            .reqwest_context("GCS: upload resumable chunk")?;
+
+        range_response_to_upload_progress(&session, response)


Here we could be nice to the user and, for example, if they submit a chunk at the wrong offset, query the offset again for them and return a 200 with the correct offset instead of an error.
Turns out that's not very easy to do, also because of the different ways that GCS can report an error that we would have to handle, so I'm not doing that for now.
Right now we recognize NOT_FOUND/GONE/CLIENT_CLOSED_REQUEST, and in other cases we'll return 500 (mapped from ErrorKind::BackendFailure).
In such cases the user is responsible to query the offset again.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit b50835d. Configure here.}

lcian · 2026-09-04T12:02:59Z

+        serialize_with = "serialize_object_id",
+        deserialize_with = "deserialize_object_id"
+    )]
+    pub(crate) object_id: ObjectId,


think if we need this

lcian · 2026-09-04T12:19:28Z

+            .map_err(|_| ErrorKind::UnknownUploadSession)?;
+        let session_uri = Url::parse(session_uri).map_err(|_| ErrorKind::UnknownUploadSession)?;
+        let session = Self::new(session_uri, total_length);
+        if session.session_uri.origin() != endpoint.origin() {


remove this check

lcian · 2026-09-04T12:41:43Z

+            .await
+            .reqwest_context("GCS: upload resumable chunk")?;
+
+        range_response_to_upload_progress(&session, response)


use an enum for completed

lcian · 2026-09-04T12:49:24Z

check if theres a higher level lib to do enc/decryption

jan-auer · 2026-09-04T11:54:34Z

        &self,
        id: ObjectId,
-        session: SessionToken,
+        session: EncryptedSessionToken,


Let's rename these parameters to token.

jan-auer · 2026-09-04T11:56:48Z

+    pub(crate) backend_token: BackendToken,
+}
+
+fn serialize_object_id<S>(id: &ObjectId, serializer: S) -> std::result::Result<S::Ok, S::Error>


Suggested change

fn serialize_object_id<S>(id: &ObjectId, serializer: S) -> std::result::Result<S::Ok, S::Error>

fn serialize_object_id<S>(id: &ObjectId, serializer: S) -> Result<S::Ok, S::Error>

jan-auer · 2026-09-04T12:06:52Z

+        header.push(key_id.len() as u8);
+        header.extend_from_slice(key_id);
+
+        let mut ciphertext = serde_json::to_vec(&session).context(


JSON escapes certain character sequences in strings. It would be good to explore other serialization formats like bincode.

At the same time we have to consider forward compatibility (if we keep the structured session token). JSON has named keys, so it's easy to add new ones, while bincode will break if the struct changes.

jan-auer · 2026-09-04T12:12:01Z

+    /// persistent keyring with [`Self::new`] when resumable sessions must survive restarts.
+    ///
+    /// Returns an error if secure random key generation fails.
+    pub fn ephemeral() -> anyhow::Result<Self> {


Two options here:

Let's try to make this infallible by using an infallible random generator and use a variation of new that always succeeds.

If this is not possible, we could maybe lift the construction of the ephemeral encryptor out of the service constructor and pass it during startup?

jan-auer · 2026-09-04T12:24:13Z

+    }
+
+    fn into_token(self) -> BackendToken {
+        format!("{}.{}", self.total_length, self.session_uri)


As a follow-up, let's investigate if we can move the responsibility of stringifying up into the encoder / the service. This all depends on how we can send a typed backend token generically through the tiered backend.

jan-auer · 2026-09-04T12:45:13Z

+    ///     key_files:
+    ///       v1: /var/run/secrets/objectstore/resumable-upload-v1
+    /// ```
+    pub resumable_token_encryption: Option<ResumableTokenEncryptionConfig>,


I'd suggest to make this configuration independent of resumable uploads. This is a key that can be used for any symmetric encryption business, which includes but does not have to be limited to resumable uploads.

jan-auer · 2026-09-04T12:46:22Z

+
+impl Service {
+    /// Loads and validates the configured resumable token encryption keys.
+    pub(crate) fn resumable_token_encryption(&self) -> Result<Option<Encryptor>> {


As per #599, there's a way to include file contents directly via config. This is also what the new kafka config uses. Let's check if we can use this instead.

jan-auer · 2026-09-04T12:52:49Z

-        Self(token)
+impl fmt::Debug for SessionToken {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.write_str("SessionToken([redacted])")


Suggested change

f.write_str("SessionToken([redacted])")

f.write_str("SessionToken")

jan-auer · 2026-09-04T12:55:09Z

+    where
+        D: Deserializer<'de>,
+    {
+        let encoded = String::deserialize(deserializer)?;


Consider deserializing into a Cow here, which can borrow and doesn't allocate.

jan-auer · 2026-09-04T12:57:56Z

-    String::from_utf8(bytes)
-        .map(SessionToken::from)
-        .map_err(|error| ApiError::map_client("session token is not valid UTF-8", error))
+    SessionToken::from_base64url(encoded)


Function can be removed and inlined.

feat(gcs): Add resumable uploads

840752a

Support resumable GCS sessions with optional service-level token encryption.

This comment has been minimized.

Sign in to view

sentry-warden Bot reviewed Sep 1, 2026

View reviewed changes

Comment thread objectstore-service/src/service.rs Outdated

This comment has been minimized.

Sign in to view

cursor Bot reviewed Sep 1, 2026

View reviewed changes

Comment thread objectstore-service/src/backend/gcs.rs Outdated

Comment thread objectstore-service/src/backend/gcs.rs

fix(resumable): Harden GCS session handling

a6411cf

Validate session origins, preserve protocol error mapping, and report committed uploads.

This comment has been minimized.

Sign in to view

cursor Bot reviewed Sep 1, 2026

View reviewed changes

Comment thread objectstore-service/src/backend/gcs.rs Outdated

sentry-warden Bot reviewed Sep 1, 2026

View reviewed changes

Comment thread objectstore-server/src/config.rs

lcian added 3 commits September 2, 2026 10:39

fix(resumable): Always encrypt session tokens

acb8bf8

Generate an ephemeral process-local key when no persistent keyring is configured, so backend capability tokens are never exposed to clients. Make service construction fallible when secure randomness is unavailable.

fix(resumable): Limit backend support to GCS

5d89b28

Remove the unintended TieredStorage forwarding implementation and restore its unsupported resumable-upload documentation.

docs(resumable): Describe protected session tokens

d9c6f82

cursor Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread objectstore-service/src/backend/gcs.rs Outdated

Comment thread objectstore-service/src/backend/gcs.rs Outdated

lcian added 12 commits September 2, 2026 11:41

fix(gcs): Honor completion after resumable offset gaps

9289e52

Return the authoritative terminal status when an offset-gap recovery query finds that the upload already completed. Preserve offset mismatch and query-failure behavior for the other outcomes.

wip

8044720

wip

62762a3

test(server): Restore resumable routing coverage

4f9e7c3

Reintroduce the offset-query Content-Length and query precedence tests. Use a protected fixture token so both requests reach the unsupported filesystem backend after session-token encryption became mandatory.

ref(resumable): Store the active encryption key directly

d5f882e

Encode the validated active-key invariant in the keyring structure so token encryption no longer needs a map lookup or expect.

ref(resumable): Encapsulate token encryption key use

c7e184c

Keep LessSafeKey behind a token-specific wrapper that always generates sealing nonces internally and centralizes authenticated decryption.

ref(resumable): Simplify token AEAD key storage

c830653

Flatten the active key into the keyring and keep one shared random generator. Name the private AEAD wrapper and its operations after their protocol-level roles.

ref(gcs): Inline resumable protocol values

9470202

Use the GCS status codes, session version, and u64 offset type directly where their meaning is already clear from context.

ref(resumable): Remove session format versioning

3049751

Keep the backend session payload and protected service envelope minimal until format versioning is needed. Allow extra backend JSON fields for forward compatibility.

ref(gcs): Name resumable session storage path

d4e0d06

Call the canonical backend object identity storage_path to match ObjectId::as_storage_path and distinguish it from a logical key.

ref(gcs): Type resumable session URI as URL

6606025

Parse and validate GCS session locations once, retain them as Url values, and reject malformed serialized sessions during deserialization.

lcian force-pushed the feat/gcs-resumable-uploads branch from aef7275 to 6606025 Compare September 3, 2026 08:42

lcian added 3 commits September 3, 2026 13:45

ref(resumable): Bind sessions at service boundary

c245f19

Store the canonical object path and backend-owned token string in the encrypted service envelope. Backends now receive their original token string and remain responsible for parsing their own typed session state.

ref(gcs): Simplify resumable session tokens

b9f9c73

Encode the declared length and typed session URI directly in the backend token instead of serializing the GCS session as JSON.

improve gcs

154fef5

lcian added 2 commits September 3, 2026 16:45

Merge branch 'main' into feat/gcs-resumable-uploads

ab2cf80

fix

c197ce1

sentry Bot reviewed Sep 3, 2026

View reviewed changes

Comment thread objectstore-service/src/backend/gcs.rs

Comment thread objectstore-service/src/resumable.rs

cursor Bot reviewed Sep 3, 2026

View reviewed changes

Comment thread objectstore-service/src/backend/gcs.rs Outdated

Comment thread objectstore-service/src/backend/gcs.rs Outdated

lcian added 8 commits September 4, 2026 10:30

improve

2c54836

improve

8ab626a

improve

9220555

improve

88c4a1c

improve

9bd703b

improve

288b1ff

improve

b95332a

cursor Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread objectstore-service/src/backend/gcs.rs

lcian commented Sep 4, 2026

View reviewed changes

improve

b50835d

lcian marked this pull request as ready for review September 4, 2026 11:41

lcian requested a review from a team as a code owner September 4, 2026 11:41

cursor Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread objectstore-service/src/backend/gcs.rs Outdated

sentry Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread objectstore-service/src/backend/gcs.rs Outdated

fix

14c71ab

lcian commented Sep 4, 2026

View reviewed changes

Comment thread objectstore-service/src/resumable.rs

lcian Sep 4, 2026

Copy link
Copy Markdown

Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check if theres a higher level lib to do enc/decryption

jan-auer reviewed Sep 4, 2026

View reviewed changes

	fn serialize_object_id<S>(id: &ObjectId, serializer: S) -> std::result::Result<S::Ok, S::Error>
	fn serialize_object_id<S>(id: &ObjectId, serializer: S) -> Result<S::Ok, S::Error>

	f.write_str("SessionToken([redacted])")
	f.write_str("SessionToken")

Uh oh!

Uh oh!

Conversation

lcian commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

lcian commented Sep 2, 2026

Uh oh!

Uh oh!

Uh oh!

lcian commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lcian commented Sep 4, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lcian commented Sep 1, 2026 •

edited

Loading