From 913a6524f1719c519ba60f39d4aebf9f14d18dc3 Mon Sep 17 00:00:00 2001 From: Lenny Chen Date: Mon, 20 Apr 2026 12:22:02 -0700 Subject: [PATCH 1/4] docs: external storage: address friction log feedback --- .../go/best-practices/data-handling/external-storage.mdx | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/develop/go/best-practices/data-handling/external-storage.mdx b/docs/develop/go/best-practices/data-handling/external-storage.mdx index d870d77d4e..1e3ac5b58f 100644 --- a/docs/develop/go/best-practices/data-handling/external-storage.mdx +++ b/docs/develop/go/best-practices/data-handling/external-storage.mdx @@ -34,12 +34,11 @@ The Go SDK includes an S3 storage driver. Follow these steps to set it up: - An Amazon S3 bucket that you have read and write access to. Refer to [lifecycle management](/external-storage#lifecycle) to ensure that your payloads remain available for the entire lifetime of the Workflow. -- Install the S3 driver module: `go get go.temporal.io/sdk/contrib/aws/s3driver` +- Install the S3 driver module and its dependencies: `go get go.temporal.io/sdk/contrib/aws/s3driver go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2 github.com/aws/aws-sdk-go-v2/config` ### Procedure -1. Load your AWS configuration and create the S3 storage driver. The driver uses your standard AWS - credentials from the environment (environment variables, IAM role, or AWS config file): +1. Load your AWS configuration and create the S3 storage driver. The driver uses your standard [AWS credentials](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html) from the environment (environment variables, IAM role, or AWS config file): From d0392b2d894403040a8cdf3a8b8556c863fe9cbc Mon Sep 17 00:00:00 2001 From: Lenny Chen Date: Mon, 20 Apr 2026 14:34:51 -0700 Subject: [PATCH 2/4] add additional dependency --- .../go/best-practices/data-handling/external-storage.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/develop/go/best-practices/data-handling/external-storage.mdx b/docs/develop/go/best-practices/data-handling/external-storage.mdx index 1e3ac5b58f..f0255a644f 100644 --- a/docs/develop/go/best-practices/data-handling/external-storage.mdx +++ b/docs/develop/go/best-practices/data-handling/external-storage.mdx @@ -34,7 +34,7 @@ The Go SDK includes an S3 storage driver. Follow these steps to set it up: - An Amazon S3 bucket that you have read and write access to. Refer to [lifecycle management](/external-storage#lifecycle) to ensure that your payloads remain available for the entire lifetime of the Workflow. -- Install the S3 driver module and its dependencies: `go get go.temporal.io/sdk/contrib/aws/s3driver go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2 github.com/aws/aws-sdk-go-v2/config` +- Install the S3 driver module and its dependencies: `go get go.temporal.io/sdk/contrib/aws/s3driver go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2 github.com/aws/aws-sdk-go-v2/config github.com/aws/aws-sdk-go-v2/service/s3` ### Procedure From 81a70becd65ab807dc5971da334e78c59e721d6f Mon Sep 17 00:00:00 2001 From: Lenny Chen Date: Mon, 20 Apr 2026 18:11:52 -0700 Subject: [PATCH 3/4] address friction log feedback --- .../data-handling/external-storage.mdx | 56 ++++++++++--------- .../data-conversion/external-storage.mdx | 4 ++ 2 files changed, 35 insertions(+), 25 deletions(-) diff --git a/docs/develop/go/best-practices/data-handling/external-storage.mdx b/docs/develop/go/best-practices/data-handling/external-storage.mdx index f0255a644f..a9914fb53c 100644 --- a/docs/develop/go/best-practices/data-handling/external-storage.mdx +++ b/docs/develop/go/best-practices/data-handling/external-storage.mdx @@ -32,13 +32,17 @@ The Go SDK includes an S3 storage driver. Follow these steps to set it up: ### Prerequisites -- An Amazon S3 bucket that you have read and write access to. Refer to [lifecycle management](/external-storage#lifecycle) - to ensure that your payloads remain available for the entire lifetime of the Workflow. -- Install the S3 driver module and its dependencies: `go get go.temporal.io/sdk/contrib/aws/s3driver go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2 github.com/aws/aws-sdk-go-v2/config github.com/aws/aws-sdk-go-v2/service/s3` +- An Amazon S3 bucket that you have read and write access to. Refer to + [lifecycle management](/external-storage#lifecycle) to ensure that your payloads remain available for the entire + lifetime of the Workflow. +- Install the S3 driver module and its dependencies: + `go get go.temporal.io/sdk/contrib/aws/s3driver go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2 github.com/aws/aws-sdk-go-v2/config github.com/aws/aws-sdk-go-v2/service/s3` ### Procedure -1. Load your AWS configuration and create the S3 storage driver. The driver uses your standard [AWS credentials](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html) from the environment (environment variables, IAM role, or AWS config file): +1. Load your AWS configuration and create the S3 storage driver. The driver uses your standard + [AWS credentials](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html) from the + environment (environment variables, IAM role, or AWS config file): @@ -217,23 +221,25 @@ A custom driver implements the `converter.StorageDriver` interface with four met reference so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored breaks retrieval. For example, two S3 drivers could be named `"s3-primary"` and `"s3-archive"`. - `Type()` returns a string that identifies the driver implementation. Unlike `Name()`, this must be the same across all - instances of the same driver type regardless of configuration. Two S3 drivers named `"s3-primary"` and `"s3-archive"` would both return - `"aws.s3driver"` as their type, while the local disk driver in the custom driver code sample returns `"local-disk"`. + instances of the same driver type regardless of configuration. Two S3 drivers named `"s3-primary"` and `"s3-archive"` + would both return `"aws.s3driver"` as their type, while the local disk driver in the custom driver code sample returns + `"local-disk"`. - `Store()` receives a slice of payloads and returns one `StorageDriverClaim` per payload. A claim is a set of string key-value pairs that the driver uses to locate the payload later. - `Retrieve()` receives the claims that `Store()` produced and returns the original payloads. ### 2. Store payloads -In `Store()`, marshal each Payload protobuf message to bytes with `proto.Marshal(payload)` and write the bytes to -your storage system. The application data has already been serialized by the [Payload Converter](/develop/go/data-handling/data-conversion) -and [Payload Codec](/develop/go/data-handling/data-encryption) before it reaches the driver. -See the [data conversion pipeline](/external-storage#data-pipeline) for more details. +In `Store()`, marshal each Payload protobuf message to bytes with `proto.Marshal(payload)` and write the bytes to your +storage system. The application data has already been serialized by the +[Payload Converter](/develop/go/data-handling/data-conversion) and +[Payload Codec](/develop/go/data-handling/data-encryption) before it reaches the driver. See the +[data conversion pipeline](/external-storage#data-pipeline) for more details. -Return a `StorageDriverClaim` for each payload with enough information to retrieve it later. The `ctx.Target` -provides identity information (namespace, Workflow ID) depending on the operation. Use a type switch on -`StorageDriverWorkflowInfo` and `StorageDriverActivityInfo` to access the concrete values. Consider structuring -your storage keys to include this information so that you can identify which Workflow owns each payload. +Return a `StorageDriverClaim` for each payload with enough information to retrieve it later. The `ctx.Target` provides +identity information (namespace, Workflow ID) depending on the operation. Use a type switch on +`StorageDriverWorkflowInfo` and `StorageDriverActivityInfo` to access the concrete values. Consider structuring your +storage keys to include this information so that you can identify which Workflow owns each payload. ### 3. Retrieve payloads @@ -279,21 +285,21 @@ c, err := client.Dial(client.Options{ ## Use multiple storage drivers When you register multiple drivers, you must provide a `DriverSelector` that implements the `StorageDriverSelector` -interface. The selector chooses which driver stores each payload. Any driver in the list that is not selected for storing is still -available for retrieval, which is useful when migrating between storage backends. Return `nil` from the selector to -keep a specific payload inline in Event History. +interface. The selector chooses which driver stores each payload. Any driver in the list that is not selected for +storing is still available for retrieval, which is useful when migrating between storage backends. Return `nil` from the +selector to keep a specific payload inline in Event History. Multiple drivers are useful in scenarios such as: -- Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the - one you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. - The old driver remains available for retrieving existing claims. -- Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For - example, use S3 for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the - appropriate driver based on the runtime environment. +- Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the one + you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. The old + driver remains available for retrieving existing claims. +- Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For example, use S3 + for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the appropriate driver + based on the runtime environment. -The following example registers two drivers but always selects `preferredDriver` for new payloads. The `legacyDriver` -is only registered so the Worker can retrieve payloads that were previously stored with it: +The following example registers two drivers but always selects `preferredDriver` for new payloads. The `legacyDriver` is +only registered so the Worker can retrieve payloads that were previously stored with it: diff --git a/docs/encyclopedia/data-conversion/external-storage.mdx b/docs/encyclopedia/data-conversion/external-storage.mdx index c2715bd48d..c084e99c26 100644 --- a/docs/encyclopedia/data-conversion/external-storage.mdx +++ b/docs/encyclopedia/data-conversion/external-storage.mdx @@ -95,6 +95,10 @@ The SDK parallelizes uploads and downloads to minimize latency. When a single Wo that exceed the threshold, the SDK uploads or downloads all of them concurrently rather than one at a time. This allows external storage operations to scale well even when a Task carries many large payloads. +When a payload is offloaded to external storage, the Temporal UI displays a reference token instead of the actual data. +This is expected. Your application code receives the fully decoded result because the SDK transparently retrieves the +payload from external storage before returning it to your Workflow or Client. + Because External Storage runs after the Payload Codec, if you use an encryption codec, payloads are already encrypted before upload to your store. From f1527948a2c8c2747a34de61a50cb8adb883792e Mon Sep 17 00:00:00 2001 From: Lenny Chen Date: Mon, 20 Apr 2026 18:16:22 -0700 Subject: [PATCH 4/4] docs: address external storage friction log feedback - Add missing Go module dependencies to prerequisites - Link to AWS SDK for Go v2 credential configuration guide - Add note about reference tokens appearing in Temporal UI Co-Authored-By: Claude Opus 4.6 (1M context) --- .../data-handling/external-storage.mdx | 56 +++++++++---------- 1 file changed, 25 insertions(+), 31 deletions(-) diff --git a/docs/develop/go/best-practices/data-handling/external-storage.mdx b/docs/develop/go/best-practices/data-handling/external-storage.mdx index a9914fb53c..f0255a644f 100644 --- a/docs/develop/go/best-practices/data-handling/external-storage.mdx +++ b/docs/develop/go/best-practices/data-handling/external-storage.mdx @@ -32,17 +32,13 @@ The Go SDK includes an S3 storage driver. Follow these steps to set it up: ### Prerequisites -- An Amazon S3 bucket that you have read and write access to. Refer to - [lifecycle management](/external-storage#lifecycle) to ensure that your payloads remain available for the entire - lifetime of the Workflow. -- Install the S3 driver module and its dependencies: - `go get go.temporal.io/sdk/contrib/aws/s3driver go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2 github.com/aws/aws-sdk-go-v2/config github.com/aws/aws-sdk-go-v2/service/s3` +- An Amazon S3 bucket that you have read and write access to. Refer to [lifecycle management](/external-storage#lifecycle) + to ensure that your payloads remain available for the entire lifetime of the Workflow. +- Install the S3 driver module and its dependencies: `go get go.temporal.io/sdk/contrib/aws/s3driver go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2 github.com/aws/aws-sdk-go-v2/config github.com/aws/aws-sdk-go-v2/service/s3` ### Procedure -1. Load your AWS configuration and create the S3 storage driver. The driver uses your standard - [AWS credentials](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html) from the - environment (environment variables, IAM role, or AWS config file): +1. Load your AWS configuration and create the S3 storage driver. The driver uses your standard [AWS credentials](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html) from the environment (environment variables, IAM role, or AWS config file): @@ -221,25 +217,23 @@ A custom driver implements the `converter.StorageDriver` interface with four met reference so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored breaks retrieval. For example, two S3 drivers could be named `"s3-primary"` and `"s3-archive"`. - `Type()` returns a string that identifies the driver implementation. Unlike `Name()`, this must be the same across all - instances of the same driver type regardless of configuration. Two S3 drivers named `"s3-primary"` and `"s3-archive"` - would both return `"aws.s3driver"` as their type, while the local disk driver in the custom driver code sample returns - `"local-disk"`. + instances of the same driver type regardless of configuration. Two S3 drivers named `"s3-primary"` and `"s3-archive"` would both return + `"aws.s3driver"` as their type, while the local disk driver in the custom driver code sample returns `"local-disk"`. - `Store()` receives a slice of payloads and returns one `StorageDriverClaim` per payload. A claim is a set of string key-value pairs that the driver uses to locate the payload later. - `Retrieve()` receives the claims that `Store()` produced and returns the original payloads. ### 2. Store payloads -In `Store()`, marshal each Payload protobuf message to bytes with `proto.Marshal(payload)` and write the bytes to your -storage system. The application data has already been serialized by the -[Payload Converter](/develop/go/data-handling/data-conversion) and -[Payload Codec](/develop/go/data-handling/data-encryption) before it reaches the driver. See the -[data conversion pipeline](/external-storage#data-pipeline) for more details. +In `Store()`, marshal each Payload protobuf message to bytes with `proto.Marshal(payload)` and write the bytes to +your storage system. The application data has already been serialized by the [Payload Converter](/develop/go/data-handling/data-conversion) +and [Payload Codec](/develop/go/data-handling/data-encryption) before it reaches the driver. +See the [data conversion pipeline](/external-storage#data-pipeline) for more details. -Return a `StorageDriverClaim` for each payload with enough information to retrieve it later. The `ctx.Target` provides -identity information (namespace, Workflow ID) depending on the operation. Use a type switch on -`StorageDriverWorkflowInfo` and `StorageDriverActivityInfo` to access the concrete values. Consider structuring your -storage keys to include this information so that you can identify which Workflow owns each payload. +Return a `StorageDriverClaim` for each payload with enough information to retrieve it later. The `ctx.Target` +provides identity information (namespace, Workflow ID) depending on the operation. Use a type switch on +`StorageDriverWorkflowInfo` and `StorageDriverActivityInfo` to access the concrete values. Consider structuring +your storage keys to include this information so that you can identify which Workflow owns each payload. ### 3. Retrieve payloads @@ -285,21 +279,21 @@ c, err := client.Dial(client.Options{ ## Use multiple storage drivers When you register multiple drivers, you must provide a `DriverSelector` that implements the `StorageDriverSelector` -interface. The selector chooses which driver stores each payload. Any driver in the list that is not selected for -storing is still available for retrieval, which is useful when migrating between storage backends. Return `nil` from the -selector to keep a specific payload inline in Event History. +interface. The selector chooses which driver stores each payload. Any driver in the list that is not selected for storing is still +available for retrieval, which is useful when migrating between storage backends. Return `nil` from the selector to +keep a specific payload inline in Event History. Multiple drivers are useful in scenarios such as: -- Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the one - you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. The old - driver remains available for retrieving existing claims. -- Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For example, use S3 - for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the appropriate driver - based on the runtime environment. +- Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the + one you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. + The old driver remains available for retrieving existing claims. +- Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For + example, use S3 for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the + appropriate driver based on the runtime environment. -The following example registers two drivers but always selects `preferredDriver` for new payloads. The `legacyDriver` is -only registered so the Worker can retrieve payloads that were previously stored with it: +The following example registers two drivers but always selects `preferredDriver` for new payloads. The `legacyDriver` +is only registered so the Worker can retrieve payloads that were previously stored with it: