diff --git a/docs/using-source/bring-your-own-bucket.md b/docs/using-source/bring-your-own-bucket.md index 9a9b1fb..f4c636a 100644 --- a/docs/using-source/bring-your-own-bucket.md +++ b/docs/using-source/bring-your-own-bucket.md @@ -105,9 +105,7 @@ account, no matter how many buckets you connect. The CLI and console retrieve th TLS thumbprint automatically: ```bash -aws iam create-open-id-connect-provider \ - --url https://data.source.coop \ - --client-id-list sts.amazonaws.com +aws iam create-open-id-connect-provider --url https://data.source.coop --client-id-list sts.amazonaws.com ``` This returns the provider ARN diff --git a/docs/using-source/data-proxy.md b/docs/using-source/data-proxy.md index aef0a44..9a7314a 100644 --- a/docs/using-source/data-proxy.md +++ b/docs/using-source/data-proxy.md @@ -53,10 +53,12 @@ Public datasets work with `--no-sign-request`. To access data that requires auth ```bash source-coop login -aws s3 ls s3://your-org/your-product --profile source-coop +aws s3 ls s3://your-org/product-id --profile source-coop ``` -See [Upload Your Data](/data-upload#get-credentials-with-the-source-cli-recommended) for the one-time install and profile setup. +See [Upload with the Source CLI](/upload-with-the-cli) for the one-time install and profile setup. + +For software that runs on its own, with nobody there to log in, use a [service account](/automated-access) instead. ## Getting Started with AWS CLI @@ -65,7 +67,3 @@ If you don't have the AWS CLI installed, follow the [AWS CLI Getting Started Gui ## Current Status The Source Data Proxy is currently in beta. We're working to make performance indistinguishable from accessing objects directly via AWS S3. - -## Uploading Data Through the Source Data Proxy - -Uploading data through the Source Data Proxy is currently disabled while we work on new, easier, and more secure ways to upload data. If you need to upload data, please contact hello@source.coop. diff --git a/docs/using-source/data-upload.md b/docs/using-source/data-upload.md deleted file mode 100644 index 119ac6a..0000000 --- a/docs/using-source/data-upload.md +++ /dev/null @@ -1,475 +0,0 @@ ---- -title: Upload Your Data -id: data-upload -slug: /data-upload -sidebar_position: 2 ---- - -This guide explains how to deliver your data to Source Cooperative in a secure and simple way. -It is written for data providers and does not require an Amazon Web Services (AWS) account or deep cloud storage knowledge. - -If you do not see the option to upload (for example, Edit Mode or View Credentials on your product page under the lock icon), contact [hello@source.coop](mailto:hello@source.coop) to request upload access. - ---- - -## How uploads work: the Source data proxy - -Unless you set up [direct bucket access](#option-3-longstanding-or-automated-access-advanced), -every upload goes through the **Source data proxy** at `https://data.source.coop`. -The proxy speaks the **S3 API**, the object storage protocol that Amazon S3 -introduced and that most storage tools now support. It is not itself an AWS service. - -Behind the proxy, a product's data may be stored with any of several object -storage providers: Amazon S3, Google Cloud Storage, Azure Blob Storage, -Cloudflare R2, and others. You don't need to know which one. You always -upload to the same proxy address, and the proxy writes to the right place. - -That is why this guide uses AWS tools: - -- **The AWS CLI and AWS SDKs (boto3, etc.) are used only as S3 clients.** You - point them at `https://data.source.coop` instead of AWS. Any S3-compatible - client that lets you set a custom endpoint URL will work. -- **Your upload credentials come from Source Cooperative, not AWS.** They use - the AWS credential format (access key ID, secret access key, session token) - because S3 clients expect it, but they are only valid at the Source data proxy. -- **Addresses are proxy addresses.** `s3://your-org/your-product/` means the - `your-product` product in the `your-org` account on Source Cooperative. It - is not an Amazon S3 bucket. Where your data is physically stored doesn't change it. - -The one exception is [Option 3](#option-3-longstanding-or-automated-access-advanced), -which writes straight to Source's Amazon S3 bucket with your own AWS identity. -In that section, "AWS" really does mean AWS. - ---- - -## The short version (what you need to do) - -You have two main ways to upload data: - -1. **Upload directly in the Source Cooperative User Interface (UI)** (drag-and-drop or file selector), or -2. **Use temporary Source credentials** to upload through the data proxy with an S3 client such as the AWS Command Line Interface (CLI) or an AWS Software Development Kit (SDK) - -For long‑term or automated access, contact the Source Cooperative team at `hello@source.coop`. - ---- - -## Option 1: Upload directly in the UI (Easiest) - -If you prefer not to use command-line tools, you can upload files directly in the web interface. - -### How to upload via the UI - -1. Go to your product page in Source Cooperative (e.g. `https://source.coop/your-org/your-product`) -2. In the top-right corner of the Product Contents card, click on the lock icon to open the dropdown menu and enable edit mode by clicking on `Edit Mode`. -3. Either: - - Drag‑and‑drop files to the `Product Contents` card - - Use `Upload Files` or `Upload Directory` options in dropdown menu to upload files via operating system's file selector - -Your files will be uploaded automatically to your product. - -This option is ideal for: - -- Small or medium uploads -- One‑time deliveries -- Non‑technical users - ---- - -## Option 2: Upload through the data proxy with temporary credentials (recommended for larger uploads) - -For larger uploads, scripting, or programmatic access, use temporary credentials -issued by Source Cooperative with an S3 client pointed at the data proxy. - -### Get credentials with the Source CLI (recommended) - -The [Source CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you with Source Cooperative and provides temporary credentials automatically, so you don't have to copy expiring credentials out of the UI. Once configured as an AWS CLI profile, the AWS CLI and SDKs refresh credentials for you, and you'll only need to re-authenticate every 30 days. - -**1. Install the CLI** - -Follow the install instructions in the [CLI README](https://github.com/source-cooperative/source-coop-cli). - -**2. Log in** - -```bash -source-coop login -``` - -This opens your browser to authenticate and caches short‑lived credentials in your OS keyring. - -**3. Configure an AWS CLI profile** - -Add a profile to `~/.aws/config` that calls the Source CLI to fetch credentials on demand and sends requests to the data proxy: - -```ini -[profile source-coop] -credential_process = source-coop creds -endpoint_url = https://data.source.coop -``` - -The `endpoint_url` line is what sends requests to the Source data proxy instead of AWS. - -**4. Use standard S3 commands with the profile** - -Pass the profile per command with `--profile`: - -```bash -aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv --profile source-coop -aws s3 sync ./data s3://your-org/your-product/ --profile source-coop -``` - -Or set it once for your shell session with the `AWS_PROFILE` environment variable: - -```bash -export AWS_PROFILE=source-coop -aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv -aws s3 sync ./data s3://your-org/your-product/ -``` - -:::tip - -`AWS_PROFILE` is also picked up automatically by the AWS SDKs (boto3, JavaScript, Go, etc.), so your scripts can use the default credential chain instead of hardcoding access keys or endpoint details. - -::: - -The AWS CLI will refresh credentials automatically; re-run `source-coop login` when your session expires. Your upload permissions are scoped to the products you have access to. - -### Get credentials from the UI - -1. Go to your product page -2. In the `Product Contents` card, open the dropdown (click on lock icon in top‑right corner) -3. Select `View Credentials` -4. Choose: - - `JSON (SDK)` credentials, or - - `Environment Variables` (shell) - -You will also see: - -- `Expiration`: the expiration time of the credentials (a specific date and time) -- `Bucket`: the bucket name on the data proxy, which is your account ID (`your-org`) -- `Prefix`: the prefix (folder) you are allowed to write to (e.g. `your-product/`) - ---- - -## What the credentials look like - -These are Source Cooperative credentials in the AWS credential format. They -only work against the data proxy, so always set the endpoint to -`https://data.source.coop`. The region value is required by S3 clients; it does -not say where your data is stored. - -### For SDK clients (boto3, AWS SDKs, etc.) - -```json -{ - "aws_access_key_id": "ASIA...", - "aws_secret_access_key": "pwEV...", - "aws_session_token": "IQoJ...", - "region_name": "us-west-2" -} -``` - -With boto3, pass the endpoint when creating the client: - -```python -import boto3 - -s3 = boto3.client("s3", endpoint_url="https://data.source.coop", **credentials) -s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") -``` - ---- - -### For terminal / shell usage - -```bash -export AWS_ACCESS_KEY_ID="ASIA..." -export AWS_SECRET_ACCESS_KEY="pwEV..." -export AWS_SESSION_TOKEN="IQoJ..." -export AWS_DEFAULT_REGION="us-west-2" -export AWS_ENDPOINT_URL="https://data.source.coop" # add this: send requests to the data proxy -``` - -These credentials are: - -- Temporary -- Scoped to your product -- Automatically revoked when they expire - ---- - -## Where to upload your data - -You may upload only to your product's prefix on the data proxy, for example: - -``` -s3://your-org/your-product/ -``` - -This matches your product page URL, `https://source.coop/your-org/your-product`. - -You may upload: - -- Files -- Folders -- Multiple objects - -You may not upload outside this path. - ---- - -## Example: Upload using the AWS CLI - -```bash -aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv --endpoint-url https://data.source.coop -``` - -Or upload a full directory: - -```bash -aws s3 sync ./data s3://your-org/your-product/ --endpoint-url https://data.source.coop -``` - -`--endpoint-url` isn't needed if you set `AWS_ENDPOINT_URL` or use the `source-coop` profile above. - ---- - -## Option 3: Long‑standing or automated access (ADVANCED) - -If you need: - -- Continuous uploads -- Scheduled pipelines -- Integration from your own AWS account - -You can use your own IAM role to write to the Source Cooperative bucket. - -:::info This option really is AWS - -Unlike Options 1 and 2, this option bypasses the data proxy. You write -directly to Source Cooperative's Amazon S3 bucket (e.g. `us-west-2.opendata.source.coop`, `eu-west-1.opendata.source.coop`) -using an identity in **your own AWS account**. It only applies to products stored -in that bucket. If your product is stored with another provider, or you don't have -an AWS account, use [Option 2](#option-2-upload-through-the-data-proxy-with-temporary-credentials-recommended-for-larger-uploads) -or contact us. - -::: - -### How this works - -- You create an IAM role (or use an existing IAM user/account) in **your own** AWS account -- You send us its ARN, and we grant it write access to your account's prefix in the Source Cooperative bucket - -No credentials are shared, and no role chaining is required. - ---- - -### Step 1: Create an IAM role (or pick an identity to use) - -Which identity you send us depends on what is doing the uploading: - -| Uploading from | Send us | -| --- | --- | -| A service (ingestion pipeline, ECS task, Lambda, EC2, GitHub Actions with OIDC) | An **IAM role** ARN | -| A person running the AWS CLI locally | An **IAM user** ARN | -| Many identities in one account | The **AWS account** ARN (we trust the whole account; your account controls who may use it) | - -For a service, create a role in your account with a trust policy for whatever assumes it, then attach a policy granting it read/write access under your account prefix. Replace `your-org` with your Source Cooperative account ID. - -:::tip - -The [IAM policy wizard](/tools/iam-policy-wizard) generates this policy for you — enter your account ID and, optionally, a product ID. - -::: - -
-Example trust policy (ECS tasks) - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { "Service": "ecs-tasks.amazonaws.com" }, - "Action": "sts:AssumeRole" - } - ] -} -``` - -
- -
-Example access policy - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "s3:PutObject", - "s3:GetObject", - "s3:DeleteObject", - "s3:AbortMultipartUpload", - "s3:ListMultipartUploadParts" - ], - "Resource": "arn:aws:s3:::us-west-2.opendata.source.coop/your-org/*" - }, - { - "Effect": "Allow", - "Action": "s3:ListBucket", - "Resource": "arn:aws:s3:::us-west-2.opendata.source.coop", - "Condition": { - "StringLike": { "s3:prefix": "your-org/*" } - } - } - ] -} -``` - -`s3:AbortMultipartUpload` and `s3:ListMultipartUploadParts` cover the multipart -uploads the AWS CLI and SDKs use automatically for large files. `s3:PutObject` -alone authorizes starting an upload and sending its parts, but without those two -a failed or resumed transfer cannot clean up after itself. Both are object-level -actions, so the prefix in `Resource` scopes them like the rest. - -:::note - -This policy deliberately omits `s3:ListBucketMultipartUploads`, which lists -in-progress uploads across the *whole* bucket. AWS does not support the -`s3:prefix` condition key on it, so it cannot be limited to your data, and no -upload path needs it — incomplete uploads are cleaned up automatically after 7 -days. Only `aws s3api list-multipart-uploads` requires it; contact us if you -have a workflow that requires this policy. - -::: - -
- -
-Creating the role with the AWS CLI - -```bash -aws iam create-role \ - --role-name source-coop-upload \ - --assume-role-policy-document file://trust-policy.json - -aws iam put-role-policy \ - --role-name source-coop-upload \ - --policy-name source-coop-write \ - --policy-document file://s3-policy.json -``` - -
- -:::note - -This policy only grants permission on *your* side. Uploads will still fail with `AccessDenied` until we grant the same role access on the bucket side (Step 2). - -::: - ---- - -### Step 2: Send us the ARN - -Email [hello@source.coop](mailto:hello@source.coop) with: - -- The ARN of the role, user, or account you want us to trust, for example: - - Role: `arn:aws:iam::123456789012:role/source-coop-upload` - - User: `arn:aws:iam::123456789012:user/data-uploader` - - Account: `arn:aws:iam::123456789012:root` -- Your Source Cooperative account ID and product ID (the `your-org/your-product` prefix you will write to) -- A short description of the workflow (e.g. "nightly ingestion pipeline running on ECS") - -We will add the ARN to the bucket policy and confirm when it is active. - ---- - -### Step 3: Upload using that identity - -Once we confirm, upload with credentials for that role, user, or account: - -```bash -aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv -``` - -Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles) need no assume-role step — the SDK picks up the role automatically. - -
-Uploading a directory - -```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ -``` - -
- -
-Assuming the role from a workstation or CI job - -Let the AWS CLI do the assume-role for you — add a profile to `~/.aws/config`: - -```ini -[profile source-coop-upload] -role_arn = arn:aws:iam::123456789012:role/source-coop-upload -source_profile = default -region = us-west-2 -``` - -```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ \ - --profile source-coop-upload -``` - -
- -
-Uploading with boto3 - -```python -import boto3 - -s3 = boto3.client("s3") -s3.upload_file( - "mydata.csv", - "us-west-2.opendata.source.coop", - "your-org/your-product/mydata.csv", -) -``` - -
- ---- - -## Why we use these controls - -These controls prevent accidental or unauthorized access and ensure: - -- Clear ownership of uploaded data -- No accidental access to other providers’ data -- Secure handling without sharing permanent credentials - -This protects both you and Source Cooperative. - ---- - -## What not to do - -Please do not: - -- Request permanent access keys -- Ask for full bucket access -- Upload outside your assigned prefix -- Reuse expired temporary credentials - ---- - -## Need help? - -If you’re unsure which option is right for you, or need help setting up automated access, contact the Source Cooperative team at `hello@source.coop`, and we will help you choose the best approach for your use case. - ---- - -Thank you for contributing data to Source Cooperative. diff --git a/docs/using-source/upload/direct-uploads.md b/docs/using-source/upload/direct-uploads.md new file mode 100644 index 0000000..3f6d95f --- /dev/null +++ b/docs/using-source/upload/direct-uploads.md @@ -0,0 +1,205 @@ +--- +title: Direct Uploads with Your Own AWS Identity +sidebar_label: Direct Uploads (Deprecated) +id: direct-uploads +slug: /direct-uploads +--- + +:::warning Deprecated + +This method is deprecated. New pipelines should use a +[service account](/automated-access) instead, which works for every product, +needs no AWS account, and doesn't need us to change a bucket policy. These +instructions stay here for pipelines already set up this way; contact +[hello@source.coop](mailto:hello@source.coop) for help moving one to a service +account. + +::: + +You can use an IAM role in your own AWS account to write to the Source +Cooperative bucket. + +This bypasses the [data proxy](/data-proxy). You write directly to Source +Cooperative's Amazon S3 bucket (e.g. `us-west-2.opendata.source.coop`, +`eu-west-1.opendata.source.coop`) using an identity in **your own AWS +account**. It only applies to products stored in that bucket. + +## How this works + +- You create an IAM role (or use an existing IAM user/account) in **your own** AWS account +- You send us its ARN, and we grant it write access to your account's prefix in the Source Cooperative bucket + +No credentials are shared, and no role chaining is required. + +--- + +## Step 1: Create an IAM role (or pick an identity to use) + +Which identity you send us depends on what is doing the uploading: + +| Uploading from | Send us | +| --- | --- | +| A service (ingestion pipeline, ECS task, Lambda, EC2, GitHub Actions with OIDC) | An **IAM role** ARN | +| A person running the AWS CLI locally | An **IAM user** ARN | +| Many identities in one account | The **AWS account** ARN (we trust the whole account; your account controls who may use it) | + +For a service, create a role in your account with a trust policy for whatever assumes it, then attach a policy granting it read/write access under your account prefix. Replace `your-org` with your Source Cooperative account ID. + +:::tip + +The [IAM policy wizard](/tools/iam-policy-wizard) generates this policy for you — enter your account ID and, optionally, a product ID. + +::: + +
+Example trust policy (ECS tasks) + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { "Service": "ecs-tasks.amazonaws.com" }, + "Action": "sts:AssumeRole" + } + ] +} +``` + +
+ +
+Example access policy + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:DeleteObject", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts" + ], + "Resource": "arn:aws:s3:::us-west-2.opendata.source.coop/your-org/*" + }, + { + "Effect": "Allow", + "Action": "s3:ListBucket", + "Resource": "arn:aws:s3:::us-west-2.opendata.source.coop", + "Condition": { + "StringLike": { "s3:prefix": "your-org/*" } + } + } + ] +} +``` + +`s3:AbortMultipartUpload` and `s3:ListMultipartUploadParts` cover the multipart +uploads the AWS CLI and SDKs use automatically for large files. `s3:PutObject` +alone authorizes starting an upload and sending its parts, but without those two +a failed or resumed transfer cannot clean up after itself. Both are object-level +actions, so the prefix in `Resource` scopes them like the rest. + +:::note + +This policy deliberately omits `s3:ListBucketMultipartUploads`, which lists +in-progress uploads across the *whole* bucket. AWS does not support the +`s3:prefix` condition key on it, so it cannot be limited to your data, and no +upload path needs it — incomplete uploads are cleaned up automatically after 7 +days. Only `aws s3api list-multipart-uploads` requires it; contact us if you +have a workflow that requires this policy. + +::: + +
+ +
+Creating the role with the AWS CLI + +```bash +aws iam create-role --role-name source-coop-upload --assume-role-policy-document file://trust-policy.json +aws iam put-role-policy --role-name source-coop-upload --policy-name source-coop-write --policy-document file://s3-policy.json +``` + +
+ +:::note + +This policy only grants permission on *your* side. Uploads will still fail with `AccessDenied` until we grant the same role access on the bucket side (Step 2). + +::: + +--- + +## Step 2: Send us the ARN + +Email [hello@source.coop](mailto:hello@source.coop) with: + +- The ARN of the role, user, or account you want us to trust, for example: + - Role: `arn:aws:iam::123456789012:role/source-coop-upload` + - User: `arn:aws:iam::123456789012:user/data-uploader` + - Account: `arn:aws:iam::123456789012:root` +- Your Source Cooperative account ID and product ID (the `your-org/product-id` prefix you will write to) +- A short description of the workflow (e.g. "nightly ingestion pipeline running on ECS") + +We will add the ARN to the bucket policy and confirm when it is active. + +--- + +## Step 3: Upload using that identity + +Once we confirm, upload with credentials for that role, user, or account: + +```bash +aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/product-id/mydata.csv +``` + +Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles) need no assume-role step — the SDK picks up the role automatically. + +
+Uploading a directory + +```bash +aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/product-id/ +``` + +
+ +
+Assuming the role from a workstation or CI job + +Let the AWS CLI do the assume-role for you — add a profile to `~/.aws/config` (`%USERPROFILE%\.aws\config` on Windows): + +```ini +[profile source-coop-upload] +role_arn = arn:aws:iam::123456789012:role/source-coop-upload +source_profile = default +region = us-west-2 +``` + +```bash +aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/product-id/ --profile source-coop-upload +``` + +
+ +
+Uploading with boto3 + +```python +import boto3 + +s3 = boto3.client("s3") +s3.upload_file( + "mydata.csv", + "us-west-2.opendata.source.coop", + "your-org/product-id/mydata.csv", +) +``` + +
diff --git a/docs/using-source/upload/in-the-browser.md b/docs/using-source/upload/in-the-browser.md new file mode 100644 index 0000000..e0c92ad --- /dev/null +++ b/docs/using-source/upload/in-the-browser.md @@ -0,0 +1,26 @@ +--- +title: Upload in the Browser +sidebar_label: In the Browser +id: in-the-browser +slug: /upload-in-the-browser +--- + +If you prefer not to use command-line tools, you can upload files directly in the web interface. + +This option is ideal for: + +- Small or medium uploads +- One‑time deliveries +- Non‑technical users + +## How to upload via the UI + +1. Go to your product page in Source Cooperative (e.g. `https://source.coop/your-org/product-id`) +2. In the top-right corner of the Product Contents card, click on the lock icon to open the dropdown menu and enable edit mode by clicking on `Edit Mode`. +3. Either: + - Drag‑and‑drop files to the `Product Contents` card + - Use `Upload Files` or `Upload Directory` options in dropdown menu to upload files via operating system's file selector + +Your files will be uploaded automatically to your product. + +For larger uploads, or to script them, [upload with the Source CLI](/upload-with-the-cli) instead. diff --git a/docs/using-source/upload/index.md b/docs/using-source/upload/index.md new file mode 100644 index 0000000..3e698b6 --- /dev/null +++ b/docs/using-source/upload/index.md @@ -0,0 +1,68 @@ +--- +title: Upload Your Data +id: data-upload +slug: /data-upload +--- + +This guide explains how to deliver your data to Source Cooperative in a secure and simple way. +It is written for data providers and does not require an Amazon Web Services (AWS) account or deep cloud storage knowledge. + +If you do not see the option to upload (for example, Edit Mode or View Credentials on your product page under the lock icon), contact [hello@source.coop](mailto:hello@source.coop) to request upload access. + +## Choose how to upload + + + + + +| | Best for | | +| --- | --- | --- | +| **[In the browser](/upload-in-the-browser)** | Small or one-time uploads, without installing anything | Easiest | +| **[With the Source CLI](/upload-with-the-cli)** | Large uploads and scripts that you run yourself | Recommended for larger uploads | +| **[With a service account](/automated-access)** | Pipelines, scheduled jobs and GitHub Actions: anything that runs with nobody at the keyboard | Advanced | + + +Already upload from your own AWS account with an IAM role? That method is +deprecated in favor of service accounts, but its instructions are still at +[Direct uploads](/direct-uploads). + +## How uploads work: the Source data proxy + +Uploads go through the **Source data proxy** at `https://data.source.coop`. +The proxy speaks the **S3 API**, the object storage protocol that Amazon S3 +introduced and that most storage tools now support. It is not itself an AWS service. + +Behind the proxy, a product's data may be stored with any of several object +storage providers: Amazon S3, Google Cloud Storage, Azure Blob Storage, +Cloudflare R2, and others. You don't need to know which one. You always +upload to the same proxy address, and the proxy writes to the right place. + +That is why these guides use AWS tools: + +- **The AWS CLI and AWS SDKs (boto3, etc.) are used only as S3 clients.** You + point them at `https://data.source.coop` instead of AWS. Any S3-compatible + client that lets you set a custom endpoint URL will work. +- **Your upload credentials come from Source Cooperative, not AWS.** They use + the AWS credential format (access key ID, secret access key, session token) + because S3 clients expect it, but they are only valid at the Source data proxy. +- **Addresses are proxy addresses.** `s3://account-id/product-id/` means the + `product-id` product in the `account-id` account on Source Cooperative. It + is not an Amazon S3 bucket. Where your data is physically stored doesn't change it. + +## Where to upload your data + +You may upload only to your product's prefix on the data proxy, for example: + +``` +s3://account-id/product-id/ +``` + +This matches your product page URL, `https://source.coop/account-id/product-id`. + +You may not upload outside this path. + +## Need help? + +If you’re unsure which option is right for you, or need help setting up automated access, contact the Source Cooperative team at `hello@source.coop`, and we will help you choose the best approach for your use case. + +Thank you for contributing data to Source Cooperative. diff --git a/docs/using-source/upload/with-service-account.md b/docs/using-source/upload/with-service-account.md new file mode 100644 index 0000000..188bde5 --- /dev/null +++ b/docs/using-source/upload/with-service-account.md @@ -0,0 +1,413 @@ +--- +title: Automated Access +sidebar_label: With a Service Account +id: with-service-account +slug: /automated-access +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Some software reaches Source Cooperative with nobody at the keyboard: a nightly +sync, a publishing pipeline, an instrument that uploads its readings. Such +software signs in as a **service account**, in one of two ways: + +- A [GitHub Actions workflow](#github-actions) uses a token GitHub issues for + each run, so there's no secret to store. +- Anything else uses one of the service account's [API keys](#api-keys). The AWS + CLI and SDKs trade the key at the [data proxy](/data-proxy) for credentials + that last an hour, and trade it again before those run out. You don't need + the Source CLI or any other helper on the machine. + +## What a service account is + +A service account is a login for software. It belongs to one account, yours or +an organization's. Only the owner of a personal account, or an owner or +maintainer of an organization, can create or edit its service accounts. Other +organization members can't. + +- **It has its own access.** You grant it products one at a time, to read or to + read and write. It can reach only products its owner owns, and it never + inherits your access or anyone else's. +- **There's no person behind it.** It has no profile, can't sign in to the + website, and can't create products or manage people. +- **You can stop it without touching a person.** Revoke its keys, or disable or + delete it, and nobody's own account changes. + +An organization can own service accounts, so a pipeline doesn't depend on the +account of whoever set it up. + +A service account's ID is its owner's ID, two hyphens, and a name of its own, +such as `your-org--nightly-sync`. It can't be changed. + +## Create a service account + +You need to be the account's owner or, for an organization, one of its owners or +maintainers. + +1. Open the profile page of the account that will own it, yours or an + organization's, click the gear icon, and choose **Service Accounts**. +2. Click **New service account**. +3. Under **Who it is**, enter a **Name**, such as `Nightly Sync`. The **Account + ID** is made from the name; click **Edit** beside it to choose another. +4. Under **How software signs in**, click **Add a GitHub workflow** to trust a + [GitHub Actions workflow](#github-actions), or **Add an API key** to issue + [a key](#issue-a-key) for anything else. Both are optional here: you can add + either later. +5. Under **What it can reach**, click **Grant a product**, choose a product and + **Read** or **Read and write**, and click **Grant**. Repeat for each + product the job needs, and no more. +6. Click **Create service account**. + +You land on the service account's page, showing the key if you added one. You +can change what it reaches at any time under **Can reach**; each change is saved +as you make it. + +## GitHub Actions + +A service account can trust a GitHub Actions workflow, which then signs in with +a token GitHub issues for each run. There is no key to store, rotate or leak. + +### Trust a workflow + +1. On the service account's page, under **Signs in with**, click **Add + sign-in** and choose **GitHub workflow**. +2. Enter the **Repository**, such as `your-org/pipelines`. Repositories created + after July 2026 carry their owner's and their own numeric IDs, as in + `your-org@123/pipelines@456`. The dialog offers that form when GitHub shows the + repository publicly; for a private one, it prints a `gh` command that finds it. +3. Under **Allow runs from**, choose one of the following: + - **Branch**, and provide the repository branch name that will trigger the the workflows run. + - **Tag**, and provide the tag name that will trigger the workflow. + - **Environment**, and provide the name of the GitHub environment the job runs in. +4. Click **Trust it**. + +The dialog shows the exact subject it trusts, such as +`repo:your-org/pipelines:ref:refs/heads/main`. + +A repository is trusted only for the ref or environment you name. To trust +another branch, add another workflow. Wildcards (`*`) are not supported. + +### Sign in from the job + +Choose **Example usage** from the workflow's menu on the service account's page +for the step to add, filled in for your service account. It looks like this: + +```yaml +jobs: + publish: + runs-on: ubuntu-latest + permissions: + id-token: write # lets the job ask GitHub for its token + contents: read + env: + AWS_ENDPOINT_URL_S3: https://data.source.coop + steps: + - name: Sign in to Source Cooperative as your-org--nightly-sync + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: arn:aws:iam::your-org--nightly-sync:role/FullAccess + audience: https://data.source.coop + sts-endpoint: https://data.source.coop/.sts + aws-region: us-west-2 + - run: aws s3 sync ./outgoing s3://your-org/product-id/outgoing/ +``` + +Unlike with an API key, the ID in `role-to-assume` matters: it names the service +account the job signs in as, and the sign-in succeeds only if that service +account trusts the workflow. End it in `role/ReadOnly` for a job that only +reads. + +The step's credentials last an hour, and nothing renews them during the job. For +a longer job, add `role-duration-seconds` to the step, up to `43200` (12 hours). + +A refused sign-in fails the step with: + +```text +AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity (request id 8f3a1c2b9d4e5f60-SEA) +``` + +The same error covers a workflow the service account doesn't trust (check the +subject, as above), a disabled service account, and an ID in `role-to-assume` +that isn't a service account. If none of those explains it, email +[hello@source.coop](mailto:hello@source.coop) and quote the request id. + +## API keys + +An API key lets software on your own server, VM, cluster or instrument sign in +as the service account. + +### Issue a key + +1. On the service account's page, under **Signs in with**, click **Add + sign-in** and choose **API key**. +2. Enter a **Label** that says where the key will live, such as `HPC cron job`, + so you know which key to revoke later. +3. Choose when it **Expires**: in 30 days, 90 days or a year, or never (until + you revoke it). +4. Click **Issue key**. + +The key is shown once. Copy it now: Source stores only a hash of it, so nobody +can show it to you again. If you lose it, issue another and revoke the lost one. +A key starts with `sck_`. + +You can't issue a key to a disabled service account. + +### Set up the machine + +Save the key in a file that only the job's user can read. A trailing newline is +fine. + + + + +```bash +mkdir -p ~/.source-coop && chmod 700 ~/.source-coop +(umask 077; cat > ~/.source-coop/nightly-sync.key) # paste the key, press Enter, then Ctrl-D +``` + + + + +```powershell +$dir = "$HOME\.source-coop" +New-Item -ItemType Directory -Force $dir | Out-Null +icacls $dir /inheritance:r /grant:r "${env:USERNAME}:(OI)(CI)F" | Out-Null +Read-Host "Paste the key" | Set-Content -NoNewline "$dir\nightly-sync.key" +``` + + + + +Then set five environment variables wherever the job runs. The key's row on the +service account's page lists them too, under **Example usage** in its menu, +filled in for your service account apart from the key file's path: + + + + +```bash +export AWS_REGION=us-west-2 +export AWS_ENDPOINT_URL_S3=https://data.source.coop +export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts +export AWS_ROLE_ARN=arn:aws:iam::your-org--nightly-sync:role/FullAccess +export AWS_WEB_IDENTITY_TOKEN_FILE=/home/your-user/.source-coop/nightly-sync.key +``` + + + + +```powershell +# Saved for your user, so scheduled tasks see them too. Open a new terminal afterwards. +$vars = @{ + AWS_REGION = "us-west-2" + AWS_ENDPOINT_URL_S3 = "https://data.source.coop" + AWS_ENDPOINT_URL_STS = "https://data.source.coop/.sts" + AWS_ROLE_ARN = "arn:aws:iam::your-org--nightly-sync:role/FullAccess" + AWS_WEB_IDENTITY_TOKEN_FILE = "C:\Users\your-user\.source-coop\nightly-sync.key" +} +$vars.GetEnumerator() | ForEach-Object { + [Environment]::SetEnvironmentVariable($_.Key, $_.Value, "User") +} +``` + + + + +| Variable | What it's for | +| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `AWS_REGION` | Unused by the data proxy, which accepts any region, but set it anyway: many SDKs won't sign requests without one. It doesn't say where your data is stored. | +| `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | +| `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | +| `AWS_ROLE_ARN` | References the service account ID and how much the credentials may do. | +| `AWS_WEB_IDENTITY_TOKEN_FILE` | The file that holds the key, as an absolute path. Write it out in full: cron, systemd and Windows Task Scheduler don't expand `$HOME` or `~`. | + +For a job that only reads, end `AWS_ROLE_ARN` in `role/ReadOnly` instead of +`role/FullAccess`. Its credentials can't write, even to products the service +account may write to. Any other role name is refused. + +That's all. The AWS CLI and SDKs read the key from the file, trade it at the +data proxy for credentials that last an hour, and trade it again before those +run out. The key file never changes, and no other program has to run alongside +the job. + +The two endpoint variables need **AWS CLI 2.13** or later, or **boto3 1.28** +(**botocore 1.31**) or later. Older releases ignore them and send the key to AWS +instead, which refuses it. + +With the AWS CLI: + +```bash +aws --version # aws-cli/2.13.0 or later +aws s3 ls s3://your-org/product-id/ +aws s3 sync ./outgoing s3://your-org/product-id/outgoing/ +``` + +With boto3: + +```python +import boto3 # boto3 1.28 (botocore 1.31) or later + +# No keys or endpoint here: boto3 reads them from the variables above. +s3 = boto3.client("s3") +s3.upload_file("mydata.csv", "account-id", "product-id/mydata.csv") +``` + +boto3 is the exception among the SDKs: it reads its region from +`AWS_DEFAULT_REGION`, not `AWS_REGION`. Set that as well +(`export AWS_DEFAULT_REGION=us-west-2`, or in PowerShell +`$env:AWS_DEFAULT_REGION = "us-west-2"`), or pass `region_name` to +`boto3.client`. + +Other AWS SDKs work the same way, provided their web identity credential +provider reads `AWS_ENDPOINT_URL_STS`. + +A cron job or a system service doesn't read your shell's profile, so set the +variables in its own environment. On Windows, a scheduled task that runs as +your user picks up the variables saved above. + +### If the key is refused + +A key that has been revoked or has expired, and a key whose service account is +disabled, are refused with the same error: + +``` +An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: API key was not accepted (request id 8f3a1c2b9d4e5f60-SEA) +``` + +The service account's page shows whether it is disabled, and each key's row +shows whether the key has been revoked or has expired. + +A key's last six characters are a checksum of the rest, so a key that was cut +short or mistyped when it was copied is refused before anything is looked up, +with an error of its own: + +```text +An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: API key is malformed; check that it was copied whole (request id 8f3a1c2b9d4e5f60-SEA) +``` + +Copy the key again from where you saved it. If you no longer have all of it, +issue a new key and revoke the old one. + +### Keep the key secret + +- Treat the key as a password for the service account. Keep it out of source + control, scripts and command lines, and give it to the job as a file. +- `aws --debug` prints the key, because it logs the request that carries it. + Don't share debug output from a machine that has a key set up. +- A key goes in a file or in the body of a request, never in a URL, because URLs + end up in logs. The data proxy refuses a key sent in a URL, with + `API key must be sent in the request body, not the URL`. If that ever happens, + replace the key. + +### Rotate a key, or change when it expires + +A service account can have several keys at once, so you can replace one without +stopping the job: issue a new key, deploy it, and revoke the old one once +nothing uses it. Each key's row shows when it was last used. + +To change when a key expires, choose **Change expiry** from its row's menu: later, for a +job that runs longer than planned, or sooner, during an incident. The new expiry +counts from today. + +## Tools that keep their first credentials + +The AWS CLI and SDKs renew credentials on their own. Tools that manage +credentials themselves often don't: given a fixed set of credentials +(`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN`, or the +same values in their own settings), they read it once and use it until they +exit. GDAL's `/vsis3/`, DuckDB and rclone are common examples; DuckDB reads a +secret's credentials when you run `CREATE SECRET`. + +Credentials from the data proxy last an hour, so with these tools a transfer +that runs longer than that fails partway through with an `ExpiredToken` (HTTP +403) error, although the key is still good. To avoid it: + +- Split long transfers into runs that each finish within the hour, and give + each run fresh credentials: start a new process, or in DuckDB, replace the + secret (`CREATE OR REPLACE SECRET`) before each batch. +- Do large copies with the AWS CLI (`aws s3 cp`, `aws s3 sync`) or an AWS SDK, + which renew credentials mid-transfer. +- Where a tool can run a credential helper each time its credentials run out, + use one. GDAL can; see the next section. + +## GDAL + +GDAL's `/vsis3/` can't use an API key directly. Given `AWS_ROLE_ARN` and +`AWS_WEB_IDENTITY_TOKEN_FILE` and no other credentials, GDAL 3.6 and later do +the exchange themselves, in a way that can't work for a key: + +- They don't read `AWS_ENDPOINT_URL_STS`, so they send the key to AWS, not to + the data proxy. +- They put the key in the URL of the request. The data proxy refuses a key in a + URL, and a key sent to AWS in a URL should be treated as exposed. + +So where GDAL runs with the five variables set, also set +`CPL_AWS_WEB_IDENTITY_ENABLE=NO` to stop it trying. If GDAL has already run with +them, replace the key. + +:::info Coming soon + +The Source CLI will be able to exchange an API key without a browser +([source-coop-cli#17](https://github.com/source-cooperative/source-coop-cli/issues/17)). +GDAL 3.12 and later can then get credentials from it as a `credential_process`, +and renew them the same way. This page will show the setup once that release is +out. + +::: + +## Revoke a key, or stop a service account + +| To | Do this | What happens | +| ---------------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Stop one key | Choose **Revoke** from the key's menu. | New exchanges with the key are refused within about a minute. | +| Stop one workflow | Choose **Remove** from the workflow's menu. | New sign-ins from the workflow are refused within about a minute. | +| Stop everything the service account does | Click **Disable** under **Danger zone**. | New exchanges with any of its keys, and new sign-ins from its workflows, are refused within about a minute. Writes stop within about a minute, and reads of restricted products within about five minutes. | + +Revoking a key doesn't recall credentials already issued with it: they keep +working until they expire, an hour after they were issued, or up to 12 hours if +the client asked for longer. Disabling the service account cuts those off too, +which makes it the emergency stop for a leaked key. Public products stay +readable by anyone, as always. + +Disabling keeps the service account's keys and grants. Enabling it again makes +every key that hasn't been revoked or expired work again, so revoke a leaked key +before you enable the account. + +### Revoke a key you found + +Anyone who holds a key can revoke it, without an account. If you come across one +that has leaked, in a repository, a log or a message, send it in the body of +this request. The commands below take the key as you paste it, so it stays out of your +shell history and process list: + + + + +```bash +cat > found.key # paste the key, press Enter, then Ctrl-D +printf '{"key":"%s"}' "$(tr -d '\n' < found.key)" | + curl -X POST https://source.coop/api/v1/service-account-keys/revocations \ + -H 'content-type: application/json' \ + --data-binary @- +rm found.key +``` + + + + +```powershell +$key = (Read-Host "Paste the key").Trim() +Invoke-RestMethod -Method Post -Uri https://source.coop/api/v1/service-account-keys/revocations ` + -ContentType "application/json" -Body (@{ key = $key } | ConvertTo-Json) +``` + + + + +For a well-formed key the answer is always `204 No Content`, whether the key was +live, already revoked or unknown. A live key is revoked, and new exchanges with +it stop within about a minute. Send the key in the body, never in the URL. + +We plan to revoke keys pushed to public GitHub repositories automatically, +through GitHub's secret scanning. diff --git a/docs/using-source/upload/with-the-cli.md b/docs/using-source/upload/with-the-cli.md new file mode 100644 index 0000000..de685d9 --- /dev/null +++ b/docs/using-source/upload/with-the-cli.md @@ -0,0 +1,174 @@ +--- +title: Upload with the Source CLI +sidebar_label: With the Source CLI +id: with-the-cli +slug: /upload-with-the-cli +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +For larger uploads, scripting, or programmatic access, use temporary credentials +issued by Source Cooperative with an S3 client pointed at the +[data proxy](/data-upload#how-uploads-work-the-source-data-proxy). + +## Get credentials with the Source CLI (recommended) + +The [Source CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you with Source Cooperative and provides temporary credentials automatically, so you don't have to copy expiring credentials out of the UI. Once configured as an AWS CLI profile, the AWS CLI and SDKs refresh credentials for you, and you'll only need to re-authenticate every 30 days. + +**1. Install the CLI** + +Follow the install instructions in the [CLI README](https://github.com/source-cooperative/source-coop-cli). + +**2. Log in** + +```bash +source-coop login +``` + +This opens your browser to authenticate and caches short‑lived credentials in your OS keyring. + +**3. Configure an AWS CLI profile** + +Add a profile to `~/.aws/config` (`%USERPROFILE%\.aws\config` on Windows) that calls the Source CLI to fetch credentials on demand and sends requests to the data proxy: + +```ini +[profile source-coop] +credential_process = source-coop creds +endpoint_url = https://data.source.coop +``` + +The `endpoint_url` line is what sends requests to the Source data proxy instead of AWS. + +**4. Use standard S3 commands with the profile** + +Pass the profile per command with `--profile`: + +```bash +aws s3 cp mydata.csv s3://your-org/product-id/mydata.csv --profile source-coop +aws s3 sync ./data s3://your-org/product-id/ --profile source-coop +``` + +Or set it once for your shell session with the `AWS_PROFILE` environment variable: + + + + +```bash +export AWS_PROFILE=source-coop +aws s3 cp mydata.csv s3://your-org/product-id/mydata.csv +aws s3 sync ./data s3://your-org/product-id/ +``` + + + + +```powershell +$env:AWS_PROFILE = "source-coop" +aws s3 cp mydata.csv s3://your-org/product-id/mydata.csv +aws s3 sync ./data s3://your-org/product-id/ +``` + + + + +:::tip + +`AWS_PROFILE` is also picked up automatically by the AWS SDKs (boto3, JavaScript, Go, etc.), so your scripts can use the default credential chain instead of hardcoding access keys or endpoint details. + +::: + +The AWS CLI will refresh credentials automatically; re-run `source-coop login` when your session expires. Your upload permissions are scoped to the products you have access to. + +## Without the CLI: get credentials from the UI + +You can also copy the same kind of temporary credentials out of the web +interface. They expire, and you'll need to copy new ones when they do. + +1. Go to your product page +2. In the `Product Contents` card, open the dropdown (click on lock icon in top‑right corner) +3. Select `View Credentials` +4. Choose: + - `JSON (SDK)` credentials, or + - `Environment Variables` (shell) + +You will also see: + +- `Expiration`: the expiration time of the credentials (a specific date and time) +- `Bucket`: the bucket name on the data proxy, which is your account ID (`your-org`) +- `Prefix`: the prefix (folder) you are allowed to write to (e.g. `product-id/`) + +### What the credentials look like + +These are Source Cooperative credentials in the AWS credential format. They +only work against the data proxy, so always set the endpoint to +`https://data.source.coop`. The data proxy doesn't use the region and accepts +any value, but set one anyway: many SDKs won't sign requests without it. It +doesn't say where your data is stored. + +For SDK clients (boto3, AWS SDKs, etc.): + +```json +{ + "aws_access_key_id": "ASIA...", + "aws_secret_access_key": "pwEV...", + "aws_session_token": "IQoJ...", + "region_name": "us-west-2" +} +``` + +With boto3, pass the endpoint when creating the client: + +```python +import boto3 + +s3 = boto3.client("s3", endpoint_url="https://data.source.coop", **credentials) +s3.upload_file("mydata.csv", "account-id", "product-id/mydata.csv") +``` + +For terminal / shell usage: + + + + +```bash +export AWS_ACCESS_KEY_ID="ASIA..." +export AWS_SECRET_ACCESS_KEY="pwEV..." +export AWS_SESSION_TOKEN="IQoJ..." +export AWS_DEFAULT_REGION="us-west-2" +export AWS_ENDPOINT_URL="https://data.source.coop" # add this: send requests to the data proxy +``` + + + + +```powershell +$env:AWS_ACCESS_KEY_ID = "ASIA..." +$env:AWS_SECRET_ACCESS_KEY = "pwEV..." +$env:AWS_SESSION_TOKEN = "IQoJ..." +$env:AWS_DEFAULT_REGION = "us-west-2" +$env:AWS_ENDPOINT_URL = "https://data.source.coop" # add this: send requests to the data proxy +``` + + + + +These credentials are: + +- Temporary +- Scoped to your product +- Automatically revoked when they expire + +### Example: Upload using the AWS CLI + +```bash +aws s3 cp mydata.csv s3://your-org/product-id/mydata.csv --endpoint-url https://data.source.coop +``` + +Or upload a full directory: + +```bash +aws s3 sync ./data s3://your-org/product-id/ --endpoint-url https://data.source.coop +``` + +`--endpoint-url` isn't needed if you set `AWS_ENDPOINT_URL` or use the `source-coop` profile above. diff --git a/sidebars.ts b/sidebars.ts index 9a34c5a..2e1cf70 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -30,7 +30,17 @@ const sidebars: SidebarsConfig = { items: [ 'using-source/create-an-account', 'using-source/create-a-data-product', - 'using-source/data-upload', + { + type: 'category', + label: 'Upload Your Data', + link: {type: 'doc', id: 'using-source/upload/data-upload'}, + items: [ + 'using-source/upload/in-the-browser', + 'using-source/upload/with-the-cli', + 'using-source/upload/with-service-account', + 'using-source/upload/direct-uploads', + ], + }, 'using-source/bring-your-own-bucket', 'using-source/data-proxy', ], diff --git a/src/css/custom.css b/src/css/custom.css index c54a6fa..47c7ccb 100644 --- a/src/css/custom.css +++ b/src/css/custom.css @@ -102,3 +102,9 @@ height: auto; } + +/* wrap plain-text blocks (error messages, output); commands keep scrolling */ +pre.language-text code { + white-space: pre-wrap; + overflow-wrap: anywhere; +} diff --git a/src/pages/tools/iam-policy-wizard.tsx b/src/pages/tools/iam-policy-wizard.tsx index 544ca34..d649d52 100644 --- a/src/pages/tools/iam-policy-wizard.tsx +++ b/src/pages/tools/iam-policy-wizard.tsx @@ -23,23 +23,34 @@ export default function IamPolicyWizard(): ReactNode { return ( + description="Generate the IAM policy your AWS role or user needs to upload to Source Cooperative." + >

IAM policy wizard

+
+ Deprecated. Uploading with an IAM role in your own + AWS account is deprecated in favor of{" "} + service accounts. Use this wizard only + for pipelines already set up this way. +

- Generate the policy to attach to the IAM role or user in your own AWS - account, so it can upload to your prefix in the Source Cooperative bucket. See{' '} - Upload Your Data for the full walkthrough. + Generate the policy to attach to the IAM role or user in{" "} + your own AWS account, so it can upload to your prefix + in the Source Cooperative bucket. See{" "} + Direct uploads for the full walkthrough.

-
+