From e6d9afcad93a90a265f0e15273cf130ba23636bc Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Fri, 25 Sep 2026 14:11:48 -0700 Subject: [PATCH 01/29] docs: add an automated access guide for service accounts Adds Automated Access (docs/using-source/automated-access.md), the guide for software that reaches Source Cooperative with nobody at the keyboard: what a service account is; creating one and granting it products; issuing an API key and the five variables the AWS CLI or an SDK needs to exchange it at the data proxy; the refusal users see and the request id to quote; keeping the key out of URLs and debug output; rotation and editable expiry; tools that keep their first credentials; GDAL; revocation, the emergency stop and revoking a found key; and GitHub Actions as coming soon. The page is added to sidebars.ts. Upload Your Data's Option 3 walked through trusting an IAM role in the uploader's own AWS account to write straight to Source's bucket. Service accounts supersede it, so the section is now a pointer to the new page, as are the two references to it earlier on that page and the "contact us" line for automated access. Access Data gains one sentence pointing unattended software at service accounts. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01R1eiTse4416N6uTgAy4Ddd --- docs/using-source/automated-access.md | 265 ++++++++++++++++++++++++++ docs/using-source/data-proxy.md | 2 + docs/using-source/data-upload.md | 219 +-------------------- sidebars.ts | 1 + 4 files changed, 275 insertions(+), 212 deletions(-) create mode 100644 docs/using-source/automated-access.md diff --git a/docs/using-source/automated-access.md b/docs/using-source/automated-access.md new file mode 100644 index 0000000..d8d0fe2 --- /dev/null +++ b/docs/using-source/automated-access.md @@ -0,0 +1,265 @@ +--- +title: Automated Access +id: automated-access +slug: /automated-access +--- + +Some software reaches Source Cooperative with nobody at the keyboard: a nightly +sync, a publishing pipeline, an instrument that uploads its readings. Give it a +**service account** and an **API key**. The AWS CLI and SDKs exchange the key +for short-lived credentials at the [data proxy](/data-proxy) and renew them on +their own, so nothing else has to run on the machine. + +## What a service account is + +A service account is a login for software. It belongs to one account, yours or +an organization's, and whoever manages that account manages it. + +- **It has its own access.** You grant it products one at a time, to read or to + read and write. It can reach only products its owner owns, and it never + inherits your access or anyone else's. +- **There's no person behind it.** It has no profile, can't sign in to the + website, and can't create products or manage people. +- **You can stop it without touching a person.** Revoke its keys, or disable or + delete it, and nobody's own account changes. + +An organization can own service accounts, so a pipeline doesn't depend on the +account of whoever set it up. + +A service account's ID is its owner's ID, two hyphens, and a name of its own, +such as `your-org--nightly-sync`. It can't be changed. + +## Create a service account + +1. Open the profile page of the account that will own it, yours or an + organization's, click the gear icon, and choose **Service Accounts**. +2. Click **New service account**. +3. Under **Who it is**, enter a **Name**, such as `Nightly Sync`. The **Account + ID** is made from the name; click **Edit** beside it to choose another. +4. Leave **How software signs in** empty. It is for GitHub Actions, which + [isn't available yet](#github-actions). +5. Under **What it can reach**, click **Grant a product**, choose a product and + **Read** or **Read and write**, and click the check mark. Repeat for each + product the job needs, and no more: a job that only downloads needs **Read**. +6. Click **Create service account**. + +You land on the service account's page. You can change what it reaches at any +time under **Can reach**; each change is saved as you make it. + +## API keys + +An API key lets software on your own server, VM, cluster or instrument sign in +as the service account. + +### Issue a key + +1. On the service account's page, under **API keys**, click **Issue an API + key**. +2. Enter a **Label** that says where the key will live, such as `HPC cron job`, + so you know which key to revoke later. +3. Choose when it **Expires**: in 30 days, 90 days or a year, or never (until + you revoke it). +4. Click **Issue key**. + +The key is shown once. Copy it now: Source stores only a hash of it, so nobody +can show it to you again. If you lose it, issue another and revoke the lost one. +A key starts with `sck_`. + +You can't issue a key to a disabled service account. + +### Set up the machine + +Save the key in a file that only the job's user can read. A trailing newline is +fine. + +```bash +mkdir -p -m 700 ~/.source-coop +cat > ~/.source-coop/nightly-sync.key # paste the key, press Enter, then Ctrl-D +chmod 600 ~/.source-coop/nightly-sync.key +``` + +Then set five environment variables wherever the job runs. The dialog that +showed you the key lists them too, filled in for your service account apart from +the key file's path: + +```bash +export AWS_ROLE_ARN=arn:aws:iam::your-org--nightly-sync:role/FullAccess +export AWS_WEB_IDENTITY_TOKEN_FILE=$HOME/.source-coop/nightly-sync.key +export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts +export AWS_ENDPOINT_URL_S3=https://data.source.coop +export AWS_REGION=us-west-2 +``` + +| Variable | What it's for | +| --- | --- | +| `AWS_ROLE_ARN` | How much the credentials may do. `FullAccess` is everything the service account may do; `ReadOnly` is reads only. The value has the shape AWS tools expect, with the service account's ID where an AWS account number would be. | +| `AWS_WEB_IDENTITY_TOKEN_FILE` | The file that holds the key, as an absolute path. | +| `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | +| `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | +| `AWS_REGION` | Required by S3 clients. It doesn't say where your data is stored. | + +That's all. The AWS CLI and SDKs read the key from the file, exchange it at the +data proxy for credentials that last an hour, and exchange it again before those +run out. Nothing else runs on the machine, and the key file never changes. + +The two endpoint variables need **AWS CLI 2.13** or later, or **boto3 1.28** +(**botocore 1.31**) or later. Older releases ignore them and send the key to AWS +instead, which refuses it. + +With the AWS CLI: + +```bash +aws --version # aws-cli/2.13.0 or later +aws s3 ls s3://your-org/your-product/ +aws s3 sync ./outgoing s3://your-org/your-product/outgoing/ +``` + +With boto3: + +```python +import boto3 # boto3 1.28 (botocore 1.31) or later + +# No keys, endpoint or region here: boto3 reads the five variables. +s3 = boto3.client("s3") +s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") +``` + +Other AWS SDKs work the same way, provided their web identity credential +provider reads `AWS_ENDPOINT_URL_STS`. + +A cron job or a system service doesn't read your shell's profile, so set the +variables in its own environment. + +### If the key is refused + +A key that has been revoked or has expired, a key whose service account is +disabled, and a value that isn't a Source Cooperative key at all are all refused +with the same error: + +```text +An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: API key was not accepted (request id 8f3a1c2b9d4e5f60-SEA) +``` + +The service account's page shows whether it is disabled, and each key's row +shows whether the key has been revoked or has expired. If none of those explains +it, email [hello@source.coop](mailto:hello@source.coop) and quote the request +id: it lets us find the reason in our logs. + +### Keep the key secret + +- Treat the key as a password for the service account. Keep it out of source + control, scripts and command lines, and give it to the job as a file. +- `aws --debug` prints the key, because it logs the request that carries it. + Don't share debug output from a machine that has a key set up. +- A key goes in a file or in the body of a request, never in a URL, because URLs + end up in logs. The data proxy refuses a key sent in a URL, with + `API key must be sent in the request body, not the URL`. If that ever happens, + replace the key. + +### Rotate a key, or change when it expires + +A service account can have several keys at once, so you can replace one without +stopping the job: issue a new key, deploy it, and revoke the old one once +nothing uses it. Each key's row shows when it was last used. + +To change when a key expires, click **Change expiry** on its row: later, for a +job that runs longer than planned, or sooner, during an incident. The new expiry +counts from today. + +## Tools that keep their first credentials + +The AWS CLI and SDKs renew credentials on their own. Tools that manage +credentials themselves often don't: given a fixed set of credentials +(`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN`, or the +same values in their own settings), they read it once and use it until they +exit. GDAL's `/vsis3/`, DuckDB and rclone are common examples; DuckDB reads a +secret's credentials when you run `CREATE SECRET`. + +Credentials from the data proxy last an hour, so with these tools a transfer +that runs longer than that fails partway through with an `ExpiredToken` (HTTP +403) error, although the key is still good. To avoid it: + +- Split long transfers into runs that each finish within the hour, and give + each run fresh credentials: start a new process, or in DuckDB, replace the + secret (`CREATE OR REPLACE SECRET`) before each batch. +- Do large copies with the AWS CLI (`aws s3 cp`, `aws s3 sync`) or an AWS SDK, + which renew credentials mid-transfer. +- Where a tool can run a credential helper each time its credentials run out, + use one. GDAL can; see the next section. + +## GDAL + +GDAL's `/vsis3/` can't use an API key directly. Given `AWS_ROLE_ARN` and +`AWS_WEB_IDENTITY_TOKEN_FILE` and no other credentials, GDAL 3.6 and later do +the exchange themselves, in a way that can't work for a key: + +- They don't read `AWS_ENDPOINT_URL_STS`, so they send the key to AWS, not to + the data proxy. +- They put the key in the URL of the request. The data proxy refuses a key in a + URL, and a key sent to AWS in a URL should be treated as exposed. + +So where GDAL runs with the five variables set, also set +`CPL_AWS_WEB_IDENTITY_ENABLE=NO` to stop it trying. If GDAL has already run with +them, replace the key. + +:::info Coming soon + +The Source CLI will be able to exchange an API key without a browser +([source-coop-cli#17](https://github.com/source-cooperative/source-coop-cli/issues/17)). +GDAL 3.12 and later can then get credentials from it as a `credential_process`, +and renew them the same way. This page will show the setup once that release is +out. + +::: + +## Revoke a key, or stop a service account + +| To | Do this | What happens | +| --- | --- | --- | +| Stop one key | Click **Revoke** on the key's row. | New exchanges with the key are refused within about a minute. | +| Stop everything the service account does | Click **Disable** under **Danger zone**. | New exchanges with any of its keys are refused within about a minute. Writes stop within about a minute, and reads of restricted products within about five minutes. | + +Revoking a key doesn't recall credentials already issued with it: they keep +working until they expire, an hour after they were issued, or up to 12 hours if +the client asked for longer. Disabling the service account cuts those off too, +which makes it the emergency stop for a leaked key. Public products stay +readable by anyone, as always. + +Disabling keeps the service account's keys and grants. Enabling it again makes +every key that hasn't been revoked or expired work again, so revoke a leaked key +before you enable the account. + +### Revoke a key you found + +Anyone who holds a key can revoke it, without an account. If you come across one +that has leaked, in a repository, a log or a message, send it in the body of +this request: + +```bash +curl -X POST https://source.coop/api/v1/service-account-keys/revocations \ + -H 'content-type: application/json' \ + -d '{"key":"sck_…"}' +``` + +For a well-formed key the answer is always `204 No Content`, whether the key was +live, already revoked or unknown. A live key is revoked, and new exchanges with +it stop within about a minute. Send the key in the body, never in the URL. + +We plan to revoke keys pushed to public GitHub repositories automatically, +through GitHub's secret scanning. + +## GitHub Actions + +:::info Coming soon + +A service account can also trust a GitHub Actions workflow, which then signs in +with a token GitHub issues for each run, with no key to store. The service +account page already lets you add a workflow, but the data proxy doesn't accept +those sign-ins yet +([data.source.coop#222](https://github.com/source-cooperative/data.source.coop/issues/222), +[data.source.coop#223](https://github.com/source-cooperative/data.source.coop/issues/223)). +Until it does, a workflow can use an API key like any other machine: keep the +key in a GitHub Actions secret, write it to a file at the start of the job, and +set the variables above. + +::: diff --git a/docs/using-source/data-proxy.md b/docs/using-source/data-proxy.md index aef0a44..98348d9 100644 --- a/docs/using-source/data-proxy.md +++ b/docs/using-source/data-proxy.md @@ -58,6 +58,8 @@ aws s3 ls s3://your-org/your-product --profile source-coop See [Upload Your Data](/data-upload#get-credentials-with-the-source-cli-recommended) for the one-time install and profile setup. +For software that runs on its own, with nobody there to log in, use a [service account](/automated-access) instead. + ## Getting Started with AWS CLI If you don't have the AWS CLI installed, follow the [AWS CLI Getting Started Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html). diff --git a/docs/using-source/data-upload.md b/docs/using-source/data-upload.md index 119ac6a..2231195 100644 --- a/docs/using-source/data-upload.md +++ b/docs/using-source/data-upload.md @@ -14,8 +14,7 @@ If you do not see the option to upload (for example, Edit Mode or View Credentia ## How uploads work: the Source data proxy -Unless you set up [direct bucket access](#option-3-longstanding-or-automated-access-advanced), -every upload goes through the **Source data proxy** at `https://data.source.coop`. +Every upload goes through the **Source data proxy** at `https://data.source.coop`. The proxy speaks the **S3 API**, the object storage protocol that Amazon S3 introduced and that most storage tools now support. It is not itself an AWS service. @@ -36,10 +35,6 @@ That is why this guide uses AWS tools: `your-product` product in the `your-org` account on Source Cooperative. It is not an Amazon S3 bucket. Where your data is physically stored doesn't change it. -The one exception is [Option 3](#option-3-longstanding-or-automated-access-advanced), -which writes straight to Source's Amazon S3 bucket with your own AWS identity. -In that section, "AWS" really does mean AWS. - --- ## The short version (what you need to do) @@ -49,7 +44,7 @@ You have two main ways to upload data: 1. **Upload directly in the Source Cooperative User Interface (UI)** (drag-and-drop or file selector), or 2. **Use temporary Source credentials** to upload through the data proxy with an S3 client such as the AWS Command Line Interface (CLI) or an AWS Software Development Kit (SDK) -For long‑term or automated access, contact the Source Cooperative team at `hello@source.coop`. +For long‑term or automated access, use a [service account](/automated-access). --- @@ -235,211 +230,11 @@ aws s3 sync ./data s3://your-org/your-product/ --endpoint-url https://data.sourc ## Option 3: Long‑standing or automated access (ADVANCED) -If you need: - -- Continuous uploads -- Scheduled pipelines -- Integration from your own AWS account - -You can use your own IAM role to write to the Source Cooperative bucket. - -:::info This option really is AWS - -Unlike Options 1 and 2, this option bypasses the data proxy. You write -directly to Source Cooperative's Amazon S3 bucket (e.g. `us-west-2.opendata.source.coop`, `eu-west-1.opendata.source.coop`) -using an identity in **your own AWS account**. It only applies to products stored -in that bucket. If your product is stored with another provider, or you don't have -an AWS account, use [Option 2](#option-2-upload-through-the-data-proxy-with-temporary-credentials-recommended-for-larger-uploads) -or contact us. - -::: - -### How this works - -- You create an IAM role (or use an existing IAM user/account) in **your own** AWS account -- You send us its ARN, and we grant it write access to your account's prefix in the Source Cooperative bucket - -No credentials are shared, and no role chaining is required. - ---- - -### Step 1: Create an IAM role (or pick an identity to use) - -Which identity you send us depends on what is doing the uploading: - -| Uploading from | Send us | -| --- | --- | -| A service (ingestion pipeline, ECS task, Lambda, EC2, GitHub Actions with OIDC) | An **IAM role** ARN | -| A person running the AWS CLI locally | An **IAM user** ARN | -| Many identities in one account | The **AWS account** ARN (we trust the whole account; your account controls who may use it) | - -For a service, create a role in your account with a trust policy for whatever assumes it, then attach a policy granting it read/write access under your account prefix. Replace `your-org` with your Source Cooperative account ID. - -:::tip - -The [IAM policy wizard](/tools/iam-policy-wizard) generates this policy for you — enter your account ID and, optionally, a product ID. - -::: - -
-Example trust policy (ECS tasks) - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { "Service": "ecs-tasks.amazonaws.com" }, - "Action": "sts:AssumeRole" - } - ] -} -``` - -
- -
-Example access policy - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "s3:PutObject", - "s3:GetObject", - "s3:DeleteObject", - "s3:AbortMultipartUpload", - "s3:ListMultipartUploadParts" - ], - "Resource": "arn:aws:s3:::us-west-2.opendata.source.coop/your-org/*" - }, - { - "Effect": "Allow", - "Action": "s3:ListBucket", - "Resource": "arn:aws:s3:::us-west-2.opendata.source.coop", - "Condition": { - "StringLike": { "s3:prefix": "your-org/*" } - } - } - ] -} -``` - -`s3:AbortMultipartUpload` and `s3:ListMultipartUploadParts` cover the multipart -uploads the AWS CLI and SDKs use automatically for large files. `s3:PutObject` -alone authorizes starting an upload and sending its parts, but without those two -a failed or resumed transfer cannot clean up after itself. Both are object-level -actions, so the prefix in `Resource` scopes them like the rest. - -:::note - -This policy deliberately omits `s3:ListBucketMultipartUploads`, which lists -in-progress uploads across the *whole* bucket. AWS does not support the -`s3:prefix` condition key on it, so it cannot be limited to your data, and no -upload path needs it — incomplete uploads are cleaned up automatically after 7 -days. Only `aws s3api list-multipart-uploads` requires it; contact us if you -have a workflow that requires this policy. - -::: - -
- -
-Creating the role with the AWS CLI - -```bash -aws iam create-role \ - --role-name source-coop-upload \ - --assume-role-policy-document file://trust-policy.json - -aws iam put-role-policy \ - --role-name source-coop-upload \ - --policy-name source-coop-write \ - --policy-document file://s3-policy.json -``` - -
- -:::note - -This policy only grants permission on *your* side. Uploads will still fail with `AccessDenied` until we grant the same role access on the bucket side (Step 2). - -::: - ---- - -### Step 2: Send us the ARN - -Email [hello@source.coop](mailto:hello@source.coop) with: - -- The ARN of the role, user, or account you want us to trust, for example: - - Role: `arn:aws:iam::123456789012:role/source-coop-upload` - - User: `arn:aws:iam::123456789012:user/data-uploader` - - Account: `arn:aws:iam::123456789012:root` -- Your Source Cooperative account ID and product ID (the `your-org/your-product` prefix you will write to) -- A short description of the workflow (e.g. "nightly ingestion pipeline running on ECS") - -We will add the ARN to the bucket policy and confirm when it is active. - ---- - -### Step 3: Upload using that identity - -Once we confirm, upload with credentials for that role, user, or account: - -```bash -aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv -``` - -Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles) need no assume-role step — the SDK picks up the role automatically. - -
-Uploading a directory - -```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ -``` - -
- -
-Assuming the role from a workstation or CI job - -Let the AWS CLI do the assume-role for you — add a profile to `~/.aws/config`: - -```ini -[profile source-coop-upload] -role_arn = arn:aws:iam::123456789012:role/source-coop-upload -source_profile = default -region = us-west-2 -``` - -```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ \ - --profile source-coop-upload -``` - -
- -
-Uploading with boto3 - -```python -import boto3 - -s3 = boto3.client("s3") -s3.upload_file( - "mydata.csv", - "us-west-2.opendata.source.coop", - "your-org/your-product/mydata.csv", -) -``` - -
+For continuous uploads, scheduled pipelines and anything else that runs without +a person at the keyboard, create a **service account**: a login for software, +with its own access to your products. Give it an API key, and the AWS CLI and +SDKs upload through the data proxy with it, renewing their credentials on their +own. See [Automated Access](/automated-access). --- diff --git a/sidebars.ts b/sidebars.ts index 9a34c5a..00cba44 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -33,6 +33,7 @@ const sidebars: SidebarsConfig = { 'using-source/data-upload', 'using-source/bring-your-own-bucket', 'using-source/data-proxy', + 'using-source/automated-access', ], }, { From 459c6aae211cb06456f9dce98f572d23283e5516 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Tue, 29 Sep 2026 16:31:27 -0700 Subject: [PATCH 02/29] docs: say what the malformed-key error means An API key now ends in a six-character checksum of the rest (ADR-013, source-cooperative/data.source.coop#242), so the data proxy and the Source CLI refuse a key that was cut short or mistyped before anything is looked up, with "API key is malformed; check that it was copied whole". The guide's "If the key is refused" section shows that error and what to do about it, and no longer says a value that isn't a key gets "API key was not accepted". Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01R1eiTse4416N6uTgAy4Ddd --- docs/using-source/automated-access.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/using-source/automated-access.md b/docs/using-source/automated-access.md index d8d0fe2..9da996b 100644 --- a/docs/using-source/automated-access.md +++ b/docs/using-source/automated-access.md @@ -132,9 +132,8 @@ variables in its own environment. ### If the key is refused -A key that has been revoked or has expired, a key whose service account is -disabled, and a value that isn't a Source Cooperative key at all are all refused -with the same error: +A key that has been revoked or has expired, and a key whose service account is +disabled, are refused with the same error: ```text An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: API key was not accepted (request id 8f3a1c2b9d4e5f60-SEA) @@ -145,6 +144,18 @@ shows whether the key has been revoked or has expired. If none of those explains it, email [hello@source.coop](mailto:hello@source.coop) and quote the request id: it lets us find the reason in our logs. +A key's last six characters are a checksum of the rest, so a key that was cut +short or mistyped when it was copied is refused before anything is looked up, +with an error of its own: + +```text +An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: API key is malformed; check that it was copied whole (request id 8f3a1c2b9d4e5f60-SEA) +``` + +Copy the key again from where you saved it. If you no longer have all of it, +issue a new key and revoke the old one. The Source CLI checks a key file the +same way before it sends the key. + ### Keep the key secret - Treat the key as a password for the service account. Keep it out of source From 02633269236d833777ce4b8497faf125c632520a Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Thu, 1 Oct 2026 20:26:36 -0700 Subject: [PATCH 03/29] docs: document GitHub Actions sign-in, ReadOnly, and the current service account UI GitHub Actions trust shipped in data.source.coop#237 (which implements #222 and #223), so the "coming soon" note becomes instructions: trusting a workflow, the step from Example usage, why a job with an environment or a pull_request trigger carries a different subject, role-duration-seconds for long jobs, and the AccessDenied a refused sign-in gets. The API-key section now says how to choose ReadOnly, and that any other role name is refused. The create and issue steps follow the UI on source.coop main: sign-ins are added under "How software signs in" at creation or from "Add sign-in" under "Signs in with", and the variables, Change expiry and Revoke live in each row's menu. The stop table gains removing a workflow, and disabling now names workflow sign-ins too (authenticateWithOidcToken refuses a disabled account). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_014rVE6YPLaG5BrUoC76r3fp --- docs/using-source/automated-access.md | 113 ++++++++++++++++++++------ 1 file changed, 90 insertions(+), 23 deletions(-) diff --git a/docs/using-source/automated-access.md b/docs/using-source/automated-access.md index 9da996b..c1fc8e2 100644 --- a/docs/using-source/automated-access.md +++ b/docs/using-source/automated-access.md @@ -36,15 +36,18 @@ such as `your-org--nightly-sync`. It can't be changed. 2. Click **New service account**. 3. Under **Who it is**, enter a **Name**, such as `Nightly Sync`. The **Account ID** is made from the name; click **Edit** beside it to choose another. -4. Leave **How software signs in** empty. It is for GitHub Actions, which - [isn't available yet](#github-actions). +4. Under **How software signs in**, click **Add a GitHub workflow** to trust a + [GitHub Actions workflow](#github-actions), or **Add an API key** to issue + [a key](#issue-a-key) for anything else. Both are optional here: you can add + either later. 5. Under **What it can reach**, click **Grant a product**, choose a product and **Read** or **Read and write**, and click the check mark. Repeat for each product the job needs, and no more: a job that only downloads needs **Read**. 6. Click **Create service account**. -You land on the service account's page. You can change what it reaches at any -time under **Can reach**; each change is saved as you make it. +You land on the service account's page, showing the key if you added one. You +can change what it reaches at any time under **Can reach**; each change is saved +as you make it. ## API keys @@ -53,8 +56,8 @@ as the service account. ### Issue a key -1. On the service account's page, under **API keys**, click **Issue an API - key**. +1. On the service account's page, under **Signs in with**, click **Add + sign-in** and choose **API key**. 2. Enter a **Label** that says where the key will live, such as `HPC cron job`, so you know which key to revoke later. 3. Choose when it **Expires**: in 30 days, 90 days or a year, or never (until @@ -78,9 +81,9 @@ cat > ~/.source-coop/nightly-sync.key # paste the key, press Enter, then Ctrl- chmod 600 ~/.source-coop/nightly-sync.key ``` -Then set five environment variables wherever the job runs. The dialog that -showed you the key lists them too, filled in for your service account apart from -the key file's path: +Then set five environment variables wherever the job runs. The key's row on the +service account's page lists them too, under **Example usage** in its menu, +filled in for your service account apart from the key file's path: ```bash export AWS_ROLE_ARN=arn:aws:iam::your-org--nightly-sync:role/FullAccess @@ -98,6 +101,10 @@ export AWS_REGION=us-west-2 | `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | | `AWS_REGION` | Required by S3 clients. It doesn't say where your data is stored. | +For a job that only reads, end `AWS_ROLE_ARN` in `role/ReadOnly` instead of +`role/FullAccess`. Its credentials can't write, even to products the service +account may write to. Any other role name is refused. + That's all. The AWS CLI and SDKs read the key from the file, exchange it at the data proxy for credentials that last an hour, and exchange it again before those run out. Nothing else runs on the machine, and the key file never changes. @@ -173,7 +180,7 @@ A service account can have several keys at once, so you can replace one without stopping the job: issue a new key, deploy it, and revoke the old one once nothing uses it. Each key's row shows when it was last used. -To change when a key expires, click **Change expiry** on its row: later, for a +To change when a key expires, choose **Change expiry** from its row's menu: later, for a job that runs longer than planned, or sooner, during an incident. The new expiry counts from today. @@ -227,8 +234,9 @@ out. | To | Do this | What happens | | --- | --- | --- | -| Stop one key | Click **Revoke** on the key's row. | New exchanges with the key are refused within about a minute. | -| Stop everything the service account does | Click **Disable** under **Danger zone**. | New exchanges with any of its keys are refused within about a minute. Writes stop within about a minute, and reads of restricted products within about five minutes. | +| Stop one key | Choose **Revoke** from the key's menu. | New exchanges with the key are refused within about a minute. | +| Stop one workflow | Choose **Remove** from the workflow's menu. | New sign-ins from the workflow are refused within about a minute. | +| Stop everything the service account does | Click **Disable** under **Danger zone**. | New exchanges with any of its keys, and new sign-ins from its workflows, are refused within about a minute. Writes stop within about a minute, and reads of restricted products within about five minutes. | Revoking a key doesn't recall credentials already issued with it: they keep working until they expire, an hour after they were issued, or up to 12 hours if @@ -261,16 +269,75 @@ through GitHub's secret scanning. ## GitHub Actions -:::info Coming soon +A service account can trust a GitHub Actions workflow, which then signs in with +a token GitHub issues for each run. There is no key to store, rotate or leak. + +### Trust a workflow + +1. On the service account's page, under **Signs in with**, click **Add + sign-in** and choose **GitHub workflow**. +2. Enter the **Repository**, such as `your-org/pipelines`. +3. Under **Pinned to**, choose **Ref** and enter the full ref the workflow runs + on, such as `refs/heads/main` or `refs/tags/v1.0`; or choose **Environment** + and enter the name of the GitHub environment the job runs in. +4. Click **Trust it**. + +The dialog shows the exact subject it trusts, such as +`repo:your-org/pipelines:ref:refs/heads/main`. A run's token has to carry that +subject exactly, so: + +- A job that names an `environment:` carries the environment's subject, not + the branch's. Pin such a job to its environment. +- Runs for pull requests carry a subject that can't be trusted, so a job + triggered by `pull_request` can't sign in. +- Repositories created after July 2026 carry their owner's and their own + numeric IDs, as in `your-org@123/pipelines@456`. The dialog offers that form + when GitHub shows the repository publicly; for a private one, it prints a `gh` + command that finds it. + +A repository is trusted only for the ref or environment you name. To trust +another branch, add another workflow. + +### Sign in from the job + +Choose **Example usage** from the workflow's menu on the service account's page +for the step to add, filled in for your service account. It looks like this: + +```yaml +jobs: + publish: + runs-on: ubuntu-latest + permissions: + id-token: write # lets the job ask GitHub for its token + contents: read + env: + AWS_ENDPOINT_URL_S3: https://data.source.coop + steps: + - name: Sign in to Source Cooperative as your-org--nightly-sync + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: arn:aws:iam::your-org--nightly-sync:role/FullAccess + audience: https://data.source.coop + sts-endpoint: https://data.source.coop/.sts + aws-region: us-west-2 + - run: aws s3 sync ./outgoing s3://your-org/your-product/outgoing/ +``` -A service account can also trust a GitHub Actions workflow, which then signs in -with a token GitHub issues for each run, with no key to store. The service -account page already lets you add a workflow, but the data proxy doesn't accept -those sign-ins yet -([data.source.coop#222](https://github.com/source-cooperative/data.source.coop/issues/222), -[data.source.coop#223](https://github.com/source-cooperative/data.source.coop/issues/223)). -Until it does, a workflow can use an API key like any other machine: keep the -key in a GitHub Actions secret, write it to a file at the start of the job, and -set the variables above. +Unlike with an API key, the ID in `role-to-assume` matters: it names the service +account the job signs in as, and the sign-in succeeds only if that service +account trusts the workflow. End it in `role/ReadOnly` for a job that only +reads. -::: +The step's credentials last an hour, and nothing renews them during the job. For +a longer job, add `role-duration-seconds` to the step, up to `43200` (12 hours). + +A refused sign-in fails the step with: + +```text +AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity (request id 8f3a1c2b9d4e5f60-SEA) +``` + +The same error covers a workflow the service account doesn't trust (check the +subject, as above), a disabled service account, and an ID in `role-to-assume` +that isn't a service account. If none of those explains it, email +[hello@source.coop](mailto:hello@source.coop) and quote the request id. From 757e190532f597309deb731c208c4c6dfb0bb293 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Thu, 1 Oct 2026 20:52:52 -0700 Subject: [PATCH 04/29] docs: list an API key's variables from where to how The example now matches the order source.coop prints (source-cooperative/source.coop#619): region, S3 endpoint, STS endpoint, role, key file. The table below it follows. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_014rVE6YPLaG5BrUoC76r3fp --- docs/using-source/automated-access.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/using-source/automated-access.md b/docs/using-source/automated-access.md index c1fc8e2..b7f731c 100644 --- a/docs/using-source/automated-access.md +++ b/docs/using-source/automated-access.md @@ -86,20 +86,20 @@ service account's page lists them too, under **Example usage** in its menu, filled in for your service account apart from the key file's path: ```bash +export AWS_REGION=us-west-2 +export AWS_ENDPOINT_URL_S3=https://data.source.coop +export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts export AWS_ROLE_ARN=arn:aws:iam::your-org--nightly-sync:role/FullAccess export AWS_WEB_IDENTITY_TOKEN_FILE=$HOME/.source-coop/nightly-sync.key -export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts -export AWS_ENDPOINT_URL_S3=https://data.source.coop -export AWS_REGION=us-west-2 ``` | Variable | What it's for | | --- | --- | +| `AWS_REGION` | Required by S3 clients. It doesn't say where your data is stored. | +| `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | +| `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | | `AWS_ROLE_ARN` | How much the credentials may do. `FullAccess` is everything the service account may do; `ReadOnly` is reads only. The value has the shape AWS tools expect, with the service account's ID where an AWS account number would be. | | `AWS_WEB_IDENTITY_TOKEN_FILE` | The file that holds the key, as an absolute path. | -| `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | -| `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | -| `AWS_REGION` | Required by S3 clients. It doesn't say where your data is stored. | For a job that only reads, end `AWS_ROLE_ARN` in `role/ReadOnly` instead of `role/FullAccess`. Its credentials can't write, even to products the service From 2c3ab9491250ceb69159f4fc9bffb5ab16963d1a Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Thu, 1 Oct 2026 21:04:39 -0700 Subject: [PATCH 05/29] docs: split Upload Your Data into browser, CLI and service account pages Upload Your Data becomes a sidebar category. Its index keeps /data-upload with what every method shares (how the proxy works, where to upload, the controls) and a table for choosing a method. Option 1 becomes Upload in the Browser (/upload-in-the-browser), Option 2 becomes Upload with the Source CLI (/upload-with-the-cli) with View Credentials as its no-CLI fallback, and Automated Access moves in as With a Service Account, keeping /automated-access. The index carries HTML anchors for the old Option headings and the CLI heading Access Data linked to, so existing links still land. Content moves unchanged apart from headings and a sentence of glue per page. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_014rVE6YPLaG5BrUoC76r3fp --- docs/using-source/data-proxy.md | 2 +- docs/using-source/data-upload.md | 270 ------------------ .../{ => upload}/automated-access.md | 1 + docs/using-source/upload/in-the-browser.md | 26 ++ docs/using-source/upload/index.md | 86 ++++++ docs/using-source/upload/with-the-cli.md | 138 +++++++++ sidebars.ts | 12 +- 7 files changed, 262 insertions(+), 273 deletions(-) delete mode 100644 docs/using-source/data-upload.md rename docs/using-source/{ => upload}/automated-access.md (99%) create mode 100644 docs/using-source/upload/in-the-browser.md create mode 100644 docs/using-source/upload/index.md create mode 100644 docs/using-source/upload/with-the-cli.md diff --git a/docs/using-source/data-proxy.md b/docs/using-source/data-proxy.md index 98348d9..94221e2 100644 --- a/docs/using-source/data-proxy.md +++ b/docs/using-source/data-proxy.md @@ -56,7 +56,7 @@ source-coop login aws s3 ls s3://your-org/your-product --profile source-coop ``` -See [Upload Your Data](/data-upload#get-credentials-with-the-source-cli-recommended) for the one-time install and profile setup. +See [Upload with the Source CLI](/upload-with-the-cli) for the one-time install and profile setup. For software that runs on its own, with nobody there to log in, use a [service account](/automated-access) instead. diff --git a/docs/using-source/data-upload.md b/docs/using-source/data-upload.md deleted file mode 100644 index 2231195..0000000 --- a/docs/using-source/data-upload.md +++ /dev/null @@ -1,270 +0,0 @@ ---- -title: Upload Your Data -id: data-upload -slug: /data-upload -sidebar_position: 2 ---- - -This guide explains how to deliver your data to Source Cooperative in a secure and simple way. -It is written for data providers and does not require an Amazon Web Services (AWS) account or deep cloud storage knowledge. - -If you do not see the option to upload (for example, Edit Mode or View Credentials on your product page under the lock icon), contact [hello@source.coop](mailto:hello@source.coop) to request upload access. - ---- - -## How uploads work: the Source data proxy - -Every upload goes through the **Source data proxy** at `https://data.source.coop`. -The proxy speaks the **S3 API**, the object storage protocol that Amazon S3 -introduced and that most storage tools now support. It is not itself an AWS service. - -Behind the proxy, a product's data may be stored with any of several object -storage providers: Amazon S3, Google Cloud Storage, Azure Blob Storage, -Cloudflare R2, and others. You don't need to know which one. You always -upload to the same proxy address, and the proxy writes to the right place. - -That is why this guide uses AWS tools: - -- **The AWS CLI and AWS SDKs (boto3, etc.) are used only as S3 clients.** You - point them at `https://data.source.coop` instead of AWS. Any S3-compatible - client that lets you set a custom endpoint URL will work. -- **Your upload credentials come from Source Cooperative, not AWS.** They use - the AWS credential format (access key ID, secret access key, session token) - because S3 clients expect it, but they are only valid at the Source data proxy. -- **Addresses are proxy addresses.** `s3://your-org/your-product/` means the - `your-product` product in the `your-org` account on Source Cooperative. It - is not an Amazon S3 bucket. Where your data is physically stored doesn't change it. - ---- - -## The short version (what you need to do) - -You have two main ways to upload data: - -1. **Upload directly in the Source Cooperative User Interface (UI)** (drag-and-drop or file selector), or -2. **Use temporary Source credentials** to upload through the data proxy with an S3 client such as the AWS Command Line Interface (CLI) or an AWS Software Development Kit (SDK) - -For long‑term or automated access, use a [service account](/automated-access). - ---- - -## Option 1: Upload directly in the UI (Easiest) - -If you prefer not to use command-line tools, you can upload files directly in the web interface. - -### How to upload via the UI - -1. Go to your product page in Source Cooperative (e.g. `https://source.coop/your-org/your-product`) -2. In the top-right corner of the Product Contents card, click on the lock icon to open the dropdown menu and enable edit mode by clicking on `Edit Mode`. -3. Either: - - Drag‑and‑drop files to the `Product Contents` card - - Use `Upload Files` or `Upload Directory` options in dropdown menu to upload files via operating system's file selector - -Your files will be uploaded automatically to your product. - -This option is ideal for: - -- Small or medium uploads -- One‑time deliveries -- Non‑technical users - ---- - -## Option 2: Upload through the data proxy with temporary credentials (recommended for larger uploads) - -For larger uploads, scripting, or programmatic access, use temporary credentials -issued by Source Cooperative with an S3 client pointed at the data proxy. - -### Get credentials with the Source CLI (recommended) - -The [Source CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you with Source Cooperative and provides temporary credentials automatically, so you don't have to copy expiring credentials out of the UI. Once configured as an AWS CLI profile, the AWS CLI and SDKs refresh credentials for you, and you'll only need to re-authenticate every 30 days. - -**1. Install the CLI** - -Follow the install instructions in the [CLI README](https://github.com/source-cooperative/source-coop-cli). - -**2. Log in** - -```bash -source-coop login -``` - -This opens your browser to authenticate and caches short‑lived credentials in your OS keyring. - -**3. Configure an AWS CLI profile** - -Add a profile to `~/.aws/config` that calls the Source CLI to fetch credentials on demand and sends requests to the data proxy: - -```ini -[profile source-coop] -credential_process = source-coop creds -endpoint_url = https://data.source.coop -``` - -The `endpoint_url` line is what sends requests to the Source data proxy instead of AWS. - -**4. Use standard S3 commands with the profile** - -Pass the profile per command with `--profile`: - -```bash -aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv --profile source-coop -aws s3 sync ./data s3://your-org/your-product/ --profile source-coop -``` - -Or set it once for your shell session with the `AWS_PROFILE` environment variable: - -```bash -export AWS_PROFILE=source-coop -aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv -aws s3 sync ./data s3://your-org/your-product/ -``` - -:::tip - -`AWS_PROFILE` is also picked up automatically by the AWS SDKs (boto3, JavaScript, Go, etc.), so your scripts can use the default credential chain instead of hardcoding access keys or endpoint details. - -::: - -The AWS CLI will refresh credentials automatically; re-run `source-coop login` when your session expires. Your upload permissions are scoped to the products you have access to. - -### Get credentials from the UI - -1. Go to your product page -2. In the `Product Contents` card, open the dropdown (click on lock icon in top‑right corner) -3. Select `View Credentials` -4. Choose: - - `JSON (SDK)` credentials, or - - `Environment Variables` (shell) - -You will also see: - -- `Expiration`: the expiration time of the credentials (a specific date and time) -- `Bucket`: the bucket name on the data proxy, which is your account ID (`your-org`) -- `Prefix`: the prefix (folder) you are allowed to write to (e.g. `your-product/`) - ---- - -## What the credentials look like - -These are Source Cooperative credentials in the AWS credential format. They -only work against the data proxy, so always set the endpoint to -`https://data.source.coop`. The region value is required by S3 clients; it does -not say where your data is stored. - -### For SDK clients (boto3, AWS SDKs, etc.) - -```json -{ - "aws_access_key_id": "ASIA...", - "aws_secret_access_key": "pwEV...", - "aws_session_token": "IQoJ...", - "region_name": "us-west-2" -} -``` - -With boto3, pass the endpoint when creating the client: - -```python -import boto3 - -s3 = boto3.client("s3", endpoint_url="https://data.source.coop", **credentials) -s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") -``` - ---- - -### For terminal / shell usage - -```bash -export AWS_ACCESS_KEY_ID="ASIA..." -export AWS_SECRET_ACCESS_KEY="pwEV..." -export AWS_SESSION_TOKEN="IQoJ..." -export AWS_DEFAULT_REGION="us-west-2" -export AWS_ENDPOINT_URL="https://data.source.coop" # add this: send requests to the data proxy -``` - -These credentials are: - -- Temporary -- Scoped to your product -- Automatically revoked when they expire - ---- - -## Where to upload your data - -You may upload only to your product's prefix on the data proxy, for example: - -``` -s3://your-org/your-product/ -``` - -This matches your product page URL, `https://source.coop/your-org/your-product`. - -You may upload: - -- Files -- Folders -- Multiple objects - -You may not upload outside this path. - ---- - -## Example: Upload using the AWS CLI - -```bash -aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv --endpoint-url https://data.source.coop -``` - -Or upload a full directory: - -```bash -aws s3 sync ./data s3://your-org/your-product/ --endpoint-url https://data.source.coop -``` - -`--endpoint-url` isn't needed if you set `AWS_ENDPOINT_URL` or use the `source-coop` profile above. - ---- - -## Option 3: Long‑standing or automated access (ADVANCED) - -For continuous uploads, scheduled pipelines and anything else that runs without -a person at the keyboard, create a **service account**: a login for software, -with its own access to your products. Give it an API key, and the AWS CLI and -SDKs upload through the data proxy with it, renewing their credentials on their -own. See [Automated Access](/automated-access). - ---- - -## Why we use these controls - -These controls prevent accidental or unauthorized access and ensure: - -- Clear ownership of uploaded data -- No accidental access to other providers’ data -- Secure handling without sharing permanent credentials - -This protects both you and Source Cooperative. - ---- - -## What not to do - -Please do not: - -- Request permanent access keys -- Ask for full bucket access -- Upload outside your assigned prefix -- Reuse expired temporary credentials - ---- - -## Need help? - -If you’re unsure which option is right for you, or need help setting up automated access, contact the Source Cooperative team at `hello@source.coop`, and we will help you choose the best approach for your use case. - ---- - -Thank you for contributing data to Source Cooperative. diff --git a/docs/using-source/automated-access.md b/docs/using-source/upload/automated-access.md similarity index 99% rename from docs/using-source/automated-access.md rename to docs/using-source/upload/automated-access.md index b7f731c..df8d849 100644 --- a/docs/using-source/automated-access.md +++ b/docs/using-source/upload/automated-access.md @@ -1,5 +1,6 @@ --- title: Automated Access +sidebar_label: With a Service Account id: automated-access slug: /automated-access --- diff --git a/docs/using-source/upload/in-the-browser.md b/docs/using-source/upload/in-the-browser.md new file mode 100644 index 0000000..bb04d3c --- /dev/null +++ b/docs/using-source/upload/in-the-browser.md @@ -0,0 +1,26 @@ +--- +title: Upload in the Browser +sidebar_label: In the Browser +id: in-the-browser +slug: /upload-in-the-browser +--- + +If you prefer not to use command-line tools, you can upload files directly in the web interface. + +This option is ideal for: + +- Small or medium uploads +- One‑time deliveries +- Non‑technical users + +## How to upload via the UI + +1. Go to your product page in Source Cooperative (e.g. `https://source.coop/your-org/your-product`) +2. In the top-right corner of the Product Contents card, click on the lock icon to open the dropdown menu and enable edit mode by clicking on `Edit Mode`. +3. Either: + - Drag‑and‑drop files to the `Product Contents` card + - Use `Upload Files` or `Upload Directory` options in dropdown menu to upload files via operating system's file selector + +Your files will be uploaded automatically to your product. + +For larger uploads, or to script them, [upload with the Source CLI](/upload-with-the-cli) instead. diff --git a/docs/using-source/upload/index.md b/docs/using-source/upload/index.md new file mode 100644 index 0000000..762cb40 --- /dev/null +++ b/docs/using-source/upload/index.md @@ -0,0 +1,86 @@ +--- +title: Upload Your Data +id: data-upload +slug: /data-upload +--- + +This guide explains how to deliver your data to Source Cooperative in a secure and simple way. +It is written for data providers and does not require an Amazon Web Services (AWS) account or deep cloud storage knowledge. + +If you do not see the option to upload (for example, Edit Mode or View Credentials on your product page under the lock icon), contact [hello@source.coop](mailto:hello@source.coop) to request upload access. + +## Choose how to upload + + + +| | Best for | | +| --- | --- | --- | +| **[In the browser](/upload-in-the-browser)** | Small or one-time uploads, without installing anything | Easiest | +| **[With the Source CLI](/upload-with-the-cli)** | Large uploads and scripts that you run yourself | Recommended for larger uploads | +| **[With a service account](/automated-access)** | Pipelines, scheduled jobs and GitHub Actions: anything that runs with nobody at the keyboard | Advanced | + +## How uploads work: the Source data proxy + +Every upload goes through the **Source data proxy** at `https://data.source.coop`. +The proxy speaks the **S3 API**, the object storage protocol that Amazon S3 +introduced and that most storage tools now support. It is not itself an AWS service. + +Behind the proxy, a product's data may be stored with any of several object +storage providers: Amazon S3, Google Cloud Storage, Azure Blob Storage, +Cloudflare R2, and others. You don't need to know which one. You always +upload to the same proxy address, and the proxy writes to the right place. + +That is why these guides use AWS tools: + +- **The AWS CLI and AWS SDKs (boto3, etc.) are used only as S3 clients.** You + point them at `https://data.source.coop` instead of AWS. Any S3-compatible + client that lets you set a custom endpoint URL will work. +- **Your upload credentials come from Source Cooperative, not AWS.** They use + the AWS credential format (access key ID, secret access key, session token) + because S3 clients expect it, but they are only valid at the Source data proxy. +- **Addresses are proxy addresses.** `s3://your-org/your-product/` means the + `your-product` product in the `your-org` account on Source Cooperative. It + is not an Amazon S3 bucket. Where your data is physically stored doesn't change it. + +## Where to upload your data + +You may upload only to your product's prefix on the data proxy, for example: + +``` +s3://your-org/your-product/ +``` + +This matches your product page URL, `https://source.coop/your-org/your-product`. + +You may upload: + +- Files +- Folders +- Multiple objects + +You may not upload outside this path. + +## Why we use these controls + +These controls prevent accidental or unauthorized access and ensure: + +- Clear ownership of uploaded data +- No accidental access to other providers’ data +- Secure handling without sharing permanent credentials + +This protects both you and Source Cooperative. + +## What not to do + +Please do not: + +- Request permanent access keys +- Ask for full bucket access +- Upload outside your assigned prefix +- Reuse expired temporary credentials + +## Need help? + +If you’re unsure which option is right for you, or need help setting up automated access, contact the Source Cooperative team at `hello@source.coop`, and we will help you choose the best approach for your use case. + +Thank you for contributing data to Source Cooperative. diff --git a/docs/using-source/upload/with-the-cli.md b/docs/using-source/upload/with-the-cli.md new file mode 100644 index 0000000..bea4c9c --- /dev/null +++ b/docs/using-source/upload/with-the-cli.md @@ -0,0 +1,138 @@ +--- +title: Upload with the Source CLI +sidebar_label: With the Source CLI +id: with-the-cli +slug: /upload-with-the-cli +--- + +For larger uploads, scripting, or programmatic access, use temporary credentials +issued by Source Cooperative with an S3 client pointed at the +[data proxy](/data-upload#how-uploads-work-the-source-data-proxy). + +## Get credentials with the Source CLI (recommended) + +The [Source CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you with Source Cooperative and provides temporary credentials automatically, so you don't have to copy expiring credentials out of the UI. Once configured as an AWS CLI profile, the AWS CLI and SDKs refresh credentials for you, and you'll only need to re-authenticate every 30 days. + +**1. Install the CLI** + +Follow the install instructions in the [CLI README](https://github.com/source-cooperative/source-coop-cli). + +**2. Log in** + +```bash +source-coop login +``` + +This opens your browser to authenticate and caches short‑lived credentials in your OS keyring. + +**3. Configure an AWS CLI profile** + +Add a profile to `~/.aws/config` that calls the Source CLI to fetch credentials on demand and sends requests to the data proxy: + +```ini +[profile source-coop] +credential_process = source-coop creds +endpoint_url = https://data.source.coop +``` + +The `endpoint_url` line is what sends requests to the Source data proxy instead of AWS. + +**4. Use standard S3 commands with the profile** + +Pass the profile per command with `--profile`: + +```bash +aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv --profile source-coop +aws s3 sync ./data s3://your-org/your-product/ --profile source-coop +``` + +Or set it once for your shell session with the `AWS_PROFILE` environment variable: + +```bash +export AWS_PROFILE=source-coop +aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv +aws s3 sync ./data s3://your-org/your-product/ +``` + +:::tip + +`AWS_PROFILE` is also picked up automatically by the AWS SDKs (boto3, JavaScript, Go, etc.), so your scripts can use the default credential chain instead of hardcoding access keys or endpoint details. + +::: + +The AWS CLI will refresh credentials automatically; re-run `source-coop login` when your session expires. Your upload permissions are scoped to the products you have access to. + +## Without the CLI: get credentials from the UI + +You can also copy the same kind of temporary credentials out of the web +interface. They expire, and you'll need to copy new ones when they do. + +1. Go to your product page +2. In the `Product Contents` card, open the dropdown (click on lock icon in top‑right corner) +3. Select `View Credentials` +4. Choose: + - `JSON (SDK)` credentials, or + - `Environment Variables` (shell) + +You will also see: + +- `Expiration`: the expiration time of the credentials (a specific date and time) +- `Bucket`: the bucket name on the data proxy, which is your account ID (`your-org`) +- `Prefix`: the prefix (folder) you are allowed to write to (e.g. `your-product/`) + +### What the credentials look like + +These are Source Cooperative credentials in the AWS credential format. They +only work against the data proxy, so always set the endpoint to +`https://data.source.coop`. The region value is required by S3 clients; it does +not say where your data is stored. + +For SDK clients (boto3, AWS SDKs, etc.): + +```json +{ + "aws_access_key_id": "ASIA...", + "aws_secret_access_key": "pwEV...", + "aws_session_token": "IQoJ...", + "region_name": "us-west-2" +} +``` + +With boto3, pass the endpoint when creating the client: + +```python +import boto3 + +s3 = boto3.client("s3", endpoint_url="https://data.source.coop", **credentials) +s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") +``` + +For terminal / shell usage: + +```bash +export AWS_ACCESS_KEY_ID="ASIA..." +export AWS_SECRET_ACCESS_KEY="pwEV..." +export AWS_SESSION_TOKEN="IQoJ..." +export AWS_DEFAULT_REGION="us-west-2" +export AWS_ENDPOINT_URL="https://data.source.coop" # add this: send requests to the data proxy +``` + +These credentials are: + +- Temporary +- Scoped to your product +- Automatically revoked when they expire + +### Example: Upload using the AWS CLI + +```bash +aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv --endpoint-url https://data.source.coop +``` + +Or upload a full directory: + +```bash +aws s3 sync ./data s3://your-org/your-product/ --endpoint-url https://data.source.coop +``` + +`--endpoint-url` isn't needed if you set `AWS_ENDPOINT_URL` or use the `source-coop` profile above. diff --git a/sidebars.ts b/sidebars.ts index 00cba44..9d62a51 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -30,10 +30,18 @@ const sidebars: SidebarsConfig = { items: [ 'using-source/create-an-account', 'using-source/create-a-data-product', - 'using-source/data-upload', + { + type: 'category', + label: 'Upload Your Data', + link: {type: 'doc', id: 'using-source/upload/data-upload'}, + items: [ + 'using-source/upload/in-the-browser', + 'using-source/upload/with-the-cli', + 'using-source/upload/automated-access', + ], + }, 'using-source/bring-your-own-bucket', 'using-source/data-proxy', - 'using-source/automated-access', ], }, { From f024ee8f9bb3a0da34d1c604f25febcc49630f8b Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Thu, 1 Oct 2026 21:17:58 -0700 Subject: [PATCH 06/29] docs: explain AWS_REGION, and drop the note that proxy uploads are disabled The data proxy verifies signatures with whatever region the client signed for and uses it for nothing else, but many SDKs refuse to sign without one, so both credential pages now say to set it anyway. Access Data's closing section said uploading through the proxy was disabled, which the Upload Your Data pages contradict; it is gone. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_014rVE6YPLaG5BrUoC76r3fp --- docs/using-source/data-proxy.md | 4 ---- docs/using-source/upload/automated-access.md | 2 +- docs/using-source/upload/with-the-cli.md | 5 +++-- 3 files changed, 4 insertions(+), 7 deletions(-) diff --git a/docs/using-source/data-proxy.md b/docs/using-source/data-proxy.md index 94221e2..4852363 100644 --- a/docs/using-source/data-proxy.md +++ b/docs/using-source/data-proxy.md @@ -67,7 +67,3 @@ If you don't have the AWS CLI installed, follow the [AWS CLI Getting Started Gui ## Current Status The Source Data Proxy is currently in beta. We're working to make performance indistinguishable from accessing objects directly via AWS S3. - -## Uploading Data Through the Source Data Proxy - -Uploading data through the Source Data Proxy is currently disabled while we work on new, easier, and more secure ways to upload data. If you need to upload data, please contact hello@source.coop. diff --git a/docs/using-source/upload/automated-access.md b/docs/using-source/upload/automated-access.md index df8d849..0b539ed 100644 --- a/docs/using-source/upload/automated-access.md +++ b/docs/using-source/upload/automated-access.md @@ -96,7 +96,7 @@ export AWS_WEB_IDENTITY_TOKEN_FILE=$HOME/.source-coop/nightly-sync.key | Variable | What it's for | | --- | --- | -| `AWS_REGION` | Required by S3 clients. It doesn't say where your data is stored. | +| `AWS_REGION` | Unused by the data proxy, which accepts any region, but set it anyway: many SDKs won't sign requests without one. It doesn't say where your data is stored. | | `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | | `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | | `AWS_ROLE_ARN` | How much the credentials may do. `FullAccess` is everything the service account may do; `ReadOnly` is reads only. The value has the shape AWS tools expect, with the service account's ID where an AWS account number would be. | diff --git a/docs/using-source/upload/with-the-cli.md b/docs/using-source/upload/with-the-cli.md index bea4c9c..fcc97bc 100644 --- a/docs/using-source/upload/with-the-cli.md +++ b/docs/using-source/upload/with-the-cli.md @@ -84,8 +84,9 @@ You will also see: These are Source Cooperative credentials in the AWS credential format. They only work against the data proxy, so always set the endpoint to -`https://data.source.coop`. The region value is required by S3 clients; it does -not say where your data is stored. +`https://data.source.coop`. The data proxy doesn't use the region and accepts +any value, but set one anyway: many SDKs won't sign requests without it. It +doesn't say where your data is stored. For SDK clients (boto3, AWS SDKs, etc.): From d581a85f3b4940f179bae873b2fa1e165b978b28 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Fri, 2 Oct 2026 14:54:56 -0700 Subject: [PATCH 07/29] docs: note that only owners and maintainers manage service accounts Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/automated-access.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/using-source/upload/automated-access.md b/docs/using-source/upload/automated-access.md index 0b539ed..0ab24e6 100644 --- a/docs/using-source/upload/automated-access.md +++ b/docs/using-source/upload/automated-access.md @@ -14,7 +14,9 @@ their own, so nothing else has to run on the machine. ## What a service account is A service account is a login for software. It belongs to one account, yours or -an organization's, and whoever manages that account manages it. +an organization's. Only the owner of a personal account, or an owner or +maintainer of an organization, can create or edit its service accounts. Other +organization members can't. - **It has its own access.** You grant it products one at a time, to read or to read and write. It can reach only products its owner owns, and it never @@ -32,6 +34,9 @@ such as `your-org--nightly-sync`. It can't be changed. ## Create a service account +You need to be the account's owner or, for an organization, one of its owners or +maintainers. + 1. Open the profile page of the account that will own it, yours or an organization's, click the gear icon, and choose **Service Accounts**. 2. Click **New service account**. From 2891cee08965fe59914a3b58a1b896a06e57f2ad Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:07:23 -0700 Subject: [PATCH 08/29] docs: keep the IAM-role upload instructions at /direct-uploads, marked deprecated Pipelines that write straight to the bucket with their own IAM role still need these instructions until they move to service accounts. The page stays out of the sidebar; the upload index links to it. Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/direct-uploads.md | 212 +++++++++++++++++++++ docs/using-source/upload/index.md | 6 +- 2 files changed, 217 insertions(+), 1 deletion(-) create mode 100644 docs/using-source/upload/direct-uploads.md diff --git a/docs/using-source/upload/direct-uploads.md b/docs/using-source/upload/direct-uploads.md new file mode 100644 index 0000000..c430031 --- /dev/null +++ b/docs/using-source/upload/direct-uploads.md @@ -0,0 +1,212 @@ +--- +title: Direct Uploads with Your Own AWS Identity +sidebar_label: Direct Uploads (Deprecated) +id: direct-uploads +slug: /direct-uploads +--- + +:::warning Deprecated + +This method is deprecated. New pipelines should use a +[service account](/automated-access) instead, which works for every product, +needs no AWS account, and doesn't need us to change a bucket policy. These +instructions stay here for pipelines already set up this way; contact +[hello@source.coop](mailto:hello@source.coop) for help moving one to a service +account. + +::: + +You can use an IAM role in your own AWS account to write to the Source +Cooperative bucket. + +This bypasses the [data proxy](/data-proxy). You write directly to Source +Cooperative's Amazon S3 bucket (e.g. `us-west-2.opendata.source.coop`, +`eu-west-1.opendata.source.coop`) using an identity in **your own AWS +account**. It only applies to products stored in that bucket. + +## How this works + +- You create an IAM role (or use an existing IAM user/account) in **your own** AWS account +- You send us its ARN, and we grant it write access to your account's prefix in the Source Cooperative bucket + +No credentials are shared, and no role chaining is required. + +--- + +## Step 1: Create an IAM role (or pick an identity to use) + +Which identity you send us depends on what is doing the uploading: + +| Uploading from | Send us | +| --- | --- | +| A service (ingestion pipeline, ECS task, Lambda, EC2, GitHub Actions with OIDC) | An **IAM role** ARN | +| A person running the AWS CLI locally | An **IAM user** ARN | +| Many identities in one account | The **AWS account** ARN (we trust the whole account; your account controls who may use it) | + +For a service, create a role in your account with a trust policy for whatever assumes it, then attach a policy granting it read/write access under your account prefix. Replace `your-org` with your Source Cooperative account ID. + +:::tip + +The [IAM policy wizard](/tools/iam-policy-wizard) generates this policy for you — enter your account ID and, optionally, a product ID. + +::: + +
+Example trust policy (ECS tasks) + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { "Service": "ecs-tasks.amazonaws.com" }, + "Action": "sts:AssumeRole" + } + ] +} +``` + +
+ +
+Example access policy + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:DeleteObject", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts" + ], + "Resource": "arn:aws:s3:::us-west-2.opendata.source.coop/your-org/*" + }, + { + "Effect": "Allow", + "Action": "s3:ListBucket", + "Resource": "arn:aws:s3:::us-west-2.opendata.source.coop", + "Condition": { + "StringLike": { "s3:prefix": "your-org/*" } + } + } + ] +} +``` + +`s3:AbortMultipartUpload` and `s3:ListMultipartUploadParts` cover the multipart +uploads the AWS CLI and SDKs use automatically for large files. `s3:PutObject` +alone authorizes starting an upload and sending its parts, but without those two +a failed or resumed transfer cannot clean up after itself. Both are object-level +actions, so the prefix in `Resource` scopes them like the rest. + +:::note + +This policy deliberately omits `s3:ListBucketMultipartUploads`, which lists +in-progress uploads across the *whole* bucket. AWS does not support the +`s3:prefix` condition key on it, so it cannot be limited to your data, and no +upload path needs it — incomplete uploads are cleaned up automatically after 7 +days. Only `aws s3api list-multipart-uploads` requires it; contact us if you +have a workflow that requires this policy. + +::: + +
+ +
+Creating the role with the AWS CLI + +```bash +aws iam create-role \ + --role-name source-coop-upload \ + --assume-role-policy-document file://trust-policy.json + +aws iam put-role-policy \ + --role-name source-coop-upload \ + --policy-name source-coop-write \ + --policy-document file://s3-policy.json +``` + +
+ +:::note + +This policy only grants permission on *your* side. Uploads will still fail with `AccessDenied` until we grant the same role access on the bucket side (Step 2). + +::: + +--- + +## Step 2: Send us the ARN + +Email [hello@source.coop](mailto:hello@source.coop) with: + +- The ARN of the role, user, or account you want us to trust, for example: + - Role: `arn:aws:iam::123456789012:role/source-coop-upload` + - User: `arn:aws:iam::123456789012:user/data-uploader` + - Account: `arn:aws:iam::123456789012:root` +- Your Source Cooperative account ID and product ID (the `your-org/your-product` prefix you will write to) +- A short description of the workflow (e.g. "nightly ingestion pipeline running on ECS") + +We will add the ARN to the bucket policy and confirm when it is active. + +--- + +## Step 3: Upload using that identity + +Once we confirm, upload with credentials for that role, user, or account: + +```bash +aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv +``` + +Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles) need no assume-role step — the SDK picks up the role automatically. + +
+Uploading a directory + +```bash +aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ +``` + +
+ +
+Assuming the role from a workstation or CI job + +Let the AWS CLI do the assume-role for you — add a profile to `~/.aws/config`: + +```ini +[profile source-coop-upload] +role_arn = arn:aws:iam::123456789012:role/source-coop-upload +source_profile = default +region = us-west-2 +``` + +```bash +aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ \ + --profile source-coop-upload +``` + +
+ +
+Uploading with boto3 + +```python +import boto3 + +s3 = boto3.client("s3") +s3.upload_file( + "mydata.csv", + "us-west-2.opendata.source.coop", + "your-org/your-product/mydata.csv", +) +``` + +
diff --git a/docs/using-source/upload/index.md b/docs/using-source/upload/index.md index 762cb40..3700ea9 100644 --- a/docs/using-source/upload/index.md +++ b/docs/using-source/upload/index.md @@ -19,9 +19,13 @@ If you do not see the option to upload (for example, Edit Mode or View Credentia | **[With the Source CLI](/upload-with-the-cli)** | Large uploads and scripts that you run yourself | Recommended for larger uploads | | **[With a service account](/automated-access)** | Pipelines, scheduled jobs and GitHub Actions: anything that runs with nobody at the keyboard | Advanced | +Already upload from your own AWS account with an IAM role? That method is +deprecated in favor of service accounts, but its instructions are still at +[Direct uploads](/direct-uploads). + ## How uploads work: the Source data proxy -Every upload goes through the **Source data proxy** at `https://data.source.coop`. +Uploads go through the **Source data proxy** at `https://data.source.coop`. The proxy speaks the **S3 API**, the object storage protocol that Amazon S3 introduced and that most storage tools now support. It is not itself an AWS service. From e8a6d81ccc99d1e8651eeca9785daaee95d91bd4 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:07:35 -0700 Subject: [PATCH 09/29] tools: mark the IAM policy wizard deprecated and link it to /direct-uploads Co-Authored-By: Claude Opus 5.5 --- src/pages/tools/iam-policy-wizard.tsx | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/pages/tools/iam-policy-wizard.tsx b/src/pages/tools/iam-policy-wizard.tsx index 544ca34..ace5401 100644 --- a/src/pages/tools/iam-policy-wizard.tsx +++ b/src/pages/tools/iam-policy-wizard.tsx @@ -26,10 +26,15 @@ export default function IamPolicyWizard(): ReactNode { description="Generate the IAM policy your AWS role or user needs to upload to Source Cooperative.">

IAM policy wizard

+
+ Deprecated. Uploading with an IAM role in your own AWS account is deprecated + in favor of service accounts. Use this wizard only for + pipelines already set up this way. +

Generate the policy to attach to the IAM role or user in your own AWS account, so it can upload to your prefix in the Source Cooperative bucket. See{' '} - Upload Your Data for the full walkthrough. + Direct uploads for the full walkthrough.

From eb36d1cfa16ede313162e0b977a181bde3900e6a Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:07:35 -0700 Subject: [PATCH 10/29] docs: drop the upload index's advice against permanent keys Service account API keys can be issued without an expiry, so the index shouldn't forbid them. Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/index.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/using-source/upload/index.md b/docs/using-source/upload/index.md index 3700ea9..e7a05bb 100644 --- a/docs/using-source/upload/index.md +++ b/docs/using-source/upload/index.md @@ -70,7 +70,6 @@ These controls prevent accidental or unauthorized access and ensure: - Clear ownership of uploaded data - No accidental access to other providers’ data -- Secure handling without sharing permanent credentials This protects both you and Source Cooperative. @@ -78,7 +77,6 @@ This protects both you and Source Cooperative. Please do not: -- Request permanent access keys - Ask for full bucket access - Upload outside your assigned prefix - Reuse expired temporary credentials From dd4c7a840829def2b8166805e06de25b909d0b34 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:07:44 -0700 Subject: [PATCH 11/29] docs: set AWS_DEFAULT_REGION, which boto3 reads, instead of AWS_REGION botocore reads only AWS_DEFAULT_REGION, so with AWS_REGION alone boto3 had no region. Note that the JavaScript and Java SDKs read AWS_REGION. Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/automated-access.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/using-source/upload/automated-access.md b/docs/using-source/upload/automated-access.md index 0ab24e6..f634b9c 100644 --- a/docs/using-source/upload/automated-access.md +++ b/docs/using-source/upload/automated-access.md @@ -92,7 +92,7 @@ service account's page lists them too, under **Example usage** in its menu, filled in for your service account apart from the key file's path: ```bash -export AWS_REGION=us-west-2 +export AWS_DEFAULT_REGION=us-west-2 export AWS_ENDPOINT_URL_S3=https://data.source.coop export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts export AWS_ROLE_ARN=arn:aws:iam::your-org--nightly-sync:role/FullAccess @@ -101,7 +101,7 @@ export AWS_WEB_IDENTITY_TOKEN_FILE=$HOME/.source-coop/nightly-sync.key | Variable | What it's for | | --- | --- | -| `AWS_REGION` | Unused by the data proxy, which accepts any region, but set it anyway: many SDKs won't sign requests without one. It doesn't say where your data is stored. | +| `AWS_DEFAULT_REGION` | Unused by the data proxy, which accepts any region, but set it anyway: many SDKs won't sign requests without one. It doesn't say where your data is stored. The AWS CLI, boto3 and the Go SDK read it; the JavaScript and Java SDKs read `AWS_REGION` instead, so for those set that too. | | `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | | `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | | `AWS_ROLE_ARN` | How much the credentials may do. `FullAccess` is everything the service account may do; `ReadOnly` is reads only. The value has the shape AWS tools expect, with the service account's ID where an AWS account number would be. | From f104eb1b5f50628875c24797e7bcaf0a34d56f37 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:07:51 -0700 Subject: [PATCH 12/29] docs: give the key file as a full path, not $HOME Cron and systemd, where the page tells readers to copy these variables, do not expand $HOME, so the job would look for a literal "$HOME/..." path. Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/automated-access.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/using-source/upload/automated-access.md b/docs/using-source/upload/automated-access.md index f634b9c..0975e13 100644 --- a/docs/using-source/upload/automated-access.md +++ b/docs/using-source/upload/automated-access.md @@ -96,7 +96,7 @@ export AWS_DEFAULT_REGION=us-west-2 export AWS_ENDPOINT_URL_S3=https://data.source.coop export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts export AWS_ROLE_ARN=arn:aws:iam::your-org--nightly-sync:role/FullAccess -export AWS_WEB_IDENTITY_TOKEN_FILE=$HOME/.source-coop/nightly-sync.key +export AWS_WEB_IDENTITY_TOKEN_FILE=/home/your-user/.source-coop/nightly-sync.key ``` | Variable | What it's for | @@ -105,7 +105,7 @@ export AWS_WEB_IDENTITY_TOKEN_FILE=$HOME/.source-coop/nightly-sync.key | `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | | `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | | `AWS_ROLE_ARN` | How much the credentials may do. `FullAccess` is everything the service account may do; `ReadOnly` is reads only. The value has the shape AWS tools expect, with the service account's ID where an AWS account number would be. | -| `AWS_WEB_IDENTITY_TOKEN_FILE` | The file that holds the key, as an absolute path. | +| `AWS_WEB_IDENTITY_TOKEN_FILE` | The file that holds the key, as an absolute path. Write it out in full: cron and systemd don't expand `$HOME` or `~`. | For a job that only reads, end `AWS_ROLE_ARN` in `role/ReadOnly` instead of `role/FullAccess`. Its credentials can't write, even to products the service From 82865c6ad485fdff8f6e936975aef202b7afa585 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:08:05 -0700 Subject: [PATCH 13/29] docs: keep a found key off the command line when revoking it The example put the key in curl's arguments, where it lands in shell history and is visible in ps, against the page's own advice. Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/automated-access.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/using-source/upload/automated-access.md b/docs/using-source/upload/automated-access.md index 0975e13..057d69a 100644 --- a/docs/using-source/upload/automated-access.md +++ b/docs/using-source/upload/automated-access.md @@ -257,13 +257,17 @@ before you enable the account. ### Revoke a key you found Anyone who holds a key can revoke it, without an account. If you come across one -that has leaked, in a repository, a log or a message, send it in the body of -this request: +that has leaked, in a repository, a log or a message, save it in a file and +send it in the body of this request, so it stays out of your shell history and +process list: ```bash -curl -X POST https://source.coop/api/v1/service-account-keys/revocations \ - -H 'content-type: application/json' \ - -d '{"key":"sck_…"}' +cat > found.key # paste the key, press Enter, then Ctrl-D +printf '{"key":"%s"}' "$(tr -d '\n' < found.key)" | + curl -X POST https://source.coop/api/v1/service-account-keys/revocations \ + -H 'content-type: application/json' \ + --data-binary @- +rm found.key ``` For a well-formed key the answer is always `204 No Content`, whether the key was From fe410d107a2ac9516195ca1e7ff9e81955490d2c Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:08:06 -0700 Subject: [PATCH 14/29] docs: create the key file private from the start mkdir -m leaves an existing directory's mode alone, and the key file was readable by others between cat and chmod. chmod the directory, and create the file under umask 077 in a subshell so the shell's umask is untouched. Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/automated-access.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/using-source/upload/automated-access.md b/docs/using-source/upload/automated-access.md index 057d69a..4b57265 100644 --- a/docs/using-source/upload/automated-access.md +++ b/docs/using-source/upload/automated-access.md @@ -82,9 +82,8 @@ Save the key in a file that only the job's user can read. A trailing newline is fine. ```bash -mkdir -p -m 700 ~/.source-coop -cat > ~/.source-coop/nightly-sync.key # paste the key, press Enter, then Ctrl-D -chmod 600 ~/.source-coop/nightly-sync.key +mkdir -p ~/.source-coop && chmod 700 ~/.source-coop +(umask 077; cat > ~/.source-coop/nightly-sync.key) # paste the key, press Enter, then Ctrl-D ``` Then set five environment variables wherever the job runs. The key's row on the From 4147f1f91316b427a67da4fa3be1868fffb784b2 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:09:03 -0700 Subject: [PATCH 15/29] docs: clarify how software signs in, name the Grant button, keep old anchors - Explain in the intro that a service account signs in with a GitHub workflow token or an API key, and what 'nothing else has to run' meant. - The grant dialog's button is labelled Grant, not a check mark. - Keep the old page's sub-section anchors so links into it land beside the page that now covers them. Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/automated-access.md | 22 +++++++++++++------- docs/using-source/upload/index.md | 9 +++++--- 2 files changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/using-source/upload/automated-access.md b/docs/using-source/upload/automated-access.md index 4b57265..d755a34 100644 --- a/docs/using-source/upload/automated-access.md +++ b/docs/using-source/upload/automated-access.md @@ -6,10 +6,15 @@ slug: /automated-access --- Some software reaches Source Cooperative with nobody at the keyboard: a nightly -sync, a publishing pipeline, an instrument that uploads its readings. Give it a -**service account** and an **API key**. The AWS CLI and SDKs exchange the key -for short-lived credentials at the [data proxy](/data-proxy) and renew them on -their own, so nothing else has to run on the machine. +sync, a publishing pipeline, an instrument that uploads its readings. Such +software signs in as a **service account**, in one of two ways: + +- A [GitHub Actions workflow](#github-actions) uses a token GitHub issues for + each run, so there's no secret to store. +- Anything else uses one of the service account's [API keys](#api-keys). The AWS + CLI and SDKs trade the key at the [data proxy](/data-proxy) for credentials + that last an hour, and trade it again before those run out. You don't need + the Source CLI or any other helper on the machine. ## What a service account is @@ -47,7 +52,7 @@ maintainers. [a key](#issue-a-key) for anything else. Both are optional here: you can add either later. 5. Under **What it can reach**, click **Grant a product**, choose a product and - **Read** or **Read and write**, and click the check mark. Repeat for each + **Read** or **Read and write**, and click **Grant**. Repeat for each product the job needs, and no more: a job that only downloads needs **Read**. 6. Click **Create service account**. @@ -110,9 +115,10 @@ For a job that only reads, end `AWS_ROLE_ARN` in `role/ReadOnly` instead of `role/FullAccess`. Its credentials can't write, even to products the service account may write to. Any other role name is refused. -That's all. The AWS CLI and SDKs read the key from the file, exchange it at the -data proxy for credentials that last an hour, and exchange it again before those -run out. Nothing else runs on the machine, and the key file never changes. +That's all. The AWS CLI and SDKs read the key from the file, trade it at the +data proxy for credentials that last an hour, and trade it again before those +run out. The key file never changes, and no other program has to run alongside +the job. The two endpoint variables need **AWS CLI 2.13** or later, or **boto3 1.28** (**botocore 1.31**) or later. Older releases ignore them and send the key to AWS diff --git a/docs/using-source/upload/index.md b/docs/using-source/upload/index.md index e7a05bb..7b75cd6 100644 --- a/docs/using-source/upload/index.md +++ b/docs/using-source/upload/index.md @@ -11,14 +11,17 @@ If you do not see the option to upload (for example, Edit Mode or View Credentia ## Choose how to upload - + + + | | Best for | | | --- | --- | --- | -| **[In the browser](/upload-in-the-browser)** | Small or one-time uploads, without installing anything | Easiest | -| **[With the Source CLI](/upload-with-the-cli)** | Large uploads and scripts that you run yourself | Recommended for larger uploads | +| **[In the browser](/upload-in-the-browser)** | Small or one-time uploads, without installing anything | Easiest | +| **[With the Source CLI](/upload-with-the-cli)** | Large uploads and scripts that you run yourself | Recommended for larger uploads | | **[With a service account](/automated-access)** | Pipelines, scheduled jobs and GitHub Actions: anything that runs with nobody at the keyboard | Advanced | + Already upload from your own AWS account with an IAM role? That method is deprecated in favor of service accounts, but its instructions are still at [Direct uploads](/direct-uploads). From ddd1afec8f2276cb02185867e030d3797c96f2a8 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:13:03 -0700 Subject: [PATCH 16/29] docs: keep AWS_REGION, and note that boto3 needs AWS_DEFAULT_REGION AWS_REGION is what most SDKs and the app's example usage read; boto3 is the one that reads only AWS_DEFAULT_REGION. Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/automated-access.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/using-source/upload/automated-access.md b/docs/using-source/upload/automated-access.md index d755a34..df374f5 100644 --- a/docs/using-source/upload/automated-access.md +++ b/docs/using-source/upload/automated-access.md @@ -96,7 +96,7 @@ service account's page lists them too, under **Example usage** in its menu, filled in for your service account apart from the key file's path: ```bash -export AWS_DEFAULT_REGION=us-west-2 +export AWS_REGION=us-west-2 export AWS_ENDPOINT_URL_S3=https://data.source.coop export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts export AWS_ROLE_ARN=arn:aws:iam::your-org--nightly-sync:role/FullAccess @@ -105,7 +105,7 @@ export AWS_WEB_IDENTITY_TOKEN_FILE=/home/your-user/.source-coop/nightly-sync.key | Variable | What it's for | | --- | --- | -| `AWS_DEFAULT_REGION` | Unused by the data proxy, which accepts any region, but set it anyway: many SDKs won't sign requests without one. It doesn't say where your data is stored. The AWS CLI, boto3 and the Go SDK read it; the JavaScript and Java SDKs read `AWS_REGION` instead, so for those set that too. | +| `AWS_REGION` | Unused by the data proxy, which accepts any region, but set it anyway: many SDKs won't sign requests without one. It doesn't say where your data is stored. | | `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | | `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | | `AWS_ROLE_ARN` | How much the credentials may do. `FullAccess` is everything the service account may do; `ReadOnly` is reads only. The value has the shape AWS tools expect, with the service account's ID where an AWS account number would be. | @@ -137,11 +137,16 @@ With boto3: ```python import boto3 # boto3 1.28 (botocore 1.31) or later -# No keys, endpoint or region here: boto3 reads the five variables. +# No keys or endpoint here: boto3 reads them from the variables above. s3 = boto3.client("s3") s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") ``` +boto3 is the exception among the SDKs: it reads its region from +`AWS_DEFAULT_REGION`, not `AWS_REGION`. Set that as well +(`export AWS_DEFAULT_REGION=us-west-2`), or pass `region_name` to +`boto3.client`. + Other AWS SDKs work the same way, provided their web identity credential provider reads `AWS_ENDPOINT_URL_STS`. From 75cdcbbc3cfc69213730044efd45b4c3a8dd207c Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:13:40 -0700 Subject: [PATCH 17/29] docs: drop the files/folders/objects list from the upload index Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/index.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/using-source/upload/index.md b/docs/using-source/upload/index.md index 7b75cd6..05b944e 100644 --- a/docs/using-source/upload/index.md +++ b/docs/using-source/upload/index.md @@ -59,12 +59,6 @@ s3://your-org/your-product/ This matches your product page URL, `https://source.coop/your-org/your-product`. -You may upload: - -- Files -- Folders -- Multiple objects - You may not upload outside this path. ## Why we use these controls From 91e4c9842db35e815ff35b7cc679298cc914ff39 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:14:02 -0700 Subject: [PATCH 18/29] docs: drop the 'Why we use these controls' section from the upload index Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/index.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/docs/using-source/upload/index.md b/docs/using-source/upload/index.md index 05b944e..0d33268 100644 --- a/docs/using-source/upload/index.md +++ b/docs/using-source/upload/index.md @@ -61,15 +61,6 @@ This matches your product page URL, `https://source.coop/your-org/your-product`. You may not upload outside this path. -## Why we use these controls - -These controls prevent accidental or unauthorized access and ensure: - -- Clear ownership of uploaded data -- No accidental access to other providers’ data - -This protects both you and Source Cooperative. - ## What not to do Please do not: From 024439f7b4da2a0e7fbb45e20f2bbe7a59752e03 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:14:10 -0700 Subject: [PATCH 19/29] docs: drop the 'What not to do' section from the upload index Co-Authored-By: Claude Opus 5.5 --- docs/using-source/upload/index.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/using-source/upload/index.md b/docs/using-source/upload/index.md index 0d33268..3a52a85 100644 --- a/docs/using-source/upload/index.md +++ b/docs/using-source/upload/index.md @@ -61,14 +61,6 @@ This matches your product page URL, `https://source.coop/your-org/your-product`. You may not upload outside this path. -## What not to do - -Please do not: - -- Ask for full bucket access -- Upload outside your assigned prefix -- Reuse expired temporary credentials - ## Need help? If you’re unsure which option is right for you, or need help setting up automated access, contact the Source Cooperative team at `hello@source.coop`, and we will help you choose the best approach for your use case. From bd6acab6db8e61f1e4e40f273b4ed7d460f5f824 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Sat, 3 Oct 2026 00:16:42 -0700 Subject: [PATCH 20/29] docs: list Direct Uploads (Deprecated) in the sidebar Co-Authored-By: Claude Opus 5.5 --- sidebars.ts | 1 + 1 file changed, 1 insertion(+) diff --git a/sidebars.ts b/sidebars.ts index 9d62a51..3d17c8d 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -38,6 +38,7 @@ const sidebars: SidebarsConfig = { 'using-source/upload/in-the-browser', 'using-source/upload/with-the-cli', 'using-source/upload/automated-access', + 'using-source/upload/direct-uploads', ], }, 'using-source/bring-your-own-bucket', From 9fa7344b0e33d019bd3d91e7f242289d277dabec Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 5 Oct 2026 17:07:28 -0700 Subject: [PATCH 21/29] Simplify --- docs/using-source/upload/automated-access.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/using-source/upload/automated-access.md b/docs/using-source/upload/automated-access.md index df374f5..e8b1038 100644 --- a/docs/using-source/upload/automated-access.md +++ b/docs/using-source/upload/automated-access.md @@ -53,7 +53,7 @@ maintainers. either later. 5. Under **What it can reach**, click **Grant a product**, choose a product and **Read** or **Read and write**, and click **Grant**. Repeat for each - product the job needs, and no more: a job that only downloads needs **Read**. + product the job needs, and no more. 6. Click **Create service account**. You land on the service account's page, showing the key if you added one. You From 85d25c03466786e02cef8c0316b355c7847a1c2e Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 5 Oct 2026 17:10:58 -0700 Subject: [PATCH 22/29] Prefer account rather than org --- docs/using-source/upload/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/using-source/upload/index.md b/docs/using-source/upload/index.md index 3a52a85..3e698b6 100644 --- a/docs/using-source/upload/index.md +++ b/docs/using-source/upload/index.md @@ -45,8 +45,8 @@ That is why these guides use AWS tools: - **Your upload credentials come from Source Cooperative, not AWS.** They use the AWS credential format (access key ID, secret access key, session token) because S3 clients expect it, but they are only valid at the Source data proxy. -- **Addresses are proxy addresses.** `s3://your-org/your-product/` means the - `your-product` product in the `your-org` account on Source Cooperative. It +- **Addresses are proxy addresses.** `s3://account-id/product-id/` means the + `product-id` product in the `account-id` account on Source Cooperative. It is not an Amazon S3 bucket. Where your data is physically stored doesn't change it. ## Where to upload your data @@ -54,10 +54,10 @@ That is why these guides use AWS tools: You may upload only to your product's prefix on the data proxy, for example: ``` -s3://your-org/your-product/ +s3://account-id/product-id/ ``` -This matches your product page URL, `https://source.coop/your-org/your-product`. +This matches your product page URL, `https://source.coop/account-id/product-id`. You may not upload outside this path. From c3193d5c249831b87d28bb6f5b6014cc5bef787a Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 5 Oct 2026 17:17:23 -0700 Subject: [PATCH 23/29] Add windows focused examples --- ...ated-access.md => with-service-account.md} | 44 ++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) rename docs/using-source/upload/{automated-access.md => with-service-account.md} (92%) diff --git a/docs/using-source/upload/automated-access.md b/docs/using-source/upload/with-service-account.md similarity index 92% rename from docs/using-source/upload/automated-access.md rename to docs/using-source/upload/with-service-account.md index e8b1038..caa6e33 100644 --- a/docs/using-source/upload/automated-access.md +++ b/docs/using-source/upload/with-service-account.md @@ -5,6 +5,9 @@ id: automated-access slug: /automated-access --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + Some software reaches Source Cooperative with nobody at the keyboard: a nightly sync, a publishing pipeline, an instrument that uploads its readings. Such software signs in as a **service account**, in one of two ways: @@ -86,15 +89,34 @@ You can't issue a key to a disabled service account. Save the key in a file that only the job's user can read. A trailing newline is fine. + + + ```bash mkdir -p ~/.source-coop && chmod 700 ~/.source-coop (umask 077; cat > ~/.source-coop/nightly-sync.key) # paste the key, press Enter, then Ctrl-D ``` + + + +```powershell +$dir = "$HOME\.source-coop" +New-Item -ItemType Directory -Force $dir | Out-Null +icacls $dir /inheritance:r /grant:r "${env:USERNAME}:(OI)(CI)F" | Out-Null +Read-Host "Paste the key" | Set-Content -NoNewline "$dir\nightly-sync.key" +``` + + + + Then set five environment variables wherever the job runs. The key's row on the service account's page lists them too, under **Example usage** in its menu, filled in for your service account apart from the key file's path: + + + ```bash export AWS_REGION=us-west-2 export AWS_ENDPOINT_URL_S3=https://data.source.coop @@ -103,13 +125,33 @@ export AWS_ROLE_ARN=arn:aws:iam::your-org--nightly-sync:role/FullAccess export AWS_WEB_IDENTITY_TOKEN_FILE=/home/your-user/.source-coop/nightly-sync.key ``` + + + +```powershell +# Saved for your user, so scheduled tasks see them too. Open a new terminal afterwards. +$vars = @{ + AWS_REGION = "us-west-2" + AWS_ENDPOINT_URL_S3 = "https://data.source.coop" + AWS_ENDPOINT_URL_STS = "https://data.source.coop/.sts" + AWS_ROLE_ARN = "arn:aws:iam::your-org--nightly-sync:role/FullAccess" + AWS_WEB_IDENTITY_TOKEN_FILE = "C:\Users\your-user\.source-coop\nightly-sync.key" +} +$vars.GetEnumerator() | ForEach-Object { + [Environment]::SetEnvironmentVariable($_.Key, $_.Value, "User") +} +``` + + + + | Variable | What it's for | | --- | --- | | `AWS_REGION` | Unused by the data proxy, which accepts any region, but set it anyway: many SDKs won't sign requests without one. It doesn't say where your data is stored. | | `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | | `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | | `AWS_ROLE_ARN` | How much the credentials may do. `FullAccess` is everything the service account may do; `ReadOnly` is reads only. The value has the shape AWS tools expect, with the service account's ID where an AWS account number would be. | -| `AWS_WEB_IDENTITY_TOKEN_FILE` | The file that holds the key, as an absolute path. Write it out in full: cron and systemd don't expand `$HOME` or `~`. | +| `AWS_WEB_IDENTITY_TOKEN_FILE` | The file that holds the key, as an absolute path. Write it out in full: cron, systemd and Windows Task Scheduler don't expand `$HOME` or `~`. | For a job that only reads, end `AWS_ROLE_ARN` in `role/ReadOnly` instead of `role/FullAccess`. Its credentials can't write, even to products the service From f29e113044ad04b15e99a9fc71f50709d40be2a1 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 5 Oct 2026 17:17:31 -0700 Subject: [PATCH 24/29] Rename file --- docs/using-source/upload/with-service-account.md | 2 +- sidebars.ts | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/using-source/upload/with-service-account.md b/docs/using-source/upload/with-service-account.md index caa6e33..c255a10 100644 --- a/docs/using-source/upload/with-service-account.md +++ b/docs/using-source/upload/with-service-account.md @@ -1,7 +1,7 @@ --- title: Automated Access sidebar_label: With a Service Account -id: automated-access +id: with-service-account slug: /automated-access --- diff --git a/sidebars.ts b/sidebars.ts index 3d17c8d..2e1cf70 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -37,7 +37,7 @@ const sidebars: SidebarsConfig = { items: [ 'using-source/upload/in-the-browser', 'using-source/upload/with-the-cli', - 'using-source/upload/automated-access', + 'using-source/upload/with-service-account', 'using-source/upload/direct-uploads', ], }, From 216ae59fcb03e138999a1b8c89c85a44cf0b8d29 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 5 Oct 2026 18:37:14 -0700 Subject: [PATCH 25/29] Reorganize, format --- .../upload/with-service-account.md | 168 +++++++++--------- 1 file changed, 81 insertions(+), 87 deletions(-) diff --git a/docs/using-source/upload/with-service-account.md b/docs/using-source/upload/with-service-account.md index c255a10..fd92d88 100644 --- a/docs/using-source/upload/with-service-account.md +++ b/docs/using-source/upload/with-service-account.md @@ -63,6 +63,75 @@ You land on the service account's page, showing the key if you added one. You can change what it reaches at any time under **Can reach**; each change is saved as you make it. +## GitHub Actions + +A service account can trust a GitHub Actions workflow, which then signs in with +a token GitHub issues for each run. There is no key to store, rotate or leak. + +### Trust a workflow + +1. On the service account's page, under **Signs in with**, click **Add + sign-in** and choose **GitHub workflow**. +2. Enter the **Repository**, such as `your-org/pipelines`. Repositories created + after July 2026 carry their owner's and their own numeric IDs, as in + `your-org@123/pipelines@456`. The dialog offers that form when GitHub shows the + repository publicly; for a private one, it prints a `gh` command that finds it. +3. Under **Allow runs from**, choose one of the following: + - **Branch**, and provide the repository branch name that will trigger the the workflows run. + - **Tag**, and provide the tag name that will trigger the workflow. + - **Environment**, and provide the name of the GitHub environment the job runs in. +4. Click **Trust it**. + +The dialog shows the exact subject it trusts, such as +`repo:your-org/pipelines:ref:refs/heads/main`. + +A repository is trusted only for the ref or environment you name. To trust +another branch, add another workflow. Wildcards (`*`) are not supported. + +### Sign in from the job + +Choose **Example usage** from the workflow's menu on the service account's page +for the step to add, filled in for your service account. It looks like this: + +```yaml +jobs: + publish: + runs-on: ubuntu-latest + permissions: + id-token: write # lets the job ask GitHub for its token + contents: read + env: + AWS_ENDPOINT_URL_S3: https://data.source.coop + steps: + - name: Sign in to Source Cooperative as your-org--nightly-sync + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: arn:aws:iam::your-org--nightly-sync:role/FullAccess + audience: https://data.source.coop + sts-endpoint: https://data.source.coop/.sts + aws-region: us-west-2 + - run: aws s3 sync ./outgoing s3://your-org/your-product/outgoing/ +``` + +Unlike with an API key, the ID in `role-to-assume` matters: it names the service +account the job signs in as, and the sign-in succeeds only if that service +account trusts the workflow. End it in `role/ReadOnly` for a job that only +reads. + +The step's credentials last an hour, and nothing renews them during the job. For +a longer job, add `role-duration-seconds` to the step, up to `43200` (12 hours). + +A refused sign-in fails the step with: + +```text +AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity (request id 8f3a1c2b9d4e5f60-SEA) +``` + +The same error covers a workflow the service account doesn't trust (check the +subject, as above), a disabled service account, and an ID in `role-to-assume` +that isn't a service account. If none of those explains it, email +[hello@source.coop](mailto:hello@source.coop) and quote the request id. + ## API keys An API key lets software on your own server, VM, cluster or instrument sign in @@ -145,13 +214,13 @@ $vars.GetEnumerator() | ForEach-Object { -| Variable | What it's for | -| --- | --- | -| `AWS_REGION` | Unused by the data proxy, which accepts any region, but set it anyway: many SDKs won't sign requests without one. It doesn't say where your data is stored. | -| `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | -| `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | -| `AWS_ROLE_ARN` | How much the credentials may do. `FullAccess` is everything the service account may do; `ReadOnly` is reads only. The value has the shape AWS tools expect, with the service account's ID where an AWS account number would be. | -| `AWS_WEB_IDENTITY_TOKEN_FILE` | The file that holds the key, as an absolute path. Write it out in full: cron, systemd and Windows Task Scheduler don't expand `$HOME` or `~`. | +| Variable | What it's for | +| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `AWS_REGION` | Unused by the data proxy, which accepts any region, but set it anyway: many SDKs won't sign requests without one. It doesn't say where your data is stored. | +| `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | +| `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | +| `AWS_ROLE_ARN` | References the service account ID and how much the credentials may do. | +| `AWS_WEB_IDENTITY_TOKEN_FILE` | The file that holds the key, as an absolute path. Write it out in full: cron, systemd and Windows Task Scheduler don't expand `$HOME` or `~`. | For a job that only reads, end `AWS_ROLE_ARN` in `role/ReadOnly` instead of `role/FullAccess`. Its credentials can't write, even to products the service @@ -290,11 +359,11 @@ out. ## Revoke a key, or stop a service account -| To | Do this | What happens | -| --- | --- | --- | -| Stop one key | Choose **Revoke** from the key's menu. | New exchanges with the key are refused within about a minute. | -| Stop one workflow | Choose **Remove** from the workflow's menu. | New sign-ins from the workflow are refused within about a minute. | -| Stop everything the service account does | Click **Disable** under **Danger zone**. | New exchanges with any of its keys, and new sign-ins from its workflows, are refused within about a minute. Writes stop within about a minute, and reads of restricted products within about five minutes. | +| To | Do this | What happens | +| ---------------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Stop one key | Choose **Revoke** from the key's menu. | New exchanges with the key are refused within about a minute. | +| Stop one workflow | Choose **Remove** from the workflow's menu. | New sign-ins from the workflow are refused within about a minute. | +| Stop everything the service account does | Click **Disable** under **Danger zone**. | New exchanges with any of its keys, and new sign-ins from its workflows, are refused within about a minute. Writes stop within about a minute, and reads of restricted products within about five minutes. | Revoking a key doesn't recall credentials already issued with it: they keep working until they expire, an hour after they were issued, or up to 12 hours if @@ -328,78 +397,3 @@ it stop within about a minute. Send the key in the body, never in the URL. We plan to revoke keys pushed to public GitHub repositories automatically, through GitHub's secret scanning. - -## GitHub Actions - -A service account can trust a GitHub Actions workflow, which then signs in with -a token GitHub issues for each run. There is no key to store, rotate or leak. - -### Trust a workflow - -1. On the service account's page, under **Signs in with**, click **Add - sign-in** and choose **GitHub workflow**. -2. Enter the **Repository**, such as `your-org/pipelines`. -3. Under **Pinned to**, choose **Ref** and enter the full ref the workflow runs - on, such as `refs/heads/main` or `refs/tags/v1.0`; or choose **Environment** - and enter the name of the GitHub environment the job runs in. -4. Click **Trust it**. - -The dialog shows the exact subject it trusts, such as -`repo:your-org/pipelines:ref:refs/heads/main`. A run's token has to carry that -subject exactly, so: - -- A job that names an `environment:` carries the environment's subject, not - the branch's. Pin such a job to its environment. -- Runs for pull requests carry a subject that can't be trusted, so a job - triggered by `pull_request` can't sign in. -- Repositories created after July 2026 carry their owner's and their own - numeric IDs, as in `your-org@123/pipelines@456`. The dialog offers that form - when GitHub shows the repository publicly; for a private one, it prints a `gh` - command that finds it. - -A repository is trusted only for the ref or environment you name. To trust -another branch, add another workflow. - -### Sign in from the job - -Choose **Example usage** from the workflow's menu on the service account's page -for the step to add, filled in for your service account. It looks like this: - -```yaml -jobs: - publish: - runs-on: ubuntu-latest - permissions: - id-token: write # lets the job ask GitHub for its token - contents: read - env: - AWS_ENDPOINT_URL_S3: https://data.source.coop - steps: - - name: Sign in to Source Cooperative as your-org--nightly-sync - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: arn:aws:iam::your-org--nightly-sync:role/FullAccess - audience: https://data.source.coop - sts-endpoint: https://data.source.coop/.sts - aws-region: us-west-2 - - run: aws s3 sync ./outgoing s3://your-org/your-product/outgoing/ -``` - -Unlike with an API key, the ID in `role-to-assume` matters: it names the service -account the job signs in as, and the sign-in succeeds only if that service -account trusts the workflow. End it in `role/ReadOnly` for a job that only -reads. - -The step's credentials last an hour, and nothing renews them during the job. For -a longer job, add `role-duration-seconds` to the step, up to `43200` (12 hours). - -A refused sign-in fails the step with: - -```text -AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity (request id 8f3a1c2b9d4e5f60-SEA) -``` - -The same error covers a workflow the service account doesn't trust (check the -subject, as above), a disabled service account, and an ID in `role-to-assume` -that isn't a service account. If none of those explains it, email -[hello@source.coop](mailto:hello@source.coop) and quote the request id. From 23e846f85ac90e31f5dbf7fe49771aace8e7b74b Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 5 Oct 2026 18:49:26 -0700 Subject: [PATCH 26/29] Simplify --- docs/using-source/upload/with-service-account.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/docs/using-source/upload/with-service-account.md b/docs/using-source/upload/with-service-account.md index fd92d88..311f163 100644 --- a/docs/using-source/upload/with-service-account.md +++ b/docs/using-source/upload/with-service-account.md @@ -269,14 +269,12 @@ variables in its own environment. A key that has been revoked or has expired, and a key whose service account is disabled, are refused with the same error: -```text +``` An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: API key was not accepted (request id 8f3a1c2b9d4e5f60-SEA) ``` The service account's page shows whether it is disabled, and each key's row -shows whether the key has been revoked or has expired. If none of those explains -it, email [hello@source.coop](mailto:hello@source.coop) and quote the request -id: it lets us find the reason in our logs. +shows whether the key has been revoked or has expired. A key's last six characters are a checksum of the rest, so a key that was cut short or mistyped when it was copied is refused before anything is looked up, @@ -287,8 +285,7 @@ An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdent ``` Copy the key again from where you saved it. If you no longer have all of it, -issue a new key and revoke the old one. The Source CLI checks a key file the -same way before it sends the key. +issue a new key and revoke the old one. ### Keep the key secret From 9701cf314c6d5240dd37e3e38870a7a3910d4f3d Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 5 Oct 2026 18:51:59 -0700 Subject: [PATCH 27/29] line-wrap plain-text code blocks --- src/css/custom.css | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/css/custom.css b/src/css/custom.css index c54a6fa..47c7ccb 100644 --- a/src/css/custom.css +++ b/src/css/custom.css @@ -102,3 +102,9 @@ height: auto; } + +/* wrap plain-text blocks (error messages, output); commands keep scrolling */ +pre.language-text code { + white-space: pre-wrap; + overflow-wrap: anywhere; +} From 78f58e438cc9085f47220fd4957d8b2c7f2bd082 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 5 Oct 2026 18:52:12 -0700 Subject: [PATCH 28/29] Add more windows friendly docs --- docs/using-source/bring-your-own-bucket.md | 4 +- docs/using-source/upload/direct-uploads.md | 15 ++------ .../upload/with-service-account.md | 27 +++++++++++--- docs/using-source/upload/with-the-cli.md | 37 ++++++++++++++++++- 4 files changed, 63 insertions(+), 20 deletions(-) diff --git a/docs/using-source/bring-your-own-bucket.md b/docs/using-source/bring-your-own-bucket.md index 9a9b1fb..f4c636a 100644 --- a/docs/using-source/bring-your-own-bucket.md +++ b/docs/using-source/bring-your-own-bucket.md @@ -105,9 +105,7 @@ account, no matter how many buckets you connect. The CLI and console retrieve th TLS thumbprint automatically: ```bash -aws iam create-open-id-connect-provider \ - --url https://data.source.coop \ - --client-id-list sts.amazonaws.com +aws iam create-open-id-connect-provider --url https://data.source.coop --client-id-list sts.amazonaws.com ``` This returns the provider ARN diff --git a/docs/using-source/upload/direct-uploads.md b/docs/using-source/upload/direct-uploads.md index c430031..0051cdf 100644 --- a/docs/using-source/upload/direct-uploads.md +++ b/docs/using-source/upload/direct-uploads.md @@ -122,14 +122,8 @@ have a workflow that requires this policy. Creating the role with the AWS CLI ```bash -aws iam create-role \ - --role-name source-coop-upload \ - --assume-role-policy-document file://trust-policy.json - -aws iam put-role-policy \ - --role-name source-coop-upload \ - --policy-name source-coop-write \ - --policy-document file://s3-policy.json +aws iam create-role --role-name source-coop-upload --assume-role-policy-document file://trust-policy.json +aws iam put-role-policy --role-name source-coop-upload --policy-name source-coop-write --policy-document file://s3-policy.json ``` @@ -179,7 +173,7 @@ aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/
Assuming the role from a workstation or CI job -Let the AWS CLI do the assume-role for you — add a profile to `~/.aws/config`: +Let the AWS CLI do the assume-role for you — add a profile to `~/.aws/config` (`%USERPROFILE%\.aws\config` on Windows): ```ini [profile source-coop-upload] @@ -189,8 +183,7 @@ region = us-west-2 ``` ```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ \ - --profile source-coop-upload +aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ --profile source-coop-upload ```
diff --git a/docs/using-source/upload/with-service-account.md b/docs/using-source/upload/with-service-account.md index 311f163..455b707 100644 --- a/docs/using-source/upload/with-service-account.md +++ b/docs/using-source/upload/with-service-account.md @@ -255,14 +255,16 @@ s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") boto3 is the exception among the SDKs: it reads its region from `AWS_DEFAULT_REGION`, not `AWS_REGION`. Set that as well -(`export AWS_DEFAULT_REGION=us-west-2`), or pass `region_name` to +(`export AWS_DEFAULT_REGION=us-west-2`, or in PowerShell +`$env:AWS_DEFAULT_REGION = "us-west-2"`), or pass `region_name` to `boto3.client`. Other AWS SDKs work the same way, provided their web identity credential provider reads `AWS_ENDPOINT_URL_STS`. A cron job or a system service doesn't read your shell's profile, so set the -variables in its own environment. +variables in its own environment. On Windows, a scheduled task that runs as +your user picks up the variables saved above. ### If the key is refused @@ -375,9 +377,12 @@ before you enable the account. ### Revoke a key you found Anyone who holds a key can revoke it, without an account. If you come across one -that has leaked, in a repository, a log or a message, save it in a file and -send it in the body of this request, so it stays out of your shell history and -process list: +that has leaked, in a repository, a log or a message, send it in the body of +this request. The commands below take the key as you paste it, so it stays out of your +shell history and process list: + + + ```bash cat > found.key # paste the key, press Enter, then Ctrl-D @@ -388,6 +393,18 @@ printf '{"key":"%s"}' "$(tr -d '\n' < found.key)" | rm found.key ``` + + + +```powershell +$key = (Read-Host "Paste the key").Trim() +Invoke-RestMethod -Method Post -Uri https://source.coop/api/v1/service-account-keys/revocations ` + -ContentType "application/json" -Body (@{ key = $key } | ConvertTo-Json) +``` + + + + For a well-formed key the answer is always `204 No Content`, whether the key was live, already revoked or unknown. A live key is revoked, and new exchanges with it stop within about a minute. Send the key in the body, never in the URL. diff --git a/docs/using-source/upload/with-the-cli.md b/docs/using-source/upload/with-the-cli.md index fcc97bc..d37bcdc 100644 --- a/docs/using-source/upload/with-the-cli.md +++ b/docs/using-source/upload/with-the-cli.md @@ -5,6 +5,9 @@ id: with-the-cli slug: /upload-with-the-cli --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + For larger uploads, scripting, or programmatic access, use temporary credentials issued by Source Cooperative with an S3 client pointed at the [data proxy](/data-upload#how-uploads-work-the-source-data-proxy). @@ -27,7 +30,7 @@ This opens your browser to authenticate and caches short‑lived credentials in **3. Configure an AWS CLI profile** -Add a profile to `~/.aws/config` that calls the Source CLI to fetch credentials on demand and sends requests to the data proxy: +Add a profile to `~/.aws/config` (`%USERPROFILE%\.aws\config` on Windows) that calls the Source CLI to fetch credentials on demand and sends requests to the data proxy: ```ini [profile source-coop] @@ -48,12 +51,27 @@ aws s3 sync ./data s3://your-org/your-product/ --profile source-coop Or set it once for your shell session with the `AWS_PROFILE` environment variable: + + + ```bash export AWS_PROFILE=source-coop aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv aws s3 sync ./data s3://your-org/your-product/ ``` + + + +```powershell +$env:AWS_PROFILE = "source-coop" +aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv +aws s3 sync ./data s3://your-org/your-product/ +``` + + + + :::tip `AWS_PROFILE` is also picked up automatically by the AWS SDKs (boto3, JavaScript, Go, etc.), so your scripts can use the default credential chain instead of hardcoding access keys or endpoint details. @@ -110,6 +128,9 @@ s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") For terminal / shell usage: + + + ```bash export AWS_ACCESS_KEY_ID="ASIA..." export AWS_SECRET_ACCESS_KEY="pwEV..." @@ -118,6 +139,20 @@ export AWS_DEFAULT_REGION="us-west-2" export AWS_ENDPOINT_URL="https://data.source.coop" # add this: send requests to the data proxy ``` + + + +```powershell +$env:AWS_ACCESS_KEY_ID = "ASIA..." +$env:AWS_SECRET_ACCESS_KEY = "pwEV..." +$env:AWS_SESSION_TOKEN = "IQoJ..." +$env:AWS_DEFAULT_REGION = "us-west-2" +$env:AWS_ENDPOINT_URL = "https://data.source.coop" # add this: send requests to the data proxy +``` + + + + These credentials are: - Temporary From 536bf837a05b089693b20f926c50fe17f621a2e9 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 5 Oct 2026 19:08:33 -0700 Subject: [PATCH 29/29] Prefer account-id / product-id --- docs/using-source/data-proxy.md | 2 +- docs/using-source/upload/direct-uploads.md | 10 ++-- docs/using-source/upload/in-the-browser.md | 2 +- .../upload/with-service-account.md | 8 +-- docs/using-source/upload/with-the-cli.md | 20 +++---- src/pages/tools/iam-policy-wizard.tsx | 58 +++++++++++-------- 6 files changed, 55 insertions(+), 45 deletions(-) diff --git a/docs/using-source/data-proxy.md b/docs/using-source/data-proxy.md index 4852363..9a7314a 100644 --- a/docs/using-source/data-proxy.md +++ b/docs/using-source/data-proxy.md @@ -53,7 +53,7 @@ Public datasets work with `--no-sign-request`. To access data that requires auth ```bash source-coop login -aws s3 ls s3://your-org/your-product --profile source-coop +aws s3 ls s3://your-org/product-id --profile source-coop ``` See [Upload with the Source CLI](/upload-with-the-cli) for the one-time install and profile setup. diff --git a/docs/using-source/upload/direct-uploads.md b/docs/using-source/upload/direct-uploads.md index 0051cdf..3f6d95f 100644 --- a/docs/using-source/upload/direct-uploads.md +++ b/docs/using-source/upload/direct-uploads.md @@ -144,7 +144,7 @@ Email [hello@source.coop](mailto:hello@source.coop) with: - Role: `arn:aws:iam::123456789012:role/source-coop-upload` - User: `arn:aws:iam::123456789012:user/data-uploader` - Account: `arn:aws:iam::123456789012:root` -- Your Source Cooperative account ID and product ID (the `your-org/your-product` prefix you will write to) +- Your Source Cooperative account ID and product ID (the `your-org/product-id` prefix you will write to) - A short description of the workflow (e.g. "nightly ingestion pipeline running on ECS") We will add the ARN to the bucket policy and confirm when it is active. @@ -156,7 +156,7 @@ We will add the ARN to the bucket policy and confirm when it is active. Once we confirm, upload with credentials for that role, user, or account: ```bash -aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv +aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/product-id/mydata.csv ``` Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles) need no assume-role step — the SDK picks up the role automatically. @@ -165,7 +165,7 @@ Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles) Uploading a directory ```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ +aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/product-id/ ``` @@ -183,7 +183,7 @@ region = us-west-2 ``` ```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ --profile source-coop-upload +aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/product-id/ --profile source-coop-upload ``` @@ -198,7 +198,7 @@ s3 = boto3.client("s3") s3.upload_file( "mydata.csv", "us-west-2.opendata.source.coop", - "your-org/your-product/mydata.csv", + "your-org/product-id/mydata.csv", ) ``` diff --git a/docs/using-source/upload/in-the-browser.md b/docs/using-source/upload/in-the-browser.md index bb04d3c..e0c92ad 100644 --- a/docs/using-source/upload/in-the-browser.md +++ b/docs/using-source/upload/in-the-browser.md @@ -15,7 +15,7 @@ This option is ideal for: ## How to upload via the UI -1. Go to your product page in Source Cooperative (e.g. `https://source.coop/your-org/your-product`) +1. Go to your product page in Source Cooperative (e.g. `https://source.coop/your-org/product-id`) 2. In the top-right corner of the Product Contents card, click on the lock icon to open the dropdown menu and enable edit mode by clicking on `Edit Mode`. 3. Either: - Drag‑and‑drop files to the `Product Contents` card diff --git a/docs/using-source/upload/with-service-account.md b/docs/using-source/upload/with-service-account.md index 455b707..188bde5 100644 --- a/docs/using-source/upload/with-service-account.md +++ b/docs/using-source/upload/with-service-account.md @@ -110,7 +110,7 @@ jobs: audience: https://data.source.coop sts-endpoint: https://data.source.coop/.sts aws-region: us-west-2 - - run: aws s3 sync ./outgoing s3://your-org/your-product/outgoing/ + - run: aws s3 sync ./outgoing s3://your-org/product-id/outgoing/ ``` Unlike with an API key, the ID in `role-to-assume` matters: it names the service @@ -239,8 +239,8 @@ With the AWS CLI: ```bash aws --version # aws-cli/2.13.0 or later -aws s3 ls s3://your-org/your-product/ -aws s3 sync ./outgoing s3://your-org/your-product/outgoing/ +aws s3 ls s3://your-org/product-id/ +aws s3 sync ./outgoing s3://your-org/product-id/outgoing/ ``` With boto3: @@ -250,7 +250,7 @@ import boto3 # boto3 1.28 (botocore 1.31) or later # No keys or endpoint here: boto3 reads them from the variables above. s3 = boto3.client("s3") -s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") +s3.upload_file("mydata.csv", "account-id", "product-id/mydata.csv") ``` boto3 is the exception among the SDKs: it reads its region from diff --git a/docs/using-source/upload/with-the-cli.md b/docs/using-source/upload/with-the-cli.md index d37bcdc..de685d9 100644 --- a/docs/using-source/upload/with-the-cli.md +++ b/docs/using-source/upload/with-the-cli.md @@ -45,8 +45,8 @@ The `endpoint_url` line is what sends requests to the Source data proxy instead Pass the profile per command with `--profile`: ```bash -aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv --profile source-coop -aws s3 sync ./data s3://your-org/your-product/ --profile source-coop +aws s3 cp mydata.csv s3://your-org/product-id/mydata.csv --profile source-coop +aws s3 sync ./data s3://your-org/product-id/ --profile source-coop ``` Or set it once for your shell session with the `AWS_PROFILE` environment variable: @@ -56,8 +56,8 @@ Or set it once for your shell session with the `AWS_PROFILE` environment variabl ```bash export AWS_PROFILE=source-coop -aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv -aws s3 sync ./data s3://your-org/your-product/ +aws s3 cp mydata.csv s3://your-org/product-id/mydata.csv +aws s3 sync ./data s3://your-org/product-id/ ``` @@ -65,8 +65,8 @@ aws s3 sync ./data s3://your-org/your-product/ ```powershell $env:AWS_PROFILE = "source-coop" -aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv -aws s3 sync ./data s3://your-org/your-product/ +aws s3 cp mydata.csv s3://your-org/product-id/mydata.csv +aws s3 sync ./data s3://your-org/product-id/ ``` @@ -96,7 +96,7 @@ You will also see: - `Expiration`: the expiration time of the credentials (a specific date and time) - `Bucket`: the bucket name on the data proxy, which is your account ID (`your-org`) -- `Prefix`: the prefix (folder) you are allowed to write to (e.g. `your-product/`) +- `Prefix`: the prefix (folder) you are allowed to write to (e.g. `product-id/`) ### What the credentials look like @@ -123,7 +123,7 @@ With boto3, pass the endpoint when creating the client: import boto3 s3 = boto3.client("s3", endpoint_url="https://data.source.coop", **credentials) -s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") +s3.upload_file("mydata.csv", "account-id", "product-id/mydata.csv") ``` For terminal / shell usage: @@ -162,13 +162,13 @@ These credentials are: ### Example: Upload using the AWS CLI ```bash -aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv --endpoint-url https://data.source.coop +aws s3 cp mydata.csv s3://your-org/product-id/mydata.csv --endpoint-url https://data.source.coop ``` Or upload a full directory: ```bash -aws s3 sync ./data s3://your-org/your-product/ --endpoint-url https://data.source.coop +aws s3 sync ./data s3://your-org/product-id/ --endpoint-url https://data.source.coop ``` `--endpoint-url` isn't needed if you set `AWS_ENDPOINT_URL` or use the `source-coop` profile above. diff --git a/src/pages/tools/iam-policy-wizard.tsx b/src/pages/tools/iam-policy-wizard.tsx index ace5401..d649d52 100644 --- a/src/pages/tools/iam-policy-wizard.tsx +++ b/src/pages/tools/iam-policy-wizard.tsx @@ -23,28 +23,34 @@ export default function IamPolicyWizard(): ReactNode { return ( + description="Generate the IAM policy your AWS role or user needs to upload to Source Cooperative." + >

IAM policy wizard

- Deprecated. Uploading with an IAM role in your own AWS account is deprecated - in favor of service accounts. Use this wizard only for - pipelines already set up this way. + Deprecated. Uploading with an IAM role in your own + AWS account is deprecated in favor of{" "} + service accounts. Use this wizard only + for pipelines already set up this way.

- Generate the policy to attach to the IAM role or user in your own AWS - account, so it can upload to your prefix in the Source Cooperative bucket. See{' '} + Generate the policy to attach to the IAM role or user in{" "} + your own AWS account, so it can upload to your prefix + in the Source Cooperative bucket. See{" "} Direct uploads for the full walkthrough.

-
+