diff --git a/README.md b/README.md index 5e87793..d599ec1 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,7 @@ share profiles # default a | **Menu bar app** | Share Bar shows what's shared and lets you drag a file onto the icon to publish it | | **Profiles** | `share --profile work ...` runs a second setup (another account or hostname) beside the first, on its own port and service | | **Private links** | `--access email:a@x,b@y`, `domain:example.com`, or `group:` puts a Cloudflare Access login (one-time PIN) on one link; the other links stay public. See [Private links](#private-links) | +| **One hostname per tenant** | local by default; the admin can turn R2 on at the origin (`setup --r2 --bucket `) and each link picks local or cloud (`add --cloud`). See [One hostname per tenant](#one-hostname-per-tenant) | | **Links with the machine off** | `setup --backend r2 --bucket `: snapshots live in a private R2 bucket, a Worker serves them, and a whole team publishes to one hostname. See [R2 backend](#r2-backend-links-that-stay-up-while-the-machine-is-off) | On the default tunnel backend, links are live only while the machine is awake; visitors get Cloudflare 530 when it sleeps. `https:///healthz` answers `ok` while share is serving; the site root 404s by design. @@ -91,6 +92,9 @@ share stop | start # take all links down / bring them back share teardown [--yes] # remove tunnel, service, and config; the shares on disk stay share --profile ... # any command against another setup (or SHARE_PROFILE=); share profiles lists them share setup --backend r2 --bucket # serve from a private R2 bucket through a Worker (see below) +share setup --r2 --bucket # on a tunnel profile's origin: R2 on, per-link storage (--alias folds an old r2 hostname in; --no-r2 rolls back) +share add --cloud|--local # R2 on: this link in the bucket, or on this machine +share migrate --to # move a tenant's origin (R2 off) to another machine, links kept ``` A share from a private GitHub repo prints a warning; the content is public to anyone with the link. @@ -150,6 +154,27 @@ One add holds at most 500 files (`SHARE_R2_MAX_FILES`), each at most 300 MiB. A `share --profile df teardown` on an r2 profile is local: it forgets this install's token and config, and the bucket, the Worker, and every share stay for the other publishers. `CLOUDFLARE_API_TOKEN= share --profile df teardown --yes --purge` deletes every share (gated ones with their Access apps), the Worker, its domain, and the bucket once it is empty, for everyone. Detail: [docs/how-it-works.md](docs/how-it-works.md#r2-backend). +## One hostname per tenant + +A tenant is one hostname with one shared list. By default every link is served from the origin machine's disk through its tunnel, exactly as above. The tenant admin can turn R2 on at the origin; after that each link picks local or cloud, and a teammate can publish from another machine. + +```sh +CLOUDFLARE_API_TOKEN= share setup s.example.com --r2 --bucket share-s # on the origin, once +share add ./report.pdf # the tenant default (local unless --storage-default cloud) +share add --cloud ./report.pdf # in the bucket: answers while the origin is off +share add --local ./notes # on the origin's disk +``` + +| | R2 off (default) | R2 on | +|---|---|---| +| Hostname | one | the same one; storage is a badge on the row | +| Link while the origin is off | Cloudflare 530 | cloud links serve; machine links answer a 503 offline page | +| Who publishes | the origin | the origin (local or cloud), members (cloud only) | +| Live dev servers, `--host` | yes | yes, on the origin | +| Rollback | none needed | `setup --no-r2` deletes the route; the tunnel serves alone again | + +`setup --r2 --bucket --alias ` folds a former r2 hostname into the tenant: the old name answers 301 to the same path and each gated link keeps its Access app. `share migrate --to ` moves an R2-off tenant to another always-on machine with every link kept, and prints the rollback line if a step fails. Share Bar shows one section per tenant, and each row carries a storage badge, a file-type icon, the link type, and a lock for gated links. Admin steps, tokens, and rollbacks: [docs/setup.md](docs/setup.md#4h-one-hostname-per-tenant-r2-on-the-origin-the-alias-fold-moving-the-origin); the model: [docs/how-it-works.md](docs/how-it-works.md#tenants-one-hostname-local-or-cloud) and [ADR-0008](docs/decisions/ADR-0008-one-host-per-tenant.md). + ## Menu bar app ```sh @@ -180,7 +205,7 @@ CLI call and action. ## Docs -- [docs/setup.md](docs/setup.md): API token setup (no browser), the R2 backend and teammate onboarding, moving machines, troubleshooting +- [docs/setup.md](docs/setup.md): API token setup (no browser), the R2 backend and teammate onboarding, a tenant's R2 and alias fold, moving machines, troubleshooting - [docs/how-it-works.md](docs/how-it-works.md): architecture, config keys, security model, the R2 data model and Worker, testing ## License diff --git a/bin/share b/bin/share index 763510c..ec5b9ac 100755 --- a/bin/share +++ b/bin/share @@ -9,6 +9,7 @@ # share add live proxy at https://// # share add ... --host . give the share its own hostname # share add ... --access group:|email:[,...]|domain: a login gate (Cloudflare Access) on this link +# share add --cloud|--local R2 on: this link in the bucket, or on this machine (default: storage_default) # share api-token [--cmd '' | --check] the Cloudflare API token --access needs: prefilled form + paste, or point at one you have # share ls list shares: link, size, source, expiry # share refresh re-copy from the source path (same link) @@ -23,6 +24,10 @@ # share serve serve in the foreground until Ctrl-C # share skill [--install] the SKILL.md that teaches an agent to use share # share teardown [--yes] delete the tunnel, DNS record, service, and stored token +# share import ... used by migrate over ssh +# share migrate --to move this tenant's origin to another machine (snapshots only, R2 off) +# share setup --r2 --bucket [--storage-default local|cloud] [--alias ] on the origin: R2 on, per-link storage +# share setup --no-r2 on the origin: R2 off again (refuses while an alias is set) # share setup --backend r2 --bucket an R2 profile: a private bucket and a Worker # serve links while this machine is off; teardown --yes --purge (admin token) deletes it all # @@ -70,6 +75,8 @@ fi hosts="${SHARE_HOSTS:-$(cfg hosts || true)}" tunnel="${SHARE_TUNNEL:-1}" # 0 = local-only (tests) mode="$(cfg mode || true)" # "quick" = trycloudflare, no domain; empty = named tunnel +r2on=0 # a tunnel profile with bucket= is a tenant origin with R2 on: local links get pointer records, --cloud adds go to the bucket +[[ $backend != r2 && -n "$(cfg bucket || true)" ]] && r2on=1 api="https://api.cloudflare.com/client/v4" this_host() { local h; h="$(uname -n)"; echo "${h%%.*}"; } @@ -79,10 +86,11 @@ running() { # the pid must still be a share serve: an unclean exit leaves the fi local pid; pid="$(cat "$pidfile")" kill -0 "$pid" 2>/dev/null && [[ $(ps -o command= -p "$pid" 2>/dev/null) == *share*serve* ]] } -rows() { # the only index reader: a forged row (bad id, main or malformed host=, a bad access= or access_rule=) never reaches a path, Caddyfile, or Cloudflare call - [[ -f $index ]] || return 0 +rows() { rows_in "$index" "$backend"; } # the only index reader +rows_in() { # rows_in : a forged row (bad id, main or malformed host=, a bad access= or access_rule=) never reaches a path, Caddyfile, or Cloudflare call + [[ -f $1 ]] || return 0 # no {n} intervals: mawk lacks them; the host regex is setup's hostname check; the rule grammar is access_parse's - awk -F'\t' -v main="$host_name" -v be="$backend" ' + awk -F'\t' -v main="$host_name" -v be="$2" ' function hostname(h) { return h ~ /^[a-z0-9]([a-z0-9.-]*[a-z0-9])?\.[a-z][a-z]+$/ } function uuid(u) { return length(u) == 36 && u !~ /[^0-9a-f-]/ && substr(u, 9, 1) == "-" && substr(u, 14, 1) == "-" && substr(u, 19, 1) == "-" && substr(u, 24, 1) == "-" && gsub(/-/, "-", u) == 4 } function rule(r, k, v, m, e, j, at) { @@ -124,9 +132,11 @@ rows() { # the only index reader: a forged row (bad id, main or malformed host=, } if (app != rl || !hp || !hb) next # a gated row carries both, or it is a hand edit; an r2 row needs its prefix and host print - }' "$index" + }' "$1" } row() { rows | awk -F'\t' -v id="$1" '$1 == id'; } +crows() { rows_in "${r2_snap_file:-/dev/null}" r2; } # the cloud rows of this process's bucket read; on an origin its own file, never $index +crow() { crows | awk -F'\t' -v id="$1" '$1 == id'; } id_of() { # accept an id or a pasted share link: https://host//name -> ; a bare https:/// resolves to the host row case ${1:-} in *://*) @@ -152,23 +162,28 @@ rand_id() { # loops past a collision with an existing row or an orphaned pub/"* ]] && continue # leftover objects under this id: never hand it out + if [[ $backend == r2 ]]; then + r2_id_taken "$id" && continue ! grep -q "^$id"$'\t' "$pending" 2>/dev/null && { echo "$id"; return; } continue fi # an id named in access-pending has an Access app waiting: a new share must never land behind it - [[ -z "$(row "$id")" && ! -e $pub/$id ]] && ! grep -q "^$id"$'\t' "$pending" 2>/dev/null && { echo "$id"; return; } + if [[ -n "$(row "$id")" || -e $pub/$id ]] || grep -q "^$id"$'\t' "$pending" 2>/dev/null; then continue; fi + [[ $r2on == 1 ]] && r2_id_taken "$id" && continue # an origin with R2 on: one namespace with the bucket + echo "$id"; return done } +r2_id_taken() { # r2_id_taken : a live GET and a prefix scan; anything but 200/404 or empty fails closed + r2_call GET "m/$1" || die "the r2 record check needs a publisher token (see above)" + case $cf_code in + 200) return 0 ;; + 404) ;; + *) die "the r2 record check for $1 failed (HTTP $cf_code); no id was picked" ;; + esac + r2_call LIST "o/$1." + [[ $cf_code == 200 ]] || die "the r2 prefix check for $1 failed (HTTP $cf_code); no id was picked" + [[ $cf_body == *""* ]] # leftover objects under this id: never hand it out +} has_opt() { case " $2 " in *" $1 "*) return 0 ;; esac; return 1; } opt_val() { # opt_val : the value of a key=value token, nothing when absent local rest="${2#*"$1"=}" @@ -222,6 +237,7 @@ release_locks() { # the EXIT trap: every held lock, a stage still outside pub, a if [[ -n $held_fqdn ]]; then f="$held_fqdn"; held_fqdn=""; host_rm "$f" || true; fi if [[ -n $held_access_id ]]; then echo "share: nothing was published; the Access app of $held_access_id waits in access-pending, the next $me add --access or $me prune with a token removes it" >&2; held_access_id=""; fi [[ -n ${r2_snap_file:-} ]] && rm -f "$r2_snap_file" + [[ -n ${import_spool:-} ]] && rm -f "$import_spool" if [[ -n ${r2_held_prefix:-} ]]; then f="$r2_held_prefix"; r2_held_prefix=""; r2_release_prefix "$r2_held_id" "$f"; fi return 0 } @@ -248,7 +264,7 @@ share_url() { # share_url [opts]: sets share_url_out; the link depen fi path="$2" local base="$pub/$1" - [[ $backend != r2 ]] || base="${r2_link_base:-}" # r2: the stage during an add; a listing has none, and the Worker 308s a folder to its slash + [[ $backend != r2 && -z ${r2_link_base:-} ]] || base="${r2_link_base:-}" # r2 or a cloud add on an origin: the stage during an add; a listing has none, and the Worker 308s a folder to its slash if [[ -z $base ]]; then : elif [[ -d $base/$2 ]]; then path="$2/" elif [[ $2 == *.md && -f $base/${2%.md}.html ]]; then path="${2%.md}.html" @@ -447,19 +463,32 @@ parse_target() { # parse_target : "live " for a port/localhost shape, } cmd_add() { - local ttl="${SHARE_TTL:-$(cfg ttl || true)}"; ttl="${ttl:-30d}" noindex="" host_fqdn="" target="" ntargets=0 access="" - local usage="usage: share add [--ttl 7d] [--host .] [--no-index] [--access ] " + local ttl="${SHARE_TTL:-$(cfg ttl || true)}"; ttl="${ttl:-30d}" noindex="" host_fqdn="" target="" ntargets=0 access="" storage="" + local usage="usage: share add [--ttl 7d] [--host .] [--no-index] [--access ] [--cloud|--local] " while [[ $# -gt 0 ]]; do case $1 in --ttl) ttl="${2:-}"; shift 2 || die "$usage" ;; --host) host_fqdn="${2:-}"; shift 2 || die "$usage" ;; --access) access="${2:-}"; shift 2 || die "$usage"; [[ -n $access ]] || die "$access_usage" ;; --no-index) noindex=1; shift ;; + --cloud|--local) [[ -z $storage || $storage == "${1#--}" ]] || die "$usage"; storage="${1#--}"; shift ;; --*) die "$usage" ;; *) target="$1"; ntargets=$((ntargets + 1)); shift ;; esac done [[ $ntargets == 1 ]] || die "$usage" + if [[ $backend != r2 && ( $r2on == 1 || -n $storage ) ]]; then # the storage of this link, settled before any write + local pt0; pt0="$(parse_target "$target")" || die "$usage" + if [[ $storage == cloud ]]; then + [[ -z $host_fqdn && $pt0 == snapshot ]] || die "--cloud: live links and --host stay on the origin's tunnel" + [[ $r2on == 1 ]] || die "R2 is off for ${host_name:-this profile}; the tenant admin enables it with '$me setup ${host_name:-} --r2 --bucket '" + elif [[ -z $storage && $r2on == 1 && -z $host_fqdn && $pt0 == snapshot ]]; then + storage="$(cfg storage_default || true)"; storage="${storage:-local}" + [[ $storage == local || $storage == cloud ]] || die "storage_default=$storage in $config is not local or cloud" + fi + elif [[ $backend == r2 && $storage == local ]]; then + die "$host_name serves local links from its origin; this machine publishes cloud links only" + fi [[ -z $host_fqdn || $mode != quick ]] || die "--host needs a named tunnel: '$me teardown', then '$me setup '" if [[ -n $access ]]; then # every refusal before any write access_parse "$access" @@ -468,7 +497,7 @@ cmd_add() { fi local pt kind tport="" src name id exp=0 url opts="" secs pt="$(parse_target "$target")" || die "$usage" - if [[ $backend == r2 ]]; then # every r2 refusal before any write + if [[ $backend == r2 || $storage == cloud ]]; then # every r2 refusal before any write; a cloud add on an origin runs the same path [[ $pt == snapshot && -z $host_fqdn ]] || die "live shares and --host need a tunnel profile; this profile serves from R2" need_host cmd_add_r2 "$(ttl_secs "$ttl")" "$noindex" "$target" "$access" @@ -502,6 +531,7 @@ cmd_add() { host_check "$host_fqdn" opts="${opts:+$opts }host=$host_fqdn" fi + if [[ $r2on == 1 ]]; then r2_token_check; fi # R2 on: a local link needs its pointer, so no token means nothing is written id="$(rand_id)" [[ $secs -gt 0 ]] && exp=$(($(date +%s) + secs)) mk_root; mkdir -p "$pub" @@ -512,9 +542,11 @@ cmd_add() { curl -s -o /dev/null --max-time 2 "http://127.0.0.1:$tport/" || echo "share: nothing answers on 127.0.0.1:$tport yet" >&2 else stage_copy "$src" "$opts" - # a gated copy stays outside pub until the gate is observed; warn_private says "public to anyone with the link", which a gated link is not - if [[ -z $access ]]; then stage_publish "$id"; warn_private "$src" "$id"; fi fi + # R2 on: the pointer reserves the id in the bucket before any byte is served and before the Access app; a 412 takes a fresh id + if [[ $r2on == 1 ]]; then r2_pointer_put "$kind" "$name" "$src" "$exp" "$opts" "$access"; fi + # a gated copy stays outside pub until the gate is observed; warn_private says "public to anyone with the link", which a gated link is not + if [[ $kind == snapshot && -z $access ]]; then stage_publish "$id"; warn_private "$src" "$id"; fi if [[ -n $access ]]; then access_create "$id" "$host_fqdn" # intent line, the app, its id into the line, read-back opts="${opts:+$opts }access=$access_app access_rule=$access_rule" @@ -539,8 +571,10 @@ cmd_add() { [[ -n $access ]] && pending_drop "$id" "$access_app" held_fqdn="" held_access_id="" lock_release index # before cmd_start: serve's startup render takes the lock itself + [[ $r2on == 0 ]] || lock_release "add-$id" share_url "$id" "$name" "$opts" url="$share_url_out" + [[ -z $access ]] || access_link_check "$url" || access_link_die "$id" "$url" # Access answers at the edge, so a stopped origin does not matter here echo "$url" if [[ ${SHARE_CLIPBOARD:-1} == 1 ]] && command -v pbcopy >/dev/null; then printf '%s' "$url" | pbcopy && echo " (link copied)" @@ -552,15 +586,24 @@ cmd_add() { cmd_ls() { access_pending_note - [[ -s $index ]] || { echo "no shares"; return 0; } - local id name src date exp opts size until gone by - while IFS=$'\t' read -r id name src date exp opts; do + [[ -s $index || $backend == r2 || $r2on == 1 ]] || { echo "no shares"; return 0; } + local id name src date exp opts storage t by size until gone tag acc us=$'\x1f' + list_rows ls + if [[ -z $list_out && ( $backend == r2 || $r2on == 1 ) ]]; then + [[ -z $list_cloud_error ]] || echo "share: cloud links not listed: $list_cloud_error" >&2 + echo "no shares"; return 0 + fi + while IFS=$us read -r id name src date exp opts storage t by; do + [[ -n $id ]] || continue until="never"; [[ ${exp:-0} -gt 0 ]] && until="$(day_of "$exp")" share_url "$id" "$name" "$opts" - echo "$share_url_out" - if [[ $backend == r2 ]]; then - by="$(opt_val by "$opts" || true)" - echo " id=$id added=$date expires=$until$(access_col "$opts") by=${by:-?} from=$src" + tag=$storage; [[ $storage == machine ]] && has_opt live "$opts" && tag=live + printf '%-7s %-8s by=%s %s\n' "$tag" "$t" "$by" "$share_url_out" + if [[ $storage == cloud ]]; then + echo " id=$id added=$date expires=$until$(access_col "$opts") by=$by from=$src" + elif [[ $backend == r2 ]]; then # a member: the origin's link, read from its pointer + acc=""; has_opt gated "$opts" && acc=" access=gated" + echo " id=$id added=$date expires=$until$acc served by the tenant's origin" elif has_opt live "$opts"; then echo " id=$id live -> $src added=$date expires=$until$(access_col "$opts")" else @@ -568,18 +611,38 @@ cmd_ls() { gone=""; [[ -e $src ]] || gone=" (source gone, snapshot still served)" echo " id=$id size=${size:-?} added=$date expires=$until$(access_col "$opts") from=$src$gone" fi - done < <(rows) # a 5-field row (v0.1.x) reads with opts empty; malformed rows are skipped + done <<<"$list_out" # a 5-field row (v0.1.x) reads with opts empty; malformed rows and records are skipped + [[ -z $list_cloud_error ]] || echo "share: cloud links not listed: $list_cloud_error" >&2 access_ls_check } +state_lines() { # state_lines: list_out as the tab lines cmd_state's jq reads (id name url kind own_host expires access storage type by), this machine's rows newest first, then the bucket's + local id name src date exp opts storage t by kind oh rule line mine=() rest=() i us=$'\x1f' + while IFS=$us read -r id name src date exp opts storage t by; do + [[ -n $id ]] || continue + kind="snapshot"; has_opt live "$opts" && [[ $storage == machine ]] && kind="live" + optv host "$opts"; oh=$optv_out + [[ $storage == cloud ]] && oh="" + optv access_rule "$opts"; rule=$optv_out + [[ $storage == machine && $backend == r2 ]] && { rule=""; has_opt gated "$opts" && rule="gated"; } # a pointer carries the flag, never the rule + share_url "$id" "$name" "$opts" + printf -v line '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s' "$id" "$name" "$share_url_out" "$kind" "$oh" "${exp:-0}" "$rule" "$storage" "$t" "$by" + if [[ $storage == machine && $backend != r2 ]]; then mine+=("$line"); else rest+=("$line"); fi + done <<<"$list_out" + for ((i = ${#mine[@]} - 1; i >= 0; i--)); do printf '%s\n' "${mine[i]}"; done + [[ ${#rest[@]} == 0 ]] || printf '%s\n' "${rest[@]}" +} access_col() { local r; r="$(opt_val access_rule "$1" || true)"; [[ -n $r ]] && printf ' access=%s' "$r"; return 0; } cmd_refresh() { [[ $backend != r2 ]] || { cmd_refresh_r2 "$@"; return; } + if [[ $r2on == 1 && -z "$(row "${1:-}")" ]]; then cmd_refresh_r2 "$@"; return; fi # an origin with R2 on: an id with no local row is a cloud link local r src opts r="$(row "${1:-}")"; [[ -n $r ]] || die "no share with id '${1:-}'" opts="$(awk -F'\t' '{print $6}' <<<"$r")" if has_opt live "$opts"; then echo "live share, nothing to refresh" >&2; return 0; fi src="$(cut -f3 <<<"$r")" + optv by "$opts" # a row import moved here keeps :; that path is on the old origin + [[ -z $optv_out || $optv_out == "$(r2_by)" || $src != "$optv_out:"* ]] || die "$1 was moved from $optv_out; re-add it from a source on this machine" [[ -e $src ]] || die "source is gone: $src (the snapshot keeps serving; $me rm $1 to unpublish)" stage_copy "$src" "$opts" # the slow copy runs outside the lock; the swap into pub waits for it lock_take index @@ -594,6 +657,7 @@ cmd_refresh() { cmd_rm() { # cmd_rm [prune]: prune passes through with no credential, so expiry is never blocked (DEC-016) [[ $backend != r2 ]] || { cmd_rm_r2 "$@"; return; } + if [[ $r2on == 1 && -z "$(row "${1:-}")" ]]; then cmd_rm_r2 "$@"; return; fi # an origin with R2 on: an id with no local row is a cloud link local r opts fqdn="" app="" tmp reloaded=1 r="$(row "${1:-}")"; [[ -n $r ]] || die "no share with id '${1:-}'" opts="$(awk -F'\t' '{print $6}' <<<"$r")" @@ -621,6 +685,7 @@ cmd_rm() { # cmd_rm [prune]: prune passes through with no credential, so ex caddy_reload || reloaded=0 lock_release index echo "unpublished $1" + if [[ ${2:-} == prune ]]; then r2_pointer_drop "$1" "${prune_mode:-soft}"; else r2_pointer_drop "$1" soft; fi [[ -n $app ]] || return 0 if [[ $reloaded == 0 ]] && has_opt live "$opts"; then # the running Caddy still proxies //*: the app is the only gate left die "Caddy did not reload, so the live route for $1 may still answer; its Access app stays in access-pending; fix Caddy (see $root/caddy.log), then: $me prune" @@ -635,14 +700,19 @@ cmd_rm() { # cmd_rm [prune]: prune passes through with no credential, so ex cmd_prune() { # cmd_prune [quiet|listing]: unpublish every share past its expiry; sweep waiting Access apps when a token resolves if [[ $backend == r2 ]]; then r2_prune "${1:-}"; return 0; fi + if [[ $r2on == 1 && ${1:-} != quiet ]]; then # an origin with R2 on, interactive: cloud expiry, the sweep, and the reconcile through its own reader + local pm=soft; [[ ${1:-} != listing ]] || pm="env" + if r2_token_ok "$pm"; then ( r2_prune "${1:-}"; r2_reconcile ) || echo "share: the cloud links of $host_name were not checked (see above); local links were" >&2 + elif [[ -z ${1:-} ]]; then echo "share: the cloud links and pointers of $host_name were not checked: no publisher token ($me api-token)" >&2; fi + fi # quiet (serve, status): no token from any source, so neither touches Access; an expired gated share defers its # app. listing (ls): only an exported CLOUDFLARE_API_TOKEN, never api_token_cmd or the Keychain, so a listing # never runs `op read`. bare (the prune verb): the full resolver, soft on a failing api_token_cmd. A token for another # account warns and skips the sweep in every mode: expiry is never blocked. case ${1:-} in - quiet) api_token="" ;; - listing) api_token_resolve env ;; - *) api_token_resolve soft ;; + quiet) api_token="" prune_mode="off" ;; + listing) api_token_resolve env; prune_mode="env" ;; + *) api_token_resolve soft; prune_mode="soft" ;; esac if [[ -n $api_token && -s $pending && -n $host_name ]]; then access_account soft && access_sweep @@ -659,6 +729,7 @@ cmd_prune() { # cmd_prune [quiet|listing]: unpublish every share past its expiry cmd_hits() { [[ $backend != r2 ]] || { cmd_hits_r2 "$@"; return; } + if [[ $r2on == 1 && -z "$(row "${1:-}")" ]]; then cmd_hits_r2 "$@"; return; fi # an origin with R2 on: an id with no local row is a cloud link local r opts fqdn="" log="$root/access.log" r="$(row "${1:-}")"; [[ -n $r ]] || die "no share with id '${1:-}'" opts="$(awk -F'\t' '{print $6}' <<<"$r")" @@ -672,6 +743,345 @@ cmd_hits() { + (if ($h | length) > 0 then ", last \($h[-1].ts | floor | strflocaltime("%F %H:%M"))" else "" end)' "$log" } +# --- import (the receiving side of migrate) --- + +iusage="usage: share import --probe | share import [:] (a tar on stdin)" +b64dec() { # b64dec : the bytes into b64_out; 1 on a non-base64 string. A decoded control byte is refused by the caller + [[ $1 =~ ^[A-Za-z0-9+/]*={0,2}$ ]] || return 1 + b64_out="$(printf '%s' "$1" | base64 -d 2>/dev/null || printf '%s' "$1" | base64 -D 2>/dev/null)" || return 1 +} +import_manifest() { # import_manifest : ":"; migrate sends the same digest of pub/ + local n sum + n="$( (cd "$1" && find . -type f) | grep -c . || true)" + sum="$(cd "$1" && find . -type f -print | LC_ALL=C sort | while IFS= read -r f; do shasum -a 256 "$f"; done | shasum -a 256 | cut -d' ' -f1)" + echo "$n:$sum" +} +import_scan() { # import_scan : every member a plain file or directory under a relative path with no .. and no dotfile; measured on bsdtar 3.5 and GNU tar 1.35 + local names modes nn nm line p seg + names="$(tar -tf "$1" 2>/dev/null)" || die "import: stdin is not a tar archive; nothing was published" + modes="$(tar -tvf "$1" 2>/dev/null)" || die "import: stdin is not a tar archive; nothing was published" + nn="$(grep -c . <<<"$names" || true)"; nm="$(grep -c . <<<"$modes" || true)" + [[ $nn -gt 0 && $nn == "$nm" ]] || die "import: the archive listing does not parse (a name with a newline?); nothing was published" + while IFS= read -r line; do # -tv: the type is the first mode character; a hardlink shows "link to", a symlink "->" + case $line in [-d]*) ;; *) die "import: the archive holds a member that is not a file or a directory ($line); nothing was published" ;; esac + case $line in *" link to "*|*" -> "*) die "import: the archive holds a link ($line); nothing was published" ;; esac + done <<<"$modes" + while IFS= read -r p; do + [[ $p != /* ]] || die "import: the archive holds an absolute path ($p); nothing was published" + p="${p#./}"; p="${p%/}" + [[ -n $p && $p != . ]] || continue + while IFS= read -r -d / seg || [[ -n $seg ]]; do + [[ $seg != .. ]] || die "import: the archive holds a .. path ($p); nothing was published" + [[ $seg != .* ]] || die "import: the archive holds a dotfile ($p); share never publishes one; nothing was published" + done < <(printf '%s' "$p") + done <<<"$names" +} +cmd_import() { # cmd_import --probe | (iusage): one share moved here by migrate, its tar on stdin; validates, stages, publishes; never creates an Access app + if [[ ${1:-} == --probe ]]; then [[ $# == 1 ]] || die "$iusage"; echo "share-import 1"; return 0; fi + [[ $# == 7 || $# == 8 ]] || die "$iusage" + local id=$1 exp=$2 by=$3 added=$4 man=${8:-} name src opts w max="${SHARE_IMPORT_MAX_BYTES:-2147483648}" size stage got xflags + [[ $backend != r2 && $r2on == 0 ]] || die "import moves links into a tunnel profile with R2 off; $me reads a bucket" + [[ $id =~ ^[0-9a-f]{6}$ ]] || die "import: bad id '$id'" + [[ $exp =~ ^[0-9]{1,11}$ ]] || die "import: bad expiry for $id" + [[ $added =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] || die "import: bad added date for $id" + [[ $by =~ ^[a-z0-9.-]+$ ]] || die "import: bad by for $id" + [[ -z $man || $man =~ ^[0-9]+:[0-9a-f]{64}$ ]] || die "import: bad manifest for $id" + b64dec "$5" || die "import: the name of $id is not base64"; name=$b64_out + b64dec "$6" || die "import: the source of $id is not base64"; src=$b64_out + b64dec "$7" || die "import: the opts of $id are not base64"; opts=$b64_out + [[ -n $name && $name != .* && $name != */* && $name != *[[:cntrl:]]* ]] || die "import: bad name for $id (empty, a dotfile, a /, or a control byte)" + [[ $src != *[[:cntrl:]]* ]] || die "import: the source path of $id holds a control byte" + for w in $opts; do # a live or host= row never arrives: migrate moves snapshots only + case $w in noindex|access=*|access_rule=*) ;; *) die "import: opt '$w' of $id is not one import takes (noindex, access=, access_rule=)" ;; esac + done + opts="${opts:+$opts }by=$by" + mk_root; mkdir -p "$pub" + local rowf; rowf="$(mktemp "$root/.import-row.XXXXXX")" # rows_in reads a regular file only + printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$id" "$name" "$by:$src" "$added" "$exp" "$opts" >"$rowf" + [[ -n "$(rows_in "$rowf" "$backend")" ]] || { rm -f "$rowf"; die "import: the row for $id does not pass the index check (an access= without its rule?)"; } + rm -f "$rowf" + if [[ -n "$(awk -F'\t' -v id="$id" '$1 == id' "$index" 2>/dev/null)" || -e $pub/$id ]] || pending_has_id "$id"; then die "import: $id is already here; nothing was published"; fi + import_spool="$(mktemp "$root/.import.XXXXXX")"; trap release_locks EXIT + head -c $((max + 1)) >"$import_spool" + size="$(wc -c <"$import_spool" | tr -d ' ')" + [[ $size -le $max ]] || die "import: $id is over $max bytes (SHARE_IMPORT_MAX_BYTES); nothing was published" + import_scan "$import_spool" + # no owner, mode bits, xattrs, ACLs, or file flags from the archive + if tar --version 2>/dev/null | grep -q 'GNU tar'; then xflags=(--no-same-owner --no-same-permissions --no-xattrs --no-acls --no-selinux) + else xflags=(--no-same-owner --no-same-permissions --no-xattrs --no-acls --no-fflags --no-mac-metadata); fi + stage="$(mktemp -d "$root/.stage.XXXXXX")"; stage_out=$stage + tar -x "${xflags[@]}" -C "$stage" -f "$import_spool" 2>/dev/null || die "import: extracting $id failed; nothing was published" + rm -f "$import_spool"; import_spool="" + [[ -z "$(find "$stage" ! -type f ! -type d)" && -z "$(find "$stage" -type f -links +1)" ]] || die "import: $id extracted something besides plain files and directories; nothing was published" + [[ -e $stage/$name ]] || die "import: the archive of $id does not hold $name at its root; nothing was published" + if [[ -n $man ]]; then + got="$(import_manifest "$stage")" + [[ $got == "$man" ]] || die "import: $id arrived as $got, not $man (a truncated copy?); nothing was published" + fi + lock_take index + [[ -z "$(awk -F'\t' -v id="$id" '$1 == id' "$index" 2>/dev/null)" && ! -e $pub/$id ]] || { lock_release index; die "import: $id appeared during the import; nothing was published"; } + stage_publish "$id" + printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$id" "$name" "$by:$src" "$added" "$exp" "$opts" >>"$index" + if running; then write_caddyfile; caddy_reload || true; fi # a stopped or not set up profile renders at its next start + lock_release index + echo "imported $id" +} + +# --- migrate (the sending side; import above is the receiving side) --- + +musage="usage: share migrate --to [--remote-profile ] [--remote-bin ] [--max-bytes ] [--yes]" +b64enc() { printf '%s' "$1" | base64 | tr -d '\n'; } # the counterpart of b64dec: never the multi-line default GNU base64 gives +migrate_target="" # the --to value; read by migrate_remote_run, never by anything else + +migrate_script() { # migrate_script : the fixed remote wrapper source; no quote or backslash, so the base64 round trip loses nothing + local bindir="${1:-}" max="${2:-}" + printf 'export PATH=%s/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin\n' "${bindir:+$bindir:}" + [[ -n $max ]] && printf 'export SHARE_IMPORT_MAX_BYTES=%s\n' "$max" + # ssh joins argv with spaces for the remote shell to reparse, so a zero-length argument (base64 of empty opts) + # would vanish and shift every argument after it; the sender replaces "" with this sentinel, undone here. + # shellcheck disable=SC2016 # literal source for the decoded remote script, not this shell's own expansion + printf 'a=(); for x in "$@"; do [ "$x" = "__share_empty_arg__" ] && x=""; a+=("$x"); done\n' + # shellcheck disable=SC2016 + printf 'exec share "${a[@]}"\n' +} +mr_out="" mr_code=0 +migrate_remote_run() { # migrate_remote_run ...: the fixed base64 template over ssh, or SHARE_MIGRATE_SSH; forwards this call's own stdin + local bindir="$1" max="$2" rprofile="$3" b64 sshcmd a; shift 3 + b64="$(migrate_script "$bindir" "$max" | base64 | tr -d '\n')" + local rargs=() + [[ -z $rprofile || $rprofile == default ]] || rargs=(--profile "$rprofile") + for a in "$@"; do rargs+=("${a:-__share_empty_arg__}"); done + if [[ -n ${SHARE_MIGRATE_SSH:-} ]]; then sshcmd="$SHARE_MIGRATE_SSH"; else sshcmd="ssh -o BatchMode=yes $migrate_target"; fi + # shellcheck disable=SC2086 # sshcmd is a fixed word list (ssh plus its flags), never a single string needing quotes + # the argv element below must CONTAIN literal " characters around $(...): ssh joins argv with spaces and the remote + # login shell reparses the joined string, so an unquoted $(...) would word-split the decoded script on its own newlines + mr_out="$($sshcmd /bin/bash -c "\"\$(printf %s $b64 | base64 -D 2>/dev/null || printf %s $b64 | base64 -d)\"" share-migrate "${rargs[@]}")" + mr_code=$? + return "$mr_code" # the line above is an assignment, always 0 on its own; every caller of "|| rc=$?" needs this function's OWN exit status to be the real one +} +migrate_remote() { migrate_remote_run "$@" /dev/null || { echo 1; return; } + local k="share-migrate-check.$$" v="probe-$RANDOM-$RANDOM" got="" + security add-generic-password -U -a share -s "$k" -w "$v" >/dev/null 2>&1 || { echo 0; return; } + got="$(security find-generic-password -a share -s "$k" -w 2>/dev/null)" + security delete-generic-password -a share -s "$k" >/dev/null 2>&1 + [[ $got == "$v" ]] && echo 1 || echo 0 +} +cmd_migrate_probe() { # one line for the sender's preflight: state, host, pub emptiness, port freedom, Keychain and launchctl + local st pempty=1 pfree=1 kok lok=1 + if running; then st=serving; elif [[ -n $host_name || $mode == quick ]]; then st=stopped; else st=not_setup; fi + [[ -z "$(find "$pub" -mindepth 1 -print -quit 2>/dev/null)" ]] || pempty=0 + listening "$port" && pfree=0 + [[ $tunnel == 0 ]] || { listening "$metrics_port" && pfree=0; } + kok="$(migrate_keychain_check /dev/null 2>&1 || lok=0; } + printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$st" "${host_name:--}" "$pempty" "$pfree" "$kok" "$lok" +} +cmd_migrate_index() { [[ -f $index ]] && awk -F'\t' '{print $1"\t"$2"\t"$4}' "$index"; return 0; } +cmd_migrate_tunnel_id() { cfg tunnel_id; } # read back after the switch, so the sender can refuse to delete a tunnel the new origin now depends on +cmd_migrate_nonce() { # a throwaway probe file only the target holds; the sender polls its URL + mk_root; mkdir -p "$pub" + local n; n="_migrate-probe-$(rand_id_raw).txt" + echo ok >"$pub/$n" + echo "$n" +} + +migrate_row_bytes() { find "$pub/$1" -type f -exec wc -c {} \; 2>/dev/null | awk '{s+=$1} END{print s+0}'; } # portable: no stat flag differs GNU/BSD +migrate_tar() { # migrate_tar : the whole pub/ tree on stdout (a bare file's generated index.html sits beside it at the id root, so the tree is more than ); bsdtar (macOS) must drop AppleDouble members or the target's dotfile check refuses them + if tar --version 2>/dev/null | grep -q 'GNU tar'; then tar -C "$pub/$1" -cf - -- . + else COPYFILE_DISABLE=1 tar -C "$pub/$1" -cf - --no-mac-metadata --no-xattrs -- .; fi +} + +migrate_retire() { # migrate_retire ...: index.migrated and migrated/ (nothing deleted), then the teardown path with no Access-app delete and no DNS delete unless it still points here; target_tid is the new origin's own tunnel id, so a collision with this machine's refuses the tunnel and DNS deletes (DEC-010: the new origin always gets its own tunnel, never this one) + local target_tid="$1"; shift + local mids=("$@") id midlist=" " line keep + for id in "${mids[@]}"; do midlist="$midlist$id "; done + lock_take index + mkdir -p "$root/migrated" + for id in "${mids[@]}"; do + line="$(awk -F'\t' -v id="$id" '$1 == id' "$index")" + [[ -n $line ]] || continue + printf '%s\n' "$line" >>"$root/index.migrated" + [[ -e $pub/$id ]] && mv "$pub/$id" "$root/migrated/$id" + done + keep=""; [[ -s $index ]] && keep="$(awk -F'\t' -v ids="$midlist" '{if (index(ids, " " $1 " ") == 0) print}' "$index")" + printf '%s\n' "$keep" | sed '/^$/d' >"$index.tmp"; mv -f "$index.tmp" "$index" + write_caddyfile; caddy_reload || true + lock_release index + svc_uninstall + cmd_stop >/dev/null 2>&1 || true + local tid; tid="$(cfg tunnel_id)" + if [[ -z $target_tid ]]; then # without the new origin's own tunnel id there is no proof the two tunnels differ + echo "dns: left alone; could not read $migrate_target's tunnel id, so deleting the DNS record or this tunnel might take down the new origin" + echo "tunnel: not deleted, for the same reason; once $migrate_target is confirmed on its own tunnel, run '$me teardown --yes' here to remove this tunnel" + elif [[ $target_tid == "$tid" ]]; then + echo "dns: left alone; $migrate_target's tunnel id matches this machine's, deleting it would take down the new origin" + echo "tunnel: not deleted, for the same reason; clean it up by hand once $migrate_target is confirmed on its own tunnel" + else + local rec rid + rec="$(cf GET "/zones/$zone/dns_records?name=$host_name")" + rid="$(jq -r --arg t "$tid.cfargotunnel.com" '.result[] | select(.content == $t) | .id' <<<"$rec" | head -1)" + if [[ -n $rid ]]; then cf DELETE "/zones/$zone/dns_records/$rid" >/dev/null; echo "dns: deleted (still pointed at this tunnel)" + else echo "dns: $host_name no longer points at this tunnel, left alone"; fi + cf DELETE "/accounts/$account/cfd_tunnel/$tid/connections" >/dev/null 2>&1 || true + cf DELETE "/accounts/$account/cfd_tunnel/$tid" >/dev/null + echo "tunnel: deleted" + fi + token_forget; api_token_forget + rm -f "$config_dir/cert.zone" + [[ -f $cert ]] && to_trash "$cert" + return 0 # the line above is a bare test && action; under set -e its own false result must not become this function's (and so cmd_migrate's) exit status +} + +cmd_migrate() { + local to="" rprofile="" rbin="" maxbytes="" yes=0 + while [[ $# -gt 0 ]]; do + case $1 in + --to) to="${2:-}"; shift 2 ;; + --remote-profile) rprofile="${2:-}"; shift 2 ;; + --remote-bin) rbin="${2:-}"; shift 2 ;; + --max-bytes) maxbytes="${2:-}"; shift 2 ;; + --yes) yes=1; shift ;; + *) die "$musage" ;; + esac + done + [[ -n $to ]] || die "$musage" + [[ $to != -* ]] || die "migrate: the target must not start with -" + [[ $to =~ ^[A-Za-z0-9_.@:-]+$ ]] || die "migrate: bad --to target '$to'" + [[ -z $rprofile || $rprofile == default || $rprofile =~ ^[a-z0-9][a-z0-9-]{0,31}$ ]] || die "migrate: bad --remote-profile '$rprofile'" + [[ -z $rbin || $rbin =~ ^[A-Za-z0-9/_.-]+$ ]] || die "migrate: bad --remote-bin '$rbin'" + [[ -z $maxbytes || $maxbytes =~ ^[0-9]+$ ]] || die "migrate: bad --max-bytes '$maxbytes'" + migrate_target="$to" + + need_host + [[ $mode != quick ]] || die "migrate needs a named tunnel; quick mode has no hostname to move to another machine" + [[ $backend != r2 && $r2on == 0 ]] || die "R2 is on for $host_name; moving its origin also moves the pointers and the publisher token, which this version does not do" + host_ok || die "$(this_host) is not $host_name's serving host ($hosts); migrate runs on the origin" + local hostrow; hostrow="$(rows | awk -F'\t' '$6 ~ /(^| )host=/ {print $1; exit}')" + if [[ -n $hostrow ]]; then + local hfqdn; hfqdn="$(opt_val host "$(row "$hostrow" | cut -f6)")" + die "$hfqdn has its own hostname on this tunnel; remove it first: $me rm $hostrow" + fi + [[ -n ${CLOUDFLARE_API_TOKEN:-} ]] || die "migrate needs CLOUDFLARE_API_TOKEN (Cloudflare Tunnel Edit, DNS Edit, Zone Read on $host_name's zone)" + auth_api + local zone account _zn; read -r zone account _zn <<<"$(cf_zone)" + + # split the rows: a live link is reported, never moved; everything else (the --host check above already refused) is a snapshot + local mids=() names=() srcs=() addeds=() exps=() optss=() live_lines="" i + while IFS=$'\t' read -r id name src added exp opts; do + if has_opt live "$opts"; then live_lines="$live_lines$id ${src##*:}"$'\n'; continue; fi + mids+=("$id"); names+=("$name"); srcs+=("$src"); addeds+=("$added"); exps+=("$exp"); optss+=("$opts") + done < <(rows) + local max="${maxbytes:-2147483648}" over="" + for i in "${!mids[@]}"; do [[ $(migrate_row_bytes "${mids[$i]}") -le $max ]] || over="$over ${mids[$i]}"; done + [[ -z $over ]] || die "migrate: over --max-bytes ($max):$over; pass --max-bytes , or move them by hand first" + + migrate_remote "$rbin" "" "$rprofile" import --probe || true # a nonzero exit is a refusal to report below, never a reason for set -e to abort here + [[ $mr_code == 0 && $mr_out == "share-import 1" ]] || die "migrate: $to's share import --probe did not answer 'share-import 1' (got '$mr_out'); check --remote-bin, or upgrade share there" + migrate_remote "$rbin" "" "$rprofile" migrate-probe || true + [[ $mr_code == 0 ]] || die "migrate: the preflight check on $to failed" + local rstate rhost rpempty rpfree rkok rlok + IFS=$'\t' read -r rstate rhost rpempty rpfree rkok rlok <<<"$mr_out" + migrate_remote "$rbin" "" "$rprofile" migrate-index || true + local target_index="$mr_out" + case $rstate in + not_setup) + if [[ $rpempty != 1 ]]; then # a bare row is a stranger; one matching this run's own ids is a rerun resuming a prior partial copy + local foreign=""; foreign="$(awk -F'\t' -v ids=" $(printf '%s ' "${mids[@]}")" '{if (index(ids, " " $1 " ") == 0) print $1}' <<<"$target_index")" + [[ -z $foreign ]] || die "migrate: $to's profile is not_setup but already has shares not in this move ($foreign); pick another --remote-profile" + fi ;; + stopped|serving) [[ $rhost == "$host_name" ]] || die "migrate: $to's profile is already set up for $rhost, not $host_name; pick another --remote-profile" ;; + *) die "migrate: unexpected state '$rstate' on $to" ;; + esac + [[ $rpfree == 1 || $rstate == serving ]] || die "migrate: $to's port for this profile is already in use" # a profile already serving this hostname (an earlier switch that failed) holds its own ports + [[ $rkok == 1 ]] || die "migrate: $to cannot write and read back a Keychain item for share" + [[ $rlok == 1 ]] || die "migrate: launchctl print gui/ on $to did not answer; share's login service needs a GUI session there" + + if [[ $yes != 1 ]]; then + [[ -t 0 ]] || die "rerun '$me migrate --to $to --yes' to confirm" + local a; read -rp "Move ${#mids[@]} link(s) of $host_name to $to and retire this machine as its origin? [y/N] " a + [[ $a == y* || $a == Y* ]] || die "aborted" + fi + echo "preflight: ok ($to, $rstate)" + + for i in "${!mids[@]}"; do + local id=${mids[$i]} name=${names[$i]} src=${srcs[$i]} added=${addeds[$i]} exp=${exps[$i]} opts=${optss[$i]} rc man by srcbare sendopts tarfile + if grep -qE "^$id $name $added\$" <<<"$target_index"; then echo "copy: $id already on $to, skipping"; continue; fi + man="$(import_manifest "$pub/$id")" + # a row already carried here by an earlier import keeps its true publisher and bare source (by= in opts, + # src stored as ":"); import appends its own by=, so that token never rides along a second time + by="$(opt_val by "$opts" 2>/dev/null || true)" + if [[ -n $by ]]; then + srcbare="${src#"$by":}" + sendopts="$(sed -E 's/(^| )by=[^ ]*//' <<<"$opts")" + else + by="$(r2_by)"; srcbare="$src"; sendopts="$opts" + fi + # a plain redirect, never a pipe: migrate_remote_run's own mr_out/mr_code would be lost in a pipeline's trailing subshell. + # "cmd || rc=$?", never a bare statement: a nonzero exit here is this loop's to report, not set -e's to abort on + tarfile="$(mktemp "$root/.migrate-tar.XXXXXX")" + migrate_tar "$id" >"$tarfile" + rc=0 + migrate_remote_run "$rbin" "$maxbytes" "$rprofile" import "$id" "$exp" "$by" "$added" \ + "$(b64enc "$name")" "$(b64enc "$srcbare")" "$(b64enc "$sendopts")" "$man" <"$tarfile" || rc=$? + rm -f "$tarfile" + [[ $rc == 0 ]] || die "migrate: copying $id to $to failed: $mr_out" + echo "copy: $id -> $to ($mr_out)" + done + if [[ -n $live_lines ]]; then + local lid lport + while IFS=' ' read -r lid lport; do + [[ -n $lid ]] && echo "not moved: live link $lid (port $lport); after the move run '$me add $lport' on the new origin" + done <<<"$live_lines" + fi + + local old_tunnel_name; old_tunnel_name="$(cfg tunnel_name)" + if [[ -z $old_tunnel_name ]]; then # an older share, or a hand-edited config, can hold tunnel_id with no tunnel_name; read it back by id so the rollback line below is never printed with an empty --tunnel-name + local old_tunnel_id; old_tunnel_id="$(cfg tunnel_id)" + [[ -n $old_tunnel_id ]] && old_tunnel_name="$(cf GET "/accounts/$account/cfd_tunnel/$old_tunnel_id" | jq -r '.result.name // empty')" + fi + [[ -n $old_tunnel_name ]] || die "migrate: this profile's config has neither tunnel_name nor a tunnel_id the API still knows; run '$me setup $host_name --tunnel-name --force' by hand first" + local new_tunnel_name="${old_tunnel_name}-m" # DEC-010: the new origin always gets its own tunnel; a name this machine already holds would reuse its tunnel id, and migrate_retire would later delete the new origin's only tunnel + local switch_rc=0 + printf '%s\n' "$CLOUDFLARE_API_TOKEN" | migrate_remote_run "$rbin" "$maxbytes" "$rprofile" setup "$host_name" --tunnel-name "$new_tunnel_name" --force --token-stdin >/dev/null || switch_rc=$? + if [[ $switch_rc != 0 ]]; then + die "migrate: the switch on $to failed; this machine may still be serving, or the DNS record may already point at the new tunnel. Rollback if $host_name is down: $me setup $host_name --tunnel-name $old_tunnel_name --force" + fi + echo "switch: $to is the new origin for $host_name" + + migrate_remote "$rbin" "" "$rprofile" migrate-tunnel-id || true + local target_tid="$mr_out" + migrate_remote "$rbin" "" "$rprofile" migrate-nonce || true + local probe="$mr_out" streak=0 code _ + for _ in $(seq 1 60); do + code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 --doh-url https://1.1.1.1/dns-query "https://$host_name/$probe" 2>/dev/null || \ + curl -s -o /dev/null -w '%{http_code}' --max-time 5 "https://$host_name/$probe" 2>/dev/null || true)" + [[ $code == 200 ]] && streak=$((streak + 1)) || streak=0 + [[ $streak -ge 3 ]] && break + sleep 1 + done + [[ $streak -ge 3 ]] || die "migrate: verify failed: $to did not answer https://$host_name/$probe three times in a row; nothing moved aside. Rollback if $host_name is down: $me setup $host_name --tunnel-name $old_tunnel_name --force" + for i in "${!mids[@]}"; do + local id=${mids[$i]} name=${names[$i]} opts=${optss[$i]} uuid aud + share_url "$id" "$name" "$opts" + uuid="$(opt_val access "$opts" || true)" + if [[ -n $uuid ]]; then + aud="$(cf GET "/accounts/$account/access/apps/$uuid" | jq -r .result.aud)" + access_probe_round "$aud" "$share_url_out" || die "migrate: verify failed: the gated link $id does not answer 302 with its app's aud; nothing moved aside" + else + code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 --doh-url https://1.1.1.1/dns-query "$share_url_out" 2>/dev/null || \ + curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$share_url_out" 2>/dev/null || true)" + [[ $code == 200 ]] || die "migrate: verify failed: $id does not answer at $share_url_out (got $code); nothing moved aside" + fi + done + echo "verify: every moved link answers from $to" + + migrate_retire "$target_tid" "${mids[@]}" + echo "retire: $host_name no longer serves from this machine; moved rows in index.migrated, trees in migrated/" +} + # --- serving --- write_caddyfile() { # rendered from index.tsv; no-store: Cloudflare must not cache past rm or stop @@ -850,10 +1260,16 @@ svc_unload() { # bootout returns before launchd lets the job go; a bootstrap rig for _ in $(seq 1 20); do launchctl print "gui/$(id -u)/$svc_label" >/dev/null 2>&1 || return 0; sleep 0.5; done } svc_load() { launchctl bootstrap "gui/$(id -u)" "$svc_plist" 2>/dev/null || { sleep 1; launchctl bootstrap "gui/$(id -u)" "$svc_plist"; }; } -svc_path() { # PATH for the service: the directories of every tool share calls - local p="" c d - for c in caddy cloudflared jq curl pandoc gh trash security; do - d="$(dirname "$(command -v "$c" 2>/dev/null || echo /usr/bin/x)")" +svc_path() { # PATH for the service: every PATH directory that holds a tool share calls, kept in the caller's own PATH order + # (never the tool list's order: a stub share's own test or CI PATH puts ahead of /usr/bin must stay ahead in the + # baked PATH too, or that PATH resolves the real /usr/bin/ once anything shells out through a plain lookup) + local tools=(caddy cloudflared jq curl pandoc gh trash security) p="" d t needed dirs + IFS=: read -ra dirs <<<"$PATH" + for d in "${dirs[@]}"; do + [[ -n $d ]] || continue + needed=0 + for t in "${tools[@]}"; do [[ -x "$d/$t" ]] && { needed=1; break; }; done + [[ $needed == 1 ]] || continue case ":$p:" in *":$d:"*) ;; *) p="${p:+$p:}$d" ;; esac done echo "$p:/usr/bin:/bin:/usr/sbin:/sbin" @@ -1053,34 +1469,26 @@ cmd_state() { # read-only JSON snapshot for the menu bar app: never prunes, writ # forks the left side, so an assignment made there never reaches this shell's copy # of the variable, and jq's --argjson would see only the local's initial "0". [[ -s $index ]] && skipped_n=$(($(awk 'length {c++} END {print c+0}' "$index") - $(rows | awk 'END {print NR}'))) - { - if [[ -s $index ]]; then - local id name src date exp opts kind own_host rule line rows=() i - while IFS=$'\t' read -r id name src date exp opts; do - kind="snapshot"; has_opt live "$opts" && kind="live" - own_host="${opts#*host=}" - [[ $own_host == "$opts" ]] && own_host="" || own_host="${own_host%% *}" - rule="${opts#*access_rule=}" - [[ $rule == "$opts" ]] && rule="" || rule="${rule%% *}" - share_url "$id" "$name" "$opts" - printf -v line '%s\t%s\t%s\t%s\t%s\t%s\t%s' "$id" "$name" "$share_url_out" "$kind" "$own_host" "${exp:-0}" "$rule" - rows+=("$line") - done < <(rows) - for ((i = ${#rows[@]} - 1; i >= 0; i--)); do printf '%s\n' "${rows[i]}"; done - fi - } | jq -nR \ + # SHARE_STATE_BRIEF (the text profiles list) reads no bucket; the menu poll reads it with a stored token only + if [[ ${SHARE_STATE_BRIEF:-0} == 1 ]]; then list_rows brief; else list_rows state; fi + state_lines | jq -nR \ --arg state "$state_val" --argjson ready "$ready_val" --arg mode "$mode_val" \ --arg host "$host_val" --arg hosts "$hosts" --argjson serves_here "$serves_val" \ - --argjson service "$service_val" --argjson skipped "$skipped_n" --argjson access_pending "$(pending_count)" ' - def row: split("\t") | {id: .[0], name: .[1], url: .[2], kind: .[3], - own_host: (if .[4] == "" then null else .[4] end), expires: (.[5] | tonumber), - access: (if .[6] == "" then null else .[6] end)}; + --argjson service "$service_val" --argjson skipped "$skipped_n" --argjson access_pending "$(pending_count)" \ + --argjson r2 "$([[ $r2on == 1 ]] && echo true || echo false)" --arg sdef "$([[ $r2on == 1 ]] && { cfg storage_default || true; })" \ + --arg cerr "$list_cloud_error" --argjson cmore "$list_cloud_more" "$state_jq_row"' {schema: 1, state: $state, ready: $ready, mode: $mode, host: (if $host == "" then null else $host end), hosts: $hosts, serves_here: $serves_here, service: $service, access_pending: $access_pending, shares: [inputs | select(length > 0) | row]} - + (if $skipped > 0 then {skipped: $skipped} else {} end)' + + (if $skipped > 0 then {skipped: $skipped} else {} end) + + {r2: $r2} + (if $r2 then {storage_default: (if $sdef == "" then "local" else $sdef end)} else {} end) + + (if $cerr == "" then {} else {cloud_error: $cerr} end) + (if $cmore > 0 then {cloud_more: $cmore} else {} end)' } +state_jq_row='def row: split("\t") | {id: .[0], name: .[1], url: .[2], kind: .[3], + own_host: (if .[4] == "" then null else .[4] end), expires: (.[5] | tonumber), + access: (if .[6] == "" then null else .[6] end), storage: .[7], type: .[8], by: .[9]};' + # --- Cloudflare setup / teardown --- # @@ -1280,6 +1688,7 @@ write_config() { # write_config ; keeps a token_cmd and [[ -n $zone_name ]] && echo "zone=$zone_name" [[ -n $keep ]] && echo "token_cmd=$keep" [[ -n $keep_api ]] && echo "api_token_cmd=$keep_api" + if [[ -f $config ]]; then grep -E '^(bucket|r2_endpoint|storage_default|aliases)=' "$config" || true; fi # an origin with R2 on stays on through a tunnel rerun } >"$config.tmp" mv -f "$config.tmp" "$config" } @@ -1449,7 +1858,7 @@ api_token_file="$config_dir/api-token" api_token_key() { echo "share-api${profile:+.$profile}:$host_name"; } dry_log() { # an r2 profile logs beside its bucket calls, so one log orders PUTs, PROBEs, and app calls [[ ${SHARE_ACCESS_DRY:-0} == 1 ]] || return 0 - if [[ $backend == r2 ]]; then echo "$*" >>"$root/r2-calls.log"; else echo "$*" >>"$root/access-calls.log"; fi + if [[ $backend == r2 || $r2on == 1 ]]; then echo "$*" >>"$root/r2-calls.log"; else echo "$*" >>"$root/access-calls.log"; fi } hostname_ok() { [[ $1 =~ ^[a-z0-9]([a-z0-9.-]*[a-z0-9])?\.[a-z][a-z]+$ ]]; } @@ -1621,6 +2030,15 @@ cf_dry() { # the SHARE_ACCESS_DRY answers: fixtures and knobs in, one log line p cf_code=201 cf_body="{\"success\":true,\"result\":$(cat "$d/$uuid.json")}" dry_log "POST app" fi ;; + "PUT "*/access/apps/*) # the full read-back with its destinations changed, as a real PUT replaces every field; a self_hosted_domains entry no longer among the new destinations answers 400, as the real API does on a shrinking PUT + uuid="${2##*/}"; dry_log "PUT app $uuid" + if [[ -f $d/$uuid.json ]]; then + if jq -e '((.self_hosted_domains // []) - [.destinations[]?.uri]) != []' <<<"$3" >/dev/null 2>&1; then + cf_code=400 cf_err=12130 cf_body='{"success":false,"errors":[{"code":12130,"message":"access.api.error.invalid_request: self_hosted_domains is inconsistent with destinations"}]}' + else + jq -c . <<<"$3" >"$d/$uuid.json"; cf_body="{\"success\":true,\"result\":$3}" + fi + else cf_code=404 cf_body='{"success":false,"errors":[{"code":12103}]}' cf_err=12103; fi ;; "GET "*/access/apps/*) uuid="${2##*/}"; dry_log "GET app $uuid" if [[ -f $d/$uuid.json ]]; then cf_body="{\"success\":true,\"result\":$(cat "$d/$uuid.json")}" @@ -1833,7 +2251,9 @@ access_delete() { # access_delete : only an app whose name says if [[ $cf_code == 404 ]]; then access_proof; return; fi # done only once Access read is proven: a scope-less token 404s on every id [[ $cf_code == 200 ]] || return 1 name="$(jq -r '.result.name // empty' <<<"$cf_body")" - if [[ $name != "share $2 $host_name "* ]]; then # a forged row or line must never delete a foreign app with an account-wide token + local a al ok=0; al="$(cfg aliases || true)" + for a in "$host_name" ${al//,/ }; do [[ $name == "share $2 $a "* ]] && ok=1; done # a folded app keeps the name it got at its alias + if [[ $ok == 0 ]]; then # a forged row or line must never delete a foreign app with an account-wide token echo "share: Access app $1 is not the app of share $2 (its name is '$name'); left alone" >&2 return 1 fi @@ -1903,11 +2323,38 @@ access_gate() { # access_gate [fqdn]: every gated URL must pass three done } +access_link_check() { # access_link_check : after publish, the link itself must answer the Access challenge (302 or 403) within SHARE_ACCESS_POST_WAIT seconds (default 30); 1 when it still answers anything else at the timeout + local wait="${SHARE_ACCESS_POST_WAIT:-30}" poll="${SHARE_ACCESS_POST_POLL:-2}" start now code n=0 + start="$(date +%s)" + while :; do + n=$((n + 1)) + if [[ ${SHARE_ACCESS_DRY:-0} == 1 ]]; then # the fixture's next line, the last one repeating; no fixture is a 302 + code="$(sed -n "${n}p" "$root/access-link-fixture" 2>/dev/null || true)" + [[ -n $code ]] || code="$(tail -1 "$root/access-link-fixture" 2>/dev/null || true)" + code="${code:-302}" + echo "$code" >>"$root/access-link-checks" + else + code="$(curl -s -o /dev/null --max-time 10 --doh-url https://1.1.1.1/dns-query -w '%{http_code}' "$1" 2>/dev/null)" || + code="$(curl -s -o /dev/null --max-time 10 -w '%{http_code}' "$1" 2>/dev/null)" || code=000 + fi + code="${code%% *}" + [[ $code == 302 || $code == 403 ]] && return 0 + access_link_code="$code" + now="$(date +%s)" + [[ $((now - start)) -lt $wait ]] || return 1 + sleep "$poll" + done +} +access_link_code="" +access_link_die() { # access_link_die : the published link answered no Access challenge in time; it stays published, the message names the way out + die "the gated link $1 answered HTTP ${access_link_code:-000} instead of the Access login after ${SHARE_ACCESS_POST_WAIT:-30}s: at some Cloudflare edges it may be readable without the login. Unpublish it now with '$me rm $1', or check $2 yourself and rerun" +} + access_sweep() { # every pending line: a live owner is skipped, a finished add's stale line dropped, the rest claimed under the lock # (orphaned bytes trashed first), then each app deleted with no lock held. A line that is not done stays claimed by # this process; once it exits, the next sweep sees a dead owner and retries. [[ -s $pending ]] || return 0 - local id app pid st created tmp reload=0 claimed="" uuid mcode="" + local id app pid st created tmp reload=0 claimed="" uuid mcode="" r2side lock_take index tmp="$(mktemp "$root/.pending.XXXXXX")" while IFS=$'\t' read -r id app pid st created; do @@ -1945,7 +2392,10 @@ access_sweep() { # every pending line: a live owner is skipped, a finished add's continue fi fi - if [[ $backend == r2 ]]; then # never the snapshot: a record naming this app is a finished add; a 404 frees the app; anything else keeps the line + # an origin with R2 on decides per id: a line with no local row may be a gated --cloud add, which takes the r2 rule + r2side=0; [[ $backend == r2 || ( $r2on == 1 && -z "$(row "$id")" ) ]] && r2side=1 + mcode="" + if [[ $r2side == 1 ]]; then # never the snapshot: a record naming this app is a finished add; a 404 frees the app; anything else keeps the line r2_call GET "m/$id"; mcode=$cf_code if [[ $cf_code == 200 && "$(jq -r '.opts // ""' <<<"$cf_body" 2>/dev/null | tr ' ' '\n' | sed -n 's/^access=//p')" == "$uuid" ]]; then lock_take index; pending_drop "$id" "$app"; lock_release index @@ -1956,7 +2406,7 @@ access_sweep() { # every pending line: a live owner is skipped, a finished add's if access_delete "$uuid" "$id"; then lock_take index; pending_drop "$id" "$app"; lock_release index # the upload of an add that never published: its own nonce only, never another publisher's prefix - if [[ $backend == r2 && $mcode == 404 && $access_deleted_nonce =~ ^[0-9a-f]{8}$ ]]; then + if [[ $r2side == 1 && $mcode == 404 && $access_deleted_nonce =~ ^[0-9a-f]{8}$ ]]; then (r2_delete_prefix "o/$id.$access_deleted_nonce/") >/dev/null 2>&1 || true fi fi @@ -1995,7 +2445,7 @@ cmd_api_token() { # cmd_api_token [--cmd '' | --check]: store a token c --cmd) [[ $# -eq 2 && -n $2 ]] || die "$usage" case $2 in *$'\n'*|*$'\r'*) die "$usage" ;; esac # a line break would inject a second config line - if [[ $backend == r2 ]]; then # the command's token is checked before the command is stored + if [[ $backend == r2 || $r2on == 1 ]]; then # the command's token is checked before the command is stored tok="$(eval "$2" 2>/dev/null)" || tok="" [[ -n $tok ]] || die "the command printed no token; run it by hand to see why; nothing was stored" r2_publisher_check "$tok"; tok="" @@ -2010,6 +2460,9 @@ cmd_api_token() { # cmd_api_token [--cmd '' | --check]: store a token c if [[ $backend == r2 ]]; then echo "Create the publisher token in the Cloudflare dashboard: the form is prefilled with Zone: Read and the name;" echo " add Workers R2 Storage Bucket Item Write with the resource bucket $(cfg bucket || true) only (a link cannot preset a bucket), set an expiry, click Create." + elif [[ $r2on == 1 ]]; then # an origin with R2 on: the Access scopes stay required, the bucket scope is added by hand + echo "Create the token in the Cloudflare dashboard: the form is prefilled with the Access scopes, pick the account that owns $host_name;" + echo " add Workers R2 Storage Bucket Item Write with the resource bucket $(cfg bucket || true) only (a link cannot preset a bucket), set an expiry, click Create." else echo "Create the token in the Cloudflare dashboard: the form is prefilled, pick the account that owns $host_name, click Create." fi @@ -2020,7 +2473,7 @@ cmd_api_token() { # cmd_api_token [--cmd '' | --check]: store a token c IFS= read -r tok || true fi [[ -n $tok ]] || die "no token entered; rerun: $me api-token" - [[ $backend != r2 ]] || r2_publisher_check "$tok" + if [[ $backend == r2 || $r2on == 1 ]]; then r2_publisher_check "$tok"; fi api_token_store "$tok" # direct call: a bad token dies here, never reaches the preflight echo "token: $api_token_store_out" tok="" ;; @@ -2033,11 +2486,38 @@ cmd_api_token() { # cmd_api_token [--cmd '' | --check]: store a token c r2_publisher_check "$api_token" fi echo "share: a publisher token for bucket $(cfg bucket || true): not an admin token, no other bucket in reach" >&2 + r2_expiry_note access_preflight || echo "share: ungated adds work with this token; gated adds and their removal also need the Access scopes above" >&2 return 0 fi + if [[ $r2on == 1 ]]; then # an origin with R2 on: the same publisher refusals, and the Access scopes stay required (SPEC-004) + if [[ ${1:-} == --check ]]; then + api_token_resolve + [[ -n $api_token ]] || die "no token for $host_name; store one: $me api-token" + r2_publisher_check "$api_token" + fi + echo "share: a publisher token for bucket $(cfg bucket || true): not an admin token, no other bucket in reach" >&2 + r2_expiry_note + fi access_preflight } +r2_expiry_note() { # r2_expiry_note: the publisher token's expires_on from /user/tokens/verify; a warning 14 days ahead, since an expired token stops every add (on an origin with R2 on, local adds too) + local e left + api_token_resolve + [[ -n $api_token ]] || return 0 + r2_api GET /user/tokens/verify + [[ $cf_code == 200 ]] || return 0 + e="$(jq -r '.result.expires_on // empty' <<<"$cf_body" 2>/dev/null)" || return 0 + [[ -n $e ]] || return 0 + left="$(jq -rn --arg e "$e" '(($e | sub("\\.[0-9]+Z$"; "Z") | fromdateiso8601) - now) / 86400 | floor' 2>/dev/null)" || return 0 + [[ $left =~ ^-?[0-9]+$ ]] || return 0 + if [[ $left -lt 0 ]]; then + echo "share: the publisher token expired on ${e%%T*}; every add to $host_name fails until a new one is stored: $me api-token" >&2 + elif [[ $left -le 14 ]]; then + echo "share: the publisher token expires on ${e%%T*} (in $left days); then every add to $host_name fails; mint a new one and store it: $me api-token" >&2 + fi + return 0 +} r2_publisher_check() { # r2_publisher_check [env]: before a publisher token is stored or used from CLOUDFLARE_API_TOKEN, refuse one that edits the Worker (admin) or lists another bucket (account-wide R2) local ep a worker="share-${host_name//./-}" bucket others tail="nothing was stored" [[ ${2:-} != env ]] || tail="the token came from CLOUDFLARE_API_TOKEN; unset it or store a publisher token with $me api-token; nothing was changed" @@ -2076,17 +2556,18 @@ access_open_url() { # the platform opener, backgrounded; skipped over ssh or wit # port=r2 sentinel: an older share dies on that port at load, so it can never # write pub/ or an Access app against an r2 profile. -WORKER_VERSION=2 # bump on every Worker source change; deploys refuse a downgrade without --force -WORKER_SHA=fdae03ca85c7 # first 12 hex of the emitted source's sha256; the suite fails when it drifts +WORKER_VERSION=4 # bump on every Worker source change; deploys refuse a downgrade without --force +WORKER_SHA=2209e2a60db0 # first 12 hex of the emitted source's sha256; the suite fails when it drifts WORKER_COMPAT_DATE=2026-09-30 # moves only with a WORKER_VERSION bump -WORKER_RECORD_V=1 # the record schema this source understands; a newer record answers 404 +WORKER_RECORD_V=2 # the record schema this source understands: v:1 cloud records, v:2 machine pointers; a newer record answers 404 worker_meta() { printf '%s\n' "$WORKER_VERSION" "$WORKER_SHA" "$WORKER_COMPAT_DATE" "$WORKER_RECORD_V"; } # one line, four fields, for the deploy and sweep paths worker_js() { # worker_js : the Worker source to $1; never a heredoc inside $( ) (macOS /bin/bash 3.2) cat >"$1" <<'SHARE_WORKER_JS' // share Worker: serves R2-backed snapshots on one hostname, a gated record only -// inside a valid Cloudflare Access session for that record's app. -const RECV = 1; +// inside a valid Cloudflare Access session for that record's app. With PASS set (a tenant +// with R2 on) it runs as a route in front of the tenant's tunnel: a cloud record is served +// from R2, a machine pointer or a miss passes through to the origin, and an alias host 301s. const TYPES = { html: "text/html", css: "text/css", js: "text/javascript", mjs: "text/javascript", json: "application/json", txt: "text/plain", md: "text/markdown", csv: "text/csv", @@ -2159,31 +2640,75 @@ const hex = async (s) => { return [...d].map((b) => b.toString(16).padStart(2, "0")).join(""); }; +const passed = new WeakSet(); // answers from the origin: no hit counted, headers already set +const OFFLINE = "the machine serving this link is offline; try again later"; + +async function pass(request) { // the route is skipped for a same-host fetch, so this reaches the tunnel (spike TASK-1a) + let r; + try { r = await fetch(request); } catch { r = null; } + if (r && r.status === 101) { passed.add(r); return r; } // a WebSocket upgrade goes back untouched + // an edge error carries no cf-cache-status; the origin's own answers do (spike TASK-1a(e)). + // 502 is never one of Cloudflare's own edge codes (521-527, 530): it is Caddy's reverse_proxy + // answer for a dead live port, a working machine, so it goes back unchanged like any other origin answer. + const down = !r || r.status === 530 || (r.status >= 520 && r.status <= 527) || + (r.status === 503 && !r.headers.has("cf-cache-status")); + const out = down + ? new Response(OFFLINE, { status: 503, headers: { "Retry-After": "60", "Content-Type": "text/plain; charset=utf-8" } }) + : new Response(r.body, r); // rewrapped: a fetch answer's headers are immutable + out.headers.set("Cache-Control", "no-store"); + out.headers.set("X-Robots-Tag", "noindex, nofollow"); + passed.add(out); + return out; +} + +const capped = (p, ms) => { // a bucket read that hangs counts as a failed one + let t; + return Promise.race([p, new Promise((_, rej) => { t = setTimeout(() => rej(new Error("slow")), ms); })]).finally(() => clearTimeout(t)); +}; + async function handle(request, env) { const u = new URL(request.url); + const tail = request.url.slice(request.url.indexOf(u.hostname) + u.hostname.length).replace(/^:\d+/, ""); + const getish = request.method === "GET" || request.method === "HEAD"; + const aliases = (env.ALIASES || "").split(",").map((a) => a.trim()).filter(Boolean); + if (aliases.includes(u.hostname)) { + if (!getish) return new Response(null, { status: 405 }); + return new Response(null, { status: 301, headers: { Location: "https://" + env.HOST + tail.split("#")[0] } }); + } if (u.hostname !== env.HOST) return new Response(null, { status: 404 }); - if (request.method !== "GET" && request.method !== "HEAD") return new Response(null, { status: 405 }); - const raw = request.url.slice(request.url.indexOf(u.hostname) + u.hostname.length).replace(/^:\d+/, "").split(/[?#]/)[0] || "/"; + if (!env.PASS && !getish) return new Response(null, { status: 405 }); + const raw = tail.split(/[?#]/)[0] || "/"; if (/%2f|%5c|%2e|%0[0-9a-f]|%1[0-9a-f]/i.test(raw) || raw.includes("//")) { return new Response(null, { status: 400 }); } if (raw === "/healthz") { - return new Response("ok", { - status: 200, - headers: { - "X-Share-Worker": (env.VERSION || "") + " " + (env.SHA || ""), - "X-Share-Gate": env.TEAM ? "1" : "0", - }, - }); + if (!getish) return new Response(null, { status: 405 }); + const headers = { + "X-Share-Worker": (env.VERSION || "") + " " + (env.SHA || ""), + "X-Share-Gate": env.TEAM ? "1" : "0", + }; + if (!env.PASS) return new Response("ok", { status: 200, headers }); + let up = false; // the tunnel leg: the origin's own /healthz, through the same skipped route + try { + const r = await fetch("https://" + env.HOST + "/healthz", { signal: AbortSignal.timeout(3000) }); + up = r.status === 200 && (await r.text()).trim() === "ok"; + } catch { up = false; } + headers["X-Share-Tunnel"] = up ? "1" : "0"; + return new Response(up ? "ok" : "tunnel down", { status: up ? 200 : 503, headers }); } const segs = u.pathname.split("/"); const id = segs[1]; - if (!/^[0-9a-f]{6}$/.test(id)) return new Response(null, { status: 404 }); - const mo = await env.BUCKET.get("m/" + id); - if (mo === null) return new Response(null, { status: 404 }); + if (!/^[0-9a-f]{6}$/.test(id)) return env.PASS ? pass(request) : new Response(null, { status: 404 }); + let mo; + try { mo = await capped(env.BUCKET.get("m/" + id), 2000); } catch { mo = null; } + if (mo === null) return env.PASS ? pass(request) : new Response(null, { status: 404 }); let rec; try { rec = await mo.json(); } catch { return new Response(null, { status: 404 }); } - if (!rec || rec.v > RECV || rec.id !== id || typeof rec.prefix !== "string" || + if (rec && typeof rec === "object" && rec.v === 2 && rec.storage === "machine") { + return env.PASS ? pass(request) : new Response(null, { status: 404 }); + } + // a cloud record is v:1 with no storage key; any other shape answers 404 and never reaches the origin + if (!rec || typeof rec !== "object" || rec.v !== 1 || "storage" in rec || rec.id !== id || typeof rec.prefix !== "string" || !new RegExp("^o/" + id + "\\.[0-9a-f]{8}/$").test(rec.prefix) || (rec.expires > 0 && Date.now() / 1000 > rec.expires)) { return new Response(null, { status: 404 }); @@ -2193,6 +2718,7 @@ async function handle(request, env) { try { ok = await gatedOk(request, env, rec); } catch { ok = false; } if (!ok) return new Response(null, { status: 404 }); } + if (!getish) return new Response(null, { status: 405 }); const rest = segs.slice(2); if (rest.length === 0) return new Response(null, { status: 308, headers: { Location: u.origin + "/" + id + "/" } }); const cond = { range: request.headers, onlyIf: request.headers }; @@ -2243,6 +2769,7 @@ export default { let resp; try { resp = await handle(request, env); } catch { resp = new Response(null, { status: 404 }); } + if (passed.has(resp)) return resp; resp.headers.set("Cache-Control", "no-store"); resp.headers.set("X-Robots-Tag", "noindex, nofollow"); if (resp.status < 400 && url.pathname !== "/healthz" && @@ -2313,6 +2840,174 @@ r2_release_prefix() { # r2_release_prefix : the EXIT trap's r2 part return 0 } +share_type() { type_of "$1"; echo "$type_out"; } +type_of() { # type_of : the file-type word for a link's source into type_out, first match wins; the extension match ignores case; no fork (state types every row) + if [[ -d $1 ]]; then # a published folder's generated listing is not a site of its own + local l=""; [[ -f $1/index.html ]] && read -r l <"$1/index.html" + if [[ -f $1/index.html && $l != *'index'* ]]; then type_out=site; else type_out=folder; fi + return 0 + fi + local nc=0; shopt -q nocasematch && nc=1; shopt -s nocasematch + case ${1##*/} in + *.html|*.htm) type_out=site ;; + *.md|*.markdown) type_out=markdown ;; + *.pdf) type_out=pdf ;; + *.png|*.jpg|*.jpeg|*.gif|*.webp|*.svg|*.heic|*.ico|*.avif) type_out=image ;; + *.mp4|*.mov|*.webm|*.m4v|*.mkv) type_out=video ;; + *.mp3|*.wav|*.m4a|*.aac|*.flac|*.ogg) type_out=audio ;; + *.zip|*.tar|*.gz|*.tgz|*.bz2|*.xz|*.7z|*.rar|*.dmg) type_out=archive ;; + *.txt|*.csv|*.json|*.yaml|*.yml|*.toml|*.xml|*.log|*.sh|*.py|*.js|*.mjs|*.ts|*.go|*.rs|*.rb|*.swift|*.java|*.c|*.h|*.css|*.sql) type_out=text ;; + *) type_out=other ;; + esac + [[ $nc == 1 ]] || shopt -u nocasematch +} +type_ok() { case ${1:-} in pdf|image|video|audio|folder|site|markdown|archive|text|other) return 0 ;; esac; return 1; } +optv() { # optv : the value of a key=value token into optv_out, empty when absent; no fork + local rest=" $2" + optv_out="" rest="${rest#*" $1="}" + [[ $rest == " $2" ]] || optv_out="${rest%% *}" +} + +r2_menu_token() { # state (the menu poll): the bucket is read only with a stored publisher token (the Keychain item or the 600 file); never a command, which may prompt, never the environment + [[ ${SHARE_API_TOKEN_OFF:-0} != 1 ]] || return 1 + [[ -z ${SHARE_R2_TOKEN:-} ]] || return 0 # the test seam stands for a stored token + [[ -z "$(cfg r2_token_cmd || true)" && -z "$(cfg api_token_cmd || true)" ]] || return 1 + api_token_resolve + [[ -n $api_token && $api_token_src != CLOUDFLARE_API_TOKEN ]] +} + +list_bucket() { # list_bucket ls|state : the bucket's display rows (fields split by \x1f), "#more " first; run in a subshell, so a die becomes cloud_error + local id name src date exp opts us=$'\x1f' + if [[ $1 == state ]]; then + r2_menu_token || die "the menu reads cloud links only with a stored token: $me api-token" + r2_read 25 "$2" + elif [[ -z ${r2_snap_file:-} ]]; then r2_read; fi # a member's ls reuses the read its prune made + echo "#more ${r2_list_more:-0}" + while IFS=$'\t' read -r id name src date exp opts; do + case $'\n'"$2" in *$'\n'"$id"$'\n'*) continue ;; esac # a shadowed id lists its local row once + optv type "$opts"; type_out=$optv_out + if ! type_ok "$type_out"; then if [[ $name == *.* ]]; then type_of "/dev/null/$name"; else type_out=folder; fi; fi + optv by "$opts" + printf '%s\n' "$id$us$name$us$src$us$date$us$exp$us$opts${us}cloud$us$type_out$us${optv_out:-?}" + done < <(crows) + [[ $backend == r2 ]] || return 0 # on the origin every machine record is its own pointer; its index is their truth + printf '%s' "${r2_mrows:-}" | tr '\t' "$us" | grep -v '^$' | awk -F"$us" -v OFS="$us" -v now="$(date +%s)" '$4 == 0 || $4 > now {print $1, $2, "", $3, $4, $5, "machine", $6, $7}' || true # an expired pointer is a prune'"'"'s, not a link +} + +list_rows() { # list_rows ls|state|brief: this profile's display rows into list_out (fields split by \x1f): id name src added expires opts storage type by + # sets list_cloud_error and list_cloud_more. Bucket rows are display data: they never reach rows(), the index, a Caddyfile, or a stage path + local id name src date exp opts lids="" f me_by us=$'\x1f' + list_out="" list_cloud_error="" list_cloud_more=0 + if [[ $backend != r2 ]]; then + me_by="$(r2_by)" + while IFS=$'\t' read -r id name src date exp opts; do + if has_opt live "$opts"; then type_out=site; else type_of "$pub/$id/$name"; fi + optv by "$opts" + list_out+="$id$us$name$us$src$us$date$us$exp$us$opts${us}machine$us$type_out$us${optv_out:-$me_by}"$'\n' + lids+="$id"$'\n' + done < <(rows) + [[ $r2on == 1 && $1 != brief ]] || return 0 + fi + [[ $1 != brief ]] || return 0 + f="$(mktemp "${TMPDIR:-/tmp}/share-list.XXXXXX")" + if ( list_bucket "$1" "$lids" ) >"$f" 2>"$f.err"; then + list_cloud_more="$(sed -n '1s/^#more //p' "$f")" + list_out+="$(sed '1d' "$f")" + [[ -z $list_out ]] || list_out+=$'\n' + else + list_cloud_error="$(awk 'NF {l = $0} END {print l}' "$f.err")"; list_cloud_error="${list_cloud_error#share: }" + list_cloud_error="${list_cloud_error:-the bucket could not be read}" + fi + rm -f "$f" "$f.err" + [[ $list_cloud_more =~ ^[0-9]+$ ]] || list_cloud_more=0 +} + +r2_token_ok() { # r2_token_ok off|env|soft: a publisher token resolves, never a die; env reads the environment only, so a listing runs no command + [[ $1 != off && ${SHARE_API_TOKEN_OFF:-0} != 1 ]] || return 1 + if [[ -n ${SHARE_R2_TOKEN:-} ]]; then : + elif [[ $1 == env ]]; then [[ -n ${CLOUDFLARE_API_TOKEN:-} ]] || return 1 + elif [[ -z "$(cfg r2_token_cmd || true)" ]]; then api_token_resolve soft; [[ -n $api_token ]] || return 1 + fi + [[ ${SHARE_R2_DRY:-0} == 1 ]] && return 0 + ( r2_creds ) >/dev/null 2>&1 || return 1 # an admin token from the environment dies in r2_publisher_check: here that only means no + r2_creds +} + +r2_pointer_write() { # r2_pointer_write : one v:2 machine record, If-None-Match: *; the answer in cf_code + local po="" w rec + for w in $5; do case $w in noindex|live) po="${po:+$po }$w" ;; esac; done + [[ -z $6 ]] || po="${po:+$po }gated" # a flag, never the rule: every member reads this object + rec="$(mktemp "$root/.r2rec.XXXXXX")" + jq -nc --arg id "$1" --arg name "$2" --arg by "$(r2_by)" --arg added "$7" --argjson exp "$4" --arg opts "$po" --arg type "$3" --arg oh "$(opt_val host "$5" || true)" \ + '{v: 2, id: $id, storage: "machine", name: $name, by: $by, added: $added, expires: $exp, opts: $opts, type: $type} + + (if $oh == "" then {} else {own_host: $oh} end)' >"$rec" # own_host: a member lists a --host row at its own name + r2_call PUT "m/$1" "$rec" -H 'If-None-Match: *' || true + rm -f "$rec" +} + +r2_pointer_put() { # r2_pointer_put [rule]: the origin's v:2 machine record for cmd_add's $id, If-None-Match: * + # a 412 means the bucket holds the id: cmd_add's id moves to a fresh one; any other answer dies before anything is served. + # The add lock on the id tells the reconcile that this pointer has a row on the way. + local t i + if [[ $1 == live ]]; then t=site; else t="$(share_type "$3")"; fi + for i in 1 2 3 4 5; do + lock_take "add-$id" 5 + r2_pointer_write "$id" "$2" "$t" "$4" "$5" "${6:-}" "$(date +%F)" + case $cf_code in + 200) return 0 ;; + 412) lock_release "add-$id"; id="$(rand_id)" || exit 1 ;; + *) die "R2 did not take the pointer record m/$id (HTTP $cf_code${cf_err:+, $cf_err}); $host_name has R2 on, so a local link needs one; nothing was published" ;; + esac + done + die "R2 answered 412 for five ids in a row; nothing was published; rerun the same $me add" +} + +add_held() { local p; p="$(readlink "$root/.lock-add-$1" 2>/dev/null)" && [[ -n $p ]] && kill -0 "$p" 2>/dev/null; } # a live add holds this id's pointer + +r2_reconcile() { # after r2_read, on the origin's interactive ls and prune: an orphan pointer goes, a row with no pointer gets one, a shadowed id is named + local now cut ids id e lm name src date exp opts t + now="$(date +%s)"; cut="$(r2_ago $((now - 600)))" + ids="$(rows | cut -f1)" + while IFS=$'\t' read -r id e; do # one origin per tenant: every machine pointer is this machine's + [[ -n $id ]] || continue + grep -qx "$id" <<<"$ids" && continue + lm="$(awk -F'\t' -v k="m/$id" '$1 == k {print $2; exit}' <<<"$r2_mmods")" + [[ -n $lm && ${lm:0:19} < $cut ]] || continue # under 10 minutes old: an add may not have written its row yet + pending_has_id "$id" && continue # a gated add inside its wait + add_held "$id" && continue + r2_call DELETE "m/$id" || true + if [[ $cf_code == 20? || $cf_code == 404 ]]; then echo "removed orphan pointer m/$id" + else echo "share: deleting the orphan pointer m/$id answered HTTP $cf_code; the next $me prune retries" >&2; fi + done <<<"$r2_machine" + while IFS=$'\t' read -r id name src date exp opts; do + [[ -n $id ]] || continue + [[ $exp == 0 || $exp -gt $now ]] || continue # an expired row is the local prune's + grep -q "^$id"$'\t' <<<"$r2_machine" && continue + if ! grep -qx "$id" <<<"$r2_mkeys"; then + if has_opt live "$opts"; then t=site; else t="$(share_type "$pub/$id/$name")"; fi + r2_pointer_write "$id" "$name" "$t" "$exp" "$opts" "$(opt_val access "$opts" || true)" "$date" + case $cf_code in + 200) echo "share: wrote the missing pointer m/$id" >&2; continue ;; + 412) ;; + *) echo "share: writing the pointer m/$id answered HTTP $cf_code; the next $me ls or prune retries" >&2; continue ;; + esac + fi + echo "share: $id is shadowed by a cloud link; rm one of them" >&2 # never deleted here: either side may be the link someone holds + done < <(rows) +} + +r2_pointer_drop() { # r2_pointer_drop off|env|soft: after a local row is gone, its machine pointer goes too; without a token it stays (the Worker passes it to Caddy, which 404s) + [[ $r2on == 1 ]] || return 0 + r2_token_ok "$2" || { [[ $2 == off ]] || echo "share: the pointer m/$1 stays until '$me prune' runs with the publisher token" >&2; return 0; } + r2_call GET "m/$1" || true + [[ $cf_code == 200 ]] || return 0 # 404: nothing to delete; anything else: the reconcile retries + # only this origin's own pointer: a cloud record on the same id is another link (a shadow), never deleted here + [[ "$(jq -r 'if .v == 2 and .storage == "machine" then "machine" else "" end' <<<"$cf_body" 2>/dev/null)" == machine ]] || return 0 + r2_call DELETE "m/$1" || true + [[ $cf_code == 20? || $cf_code == 404 ]] || echo "share: deleting the pointer m/$1 answered HTTP $cf_code; '$me prune' retries it" >&2 + return 0 +} + cmd_add_r2() { # cmd_add_r2 [rule]: stage, upload to a fresh prefix, then PUT m/ with If-None-Match: * is the publish # gated (a rule, already parsed): the app and its gate sit between the upload and the record, and the record carries the app's aud local secs=$1 opts="" src name id exp=0 prefix rec url nonce gated="${4:-}" @@ -2351,8 +3046,8 @@ cmd_add_r2() { # cmd_add_r2 [rule]: stage, upload to a fi rec="$(mktemp "$root/.r2rec.XXXXXX")" jq -nc --arg id "$id" --arg name "$name" --arg src "$src" --arg added "$(date +%F)" --argjson exp "$exp" \ - --arg opts "$opts" --arg prefix "$prefix" --arg by "$(r2_by)" --arg aud "${gated:+$access_aud}" \ - '{v: 1, id: $id, name: $name, src: $src, added: $added, expires: $exp, opts: $opts, prefix: $prefix, by: $by} + --arg opts "$opts" --arg prefix "$prefix" --arg by "$(r2_by)" --arg aud "${gated:+$access_aud}" --arg type "$(share_type "$src")" \ + '{v: 1, id: $id, name: $name, src: $src, added: $added, expires: $exp, opts: $opts, prefix: $prefix, by: $by, type: $type} + (if $aud == "" then {} else {aud: $aud} end)' >"$rec" r2_call PUT "m/$id" "$rec" -H 'If-None-Match: *' rm -f "$rec" @@ -2375,6 +3070,7 @@ cmd_add_r2() { # cmd_add_r2 [rule]: stage, upload to a r2_link_base="$stage_out"; share_url "$id" "$name" "$opts"; r2_link_base="" rm -rf "$stage_out"; stage_out="" url="$share_url_out" + [[ -z $gated ]] || access_link_check "$url" || access_link_die "$id" "$url" echo "$url" if [[ ${SHARE_CLIPBOARD:-1} == 1 ]] && command -v pbcopy >/dev/null; then printf '%s' "$url" | pbcopy && echo " (link copied)" @@ -2487,12 +3183,13 @@ r2_record_row() { # r2_record_row : the index.tsv row for a sound and (.name | clean) and (.src | clean) and (.opts | clean) and (.added | clean) and (.expires | type == "number" and . >= 0 and . < 1e11) # epoch seconds; 1e300 would break bash arithmetic - and (.opts | test("(^| )(prefix|by)=") | not) # the row gains its own prefix= and by=; a record never supplies them + and (.opts | test("(^| )(prefix|by|type)=") | not) # the row gains its own prefix=, by=, and type=; a record never supplies them and (.prefix | type == "string" and test("^o/" + $id + "\\.[0-9a-f]{8}/$")) and (.by | type == "string" and test("^[a-z0-9.-]+$")) and ((.aud // "") | test("^[0-9a-f]{64}$|^$"))) | [.id, .name, .src, .added, (.expires | floor | tostring), - (if .opts == "" then "" else .opts + " " end) + "prefix=" + .prefix + " by=" + .by] + (if .opts == "" then "" else .opts + " " end) + "prefix=" + .prefix + " by=" + .by + + ((.type // "") as $t | if (["pdf", "image", "video", "audio", "folder", "site", "markdown", "archive", "text", "other"] | index($t)) then " type=" + $t else "" end)] | @tsv' <<<"$2" 2>/dev/null || true # a corrupt or forged record is dropped, never fatal } @@ -2521,7 +3218,7 @@ r2_fetch_records() { # r2_fetch_records : parallel-GET every key in r2_lis [[ $code == 200 ]] && printf '%s' "$cf_body" >"$f" fi case $code in - 200) r2_record_row "$key" "$(cat "$f")" >>"$snap"; r2_record_raw "$key" "$(cat "$f")" ;; + 200) r2_take "$snap" "$key" "$(cat "$f")" ;; 404) ;; *) rm -rf "$wd"; die "r2 snapshot: GET $key answered HTTP $code" ;; esac @@ -2529,19 +3226,30 @@ r2_fetch_records() { # r2_fetch_records : parallel-GET every key in r2_lis rm -rf "$wd" } -r2_snapshot() { # r2_snapshot: every m/ record into a mktemp file that $index then names, so rows() stays the only reader - local snap k +r2_snapshot() { r2_read; index="$r2_snap_file"; } # an r2 profile: the bucket read is its index, so rows() stays the only reader + +r2_read() { # r2_read [ ]: every m/ record into this process's own file (r2_snap_file, read through crows), the sweep view, and the machine pointers; never assigns $index + # with (the menu poll): the local ids' keys are skipped and only the newest n others by LastModified are fetched; r2_list_more counts the rest + local snap k keep mkdir -p "$root" snap="$(mktemp "$root/.r2-index.XXXXXX")" r2_snap_file="$snap"; trap release_locks EXIT - r2_refs="" r2_sweep_block="" + r2_refs="" r2_sweep_block="" r2_machine="" r2_mrows="" r2_list_more=0 r2_list "m/" + r2_list_out="$(grep -E '^m/[0-9a-f]{6}$' <<<"$r2_list_out" || true)" # a key is read, shown, or deleted only in the shape share writes + r2_mkeys="${r2_list_out//m\//}" r2_mmods="$r2_list_mods" + if [[ -n ${1:-} ]]; then + keep="$(awk -F'\t' 'NR == FNR {if ($0 != "") own["m/" $0]; next} $1 ~ /^m\/[0-9a-f]+$/ && length($1) == 8 && !($1 in own) {print $2 "\t" $1}' \ + <(printf '%s\n' "${2:-}") <(printf '%s\n' "$r2_list_mods") | LC_ALL=C sort -r | cut -f2)" + r2_list_out="$(head -n "$1" <<<"$keep")" + r2_list_more=$(( $(grep -c . <<<"$keep" || true) - $(grep -c . <<<"$r2_list_out" || true) )) + fi if [[ ${SHARE_R2_DRY:-0} == 1 ]]; then while IFS= read -r k; do - [[ $k == m/* ]] || continue + [[ -n $k ]] || continue r2_call GET "$k" case $cf_code in - 200) r2_record_row "$k" "$cf_body" >>"$snap"; r2_record_raw "$k" "$cf_body" ;; + 200) r2_take "$snap" "$k" "$cf_body" ;; 404) ;; *) die "r2 snapshot: GET $k answered HTTP $cf_code" ;; esac @@ -2549,14 +3257,34 @@ r2_snapshot() { # r2_snapshot: every m/ record into a mktemp file that $index th else r2_fetch_records "$snap" || die "r2 snapshot: a record fetch failed; nothing was listed" fi - index="$snap" +} + +r2_take() { # r2_take : one record into the cloud rows, the sweep view, and r2_machine ("\t" per v:2 machine pointer) + local e + r2_record_row "$2" "$3" >>"$1"; r2_record_raw "$2" "$3" + e="$(jq -r --arg id "${2#m/}" 'select(.v == 2 and .storage == "machine" and .id == $id and (.expires | type == "number" and . >= 0 and . < 1e11)) | .expires | floor' <<<"$3" 2>/dev/null)" || return 0 + [[ -z $e ]] || r2_machine+="${2#m/}"$'\t'"$e"$'\n' + # the v:2 display reader: every field checked before shell arithmetic or a URL sees it; a record that fails is skipped like a forged row + r2_mrows+="$(jq -r --arg id "${2#m/}" ' + def clean: type == "string" and (explode | all(. >= 32 and . != 127)); + select(type == "object" and .v == 2 and .storage == "machine" and .id == $id + and (.expires | tostring | test("^[0-9]{1,11}$")) and (.added | type == "string" and test("^[0-9]{4}-[0-9]{2}-[0-9]{2}$")) + and (.name | clean and length > 0) and (.by | clean and test("^[a-z0-9.-]+$")) + and (.opts | type == "string" and (split(" ") | all(. == "" or . == "noindex" or . == "live" or . == "gated"))) + and ((.type // "") as $t | ["pdf", "image", "video", "audio", "folder", "site", "markdown", "archive", "text", "other"] | index($t)) + and ((.own_host // "") | type == "string" and test("^$|^[a-z0-9]([a-z0-9.-]*[a-z0-9])?\\.[a-z][a-z]+$"))) + | [.id, .name, .added, (.expires | tostring), (.opts + (if (.own_host // "") == "" then "" else " host=" + .own_host end)), .type, .by] | @tsv' <<<"$3" 2>/dev/null)"$'\n' || true + return 0 } r2_record_raw() { # r2_record_raw : the orphan sweep's own view; the prefix into r2_refs, or the first key this CLI cannot read into r2_sweep_block + # a cloud record (v:1) names its upload; an origin's machine pointer (v:2, storage machine, no prefix) is readable and names none local p - p="$(jq -r --argjson v "$WORKER_RECORD_V" 'if type == "object" and (.prefix | type == "string") and (.v | type == "number") and .v <= $v then .prefix else error("unreadable") end' <<<"$2" 2>/dev/null)" || + p="$(jq -r 'if type == "object" and .v == 1 and (.prefix | type == "string") then .prefix + elif type == "object" and .v == 2 and .storage == "machine" and (has("prefix") | not) then "" + else error("unreadable") end' <<<"$2" 2>/dev/null)" || { r2_sweep_block="${r2_sweep_block:-$1}"; return 0; } - r2_refs+="$p"$'\n' + [[ -z $p ]] || r2_refs+="$p"$'\n' } r2_ago() { date -u -r "$1" +%Y-%m-%dT%H:%M:%S 2>/dev/null || date -u -d "@$1" +%Y-%m-%dT%H:%M:%S; } # epoch -> the ISO form S3's LastModified starts with @@ -2586,15 +3314,16 @@ r2_sweep_orphans() { # bare prune: every o/./ no record names, whose } r2_prune() { # r2_prune [quiet|listing]: the snapshot; unless quiet, every expired row through rm, a gated one only with an Access-proven token; a bare prune then sweeps orphan uploads - r2_snapshot + # on an origin with R2 on (cmd_prune calls it first) the cloud rows come from its own reader and never touch $index; cmd_prune runs the access sweep there + if [[ $backend == r2 ]]; then r2_snapshot; else r2_read; fi [[ ${1:-} != quiet ]] || return 0 - local now id opts expired gate_ok=0 + local now id opts expired gate_ok=0 e now="$(date +%s)" - expired="$(rows | awk -F'\t' -v now="$now" '$5 > 0 && $5 < now {print $1 "\t" $6}')" + expired="$(crows | awk -F'\t' -v now="$now" '$5 > 0 && $5 < now {print $1 "\t" $6}')" if [[ -s $pending || $expired == *access=* ]]; then # listing: the environment only, never op read; bare: the full resolver, soft if [[ ${1:-} == listing ]]; then api_token_resolve env; else api_token_resolve soft; fi [[ -n $api_token ]] && access_account soft && access_apps_edit_ok && gate_ok=1 - [[ $gate_ok == 0 ]] || access_sweep + [[ $gate_ok == 0 || $backend != r2 ]] || access_sweep fi while IFS=$'\t' read -r id opts; do [[ -n $id ]] || continue @@ -2602,16 +3331,27 @@ r2_prune() { # r2_prune [quiet|listing]: the snapshot; unless quiet, every expir echo "share: expired gated share $id waits for a publisher with the Access token" >&2 continue fi - cmd_rm "$id" prune | sed 's/$/ (expired)/' || true # a listing never dies on one row + cmd_rm_r2 "$id" prune | sed 's/$/ (expired)/' || true # a listing never dies on one row done <<<"$expired" + if [[ $backend == r2 ]]; then # a member: a machine pointer past its expiry goes; the origin's own prune unpublishes the link + while IFS=$'\t' read -r id e; do + [[ -n $id && $e -gt 0 && $e -lt $now ]] || continue + r2_call DELETE "m/$id" || true + if [[ $cf_code == 20? || $cf_code == 404 ]]; then echo "removed expired pointer m/$id"; fi + done <<<"$r2_machine" + fi [[ -n ${1:-} ]] || r2_sweep_orphans } cmd_rm_r2() { # cmd_rm_r2 [prune]: the record, a fresh read proving it gone, then every nonce's objects, then a gated share's app # any publisher may rm any share; a gated one needs a token that passes the Apps Edit probe first (prune: r2_prune proved it) local r opts tmp app - [[ -n ${r2_snap_file:-} ]] || r2_snapshot - r="$(row "${1:-}")"; [[ -n $r ]] || die "no share with id '${1:-}'" + [[ -n ${r2_snap_file:-} ]] || { if [[ $backend == r2 ]]; then r2_snapshot; else r2_read; fi; } + r="$(crow "${1:-}")" + if [[ -z $r ]]; then # a member never removes the origin's machine link: the origin keeps serving it and its reconcile writes the pointer back + [[ $backend == r2 ]] && grep -q "^${1:-x}"$'\t' <<<"$r2_machine" && die "$1 is served from the tenant's origin; remove it there" + die "no share with id '${1:-}'" + fi opts="$(awk -F'\t' '{print $6}' <<<"$r")" app="$(opt_val access "$opts" || true)" if [[ -n $app ]]; then @@ -2628,7 +3368,7 @@ cmd_rm_r2() { # cmd_rm_r2 [prune]: the record, a fresh read proving it gone r2_call GET "m/$1" # a lost DELETE must not look done; a gated line and its app stay [[ $cf_code == 404 ]] || die "m/$1 still answers HTTP $cf_code after its delete (HTTP $dcode); its objects${app:+ and its Access app} were kept; rerun $me rm $1" tmp="$(mktemp "$root/.r2-index.XXXXXX")" - awk -F'\t' -v id="$1" '$1 != id' "$index" >"$tmp" && mv -f "$tmp" "$index" # this process's listing drops the row too + awk -F'\t' -v id="$1" '$1 != id' "$r2_snap_file" >"$tmp" && mv -f "$tmp" "$r2_snap_file" # this process's listing drops the row too; on an origin never $index r2_own_drop "$1" (r2_delete_prefix "o/$1.") || echo "share: some objects under o/$1. remain; nothing serves them, and a $me prune removes them after 24 h" >&2 echo "unpublished $1" @@ -2649,6 +3389,8 @@ cmd_refresh_r2() { # cmd_refresh_r2 : this install's own share only; a new p *) die "reading m/$1 answered HTTP $cf_code; nothing was changed" ;; esac body=$cf_body etag=$r2_etag + [[ $backend != r2 || "$(jq -r 'if .v == 2 and .storage == "machine" then 1 else 0 end' <<<"$body" 2>/dev/null)" != 1 ]] || + die "$1 is served from the tenant's origin; remove it there" [[ -n "$(r2_record_row "m/$1" "$body")" && -n $etag ]] || die "no share with id '$1'" oprefix="$(jq -r '.prefix' <<<"$body")" own="$(r2_own_get "$1" || true)" @@ -2681,6 +3423,10 @@ cmd_refresh_r2() { # cmd_refresh_r2 : this install's own share only; a new p cmd_hits_r2() { # cmd_hits_r2 : one Analytics Engine SQL call; the id is checked before it can reach the query [[ ${1:-} =~ ^[0-9a-f]{6}$ ]] || die "no share with id '${1:-}'" local ds="share_${host_name//[.-]/_}" ep + if [[ $backend != r2 ]]; then # an origin: only a sound cloud record reaches the query; a member keeps one account call and no bucket read + r2_call GET "m/$1" || true + [[ $cf_code == 200 && -n "$(r2_record_row "m/$1" "$cf_body")" ]] || die "no share with id '$1'" + fi [[ $ds =~ ^[a-z0-9_]+$ ]] || die "bad hostname '$host_name' for a dataset name" ep="$(cfg r2_endpoint || true)"; acct="${ep#https://}"; acct="${acct%%.*}" [[ -n $ep && $acct =~ ^[A-Za-z0-9-]+$ ]] || die "the profile config has no r2_endpoint; rerun $me setup $host_name --backend r2 --bucket $(cfg bucket || true)" @@ -2693,6 +3439,10 @@ cmd_hits_r2() { # cmd_hits_r2 : one Analytics Engine SQL call; the id is che (.data[0] // {}) as $r | ($r.hits // 0 | tonumber) as $h | ($r.visitors // 0 | tonumber) as $v | n($h; "hit") + ", " + n($v; "visitor") + (if $h > 0 and ($r.last // "") != "" then ", last " + ($r.last | strptime("%Y-%m-%d %H:%M:%S") | mktime | strflocaltime("%F %H:%M")) else "" end)' <<<"$cf_body" + # a member reads no bucket here, so a 0 for an id it did not add may be a machine link the Worker never counts + [[ $backend == r2 && "$(jq -r '.data[0].hits // 0 | tonumber' <<<"$cf_body")" == 0 && -z "$(r2_own_get "$1")" ]] && + echo "this machine counts cloud links only; if $1 is a machine link, its stats are on the tenant's origin: $me hits $1 there" + return 0 } cmd_status_r2() { # read-only: the Worker's health and version, then the listing without its prune (status never deletes) @@ -2712,26 +3462,19 @@ cmd_status_r2() { # read-only: the Worker's health and version, then the listing } cmd_state_r2() { # the Share Bar snapshot for an r2 profile: schema 1 plus "backend"; SHARE_STATE_BRIEF=1 (profiles) skips the share list and its m/ reads - local state_val="not_setup" ready="false" id name src date exp opts + local state_val="not_setup" ready="false" if [[ -n $host_name ]]; then state_val="serving" # nothing runs here; the Worker serves while the account does r2_healthz 2; [[ $hz_code == 200 ]] && ready="true" fi - [[ -z $host_name || ${SHARE_STATE_BRIEF:-0} == 1 ]] || r2_snapshot - { - if [[ -z $host_name || ${SHARE_STATE_BRIEF:-0} == 1 ]]; then : - else - while IFS=$'\t' read -r id name src date exp opts; do - share_url "$id" "$name" "$opts" - printf '%s\t%s\t%s\t%s\t%s\n' "$id" "$name" "$share_url_out" "${exp:-0}" "$(opt_val access_rule "$opts" || true)" - done < <(rows) - fi - } | jq -nR --arg state "$state_val" --argjson ready "$ready" --arg host "$host_name" --argjson access_pending "$(pending_count)" ' - def row: split("\t") | {id: .[0], name: .[1], url: .[2], kind: "snapshot", own_host: null, - expires: (.[3] | tonumber), access: (if .[4] == "" then null else .[4] end)}; + list_out="" list_cloud_error="" list_cloud_more=0 + [[ -z $host_name || ${SHARE_STATE_BRIEF:-0} == 1 ]] || list_rows state + state_lines | jq -nR --arg state "$state_val" --argjson ready "$ready" --arg host "$host_name" --argjson access_pending "$(pending_count)" \ + --arg cerr "$list_cloud_error" --argjson cmore "$list_cloud_more" "$state_jq_row"' {schema: 1, state: $state, ready: $ready, mode: "named", host: (if $host == "" then null else $host end), hosts: "", serves_here: false, service: false, access_pending: $access_pending, backend: "r2", - shares: [inputs | select(length > 0) | row]}' + shares: [inputs | select(length > 0) | row], r2: true} + + (if $cerr == "" then {} else {cloud_error: $cerr} end) + (if $cmore > 0 then {cloud_more: $cmore} else {} end)' } r2_own_get() { # r2_own_get : the "\t\t" line for one of this install's own adds @@ -2774,7 +3517,7 @@ r2_creds() { # r2_creds: the publisher pair into r2_key_id/r2_secret; the token r2_call() { # r2_call METHOD KEY [ARG] [curl args...]: one S3 call on cf_try's globals; LIST takes [continuation]; 429 and 5xx retried, Retry-After capped at 30s cf_body="" cf_code=000 cf_err="" r2_etag="" - [[ $backend == r2 ]] || { cf_err=backend; return 1; } + [[ $backend == r2 || $r2on == 1 ]] || { cf_err=backend; return 1; } if [[ ${SHARE_R2_DRY:-0} == 1 ]]; then if [[ $tunnel != 0 || $host_name != *.test ]]; then echo "share: SHARE_R2_DRY=1 is a test seam for SHARE_TUNNEL=0 and a .test hostname only; the r2 call was refused; unset it" >&2 @@ -2812,6 +3555,7 @@ r2_dry() { # r2_dry METHOD KEY [ARG]: a directory-backed bucket for tests; one " mkdir -p "$d" "$root" if [[ ${SHARE_R2_DRY_PAUSE:-} == "$meth $key" ]]; then local w=0 + : >"$d/.paused" # a test waits for this, then looks at the world while the call is held while [[ ! -f $d/.resume && $w -lt 600 ]]; do sleep 0.05; w=$((w + 1)); done fi printf '%s %s\n' "$meth" "$key" >>"$root/r2-calls.log" @@ -2819,6 +3563,11 @@ r2_dry() { # r2_dry METHOD KEY [ARG]: a directory-backed bucket for tests; one " f="$d/$key" case $meth in PUT) + [[ ${SHARE_R2_DRY_PUT:-} == 500 || ${SHARE_R2_DRY_FAIL:-} == "PUT $key" ]] && { cf_code=500 cf_err=InternalError; return 0; } + if [[ ${SHARE_R2_DRY_PUT:-} == race-once && $key == m/* && ! -f $d/.fx/race.done ]]; then # another publisher takes this id just before the PUT + mkdir -p "$d/.fx" "${f%/*}"; : >"$d/.fx/race.done" + printf '{"v":1,"id":"%s","name":"theirs.txt","src":"/x","added":"2026-10-01","expires":0,"opts":"","prefix":"o/%s.00000000/","by":"peer"}\n' "${key#m/}" "${key#m/}" >"$f" + fi for h in "${@:4}"; do case "$(printf '%s' "$h" | tr '[:upper:]' '[:lower:]')" in -h) ;; @@ -2845,7 +3594,7 @@ r2_dry() { # r2_dry METHOD KEY [ARG]: a directory-backed bucket for tests; one " [[ -n $arg3 ]] && { urldec "$arg3"; want="$urldec_out"; } # the continuation token arrives url-encoded, as S3 serves it while IFS= read -r k; do [[ $k == "$key"* ]] && [[ -z $want || $k > $want ]] && all+="$k"$'\n' - done < <(cd "$d" && find . -type f ! -name '.*' ! -path './.*' ! -name 'r2-calls.log' | sed 's|^\./||' | LC_ALL=C sort) + done < <({ cd "$d" && find . -type f ! -name '.*' ! -path './.*' ! -name 'r2-calls.log' | sed 's|^\./||'; [[ -z ${SHARE_R2_DRY_LIST_EXTRA:-} ]] || echo "$SHARE_R2_DRY_LIST_EXTRA"; } | LC_ALL=C sort) # EXTRA: a key no file holds (m/../share.json) cf_body='dry'"$key"'1000url' while IFS= read -r k; do [[ -n $k ]] || continue @@ -2871,18 +3620,40 @@ r2_api() { # r2_api METHOD PATH [JSON]: cf_try, or the dry account under SHARE_R cf_try "$@" } +r2_api_retry() { # r2_api_retry METHOD PATH [JSON]: r2_api again on a Cloudflare 5xx, 3 tries in all, SHARE_CF_RETRY_SLEEP seconds (default 2) doubling between them; the last answer stays in cf_code + local try=1 nap="${SHARE_CF_RETRY_SLEEP:-2}" + while :; do + r2_api "$@" + [[ $cf_code == 5?? && $try -lt 3 ]] || return 0 + echo "share: Cloudflare answered HTTP $cf_code to $1 $2; retrying in ${nap}s (try $try of 3)" >&2 + sleep "$nap"; nap=$((nap * 2)); try=$((try + 1)) + done +} + r2_api_dry() { # the account API under SHARE_R2_DRY: state files in $SHARE_R2_DRY_DIR/.cf, one "API METHOD path" line per call local d="${SHARE_R2_DRY_DIR:-$root/r2-dry}/.cf" p=$2 n deny='{"success":false,"errors":[{"code":10000}]}' mkdir -p "$d" "$root" printf 'API %s %s\n' "$1" "$p" >>"$root/r2-calls.log" cf_code=200 cf_err="" cf_body='{"success":true,"result":[]}' + # SHARE_R2_DRY_FAIL="API ": that call answers 500 and changes nothing; a die-injection seam + # shellcheck disable=SC2053 # the seam is a glob on purpose + # SHARE_R2_DRY_FAIL_TIMES= limits the failure to the first n matching calls, so a retry can succeed + if [[ -n ${SHARE_R2_DRY_FAIL:-} && "API $1 $p" == $SHARE_R2_DRY_FAIL ]]; then + n="$(cat "$d/fail.count" 2>/dev/null || echo 0)" + if [[ -z ${SHARE_R2_DRY_FAIL_TIMES:-} || $n -lt $SHARE_R2_DRY_FAIL_TIMES ]]; then + echo $((n + 1)) >"$d/fail.count" + cf_code=500 cf_err=10013 cf_body='{"success":false,"errors":[{"code":10013}]}'; return 0 + fi + fi case "$1 $p" in - "GET /user/tokens/verify") cf_body='{"success":true,"result":{"id":"dry-token-id","status":"active"}}' ;; + "GET /user/tokens/verify") cf_body="$(jq -nc --arg e "${SHARE_R2_DRY_EXPIRES:-}" '{success: true, result: ({id: "dry-token-id", status: "active"} + (if $e == "" then {} else {expires_on: $e} end))}')" ;; "GET /zones?name="*) n="${p#*name=}"; n="${n%%&*}" [[ $n == "${host_name#*.}" ]] && cf_body="{\"success\":true,\"result\":[{\"id\":\"zone-dry\",\"name\":\"$n\",\"account\":{\"id\":\"acct-dry\"}}]}" ;; "GET "*/workers/scripts/*/settings) + n="${p%/settings}"; n="${n##*/}" if [[ ${SHARE_R2_DRY_ROLE:-} == deny ]]; then cf_code=403 cf_body=$deny + elif [[ -f $d/script-$n.json ]]; then cf_body="{\"success\":true,\"result\":$(cat "$d/script-$n.json")}" # a second Worker on the account (an alias's) elif [[ -f $d/script.json ]]; then cf_body="{\"success\":true,\"result\":$(cat "$d/script.json")}" else cf_code=404 cf_body='{"success":false,"errors":[{"code":10007}]}'; fi ;; "PUT "*/workers/scripts/*) cf_body='{"success":true,"result":{}}' ;; # r2_deploy writes script.json itself @@ -2907,6 +3678,17 @@ r2_api_dry() { # the account API under SHARE_R2_DRY: state files in $SHARE_R2_DR if [[ ${SHARE_R2_DRY_DOMAINS:-} == 500 ]]; then cf_code=500 cf_body='{"success":false,"errors":[{"code":10013}]}' elif [[ -f $d/domain.json ]]; then cf_body="{\"success\":true,\"result\":[$(jq -c '{id: "dom-dry"} + .' "$d/domain.json")]}"; fi ;; "DELETE "*/workers/domains/*) rm -f "$d/domain.json" ;; + "GET /zones/"*/workers/routes) + if [[ ${SHARE_R2_DRY_ROUTES:-} == deny ]]; then cf_code=403 cf_body=$deny + else cf_body="{\"success\":true,\"result\":$(cat "$d/routes.json" 2>/dev/null || echo '[]')}"; fi ;; + "POST /zones/"*/workers/routes) + jq -c --argjson r "$3" '. + [$r + {id: "route-\(length + 1)"}]' "$d/routes.json" 2>/dev/null >"$d/routes.tmp" || jq -nc --argjson r "$3" '[$r + {id: "route-1"}]' >"$d/routes.tmp" + mv -f "$d/routes.tmp" "$d/routes.json"; cf_body='{"success":true,"result":{}}' ;; + "PUT /zones/"*/workers/routes/*) + jq -c --argjson r "$3" --arg id "${p##*/}" 'map(if .id == $id then $r + {id: $id} else . end)' "$d/routes.json" >"$d/routes.tmp" && mv -f "$d/routes.tmp" "$d/routes.json" + cf_body='{"success":true,"result":{}}' ;; + "DELETE /zones/"*/workers/routes/*) + jq -c --arg id "${p##*/}" 'map(select(.id != $id))' "$d/routes.json" >"$d/routes.tmp" && mv -f "$d/routes.tmp" "$d/routes.json" ;; "DELETE "*/workers/scripts/*) rm -f "$d/script.json" ;; "DELETE "*/r2/buckets/*) rm -f "$d/bucket" ;; "PUT "*/workers/domains) jq -c '{hostname, service}' <<<"$3" >"$d/domain.json"; cf_body='{"success":true,"result":{}}' ;; @@ -2921,27 +3703,39 @@ r2_api_dry() { # the account API under SHARE_R2_DRY: state files in $SHARE_R2_DR cf_err="$(jq -r '.errors[0].code // empty' <<<"$cf_body" 2>/dev/null || true)" } -r2_healthz() { # r2_healthz [secs]: hz_code (200 only for a "200 ok" body), hz_pair (X-Share-Worker), hz_gate (X-Share-Gate: 1 when TEAM is set) from https:///healthz - hz_code=000 hz_pair="" hz_gate=0 - if [[ ${SHARE_R2_DRY:-0} == 1 ]]; then - local d="${SHARE_R2_DRY_DIR:-$root/r2-dry}/.cf" - printf 'HEALTHZ\n' >>"$root/r2-calls.log" - [[ ${SHARE_R2_DRY_HEALTHZ:-} != down && -f $d/domain.json && -f $d/script.json ]] || return 0 - hz_code=200 - hz_pair="$(jq -r '[(.bindings[] | select(.name == "VERSION") | .text), (.bindings[] | select(.name == "SHA") | .text)] | join(" ")' "$d/script.json")" - hz_gate="$(jq -r 'if any(.bindings[]; .name == "TEAM" and (.text // "") != "") then 1 else 0 end' "$d/script.json")" - return 0 - fi +r2_healthz() { # r2_healthz [secs]: hz_code (200 only for a "200 ok" body), hz_pair (X-Share-Worker), hz_gate (X-Share-Gate: 1 when TEAM is set), hz_tunnel (X-Share-Tunnel) from https:///healthz + # a tenant Worker answers 503 with its pair and X-Share-Tunnel: 0 while the origin is off: the Worker is up, so hz_code is 200 and hz_tunnel 0 + hz_code=000 hz_pair="" hz_gate=0 hz_tunnel="" local h b h="$(mktemp "${TMPDIR:-/tmp}/share-hz.XXXXXX")" - # DNS over HTTPS first, as live_check does: a new name's NXDOMAIN from before its record existed sits in the local cache - b="$(curl -sS --max-time "${1:-5}" --doh-url https://1.1.1.1/dns-query -D "$h" "https://$host_name/healthz" 2>/dev/null)" || - b="$(curl -sS --max-time "${1:-5}" -D "$h" "https://$host_name/healthz" 2>/dev/null)" || true - hz_code="$(awk 'NR == 1 {print $2}' "$h")"; hz_code="${hz_code:-000}" - [[ $hz_code != 200 || $b == ok ]] || hz_code=000 + if [[ ${SHARE_R2_DRY:-0} == 1 ]]; then # the dry Worker's answer as a header file and a body, read by the same lines below + local d="${SHARE_R2_DRY_DIR:-$root/r2-dry}/.cf" pair gate + printf 'HEALTHZ\n' >>"$root/r2-calls.log" + local routed=0 # a tenant Worker answers through its route on /*, an r2 profile's through its custom domain + [[ -f $d/routes.json ]] && jq -e --arg p "$host_name/*" 'any(.[]; .pattern == $p)' "$d/routes.json" >/dev/null 2>&1 && routed=1 + if [[ ${SHARE_R2_DRY_HEALTHZ:-} != down && ( -f $d/domain.json || $routed == 1 ) && -f $d/script.json ]]; then + pair="$(jq -r '[(.bindings[] | select(.name == "VERSION") | .text), (.bindings[] | select(.name == "SHA") | .text)] | join(" ")' "$d/script.json")" + gate="$(jq -r 'if any(.bindings[]; .name == "TEAM" and (.text // "") != "") then 1 else 0 end' "$d/script.json")" + case ${SHARE_R2_DRY_HEALTHZ:-} in + tunnel-down) printf 'HTTP/2 503 \r\nx-share-worker: %s\r\nx-share-gate: %s\r\nx-share-tunnel: 0\r\n\r\n' "$pair" "$gate" >"$h"; b="tunnel down" ;; + 503-bare) printf 'HTTP/2 503 \r\n\r\n' >"$h"; b="" ;; + *) local tl=""; jq -e 'any(.bindings[]; .name == "PASS" and .text == "1")' "$d/script.json" >/dev/null && tl=$'x-share-tunnel: 1\r\n' + printf 'HTTP/2 200 \r\nx-share-worker: %s\r\nx-share-gate: %s\r\n%s\r\n' "$pair" "$gate" "$tl" >"$h"; b="ok" ;; + esac + fi + else + # DNS over HTTPS first, as live_check does: a new name's NXDOMAIN from before its record existed sits in the local cache + b="$(curl -sS --max-time "${1:-5}" --doh-url https://1.1.1.1/dns-query -D "$h" "https://$host_name/healthz" 2>/dev/null)" || + b="$(curl -sS --max-time "${1:-5}" -D "$h" "https://$host_name/healthz" 2>/dev/null)" || true + fi + hz_code="$(awk 'NR == 1 {sub(/\r$/, ""); print $2}' "$h")"; hz_code="${hz_code:-000}" hz_pair="$(awk 'tolower($1) == "x-share-worker:" {sub(/\r$/, ""); print $2 " " $3; exit}' "$h")" [[ "$(awk 'tolower($1) == "x-share-gate:" {sub(/\r$/, ""); print $2; exit}' "$h")" == 1 ]] && hz_gate=1 + hz_tunnel="$(awk 'tolower($1) == "x-share-tunnel:" {sub(/\r$/, ""); print $2; exit}' "$h")" rm -f "$h" + if [[ $hz_code == 503 && $hz_tunnel == 0 && $hz_pair =~ ^[0-9]+\ [0-9a-f]{12}$ ]]; then hz_code=200 # read from the headers, never the status alone + elif [[ $hz_code == 200 && ${b:-} != ok ]]; then hz_code=000; fi + return 0 } r2_wait_healthz() { # r2_wait_healthz admin|join: three 200s in a row within SHARE_R2_WAIT; admin also needs this CLI's pair; a new name waits for its certificate @@ -2949,11 +3743,15 @@ r2_wait_healthz() { # r2_wait_healthz admin|join: three 200s in a row within SHA [[ ${SHARE_R2_DRY:-0} == 1 ]] && poll=0 while :; do r2_healthz - if [[ $hz_code == 200 && ( $1 == join || $hz_pair == "$WORKER_VERSION $WORKER_SHA" ) ]]; then + if [[ $hz_code == 200 && ( $1 == join || $hz_pair == "$WORKER_VERSION $WORKER_SHA" ) && ( $1 != tenant || $hz_tunnel == 1 ) ]]; then streak=$((streak + 1)); [[ $streak -ge 3 ]] && return 0 else streak=0 fi + if [[ $1 == tenant && $((SECONDS - start)) -ge $wait ]]; then # the config is written by now: the rollback is --no-r2 + local leg="the Worker leg"; [[ $hz_code == 200 && $hz_pair == "$WORKER_VERSION $WORKER_SHA" ]] && leg="the tunnel leg (X-Share-Tunnel: ${hz_tunnel:-none})" + tenant_die "https://$host_name/healthz did not answer 200 ok with this share's Worker and X-Share-Tunnel: 1 three times in a row within ${wait}s (last: $hz_code ${hz_pair:-no pair}); $leg is down" + fi [[ $((SECONDS - start)) -lt $wait ]] || die "https://$host_name/healthz did not answer 200 ok three times in a row within ${wait}s (last: $hz_code ${hz_pair:-no pair}); the config was not written; rerun setup once the certificate is issued" sleep "$poll" @@ -2963,8 +3761,10 @@ r2_wait_healthz() { # r2_wait_healthz admin|join: three 200s in a row within SHA r2_deploy_meta() { # r2_deploy_meta [salt]: the multipart metadata; printf only, so the salt never reaches an argv local salt="" [[ -n ${3:-} ]] && salt=',{"type":"secret_text","name":"SALT","text":"'"$3"'"}' - printf '{"main_module":"share.js","compatibility_date":"%s","bindings":[{"type":"r2_bucket","name":"BUCKET","bucket_name":"%s"},{"type":"analytics_engine","name":"HITS","dataset":"%s"},{"type":"plain_text","name":"HOST","text":"%s"},{"type":"plain_text","name":"TEAM","text":"%s"},{"type":"plain_text","name":"VERSION","text":"%s"},{"type":"plain_text","name":"SHA","text":"%s"}%s],"keep_bindings":["secret_text"]}\n' \ - "$WORKER_COMPAT_DATE" "$r2_bucket" "$2" "$host_name" "$1" "$WORKER_VERSION" "$WORKER_SHA" "$salt" + local tenant="" # a tenant Worker (setup --r2) passes misses to the origin and 301s its aliases; an r2-only profile binds neither + [[ -z ${r2_pass:-} ]] || tenant=',{"type":"plain_text","name":"PASS","text":"1"},{"type":"plain_text","name":"ALIASES","text":"'"${r2_aliases:-}"'"}' + printf '{"main_module":"share.js","compatibility_date":"%s","bindings":[{"type":"r2_bucket","name":"BUCKET","bucket_name":"%s"},{"type":"analytics_engine","name":"HITS","dataset":"%s"},{"type":"plain_text","name":"HOST","text":"%s"},{"type":"plain_text","name":"TEAM","text":"%s"},{"type":"plain_text","name":"VERSION","text":"%s"},{"type":"plain_text","name":"SHA","text":"%s"}%s%s],"keep_bindings":["secret_text"]}\n' \ + "$WORKER_COMPAT_DATE" "$r2_bucket" "$2" "$host_name" "$1" "$WORKER_VERSION" "$WORKER_SHA" "$tenant" "$salt" } r2_deploy() { # r2_deploy : one multipart PUT; the first deploy adds the SALT secret, later ones keep it @@ -3024,12 +3824,404 @@ write_config_r2() { # write_config_r2 : the r2 profile config; keeps api echo "bucket=$1" echo "port=r2" echo "r2_endpoint=$r2_endpoint" + [[ -n ${r2_join_aliases:-} ]] && echo "aliases=$r2_join_aliases" # a member's rm and expiry of a folded share delete its app [[ -n $keep_api ]] && echo "api_token_cmd=$keep_api" [[ -n $keep_r2 ]] && echo "r2_token_cmd=$keep_r2" } >"$config.tmp" mv -f "$config.tmp" "$config" } +tenant_admin() { # tenant_admin : setup --r2 and --no-r2 on the origin: the profile checks, the admin token, zone, and the admin's own S3 pair + [[ $mode != quick ]] || die "$host_name is a quick tunnel; R2 needs a named tunnel tenant: '$me teardown', then '$me setup '" + [[ -n $(cfg tunnel_id || true) && "$(cfg hostname || true)" == "$host_name" && $backend != r2 ]] || + die "this profile is not the tunnel origin of $host_name; run '$me setup $host_name' on the origin first" + host_ok || die "run this on the origin, $hosts" + local c + for c in jq curl; do command -v "$c" >/dev/null || die "missing: $c (brew install $c)"; done + api_token="${CLOUDFLARE_API_TOKEN:-}" + [[ -n $api_token ]] || die "setup --r2 and --no-r2 read the tenant admin token from CLOUDFLARE_API_TOKEN only (Workers Scripts Edit, Workers R2 Storage Edit, Workers Routes Edit, Zone Read, DNS Read, Access Organizations Read); nothing was changed" + r2_setup_zone + r2_endpoint="https://$acct.r2.cloudflarestorage.com" r2_bucket=$1 + r2_api GET /user/tokens/verify # the admin token's own S3 pair, as SPEC-007's setup uses it + r2_key_id="$(jq -r '.result.id // empty' <<<"$cf_body" 2>/dev/null || true)" + [[ $cf_code == 200 && -n $r2_key_id ]] || die "the token did not verify (HTTP $cf_code); nothing was changed" + r2_secret="$(printf '%s' "$api_token" | shasum -a 256 | cut -d' ' -f1)" + r2on=1 +} + +tenant_route() { # tenant_route : the routes on the zone; sets route_id and route_open for /* naming the worker; dies on a route of another script on this host + local w=$1 p sc + route_id="" route_open="" + r2_api GET "/zones/$zone_id/workers/routes" + [[ $cf_code == 200 ]] || die "reading Worker routes answered HTTP $cf_code (the token needs Workers Routes Edit); nothing was changed" + while IFS=$'\t' read -r p sc; do # any pattern whose host part matches this host and names another script would shadow or be shadowed by ours + [[ -n $p ]] || continue + # shellcheck disable=SC2053 # the route's host part is a glob on purpose + [[ $host_name == ${p%%/*} ]] || continue + [[ $sc == "$w" ]] || die "Worker route $p names script ${sc:-none}, not $w; refused, nothing was changed" + done < <(jq -r '.result[]? | [.pattern, (.script // "")] | @tsv' <<<"$cf_body") + route_id="$(jq -r --arg p "$host_name/*" --arg w "$w" '[.result[]? | select(.pattern == $p and .script == $w) | .id] | first // empty' <<<"$cf_body")" + route_open="$(jq -r --arg p "$host_name/*" --arg w "$w" '[.result[]? | select(.pattern == $p and .script == $w) | .request_limit_fail_open] | first // empty' <<<"$cf_body")" +} + +tenant_config() { # tenant_config [ ]: the four tenant keys in the tunnel config; none given drops them + local tmp; tmp="$(mktemp "$config_dir/.config.XXXXXX")" + grep -vE '^(bucket|r2_endpoint|storage_default|aliases)=' "$config" >"$tmp" || true + [[ $# == 0 ]] || printf 'bucket=%s\nr2_endpoint=%s\nstorage_default=%s\naliases=%s\n' "$1" "$2" "$3" "$4" >>"$tmp" + mv -f "$tmp" "$config" + if [[ ${SHARE_R2_DRY:-0} == 1 ]]; then if [[ $# == 0 ]]; then echo "CONFIG r2 off"; else echo "CONFIG r2 on"; fi >>"$root/r2-calls.log"; fi +} + +alias_scan() { # alias_scan strict|list: after r2_read, every gated cloud record by its Access app's name: fold_apps (" " per app named for $alias), fold_late (ids gated at this host already: added since the fold); strict dies on any other app, before any write + local cid copts uuid nm checked=0 + fold_apps="" fold_late="" + while IFS=$'\t' read -r cid _ _ _ _ copts; do + uuid="$(opt_val access "$copts" || true)" + [[ -n $uuid ]] || continue + if [[ $1 == strict && $checked == 0 ]]; then + access_apps_edit_ok || die "the token lacks Access: Apps and Policies Edit, which --alias needs to move each gated link's app to $host_name; nothing was changed" + checked=1 + fi + cf_try GET "/accounts/$acct/access/apps/$uuid" + nm=""; [[ $cf_code == 200 ]] && nm="$(jq -r '.result.name // empty' <<<"$cf_body")" + case $nm in + "share $cid $alias "*) fold_apps+="$uuid $cid"$'\n' ;; + "share $cid $host_name "*) fold_late+="${fold_late:+ }$cid" ;; + *) [[ $1 != strict ]] || die "the Access app $uuid of cloud link $cid answered HTTP $cf_code named '$nm', not 'share $cid $alias '; refused, nothing was changed" ;; + esac + done < <(crows) +} + +alias_dests() { # alias_dests add|undo|drop: each fold app gains /{,/*}, loses them again, or loses /{,/*}; the PUT sends the full read-back (an omitted field resets to its default) and is skipped when nothing changes; alias_moved gets " " per changed app + local uuid id have want aud body + alias_moved="" + while read -r uuid id; do + [[ -n $uuid ]] || continue + cf_try GET "/accounts/$acct/access/apps/$uuid" + [[ $cf_code == 200 ]] || return 1 + aud="$(jq -r '.result.aud // empty' <<<"$cf_body")" + have="$(jq -c '[.result.destinations[]?]' <<<"$cf_body")" + case $1 in + add) want="$(jq -c --arg u "$host_name/$id" 'reduce ($u, $u + "/*") as $x (.; if any(.[]; .uri == $x) then . else . + [{type: "public", uri: $x}] end)' <<<"$have")" ;; + undo) want="$(jq -c --arg u "$host_name/$id" 'map(select(.uri != $u and .uri != $u + "/*"))' <<<"$have")" ;; + drop) want="$(jq -c --arg u "$alias/$id" 'map(select(.uri != $u and .uri != $u + "/*"))' <<<"$have")" ;; + esac + [[ $want != "$have" ]] || continue + # del(.self_hosted_domains, .domain): these legacy mirrors of an app's original single destination + # are not kept in sync by a later PUT. Echoing one back stale, now naming a destination this PUT + # removes (drop's own case), answers 400; dropping them lets the API recompute them from destinations. + body="$(jq -c --argjson d "$want" '.result | .destinations = $d | del(.self_hosted_domains, .domain)' <<<"$cf_body")" + cf_try PUT "/accounts/$acct/access/apps/$uuid" "$body" + [[ $cf_code == 200 && "$(jq -r '.result.aud // empty' <<<"$cf_body")" == "$aud" ]] || return 1 + alias_moved+="$id $aud"$'\n' + done <<<"${fold_apps:-}" +} + +alias_wait_301() { # alias_wait_301: https:///healthz answers 301 to https:///healthz three times in a row within SHARE_R2_WAIT + local wait="${SHARE_R2_WAIT:-300}" start=$SECONDS streak=0 out d="${SHARE_R2_DRY_DIR:-$root/r2-dry}/.cf" + while :; do + out="000 " + if [[ ${SHARE_R2_DRY:-0} == 1 ]]; then # the dry alias answers 301 once its domain names this host's Worker and that Worker binds it in ALIASES + printf 'ALIAS-301 %s\n' "$alias" >>"$root/r2-calls.log" + if [[ ${SHARE_R2_DRY_FAIL:-} != ALIAS-301 && "$(jq -r .service "$d/domain.json" 2>/dev/null)" == "share-${host_name//./-}" ]] && + jq -e --arg a "$alias" 'any(.bindings[]; .name == "ALIASES" and (.text | split(",") | index($a)))' "$d/script.json" >/dev/null 2>&1; then + out="301 https://$host_name/healthz" + fi + else + out="$(curl -s -o /dev/null --max-time 10 --doh-url https://1.1.1.1/dns-query -w '%{http_code} %{redirect_url}' "https://$alias/healthz" 2>/dev/null)" || + out="$(curl -s -o /dev/null --max-time 10 -w '%{http_code} %{redirect_url}' "https://$alias/healthz" 2>/dev/null)" || out="000 " + fi + if [[ $out == "301 https://$host_name/healthz" ]]; then streak=$((streak + 1)); [[ $streak -ge 3 ]] && return 0; else streak=0; fi + [[ $((SECONDS - start)) -lt $wait ]] || return 1 + if [[ ${SHARE_R2_DRY:-0} == 1 ]]; then sleep 0.2; else sleep 2; fi + done +} + +alias_rollback() { # alias_rollback: the alias rollback list on stderr, in the order R1 rehearses it (the app destinations come back before the rebind) + local apps="" ptrs cl uuid id + while read -r uuid id; do [[ -n $uuid ]] && apps+="${apps:+, }$uuid ($id)"; done <<<"${fold_apps:-}" + ptrs="$({ rows | cut -f1; cut -f1 <<<"${r2_machine:-}"; } | { grep -E '^[0-9a-f]{6}$' || true; } | sort -u | sed 's|^|m/|' | tr '\n' ' ')" # no local rows and no pointers is a valid state: a grep with no match must not end the run under set -e before the list prints + cl="$(crows | cut -f1 | tr '\n' ' ')" + { + echo "alias rollback for $alias, in this order, each with the tenant admin token:" + echo " 1. each folded Access app gets $alias/ and $alias//* back (PUT the full GET result, only destinations changed): ${apps:-none}" + echo " 2. PUT /accounts/$acct/workers/domains {\"hostname\":\"$alias\",\"service\":\"share-${alias//./-}\",\"zone_id\":\"$zone_id\",\"environment\":\"production\",\"override_existing_origin\":true}" + echo " 3. PUT share.json in bucket $bucket with If-Match on its ETag: {\"v\":1,\"host\":\"$alias\"}" + echo " 4. remove the aliases= line from $config" + echo " 5. DELETE every v:2 machine record: ${ptrs:-none}" + echo " 6. $me setup $host_name --no-r2" + echo " cloud links ${cl:-none }answer at https://$alias// again, never at https://$host_name//; gated ones added since the fold (${fold_late:-none}) have no app on $alias: re-add them or tell their readers first" + } >&2 +} + +tenant_die() { # tenant_die : a die after the config write names the rollback: the alias list when an alias is folded, --no-r2 otherwise + if [[ -n ${alias:-} ]]; then alias_rollback; die "$1; roll back with the alias rollback list above, or rerun the same setup"; fi + die "$1; roll back with: $me setup $host_name --no-r2" +} + +alias_own_handoff() { # the local profile serving $alias hands its r2-own lines to this profile, so refresh keeps working on the shares it added; its own file stays + local c n r id pfx src + for c in "${XDG_CONFIG_HOME:-$HOME/.config}/share/config" "$profiles_dir"/*/config; do + [[ -f $c && $c != "$config" ]] || continue + [[ "$(awk 'index($0, "hostname=") == 1 {print substr($0, 10); exit}' "$c")" == "$alias" ]] || continue + n="${c%/config}"; n="${n##*/}" + if [[ $c == */profiles/*/config ]]; then r="$HOME/share/profiles/$n"; else r="$HOME/share"; fi + [[ -f $r/r2-own ]] || continue + while IFS=$'\t' read -r id pfx src; do + [[ $id =~ ^[0-9a-f]{6}$ && $pfx == "o/$id."*/ ]] && r2_own_put "$id" "$pfx" "$src" + done <"$r/r2-own" + echo "r2-own: the shares profile ${n:-default} added at $alias are this profile's to refresh" + done +} + +cmd_setup_tenant() { # cmd_setup_tenant [alias]: setup --r2 on the origin; every read and refusal before the first write; a rerun converges + local bucket=$1 sdef=$2 force=$3 alias=${4:-} worker="share-${host_name//./-}" dataset="share_${host_name//[.-]/_}" calias + calias="$(cfg aliases || true)"; alias="${alias:-$calias}" # a rerun without --alias keeps the folded alias, so the redeploy keeps ALIASES + if [[ -n $alias ]]; then # step 1: one alias, another hostname + if [[ $alias == "$host_name" ]] || ! hostname_ok "$alias"; then die "--alias needs another hostname, not '$alias'; nothing was changed"; fi + [[ -z $calias || $calias == "$alias" ]] || die "$host_name already folds $calias; one alias per tenant; nothing was changed" + fi + [[ $bucket =~ ^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ ]] || die "bad bucket name '$bucket' (3-63 chars of a-z 0-9 -; no leading or trailing -)" + [[ $dataset =~ ^[a-z0-9_]+$ ]] || die "bad hostname '$host_name' for a dataset name" + local cur; cur="$(cfg bucket || true)" + [[ -z $cur || $cur == "$bucket" ]] || die "R2 is on for $host_name with bucket $cur; '$me setup $host_name --no-r2' first" + [[ $tunnel == 0 ]] || running || die "$host_name is not serving on this machine; '$me start', then rerun; nothing was changed" + sdef="${sdef:-$(cfg storage_default || true)}"; sdef="${sdef:-local}" + tenant_admin "$bucket" + [[ -z $alias || $alias == *".$zone_name" ]] || die "--alias $alias is not in zone $zone_name, where the tenant Worker lives; nothing was changed" + + # step 2: only an admin token reads the Worker's settings + local first=0 dep_v=0 dep_sha="" dep_team="" dep_pass="" dep_aliases="" + r2_api GET "/accounts/$acct/workers/scripts/$worker/settings" + case $cf_code in + 200) ;; + 404) first=1 ;; + 401|403) die "the token cannot read Worker $worker's settings: only the tenant admin enables R2 (an admin token in CLOUDFLARE_API_TOKEN); nothing was changed" ;; + *) die "reading Worker $worker answered HTTP $cf_code; nothing was changed" ;; + esac + echo "deploying as admin" + if [[ $first == 0 ]]; then # step 3, SPEC-007 step 4: share's Worker for this host and bucket, never a downgrade + jq -e --arg h "$host_name" --arg b "$bucket" '.result.bindings as $bs + | ($bs | any(.name == "HOST" and .text == $h)) and ($bs | any(.name == "BUCKET" and .bucket_name == $b)) + and ($bs | any(.name == "VERSION" and (.text | test("^[0-9]+$"))))' <<<"$cf_body" >/dev/null || + die "Worker $worker exists but is not share's Worker for $host_name on bucket $bucket (its bindings differ); refused, nothing was changed" + dep_v="$(jq -r '.result.bindings[] | select(.name == "VERSION") | .text' <<<"$cf_body")" + dep_sha="$(jq -r '[.result.bindings[] | select(.name == "SHA") | .text] | first // ""' <<<"$cf_body")" + dep_team="$(jq -r '[.result.bindings[] | select(.name == "TEAM") | .text] | first // ""' <<<"$cf_body")" + dep_pass="$(jq -r '[.result.bindings[] | select(.name == "PASS") | .text] | first // ""' <<<"$cf_body")" + dep_aliases="$(jq -r '[.result.bindings[] | select(.name == "ALIASES") | .text] | first // ""' <<<"$cf_body")" + [[ $dep_v -le $WORKER_VERSION || $force == 1 ]] || + die "Worker $worker runs version $dep_v, newer than this share's $WORKER_VERSION; upgrade share, or pass --force to downgrade it; nothing was changed" + fi + + # step 3, SPEC-007 step 5: a private bucket, empty or holding this host's marker + local bucket_new=0 marker="" marker_etag="" + r2_api GET "/accounts/$acct/r2/buckets/$bucket" + case $cf_code in + 404) bucket_new=1 ;; + 200) + r2_api GET "/accounts/$acct/r2/buckets/$bucket/domains/managed" + [[ $cf_code == 200 ]] || die "reading bucket $bucket's r2.dev setting answered HTTP $cf_code; nothing was changed" + [[ "$(jq -r '.result.enabled' <<<"$cf_body")" == false ]] || die "bucket $bucket has its public r2.dev URL on; a share bucket must be private; refused, nothing was changed" + r2_api GET "/accounts/$acct/r2/buckets/$bucket/domains/custom" + [[ $cf_code == 200 ]] || die "reading bucket $bucket's custom domains answered HTTP $cf_code; nothing was changed" + [[ "$(jq '.result.domains | length' <<<"$cf_body")" == 0 ]] || die "bucket $bucket has a public custom domain; a share bucket must be private; refused, nothing was changed" + r2_call LIST "" + [[ $cf_code == 200 ]] || die "listing bucket $bucket answered HTTP $cf_code; nothing was changed" + local has_objects=0; [[ $cf_body == *""* ]] && has_objects=1 + r2_call GET share.json + case $cf_code in + 200) marker="$cf_body" marker_etag=$r2_etag # the fold: a marker naming the alias is the alias's bucket, adopted in place + jq -e --arg h "$host_name" --arg a "$alias" '.v == 1 and (.host == $h or ($a != "" and .host == $a)) + and ((.aliases // []) - [$a] | length == 0)' <<<"$marker" >/dev/null 2>&1 || + die "bucket $bucket's share.json names another hostname or alias; refused, nothing was changed" ;; + 404) [[ $has_objects == 0 ]] || die "bucket $bucket holds objects and no share.json; it is not a share bucket; refused, nothing was changed" ;; + *) die "reading $bucket/share.json answered HTTP $cf_code; nothing was changed" ;; + esac ;; + *) die "reading bucket $bucket answered HTTP $cf_code (the token needs Workers R2 Storage: Edit); nothing was changed" ;; + esac + + # step 4: the alias's custom domain is its own share Worker on this bucket, or this one's after an earlier fold + local alias_svc="" + if [[ -n $alias ]]; then + local aw="share-${alias//./-}" + r2_api GET "/accounts/$acct/workers/domains?hostname=$alias" + [[ $cf_code == 200 ]] || die "reading Worker custom domains answered HTTP $cf_code; nothing was changed" + alias_svc="$(jq -r --arg h "$alias" '[.result[]? | select(.hostname == $h) | .service] | first // empty' <<<"$cf_body")" + if [[ -n $alias_svc && $alias_svc != "$worker" ]]; then + [[ $alias_svc == "$aw" ]] || die "$alias is the custom domain of Worker $alias_svc, not $aw; refused, nothing was changed" + r2_api GET "/accounts/$acct/workers/scripts/$aw/settings" + [[ $cf_code == 200 ]] || cf_body='{}' + jq -e --arg h "$alias" --arg b "$bucket" '.result.bindings as $bs + | ($bs | any(.name == "HOST" and .text == $h)) and ($bs | any(.name == "BUCKET" and .bucket_name == $b))' <<<"$cf_body" >/dev/null 2>&1 || + die "Worker $aw is not share's Worker for $alias on bucket $bucket (HTTP $cf_code, or its bindings differ); refused, nothing was changed" + fi + fi + + # step 5: no route of another script on this host + tenant_route "$worker" + + local team="" # SPEC-007 step 7: the Access team for gated cloud links + r2_api GET "/accounts/$acct/access/organizations" + if [[ $cf_code == 200 ]]; then + team="$(jq -r '.result.auth_domain // empty' <<<"$cf_body")" + [[ $team =~ ^[a-z0-9-]+\.cloudflareaccess\.com$ ]] || team="" + else + team="$dep_team" + fi + + # step 6, under the index lock through step 11 so a concurrent add waits: the bucket snapshot against the local index + mkdir -p "$root" + lock_take index + local ids id clash="" + ids="$(rows | cut -f1)" + if [[ $bucket_new == 0 ]]; then + r2_read + for id in $ids; do + grep -qx "$id" <<<"$r2_mkeys" || continue + grep -q "^$id"$'\t' <<<"$r2_machine" || clash="${clash:+$clash }$id" + done + [[ -z $clash ]] || die "local links $clash share an id with a record in bucket $bucket (a cloud link); rm one of each pair, then rerun; nothing was changed" + fi + fold_apps="" fold_late="" + [[ -z $alias || $bucket_new == 1 ]] || alias_scan strict # step 7: every gated cloud link's app is the alias's (or this host's after a fold) + + # writes. A new bucket gets its marker before any pointer, so a die below leaves a bucket that names this host + if [[ $bucket_new == 1 ]]; then + r2_api POST "/accounts/$acct/r2/buckets" "{\"name\":\"$bucket\"}" + [[ $cf_code == 200 ]] || die "creating bucket $bucket answered HTTP $cf_code" + fi + if [[ -z $marker ]]; then + local mf; mf="$(mktemp "$root/.marker.XXXXXX")" + printf '{"v":1,"host":"%s"}\n' "$host_name" >"$mf" + r2_call PUT share.json "$mf" -H 'If-None-Match: *' + [[ $cf_code == 200 || $cf_code == 412 ]] || { rm -f "$mf"; die "writing $bucket/share.json answered HTTP $cf_code"; } + if [[ $cf_code == 412 ]]; then + r2_call GET share.json + jq -e --arg h "$host_name" '.v == 1 and .host == $h' <<<"$cf_body" >/dev/null 2>&1 || { rm -f "$mf"; die "bucket $bucket's share.json names another hostname"; } + marker="$cf_body" marker_etag=$r2_etag + else + marker="$(cat "$mf")" marker_etag=$r2_etag + fi + rm -f "$mf" + fi + # step 8: the Worker with PASS and ALIASES, workers.dev off + r2_pass=1 r2_aliases="$alias" + if [[ $first == 1 || $dep_v != "$WORKER_VERSION" || $dep_sha != "$WORKER_SHA" || $dep_team != "$team" || $dep_pass != 1 || $dep_aliases != "$r2_aliases" ]]; then + r2_deploy "$worker" "$team" "$dataset" "$first" + echo "worker: $worker $WORKER_VERSION $WORKER_SHA deployed (passes misses to the tunnel)" + else + echo "worker: $worker $WORKER_VERSION $WORKER_SHA already deployed" + fi + r2_api POST "/accounts/$acct/workers/scripts/$worker/subdomain" '{"enabled":false,"previews_enabled":false}' + [[ $cf_code == 200 ]] || die "turning off $worker's workers.dev route answered HTTP $cf_code; no route exists yet, nothing is exposed" + r2_api GET "/accounts/$acct/workers/scripts/$worker/subdomain" + [[ $cf_code == 200 && "$(jq -r '"\(.result.enabled),\(.result.previews_enabled)"' <<<"$cf_body")" == false,false ]] || + die "$worker's workers.dev route still reads on; no route exists yet, nothing is exposed; rerun setup" + + # step 9: a pointer for every local row; one that holds this origin's pointer already is done + local n=0 ptr_hint="" + [[ -z $alias ]] || ptr_hint=" (or DELETE the v:2 pointers written so far: an older member sweep cannot read them)" + for id in $ids; do + grep -q "^$id"$'\t' <<<"${r2_machine:-}" && continue + local name src date exp opts t + IFS=$'\t' read -r _ name src date exp opts <<<"$(row "$id")" + if has_opt live "$opts"; then t=site; else t="$(share_type "$pub/$id/$name")"; fi + r2_pointer_write "$id" "$name" "$t" "$exp" "$opts" "$(opt_val access "$opts" || true)" "$date" + case $cf_code in + 200) n=$((n + 1)) ;; + 412) die "another publisher took id $id in bucket $bucket during setup; the route is not attached; rm one of them, then rerun$ptr_hint" ;; + *) die "writing the pointer m/$id answered HTTP $cf_code; the route is not attached; rerun the same setup$ptr_hint" ;; + esac + done + echo "pointers: $n written" + + # step 10: each folded app gains this host's destinations (AUD kept); Access must enforce there before anything points visitors at it + if [[ -n $fold_apps ]]; then + alias_dests add || { alias_dests undo || true; die "moving the Access apps to $host_name answered HTTP $cf_code; the added destinations were removed; the route is not attached; rerun the same setup (or DELETE the v:2 pointers written so far)"; } + local gid gaud + while read -r gid gaud; do + [[ -n $gid ]] || continue + access_gate "$gid" "$gaud" || { alias_dests undo || true; die "Cloudflare Access did not enforce on $host_name/$gid within ${access_wait}s; the added destinations were removed; the route is not attached; rerun the same setup (or DELETE the v:2 pointers written so far)"; } + done <<<"$alias_moved" + echo "access: $(grep -c . <<<"$fold_apps") gated link(s) also gated at $host_name" + fi + + # step 11: the config, before the marker and the route, so every later die leaves a state a rerun and the rollback can read + tenant_config "$bucket" "$r2_endpoint" "$sdef" "$alias" + lock_release index + + # step 12: the marker names this host and its alias, written only over the copy read above + if [[ -n $alias ]]; then + local want; want="{\"v\":1,\"host\":\"$host_name\",\"aliases\":[\"$alias\"]}" + if [[ "$(jq -c . <<<"${marker:-null}" 2>/dev/null)" != "$want" ]]; then + local mf2; mf2="$(mktemp "$root/.marker.XXXXXX")"; printf '%s\n' "$want" >"$mf2" + if [[ -n $marker ]]; then r2_call PUT share.json "$mf2" -H "If-Match: \"$marker_etag\""; else r2_call PUT share.json "$mf2"; fi + rm -f "$mf2" + [[ $cf_code == 200 ]] || tenant_die "writing $bucket/share.json answered HTTP $cf_code (412: it changed since it was read)" + fi + fi + + # step 13: the route; fail open at a request limit (a drop to the free plan) keeps machine links up + local body="{\"pattern\":\"$host_name/*\",\"script\":\"$worker\",\"request_limit_fail_open\":true}" + if [[ -z $route_id ]]; then + r2_api POST "/zones/$zone_id/workers/routes" "$body" + [[ $cf_code == 200 ]] || tenant_die "attaching the route $host_name/* answered HTTP $cf_code; links still serve from the tunnel" + echo "route: $host_name/* -> $worker" + elif [[ $route_open != true ]]; then + r2_api PUT "/zones/$zone_id/workers/routes/$route_id" "$body" + [[ $cf_code == 200 ]] || tenant_die "setting the route $host_name/* to fail open answered HTTP $cf_code" + echo "route: $host_name/* -> $worker (now fails open)" + else + echo "route: $host_name/* -> $worker already attached" + fi + + # step 14: the alias's custom domain moves to this Worker in place (the plain PUT refuses a bound name) + if [[ -n $alias && $alias_svc != "$worker" ]]; then + r2_api_retry PUT "/accounts/$acct/workers/domains" "{\"hostname\":\"$alias\",\"service\":\"$worker\",\"zone_id\":\"$zone_id\",\"environment\":\"production\",\"override_existing_origin\":true}" + [[ $cf_code == 200 ]] || tenant_die "binding $alias to $worker answered HTTP $cf_code (after up to 3 tries); ${alias_svc:-no Worker} still serves it" + echo "alias: $alias -> $worker (answers 301 to $host_name)" + fi + + # step 15: both legs, then a probe file through the tunnel leg, then the alias's 301 + r2_wait_healthz tenant + if [[ ${SHARE_R2_DRY:-0} == 1 ]]; then echo "TUNNEL-PROBE" >>"$root/r2-calls.log" + else live_check 60 "$host_name" || tenant_die "a probe file did not answer through https://$host_name/ three times in a row; the tunnel leg is down"; fi + [[ -z $alias ]] || alias_wait_301 || tenant_die "https://$alias/healthz did not answer 301 to https://$host_name/healthz three times in a row within ${SHARE_R2_WAIT:-300}s" + + # step 16: the folded apps drop the alias's destinations; this profile takes over the alias profile's own shares + if [[ -n $alias ]]; then + alias_dests drop || die "dropping $alias from the folded Access apps answered HTTP $cf_code; the fold serves; rerun the same setup to finish" + alias_own_handoff + fi + echo "ready: R2 is on for https://$host_name/ (bucket $bucket, storage default $sdef${alias:+, $alias redirects here}). Store this origin's publisher token: $me api-token" +} + +cmd_setup_no_r2() { # cmd_setup_no_r2: the rollback without an alias: the route goes, then the four config keys; the bucket, records, and Worker stay + local bucket worker="share-${host_name//./-}" + bucket="$(cfg bucket || true)" + local alias; alias="$(cfg aliases || true)" + tenant_admin "${bucket:-none}" + if [[ -z $alias && -n $bucket ]]; then # the marker's aliases count too: a 301 landing on a plain tunnel turns every folded cloud link into a 404 + r2_call GET share.json + [[ $cf_code == 200 ]] && alias="$(jq -r '(.aliases // []) | join(",")' <<<"$cf_body" 2>/dev/null || true)" + fi + if [[ -n $alias ]]; then + fold_apps="" fold_late="" + r2_read; alias_scan list + alias_rollback + die "$alias redirects here; run the alias rollback above first; nothing was changed" + fi + tenant_route "$worker" + if [[ -n $route_id ]]; then + r2_api DELETE "/zones/$zone_id/workers/routes/$route_id" + [[ $cf_code == 200 || $cf_code == 404 ]] || die "deleting the route $host_name/* answered HTTP $cf_code; R2 stays on; rerun" + echo "route: $host_name/* removed" + fi + tenant_config + echo "ready: R2 is off for https://$host_name/: the tunnel serves it alone. The bucket, its records, and Worker $worker stay" +} + cmd_setup_r2() { # cmd_setup_r2 : every refusal before the first write local force=$1 bucket=$2 tname=$3 login=$4 service=$5 quick=$6 local rusage="usage: share setup --backend r2 --bucket [--force]" @@ -3077,6 +4269,9 @@ cmd_setup_r2() { # cmd_setup_r2 /dev/null && + die "$host_name is a tenant with R2 on (its Worker passes misses to a tunnel); refused even with --force, nothing was changed; run '$me setup $host_name --r2 --bucket $bucket' on the origin" fi local bucket_new=0 marker="" skipped=0 @@ -3105,7 +4300,9 @@ cmd_setup_r2() { # cmd_setup_r2 /dev/null 2>&1 || - die "bucket $bucket's share.json names another hostname; refused, nothing was changed" ;; + die "bucket $bucket's share.json names another hostname; refused, nothing was changed" + [[ $role != admin ]] || ! jq -e '(.aliases // []) | length > 0' <<<"$marker" >/dev/null 2>&1 || + die "$host_name is a tenant with R2 on (its marker holds aliases); refused even with --force, nothing was changed; run '$me setup $host_name --r2 --bucket $bucket' on the origin" ;; 404) [[ $role == admin ]] || die "bucket $bucket has no share.json; whoever holds the admin token runs '$me setup $host_name --backend r2 --bucket $bucket' first" [[ $has_objects == 0 ]] || die "bucket $bucket holds objects and no share.json; it is not a share bucket; refused, nothing was changed" ;; *) die "reading $bucket/share.json answered HTTP $cf_code; nothing was changed" ;; @@ -3115,13 +4312,29 @@ cmd_setup_r2() { # cmd_setup_r2 /dev/null || true); do + if hostname_ok "$alias" && alias_wait_301; then r2_join_aliases+="${r2_join_aliases:+,}$alias" + else echo "alias $alias: the marker names it, but https://$alias/healthz does not answer 301 here; not kept"; fi + done + if [[ $hz_pair != "$WORKER_VERSION $WORKER_SHA" && -n $hz_tunnel ]]; then # a tenant Worker: only its origin redeploys it + echo "the Worker at $host_name runs $hz_pair; this share ships $WORKER_VERSION $WORKER_SHA; whoever holds the admin token runs '$me setup $host_name --r2 --bucket $bucket' on the tenant's origin" + elif [[ $hz_pair != "$WORKER_VERSION $WORKER_SHA" ]]; then echo "the Worker at $host_name runs $hz_pair; this share ships $WORKER_VERSION $WORKER_SHA; whoever holds the admin token reruns '$me setup $host_name --backend r2 --bucket $bucket'" + fi write_config_r2 "$bucket" echo "ready: https://$host_name/ serves from bucket $bucket. Store your publisher token: $me api-token; then: $me add " return 0 fi + r2_api GET "/zones/$zone_id/workers/routes" # a route on the hostname means a tenant; SPEC-007's minimal admin set may answer 403, which the PASS check covers + case $cf_code in + 200) [[ "$(jq --arg p "$host_name/*" '[.result[]? | select(.pattern == $p)] | length' <<<"$cf_body")" == 0 ]] || + die "$host_name has a Worker route on $host_name/* (a tenant with R2 on); refused even with --force, nothing was changed; run '$me setup $host_name --r2 --bucket $bucket' on the origin" ;; + 403) ;; + *) die "reading Worker routes answered HTTP $cf_code; nothing was changed" ;; + esac r2_api GET "/zones/$zone_id/dns_records?name=$host_name" [[ $cf_code == 200 ]] || die "reading DNS for $host_name answered HTTP $cf_code (the token needs DNS: Read); nothing was changed" local ndns ours theirs @@ -3172,8 +4385,8 @@ cmd_setup_r2() { # cmd_setup_r2 " } +tusage="usage: share setup --r2 --bucket [--storage-default local|cloud] [--alias ] | share setup --no-r2" cmd_setup() { - local tunnel_name="" force=0 service=1 login=0 quick_setup=0 had_host_arg=0 backend_arg="" bucket="" + local tunnel_name="" force=0 service=1 login=0 quick_setup=0 had_host_arg=0 backend_arg="" bucket="" r2flag="" sdef="" alias_arg="" token_stdin=0 [[ -n ${1:-} && ${1:-} != --* ]] && { host_name="$1"; had_host_arg=1; shift; } while [[ $# -gt 0 ]]; do case $1 in @@ -3194,9 +4408,29 @@ cmd_setup() { --quick) quick_setup=1; shift ;; --backend) backend_arg="${2:-}"; shift 2 ;; --bucket) bucket="${2:-}"; shift 2 ;; - *) die "usage: share setup [hostname] [--tunnel-name N] [--force] [--no-service] [--login] [--backend tunnel|r2 --bucket ] | share setup --quick [--no-service]" ;; + --r2|--no-r2) [[ -z $r2flag ]] || die "$tusage"; r2flag="$1"; shift ;; + --storage-default) sdef="${2:-}"; shift 2 ;; + --alias) alias_arg="${2:-}"; [[ -n $alias_arg ]] || die "$tusage"; shift 2 ;; + --token-stdin) token_stdin=1; shift ;; + *) die "usage: share setup [hostname] [--tunnel-name N] [--force] [--no-service] [--login] [--backend tunnel|r2 --bucket ] | share setup --quick [--no-service] | $tusage" ;; esac done + if [[ $token_stdin == 1 ]]; then # migrate's switch: one line on stdin, into this process only; never argv, a file, or a child's environment + CLOUDFLARE_API_TOKEN=""; IFS= read -r CLOUDFLARE_API_TOKEN || true + [[ -n $CLOUDFLARE_API_TOKEN ]] || die "--token-stdin read no token" + fi + if [[ -n $r2flag ]]; then # R2 on a tunnel tenant: the origin's own setup, never a new tunnel + [[ $had_host_arg == 1 && -z $backend_arg && -z $tunnel_name && $service == 1 && $login == 0 && $quick_setup == 0 ]] || die "$tusage" + if [[ $r2flag == --r2 ]]; then + [[ -n $bucket && ( -z $sdef || $sdef == local || $sdef == cloud ) ]] || die "$tusage" + cmd_setup_tenant "$bucket" "$sdef" "$force" "$alias_arg" + else + [[ -z $bucket && -z $sdef && $force == 0 && -z $alias_arg ]] || die "$tusage" + cmd_setup_no_r2 + fi + return + fi + [[ -z $sdef && -z $alias_arg ]] || die "--storage-default and --alias need --r2" [[ -z $backend_arg || $backend_arg == tunnel || $backend_arg == r2 ]] || die "--backend must be 'tunnel' or 'r2'" [[ -z $bucket || $backend_arg == r2 ]] || die "--bucket needs --backend r2" if [[ $backend_arg == r2 ]]; then @@ -3215,6 +4449,9 @@ cmd_setup() { if [[ -n $old_host && $old_host != "$host_name" ]]; then # the main site would answer the new name for the whole pub tree; the apps cover only the old one [[ -z "$(host_name="$old_host" rows | awk -F'\t' '$6 ~ /(^| )access=/')" && $(pending_count) == 0 ]] || die "$old_host has gated shares or Access apps awaiting deletion; under $host_name they would be served with no gate; remove them first ('$me rm ', see '$me ls', then '$me prune' with the token; a line prune reports as malformed is yours to delete from $pending) or run '$me teardown'" + elif [[ -z $old_host && $token_stdin == 0 ]]; then # a not_setup profile can still hold rows from an earlier import; the first setup must not start serving a gated one with no gate. migrate's own switch (--token-stdin) is exempt: it never renames the hostname, so a row's existing Access app still covers it + [[ -z "$(rows | awk -F'\t' '$6 ~ /(^| )access=/')" ]] || + die "this not-yet-set-up profile already holds gated shares; under $host_name they would be served with no gate; remove them first ('$me rm ', see '$me ls') or run '$me teardown'" fi tunnel_name="${tunnel_name:-share-${host_name//./-}}" local other # two profiles on one hostname or one tunnel name would reuse, then delete, one tunnel @@ -3438,6 +4675,8 @@ r2_purge() { # r2_purge : the admin token from the environment; every ch jq -e --arg h "$host_name" --arg b "$bucket" '.result.bindings as $bs | ($bs | any(.name == "HOST" and .text == $h)) and ($bs | any(.name == "BUCKET" and .bucket_name == $b))' <<<"$cf_body" >/dev/null || die "Worker $worker is not share's Worker for $host_name on bucket $bucket (its bindings differ); refused, nothing was changed" + ! jq -e '.result.bindings | any(.name == "PASS" and .text == "1")' <<<"$cf_body" >/dev/null || + die "$host_name is a tenant with R2 on: its origin runs '$me setup $host_name --no-r2' first; refused, nothing was changed" r2_call GET share.json if [[ $cf_code != 200 ]] || ! jq -e --arg h "$host_name" '.v == 1 and .host == $h' <<<"$cf_body" >/dev/null 2>&1; then die "bucket $bucket's share.json does not name $host_name (HTTP $cf_code); refused, nothing was changed" @@ -3491,8 +4730,8 @@ r2_purge_apps() { # every Access app named "share <6 hex> <8 hex>", read while :; do cf_try GET "/accounts/$acct/access/apps?page=$page&per_page=100" [[ $cf_code == 200 ]] || die "listing Access apps answered HTTP $cf_code; the Worker, domain, and bucket stay" - ids="$ids$(jq -r --arg h "$host_name" '.result[] | (.name | split(" ")) as $w - | select(($w | length) == 4 and $w[0] == "share" and ($w[1] | test("^[0-9a-f]{6}$")) and $w[2] == $h and ($w[3] | test("^[0-9a-f]{8}$"))) + ids="$ids$(jq -r --arg h "$host_name" --arg a "$(cfg aliases || true)" '.result[] | (.name | split(" ")) as $w + | select(($w | length) == 4 and $w[0] == "share" and ($w[1] | test("^[0-9a-f]{6}$")) and ([$h] + ($a | split(",")) | index($w[2])) and ($w[3] | test("^[0-9a-f]{8}$"))) | "\(.id) \($w[1])"' <<<"$cf_body")"$'\n' pages="$(jq -r '.result_info.total_pages // 1' <<<"$cf_body")" [[ $page -lt $pages ]] || break @@ -3561,6 +4800,12 @@ case "${1:-status}" in teardown) shift; cmd_teardown "$@" ;; service) cmd_service "${2:-}" ;; add) shift; cmd_add "$@" ;; + import) shift; cmd_import "$@" ;; + migrate) shift; cmd_migrate "$@" ;; + migrate-probe) cmd_migrate_probe ;; + migrate-index) cmd_migrate_index ;; + migrate-tunnel-id) cmd_migrate_tunnel_id ;; + migrate-nonce) cmd_migrate_nonce ;; api-token) shift; cmd_api_token "$@" ;; ls) cmd_prune listing; cmd_ls ;; skill) cmd_skill "${2:-}" ;; diff --git a/docs/decisions/ADR-0008-one-host-per-tenant.md b/docs/decisions/ADR-0008-one-host-per-tenant.md new file mode 100644 index 0000000..df055f1 --- /dev/null +++ b/docs/decisions/ADR-0008-one-host-per-tenant.md @@ -0,0 +1,41 @@ +# ADR-0008: a tenant is one hostname; storage is local by default and cloud per link + +Status: accepted +Date: 2026-10-05 + +## Context + +Dwarves ran one team as two hostnames: machine-local links at `s.d.foundation` (a tunnel profile on the Mac Mini) and cloud links at `f.d.foundation` (an r2 profile, ADR-0007). People had to know which hostname held which link, and Share Bar showed two sections for one team. The personal tenant `s.han.ws` ran from a laptop, so every link answered 530 while it slept. + +The operator direction: a tenant is one hostname with one shared list; storage is local by default exactly as share works today; R2 is optional per tenant and per link; one always-on machine is the origin. The design, the test plan, and the live probes are in `docs/specs/SPEC-008-one-host-per-tenant.md` and `docs/implementation-notes/one-host-per-tenant.md`. + +## Decision + +- One hostname per tenant. Storage shows as a badge on a row, never as a subdomain. +- Local by default. A tenant with R2 off runs today's code paths: no Worker, no bucket call, files byte-identical. `--cloud` is refused there with the line that names the admin command. +- R2 is turned on by the admin on the origin: `setup --r2 --bucket [--storage-default local|cloud] [--alias ]`. It adds a route Worker on `/*` over the existing tunnel CNAME, so `--no-r2` (delete the route) is the whole rollback. +- The Worker reads one bucket key per request. A cloud record (`v:1`) is served from R2 and never reaches the tunnel. A machine pointer (`v:2`, `storage: machine`), no record, or an R2 error passes the request through to the origin on the same hostname. Any other record shape answers 404, so a stale or forged cloud id never falls back to a file on the origin. +- The origin writes a pointer record for every local link while R2 is on. Pointers feed one shared list on every machine and make the id namespace collision-proof (`If-None-Match: *`). A local add fails closed when the pointer cannot be written. +- Bucket rows are display data. A record read from the bucket never enters `rows()`, the index, the Caddyfile, a stage path, or shell arithmetic. +- A member machine publishes cloud links only. `--local`, live ports, and `--host` stay on the origin; a member cannot remove or refresh a machine row. +- `--alias ` folds a former r2 hostname into the tenant: the tenant Worker takes the alias's custom domain and answers 301 to the same path. The bucket and each gated link's Access app (name, AUD) are adopted in place, so every old link keeps working. +- `share migrate --to ` moves an R2-off origin between machines over ssh with the receiver's own `share import`, keeping ids, names, dates, expiries, and gates. The new origin always gets its own tunnel (`-m`); nothing on the old machine is deleted, only moved aside. +- The Worker passes an origin's 502 through unchanged. Only 530, 520 to 527, and a 503 with no `cf-cache-status` mean the machine is offline and answer the 503 offline page. A dead live port is a working machine, and its 502 is the origin's own answer. + +Alternatives rejected: + +- Two hostnames (the starting point): the operator direction rules it out. +- R2 as the default storage: every tenant would pay for and depend on Cloudflare storage it did not ask for. +- A second internal origin hostname behind an Access service token: a hostname, a secret with an expiry, and a token that would open every machine link to its holder. The route pass-through needs none of them. +- Tunnel first, R2 on a 404: cloud links would need the origin up, which defeats them. +- Reusing the old tunnel on the new origin: two connectors until the old machine stops, and a later `teardown` on the old machine would delete the new origin's tunnel. +- A Cloudflare redirect rule for the old hostname: the 301 stays in code the suite tests and the CLI deploys. + +## Consequences + +- With R2 on, every request to the tenant runs the Worker and one bucket read (two for a cloud hit). The origin's local publishing depends on R2 answering and on a stored publisher token; the gain is that no member's cloud link can collide with an origin link. +- Cloud links answer while the origin is off; machine links answer the 503 offline page. `/healthz` reports both legs through headers, so members join and publish while the origin is down. +- A Worker change ships only with a `WORKER_VERSION` bump, as ADR-0007 fixes. This work moved the Worker to version 4. +- A tenant moves between machines in one command, but only with R2 off. Moving an R2-on origin also moves the pointers and the publisher token, which is out of scope. +- An older share CLI on an origin with R2 on still serves and adds local links, without pointers. The next upgraded `ls` writes them and names any collision. +- Share Bar reads one list per tenant, with a storage badge, file-type icon, link type, and lock on each row. diff --git a/docs/how-it-works.md b/docs/how-it-works.md index dfc15dd..4aa1ddd 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -350,6 +350,114 @@ pseudonymizes visitors, it does not anonymize them. `share hits ` checks the over the profile's dataset for that id. It needs a token with Account Analytics: Read, which reads every dataset on the account. +## Tenants: one hostname, local or cloud + +A tenant is one hostname with one shared list. Its origin is the one machine in the tunnel profile's `hosts`; every link is served from that machine's disk through its tunnel unless the admin turns R2 on. The mechanism and its trade-offs: [ADR-0008](decisions/ADR-0008-one-host-per-tenant.md). + +``` + visitor https://s.example.com// https://f.example.com/ (alias) + | | 301 to the tenant host + v v + Cloudflare edge: TLS; an Access app on / and //* per gated link + | + R2 off | R2 on + CNAME to the tunnel | Worker share- on route /* + | GET m/ from the bucket + | cloud record (v:1) -> R2 bytes + | machine pointer (v:2), none -> fetch(request): the route is + | or an R2 error skipped, the request goes on + v to the origin + tunnel --> cloudflared on the origin --> caddy --> pub/ or a live port +``` + +| Tenant state | Origin | Other machines | +|---|---|---| +| R2 off (default) | publishes everything locally, today's code paths and files | none; `setup --quick` stays available | +| R2 on | publishes local or cloud per link; keeps the tunnel and gains `bucket=`, `r2_endpoint=`, `storage_default=` (and `aliases=` after a fold) in its config | members: an r2 profile joined with a bucket-scoped publisher token; they publish cloud links only | + +### Per-link storage + +| `add` form | R2 off | R2 on, `storage_default=local` | R2 on, `storage_default=cloud` | +|---|---|---|---| +| `add ` | local | local | cloud | +| `add --cloud ` | refused, naming the admin command | cloud | cloud | +| `add --local ` | local | local | local | +| `add `, `--host` | local | local | local | + +`--cloud` with a port, `--host`, or `--local` is refused. A member's `--local` and `add ` die with ` serves local links from its origin`. `rm`, `refresh`, and `hits` dispatch on the row's storage; a member's `rm` or `refresh` of a machine row dies before any call, because only the origin serves it. + +### Enabling and leaving R2 + +`CLOUDFLARE_API_TOKEN= share setup --r2 --bucket [--storage-default local|cloud] [--alias ]` runs on the origin only. Every read and refusal comes before the first write, and a rerun after a die converges. The order matters: the Worker is deployed with subdomain and previews off, pointer records are written for every local row, the config keys are written, the marker moves, the route is attached, and live checks (`/healthz` three times with this CLI's version pair, a probe file through the tunnel leg) pass last. + +With `--alias ` the tenant Worker takes the alias's custom domain and answers 301 to the same path on the tenant host. Each gated cloud link's Access app gains the tenant destinations first (the PUT keeps the AUD), passes the gate probe, and drops the alias destinations last. A folded app keeps its name `share `, so `rm` and expiry still delete it. + +`share setup --no-r2` is the rollback for a tenant without an alias: the route goes, then the four config keys. The tunnel serves the hostname alone again; pointers, cloud records, the bucket, and the Worker stay. It refuses while an alias is set, because an alias whose 301 lands on a plain tunnel turns every folded cloud link into a 404. Every die after the marker moved prints the alias rollback list instead. + +### Pointer records + +With R2 on, a local add on the origin writes `m/` as `{"v":2,"id","storage":"machine","name","by","added","expires","opts","type"}` after the stage is built and before the file is published (before the Access app for a gated add). The write carries `If-None-Match: *`, so one namespace spans both storages and a cloud add can never take an id the origin holds. `opts` keeps only `noindex`, `live`, and `gated`, never the rule. A failed pointer write dies before anything is served: a local add fails closed while R2 is on. + +`rm` and `prune` of a local row delete its pointer when a token resolves. The origin's interactive `ls` and `prune` reconcile: a pointer with no local row and a `LastModified` over 10 minutes old is deleted (an add that died), and a local row with no pointer gets one. A local row whose id already holds a cloud record prints ` is shadowed by a cloud link; rm one of them` and nothing is deleted. + +Cloud records stay `v:1` (with an optional `type`), so a v0.8.0 CLI and Worker keep reading every cloud link. A machine pointer is `v:2` and `WORKER_RECORD_V` is 2; an older Worker answers 404 for it. + +### One shared list and its fields + +`ls`, `state`, and `profiles --json` read one row set per profile: the local index on an origin with R2 off, the local index plus every cloud record on an origin with R2 on, and every bucket record on a member. A record read from the bucket is display data only: it never enters `rows()`, the index, the Caddyfile, a stage path, `refresh`, or shell arithmetic, and every field is checked before it is shown. A bucket read that fails leaves the local rows and adds `cloud_error`. + +`state` and `profiles --json` read the bucket only with a token from the Keychain item or the mode-600 file (never an `api_token_cmd`, which may prompt during a poll) and fetch the newest 25 records; `cloud_more` counts the rest. `share ls` prints one tag before each link (`cloud`, `machine`, or `live`), then the type and `by=`. + +| `type` | First match wins | +|---|---| +| `site` | a live link; a folder with `index.html` at its root; a single `.html` or `.htm` file | +| `folder` | any other folder | +| `markdown`, `pdf` | `.md` `.markdown`; `.pdf` | +| `image`, `video`, `audio` | `.png .jpg .jpeg .gif .webp .svg .heic .ico .avif`; `.mp4 .mov .webm .m4v .mkv`; `.mp3 .wav .m4a .aac .flac .ogg` | +| `archive`, `text` | `.zip .tar .gz .tgz .bz2 .xz .7z .rar .dmg`; common text and source extensions | +| `other` | anything else | + +The extension match ignores case. Local rows are typed at read time; new cloud records and pointers store `type`; a `v:1` record without it is typed by its name. + +### The Worker on a tenant + +Bindings are SPEC-007's set plus `PASS="1"` (pass misses to the origin) and `ALIASES` (comma list). Checks run in this order: + +| Request | Answer | +|---|---| +| Host in `ALIASES` | GET or HEAD: 301 to the tenant host, same path and query; other methods 405 | +| Host not exactly `HOST` | 404 | +| raw path holds `%2F`, `%5C`, `%2E`, a low escape, or `//` | 400, in front of the tunnel leg too | +| `/healthz` | the Worker's own answer; with `PASS`, also a 3 s probe of the origin's `/healthz`: `200` and `X-Share-Tunnel: 1`, else `503` and `X-Share-Tunnel: 0` | +| `v:1` record, no `storage` | SPEC-007's checks, then R2 bytes | +| `v:2` machine pointer, no record, or `BUCKET.get` throwing or over 2 s | `PASS` set: pass through; else 404 | +| any other record shape | 404, never passed through | + +A pass-through goes back with its status and body unchanged and `no-store` and `noindex` set. Only a failed fetch, 530, 520 to 527, or a 503 without `cf-cache-status` (the origin's own answers carry it, edge errors do not) becomes the 503 offline page with `Retry-After: 60`. A 502 is the origin's own answer, such as a dead live port, and passes through. A 101 WebSocket answer goes back untouched. + +Callers read the cloud leg from the `/healthz` headers, never the status alone: a `503` with a valid `X-Share-Worker` pair and `X-Share-Tunnel: 0` means the Worker is up and the origin is down, so a member's join, a gated member add, and an r2 `state` keep working while the origin is off. + +### Moving an origin: `migrate` and `import` + +`share migrate --to [--remote-profile ] [--remote-bin ] [--max-bytes ] [--yes]` runs on the current origin while it is online, with `CLOUDFLARE_API_TOKEN` in the environment. It moves a tenant with R2 off and refuses one with `bucket=`. + +``` + old origin new origin, over ssh + 1 preflight: named tunnel profile, this host import --probe; the profile is not_setup or set + in hosts, no --host row, the token resolves up for the same hostname; Keychain round trip + 2 per snapshot row: tar | ssh -------------------> import ... validates, stages, publishes + 3 the switch, token on ssh stdin ----------------> setup --tunnel-name -m --force + --token-stdin (a new tunnel, CNAME repointed) + 4 verify through https:///: a probe file only the new origin holds answers 200 three times, + then every moved link answers 200 and every gated one 302 with kid == its app's AUD + 5 retire: rows to index.migrated, pub/ to migrated/ (nothing deleted), then the + teardown path with no Access delete and no DNS delete unless it still points here +``` + +`import` is the receiving verb and validates every field and every tar member (no `..`, absolute path, symlink, or hardlink) before it publishes. Live rows are listed as not moved and a `--host` row blocks the move. The new origin always gets its own tunnel: a name the old machine already holds would reuse its tunnel id, and the retire step would then delete the new origin's only tunnel (`migrate_retire` also refuses that delete if the ids ever match). A failure in the switch prints the rollback, `setup --tunnel-name --force` on the old machine; a tunnel name missing from the config is read back from the API by id. `SHARE_MIGRATE_SSH` replaces the ssh command with a local one so the suite runs both sides on one machine. + +Files `migrate` adds on the old origin: `index.migrated` and `migrated/`. A moved row's `src` becomes `:`, and `refresh` of such a row dies with `re-add it from a source on this machine`. + ## `share state` The menu bar app never reads share's files. It runs `share profiles --json`, a read-only @@ -398,6 +506,13 @@ The app changes anything only by running share's normal verbs (`add`, `rm`, `ref | `shares[].expires` | epoch seconds; `0` means never | | `shares[].url` | exactly what `share ls` prints for that row | | `shares[].access` | the `--access` rule of a gated share (`group:...`, `email:...`, `domain:...`), else `null` | +| `shares[].storage` | `machine` (served by the tenant's origin through its tunnel) or `cloud` (served from the bucket) | +| `shares[].type` | the file type: `pdf`, `image`, `video`, `audio`, `folder`, `site`, `markdown`, `archive`, `text`, or `other` | +| `shares[].by` | the machine that published the share | +| `r2` | `true` when the profile reads a bucket (an origin with R2 on, or an r2 profile), else `false` | +| `storage_default` | an origin with R2 on only: `local` or `cloud`, the storage of an `add` with neither `--local` nor `--cloud` | +| `cloud_error` | why the bucket's rows are missing (the menu reads them only with a stored token); the local rows still list | +| `cloud_more` | how many more bucket rows exist past the newest 25 that `state` reads; about, since it counts orphan pointers too | | `access_pending` | count of Access apps waiting for deletion (`access-pending` lines) | | `skipped` | count of malformed index rows; present only when greater than zero | | `backend` | `"r2"` on an r2 profile only, absent on a tunnel profile. There `state` is `serving` once set up, `ready` comes from one `/healthz` probe (2 s), `mode` is `named`, `hosts` is `""`, `serves_here` and `service` are `false`, every share is a `snapshot`, and `skipped` is never set. Share Bar shows neither Start nor Stop for it | diff --git a/docs/implementation-notes/one-host-per-tenant.md b/docs/implementation-notes/one-host-per-tenant.md new file mode 100644 index 0000000..27e40ee --- /dev/null +++ b/docs/implementation-notes/one-host-per-tenant.md @@ -0,0 +1,264 @@ +# One hostname per tenant: implementation notes + +The delta from SPEC-008 for the builder. Spike answers (TASK-1a, TASK-1b, TASK-7a's tar step) land here too. + +## Validation warnings carried to the build + +Round 1 warnings the spec did not fold. Each one is a build rule unless a task's spike answer overrides it. + +| Warning | Build rule | Task | +|---|---|---| +| Every request to an R2-on tenant runs the Worker plus one bucket read; dev-server HMR multiplies both | measure live-link request volume in L3; if it matters, memo "machine or miss" per id in the Cache API for a few seconds; add a Worker-requests or Class B threshold to the vps-mon catalog | TASK-3, TASK-9a | +| The route fails closed at the Workers request limit, taking machine links down | TASK-1b(k) reads the plan; on a free plan set the route to fail open (Caddy 404s cloud ids, edge Access still gates) | TASK-1b, TASK-4 | +| A publisher token reaching its `expires_on` silently blocks local adds on an R2-on origin | `api-token --check` warns 14 days ahead; vps-mon expiry coverage for the origin token | TASK-2, TASK-11a | +| `setup --r2` holds no index lock while pointers backfill | take the index lock from step 6 through step 11, so a concurrent local add waits and then writes its pointer | TASK-4 | +| A dry trace exists for one negative control only | write the dry trace for each control into the verification record as it goes red | TASK-10 | +| A tar truncated at a member boundary can publish a partial tree | `migrate` sends a per-id file count and sha256 manifest; `import` verifies it before publishing; the rerun skip compares the manifest | TASK-7a, TASK-7b | +| A member could take a machine link's URL by deleting its pointer and writing a cloud record (accepted trust, SPEC-007 DEC-011) | `state` reports `shadowed` ids so Share Bar can show them | TASK-6 | +| The migrate switch token can be broad if TASK-1b(i) picks the Toolkit token | prefer a run-scoped one-day token for the switch | TASK-11b | +| The Worker's `//` and low-escape 400s now cover live dev-server paths too | document it in how-it-works; scope them to cloud ids only if a real dev server needs `//` | TASK-3, TASK-10 | +| `brew pin` is no downgrade | the D1 and P1 rollback is an install of the previous tag from the tap's git history | TASK-11a, TASK-11b | +| After P2 nothing watches `s.han.ws` | add `s.han.ws/healthz` to vps-mon in TASK-11b | TASK-11b | +| After P2, a local restore on the Air (rows from `index.migrated`, trees from `migrated/`) has no command | rehearse the manual restore once in R2 and write it into the verification record | TASK-9b | +| `--alias` adds permanent surface for a one-time fold | keep the alias code small and covered; it retires with the alias rollback and `--no-r2` | TASK-5 | + +Round 2 warnings, same rule: + +| Warning | Build rule | Task | +|---|---|---| +| A D3 rollback breaks cloud links added after the fold | the alias rollback list prints the cloud ids added since the fold, so Han re-adds or warns first | TASK-5 | +| Teammates still joined at `f.d.foundation` are not told | D4 lists the bucket's `by` values and tells each to rejoin at `s.d.foundation` | TASK-11a | +| `migrate` holds no index lock; an add on the old origin mid-run is never moved | hold the index lock from step 1 to step 5, or re-diff the index before retiring and stop on new rows | TASK-7b | +| `migrate` on a laptop can sleep mid-retire | run under `caffeinate -i` on macOS; make the retire idempotent on rerun | TASK-7b | +| The 2 GiB import cap has no override | the preflight lists over-cap ids; add `--max-bytes` | TASK-7b | +| A fresh-bucket `setup --r2` can die between pointers and the marker | write `share.json` before step 9 when the bucket is new; add a fresh-bucket variant of row 11 | TASK-4 | +| A slow R2 adds up to 2 s to every machine request | add a failure-mode line with Worker wall-time as the signal; consider a lower read cap after L3 measures it | TASK-3 | +| A DELETE-then-PUT rebind can leave NXDOMAIN cached past the API gap | TASK-1b(f) measures through a public resolver queried during the gap | TASK-1b | +| A die at steps 9 or 10 with `--alias` leaves `v:2` pointers that block a v0.8.0 member's sweep | that die names the rerun or the pointer cleanup | TASK-5 | +| The Mini carries both tenants; a reboot stalled at the login window takes both down | vps-mon `healthz` on both hostnames is the signal; state it in the failure table of the ADR | TASK-10, TASK-11b | +| `hits` of a machine row on a member is unspecified; members see `--host` rows at the wrong URL | `hits` on a member refuses a machine row; pointers carry an `own_host` display field | TASK-2b, TASK-6 | +| The member purge is undefined on a tenant (route, not custom domain; machine rows refused on members) | a member purge refuses while the Worker binds `PASS="1"`; the origin runs `--no-r2` first; tenant purge order is route, pointers, records, script, bucket | TASK-4 | +| The migrate gated check needs Access read; the old origin's tunnel name may be missing | step 4 uses the old origin's stored Access token; before computing the new origin's tunnel name and the rollback line, `cmd_migrate` reads `tunnel_name` from the API by `tunnel_id` when the config key is absent, and dies rather than print an empty `--tunnel-name` if neither is known | TASK-7b | +| bsdtar lists a hardlink as a regular mode with `link to`; archive flags can pin files | the pre-scan rejects any `link to`; extract with no file flags or xattrs; row 18 gains a sibling hardlink and an absolute one | TASK-7a | +| Pointers expose every machine link's URL to every member | `docs/setup.md` says so in the onboarding | TASK-10 | +| The D3 rollback rebinds before restoring app destinations | restore destinations first, then rebind | TASK-5 | + +## TASK-1a: edge spike (2026-10-01) + +Throwaway objects on the Dwarves account, all created with the Toolkit admin token: hostname `share-e2e-spk584462.d.foundation` (proxied CNAME to tunnel `share-e2e-spk584462`), a route Worker of the same name on `/*`, a service token, and one Access app on `/gated{,/*}` with a `non_identity` service-token policy plus an email include (so an unauthenticated browser gets 302, not 403). The origin ran on the Mini: `cloudflared tunnel run` (token from the environment) to Caddy on 127.0.0.1:28990, which proxied to a node echo server that logged every request it received and spoke a WebSocket echo. The spike Worker answered 418 `REENTERED` for any request carrying `x-spike-pass`, answered `/__w` itself, fetched `https:///healthz` on `/__hz`, and otherwise ran `fetch(request)`; `?mark=1` cloned the request with `x-spike-pass: 1` first, and `x-spike-raw: 1` returned the subrequest's status and headers as JSON. Every probe used `curl --doh-url https://1.1.1.1/dns-query`. + +| Q | Command (shape) | Observed | +|---|---|---| +| (a) | `GET https:///abc123/f?x=1` | 200 with `x-spike-worker: 1` and `x-origin: 1`; the origin logged the request with `cf-worker: d.foundation` | +| (a) | `GET https:///abc123/f?mark=1` | 200 from the origin, which logged `x-spike-pass: 1`; no 418, so the subrequest skipped the route Worker | +| (a) | `GET https:///__hz` (Worker fetches `https:///healthz` with `x-spike-pass`, `AbortSignal.timeout(3000)`) | `{"status":200,"body":"ok","reentered":null}`; the origin logged `GET /healthz` | +| (b) | unauthenticated `GET /gated/x`, `/gated/__w`, `/gated` | 302 to `https://dwarves.cloudflareaccess.com/cdn-cgi/access/login/?kid=`, `kid` equal to the app's AUD; no `x-spike-worker` header on any; the origin log stayed empty, and `/gated/__w` (answered by the Worker itself) also got the 302, so Access runs before the route Worker | +| (b) | `GET /gated/__w` with `CF-Access-Client-Id`/`-Secret` | 200 from the Worker, `{"worker":true,"jwt":true,"cookie":true}`: the admitted request carries `Cf-Access-Jwt-Assertion` and a `CF_Authorization` cookie | +| (b) | the same with a wrong secret | 302 to the login flow, never admitted | +| (c) | `GET /gated/x` with the service token | 200 from the origin; the origin logged `jwt=1` and `cookie` present: `fetch(request)` forwards the JWT and the cookie | +| (d) | `POST /live/api` with 100000 random bytes; `PUT`; `DELETE` | 200 each; the origin logged `POST body=100000`, `PUT body=1`, `DELETE` | +| (d) | node 22 `new WebSocket("wss:///live/ws")`, send `ping1` | `["hello-from-origin","echo:ping1"]`; the origin logged `UPGRADE /live/ws cfworker=d.foundation`; the Worker returned the 101 response object as is | +| (d) | the same on `/gated/ws` without, then with, the service token | without: handshake refused (non-101); with: echo works, the origin logged `jwt=1` | +| (e) | Caddy up, `/dead/x` proxied to a closed port (Caddy answers `502`, `Server: Caddy`, empty body, checked locally) | the Worker's subrequest sees `502` with Cloudflare's own HTML page (`d.foundation \| 502: Bad gateway`), `server: cloudflare`, `retry-after: 60`, no `cf-cache-status` | +| (e) | Caddy `respond "caddy 502 with body" 502` | the same Cloudflare 502 page; Cloudflare replaces every origin 502, body or not | +| (e) | Caddy stopped, cloudflared up | `502`, the same Cloudflare page and headers as above; `/__hz` got `502 error code: 502` | +| (e) | cloudflared stopped | `530` (`error code: 1033`, the Cloudflare Tunnel error page) to both the pass-through and `/__hz`; `retry-after` present, no `cf-cache-status`; the client saw the 530 through the Worker | +| (e) | Caddy `respond "caddy 503" 503`; Caddy `respond 404`; the node origin's own `503` | `503` with body `caddy 503` and `cf-cache-status: DYNAMIC`; `404` with `cf-cache-status: DYNAMIC`; `503` with `cf-cache-status: DYNAMIC` and `via: 1.1 Caddy` | + +Answers: + +- (a) holds: a route Worker's `fetch(request)` and its `fetch("https:///healthz")` both reach the tunnel origin and never re-enter the Worker. Approach (A) stands. +- (b) holds: Access answers an unauthenticated gated path with its 302 before the route Worker runs; an admitted request reaches the Worker with `Cf-Access-Jwt-Assertion`. +- (c) holds: the pass-through keeps the JWT header and the `CF_Authorization` cookie. +- (d) holds: POST, PUT, DELETE with bodies and a WebSocket upgrade pass through; `return r` on a 101 is enough. +- (e) the pass-through sees 530 with the connector down and 502 with the connector up but nothing answering. An origin's own 502 never reaches the Worker: Cloudflare swaps it for its own page, so a Caddy 502 and an edge 502 look the same and both get the offline page (as the spec wants). The header that tells an origin answer apart is `cf-cache-status`: present on every response the origin produced (503, 404, 200), absent on every edge-generated error (502, 530). `via: 1.1 Caddy` appears only on responses Caddy proxied, so it is not a general marker. Build rule for TASK-3: offline page when the status is 530, 520 to 527, or 502; or 503 without `cf-cache-status`. Superseded in the review fix pass (DEC-013): 502 is no longer in the set, because a dead live port and a dead Caddy both arrive as the same 502 and the first is a working machine; the rule is now 530, 520 to 527, or 503 without `cf-cache-status`, and `WORKER_VERSION` is 4. Not measured: 520 to 527 and an edge-generated 503 (no way to force them with a tunnel origin); the rule keys them on the same header. + +Side findings for later tasks: + +- A route object carries `request_limit_fail_open` (default `false`); `PUT zones//workers/routes/` with `true` was accepted and read back. TASK-1b(k) uses this. +- After the DNS record was deleted, the authoritative server answered NXDOMAIN at once, while 1.1.1.1 still served its cached proxied A records (the edge answered 530) until the TTL ran out. + +Cleanup, checked through the API afterwards: route 0, Worker settings 404, DNS records for the name 0, Access apps and service tokens named `share-e2e-spk*` 0, tunnels named `share-e2e-spk*` (not deleted) 0, no spike process left on the Mini; `dig @david.ns.cloudflare.com ` NXDOMAIN. + +## TASK-1b: account spike (2026-10-01) + +Throwaway objects: a token `share-e2e-spk13b60d-admin` minted through `op://Toolkit/cf-tokens-admin` (`POST /user/tokens`, one-day `expires_on`) with the tenant admin set (account: Workers Scripts Edit, Workers R2 Storage Edit, Access Apps and Policies Edit, Access Organizations Read; zone `d.foundation`: Zone Read, DNS Read, Workers Routes Edit); Workers `share-e2e-spk13b60d-a` and `-b` (each answers `w1` or `w2`); custom domain `share-e2e-spk13b60d.d.foundation`; one Access app; on the personal account one tunnel and one CNAME `share-e2e-spk13b60d.han.ws`. The minted token ran the deploys, the first bind, the refused in-place PUT, the DELETE then PUT of the first pass, and every (g) call; the Toolkit token ran the `override_existing_origin` passes, the timed DELETE then PUT, (i), and the cleanup. Edge probes: node 22 `fetch` every 100 ms with keep-alive (about 160 ms per sample), plus `dig` against 1.1.1.1 and the zone's authoritative server every 200 ms during the first pass. + +| Q | Command (shape) | Observed | +|---|---|---| +| (f) | `PUT accounts//workers/domains {hostname, service:"-b", zone_id, environment:"production"}` while the hostname is bound to `-a` | refused: `success:false`, code `100116`, "Hostname ... already in use by other custom domain. Either delete it, try a different hostname, or use the option 'override_existing_origin' to override."; the binding stays on `-a` | +| (f) | the same PUT with `"override_existing_origin":true`, three times (`-a` to `-b`, back, again) | `success:true`, same domain id, `service` switched; 308 samples over 50 s, 0 non-200; the new Worker answered 1.1 s, 1.1 s, and 3.6 s after the PUT returned | +| (f) | `DELETE workers/domains/`, then the plain PUT to `-a` | DELETE 200, PUT 200 (same domain id came back); the old Worker kept answering about 1.1 s after the DELETE returned, then 4 samples of `522` over 0.7 s, then `w1`; total failed window under 1 s; 210 samples, 4 non-200 | +| (f) | `dig @1.1.1.1` and `dig @.ns.cloudflare.com` on the hostname during the first DELETE then PUT | `NOERROR` on every sample, never `NXDOMAIN`: the domain's DNS record outlives the API gap, so no negative cache forms | +| (g) | `POST access/apps` with destinations `/ab12cd{,/*}`, then `PUT access/apps/` with the full `GET` result plus two destinations on `x.` | `success:true`; `aud` and `id` unchanged; four destinations; `self_hosted_domains` follows the destinations; policies unchanged (only their `updated_at` moved) | +| (g) | `PUT access/apps/` with only `name`, `type`, `destinations` | `success:true`, `aud` unchanged, the policy kept, but `app_launcher_visible` went from `false` back to the default `true`: a PUT resets every field it omits | +| (h) | the Toolkit token's id from `GET /user/tokens/verify`, its policies through the minter (`GET /user/tokens/`) | token `all-wuandkin-admin`, no expiry, four accounts; on the Dwarves account: Workers Scripts Write, Workers R2 Storage Write, Workers Routes Write, DNS Read and Write, Zone Read, Access: Apps and Policies Write, Access: Organizations Read, Cloudflare Tunnel Write, Account Analytics Read (zone groups on `zone.*`). TASK-1a already used it for the route POST, PUT, DELETE and the Access app POST, DELETE | +| (i) | with the Toolkit token on the personal account: `POST cfd_tunnel`, `PUT .../configurations`, `GET .../token`, `POST zones//dns_records` CNAME, `PUT` of that record, both DELETEs | every call `success:true`; the same token policy covers the personal account with Cloudflare Tunnel Write and DNS Write | +| (k) | `GET accounts//subscriptions` on both accounts; `GET zones//workers/routes` | Dwarves LLC: `Workers Paid`, `R2 Paid`; Han Ngo: `Workers Paid`, `R2 Paid`, Analytics Engine; both zones `Free Website`. Every route object carries `request_limit_fail_open` (both existing routes `false`); TASK-1a set it to `true` with a route PUT and read it back | + +Answers: + +- (f) The plain PUT never rebinds. `PUT workers/domains` with `override_existing_origin: true` rebinds in place with no failed request at a 160 ms sample rate, so step 14 uses it and no gap exists to bound. DELETE then PUT costs under 1 s of 522 and never NXDOMAIN; it stays the fallback only if the override flag ever stops working. +- (g) A PUT keeps the AUD, the id, and the policies when it sends the full `GET` result. It resets any omitted field to its default, so step 10, step 16, and the alias rollback must PUT the full read-back with only `destinations` changed. +- (h) Yes. The Toolkit token holds Workers Routes Edit and Access Apps Edit on the Dwarves account, plus every other scope of the tenant admin set. The admin role uses it. It is broad (four accounts, no expiry); the minted one-day token with only the tenant admin set also ran every (f) and (g) call, so the run-scoped fallback in the token table works. +- (i) Yes. The Toolkit token edits tunnels and DNS on the personal account. The round-1 warning still prefers a run-scoped one-day token for the migrate switch (TASK-11b). +- (k) Both accounts are on Workers Paid, so no daily request limit applies and the route never hits the free plan's failure mode. TASK-4 sets `request_limit_fail_open: true` on the route anyway: it costs nothing on a paid plan and keeps machine links up if an account ever drops to free. + +Cleanup, checked through the API afterwards on both accounts: Workers, custom domains, Access apps, service tokens, tunnels, routes, and DNS records with names starting `share-e2e-spk` all 0; buckets starting `share-e2e` 0; user tokens starting `share-e2e-spk` 0 and the minted token's id answers 404; `dig` at each zone's authoritative server answers NXDOMAIN for both throwaway names. + +## Build deltas + +Deviations from the spec, per task. A task with none is not listed. + +### TASK-2a (per-link storage on the origin) + +- An origin with R2 on is a tunnel profile whose config holds `bucket=`; the process flag `r2on` keys every bucket call (`r2_call`), the dry log (`dry_log` writes `r2-calls.log`, so one log orders bucket PUTs, probes, and Access calls), `rand_id`'s bucket leg, and `api-token`'s publisher refusals. `backend` stays the tunnel, so a cloud add on the origin runs `cmd_add_r2` unchanged; `share_url` reads `r2_link_base` whenever it is set, and the origin keeps its own `r2-own` file. +- `--cloud` with a port or `--host` dies with `--cloud: live links and --host stay on the origin's tunnel` on every tunnel profile, R2 on or off, before the R2-off line. `--cloud --local` is the usage line. A bad `storage_default` dies naming the config. +- A local add on an R2-on origin runs `r2_token_check` before `rand_id`, so no token means no stage, no id, no write. +- The pointer PUT retries a 412 with a fresh `rand_id` up to five ids, then dies. The `type` is `site` for a live link, else the source's type. +- Cloud records written by `cmd_add_r2` now carry `type` too (the spec's "new cloud records store `type` at add time"); `v:1` readers ignore it. +- `rm` deletes the pointer only after a `GET m/` shows `v:2` with `storage:"machine"`: a cloud record on the same id (a shadow) is never deleted from the origin. Token modes: `rm` and bare `prune` use the soft resolver; `ls`'s listing prune only an environment token; serve's hourly prune never. An admin token from the environment counts as no token there (its `r2_publisher_check` refusal is probed in a subshell), so a listing never dies on it. +- `api-token` on an R2-on origin: `--cmd`, the paste, and `--check` run `r2_publisher_check`; the paste names the bucket scope beside the prefilled Access form; `access_preflight` stays required. Not built yet: the 14-day `expires_on` warning (round-1 warning), left for TASK-11a's token work. +- Test seams: `SHARE_R2_DRY_PUT=500` (every PUT answers 500), `SHARE_R2_DRY_PUT=race-once` (the first conditional PUT of an `m/` key finds a peer's record written just before it, so `If-None-Match: *` answers 412), and `SHARE_R2_DRY_PAUSE` now drops `.paused` in the dry bucket, so row 4 inspects `pub/` while the pointer PUT is held instead of racing a watcher. +- Left for TASK-2b: on the origin, `rm`, `refresh`, and `hits` of a cloud id still read the local index only (`no share with id`). + +### TASK-2b (dispatch, reconcile, sweep, origin cloud expiry) + +- Fixes TASK-2a's leftover: on an origin with R2 on, `rm`, `refresh`, and `hits` of an id with no local row take the r2 path. A local row wins on a shadowed id. +- The bucket reader is `r2_read`: one mktemp file per process (`r2_snap_file`), read through `crows`; it never assigns or writes `$index`. `r2_snapshot` (members) is `r2_read` plus the old `index` rebind. `cmd_rm_r2` and `r2_prune` read and rewrite the reader's file only. Keys outside `^m/[0-9a-f]{6}$` are dropped before any GET, on members too. +- `hits` on a member does not refuse a machine row. SPEC-007 row 13 pins member `hits` to one account call and no bucket read, and a refusal needs a `GET m/`. The spec test wins over the round-2 build rule. The origin's `hits` of a cloud id reads `m/` first, as row 27 asks. Batch 3 settles the open item (DEC-012): a 0 count on an id outside `r2-own` adds the machine-link line naming the origin's `hits`. +- A member's `rm` and `refresh` of a machine row die after the read that identifies the row (the snapshot for `rm`, `GET m/` for `refresh`) and before any write. +- No add lock existed. `r2_pointer_put` now takes `.lock-add-` before the pointer PUT; `cmd_add` releases it after the row write, the EXIT trap on a die. The reconcile skips an id whose lock names a live pid. +- The origin's interactive pass runs in a subshell before the local expiry: cloud expiry, the orphan sweep (bare prune only, as SPEC-007), then the reconcile. A bucket failure prints one line and the local expiry still runs. The reconcile skips expired local rows (the local prune's) and prints its shadow and backfill lines on stderr. `access_sweep` runs once, in `cmd_prune`. +- `r2_record_raw` reads the cloud shape as `v:1` exactly instead of `v <= WORKER_RECORD_V`, so TASK-3's bump to 2 keeps a `v:2` record without `storage:"machine"` unreadable for the sweep. +- A member's bare prune and `ls` delete a machine pointer whose `expires` is past (pointer rule 3). +- The 14-day warning: `api-token` (`--check`, the paste, `--cmd`) on an r2 member and on an R2-on origin reads `expires_on` from `/user/tokens/verify` and warns at 14 days or fewer, or once expired. Not on every add (`r2_key_id` in the config skips the verify call there). Test seam: `SHARE_R2_DRY_EXPIRES`. + + +### TASK-3 (Worker v3, r2_healthz) + +- The method rule: with `PASS` empty the Worker keeps SPEC-007's early 405 for any method but GET and HEAD, so SPEC-007 rows 17 to 19 and 28 run unchanged. With `PASS` set, the 405 applies only to what the Worker answers itself (`/healthz`, a cloud record, after its record and JWT checks); a pass-through takes any method. +- The record branches read `v == 1` with no `storage` key as a cloud record and `v == 2` with `storage == "machine"` as a pointer; everything else (a missing `v` included, which SPEC-007's `v > 1` test let through) answers 404 and never passes through. +- The offline page also answers a thrown `fetch`. Its body is plain text with `Content-Type: text/plain; charset=utf-8`. +- Hits: a pass-through is marked in a `WeakSet` and the outer handler returns it untouched, so no data point is written and a 101 keeps its `webSocket`. +- `r2_healthz` reads one header file for both the live call and the dry seam (`SHARE_R2_DRY_HEALTHZ=tunnel-down` and `503-bare` write the 503 a tenant Worker sends with and without its headers), and strips the status line's carriage return. +- `tests/worker.mjs`'s stub origin returns a real `Response` whose `headers` throw on `set`, `append`, and `delete`. Node unrefs `AbortSignal.timeout`'s timer, so the hang case keeps the event loop alive with its own timer. +- Not measured or built: the slow-R2 2 s cap has no test row (only the throw); the Cache API memo and the Worker-wall-time failure line from the round-1 and round-2 warnings wait for L3's numbers (TASK-9a). + +### TASK-4 (setup --r2, --no-r2, no alias) + +- `--alias` is not parsed yet (TASK-5); `setup --r2` takes `--bucket`, `--storage-default`, and `--force` (a Worker version downgrade, as SPEC-007). A rerun keeps the stored `storage_default` unless the flag is given. +- Step 1 adds two refusals: a profile whose config holds another bucket (`--no-r2` first), and an origin that is not serving (`share start` first), since step 15 probes a file through the tunnel. The dry seam skips the probe and logs `TUNNEL-PROBE`. +- Step 3 keeps SPEC-007 step 7 (`GET access/organizations`) so the tenant Worker's `TEAM` binding serves gated cloud links; it runs after the routes read. +- Step 5 refuses any route whose host part (a glob) matches the tenant host and names another script, narrower patterns included, not only `/*` and wider ones. +- Step 6 runs only on an existing bucket; on a new one the bucket POST and the `{"v":1,"host"}` marker PUT come before the Worker deploy and every pointer (the round-2 fresh-bucket rule). The index lock is held from step 6 through step 11. Residual: an add that read its config before step 11 writes no pointer; the next reconcile writes it. +- Step 9 skips an id that already holds this origin's machine pointer (the rerun's convergence); a 412 on any other id dies before the route. +- Step 13 PUTs `request_limit_fail_open: true` onto an existing route of this Worker that lacks it; a rerun with the flag set attaches nothing. +- The step-15 die names the leg (Worker or tunnel) and the `--no-r2` rollback. +- `write_config` (a tunnel setup rerun on the origin) keeps the four tenant keys, so a rerun never turns R2 off in the config while the route stays. +- `--no-r2` refuses while the config's `aliases=` is set; the marker's `aliases` and the printed rollback list wait for TASK-5. +- SPEC-007's admin path now reads `zones//workers/routes` before its DNS read; a 403 there passes (round-3 rule), any other non-200 dies. A member purge refuses a Worker that binds `PASS="1"` (round-2 rule). The origin-side purge of a tenant (route, pointers, records, script, bucket) is not built. +- Round-3 rule on a marker field that v0.8.0's admin setup refuses: v0.8.0 checks the marker with one test (`.v == 1 and .host == `) for both roles, so any field that fails its admin path fails its join too. None was added; D1's check that every machine holding the admin token runs the release stays the guard. +- Dry seam: `zones//workers/routes` GET, POST, PUT, DELETE on `.cf/routes.json`; `SHARE_R2_DRY_ROUTES=deny` answers 403; the dry `/healthz` answers through a route as well as a custom domain and adds `x-share-tunnel: 1` for a `PASS` Worker. + +### TASK-5 (--alias, the alias rollback list) + +- One alias per tenant. A rerun without `--alias` takes the config's `aliases=`, so a redeploy keeps the `ALIASES` binding; a different `--alias` beside a stored one is refused. `--alias` must sit in the tenant's zone (step 14 binds it with that zone's id). +- Step 3 accepts a marker naming the tenant host or the alias, with `aliases` empty or exactly `[]` (a rerun after step 12). +- Step 4 also accepts the alias's domain already bound to the tenant Worker (a rerun after step 14), and no domain at all (step 14 creates it). +- Step 7 checks Apps Edit (the `{}` probe) only when a gated cloud record exists. An app named `share ` is not refused: it is a gated cloud link added on the tenant after an earlier fold. Its id is listed in the rollback list as added since the fold. +- Step 10 runs inside the index lock (step 6 to step 11), so a concurrent add can wait out `access_gate`. Only apps whose PUT changed something are probed; a rerun probes nothing. +- Step 12 writes the marker with `If-Match` on the ETag read in step 3. On a fresh bucket the early `{"v":1,"host"}` marker's ETag is kept for it. +- Step 15 waits for three 301 answers in a row, like the other live checks. The step-15 and step-13 dies and the healthz die go through `tenant_die`, which prints the alias list when an alias is folded and `--no-r2` otherwise. +- Step 16 copies the alias profile's `r2-own` lines into this profile's file and leaves the alias profile's own file in place (nothing deleted; D4's teardown removes it). +- The rollback list puts the app destinations first, then the rebind (the round-2 rule), unlike the spec's D3 prose order. It names every cloud id (they answer at the alias again, not at the tenant host) and, apart, the gated ones added since the fold (no app on the alias). It is a list of exact calls, not a command: R1 runs it by hand, and row 30 applies it to the dry account in the suite. +- `--no-r2` now resolves the admin token and reads the marker before its alias refusal, so the refusal can print the list (a marker with `aliases` refuses too). +- A member's join reads the marker's `aliases` once and keeps each one only after `https:///healthz` answers 301 to the tenant three times; it stores them as `aliases=` in its r2 config. `access_delete` and the purge's by-name pass accept an app named for the host or any stored alias. +- Dry seams: `SHARE_R2_DRY_FAIL` (`API `, `PUT `, or `ALIAS-301`) fails one call with no state change; `.cf/script-.json` answers the settings of a second Worker; the Access dry answers `PUT access/apps/`; the dry alias answers 301 once its domain names the tenant Worker and that Worker binds it. + +### TASK-6 (one shared list) + +- `list_rows` builds the display rows for `ls` and `state`: the local index (typed at read time, `by` from a migrated row's `by=` or `r2_by`), then the bucket through `list_bucket` in a subshell, so any die there becomes `cloud_error` and the local rows still print. Bucket rows live only in that list; `rows()`, `index.tsv`, the Caddyfile, and stage paths never see them. Fields are split by `\x1f`, because bash `read` merges empty tab-separated fields. +- On the origin, `ls` reads the bucket a second time after its prune (the prune runs in a subshell). A member's `ls` reuses the read its prune made. +- `ls` prints ` by= ` on the link line (`cloud`, `machine`, or `live`); the detail line keeps its old shape. A member's machine row says `served by the tenant's origin`, and `access=gated` for a gated pointer. A shadowed id lists its local row once. An expired pointer is not listed. +- A cloud record's stored `type` rides into its row as a `type=` token (a record's own `opts` may not carry `type=`, `prefix=`, or `by=`). A record with no valid `type` is typed by its name's extension, and a name with no extension is a `folder`. +- The `v:2` display reader keeps a record only when every field passes (`expires` as `^[0-9]{1,11}$`, `added` as a date, `name` and `by` free of control bytes, `opts` only `noindex`, `live`, `gated`, `type` from the table, an optional `own_host` as a hostname). Pointers now carry `own_host` for a `--host` row (round-2 rule), so a member lists that link at its own name. +- A local folder whose `index.html` is share's generated listing (its first line holds `index`) types as `folder`, not `site`. `type_of` is fork-free (`nocasematch`), so the 500-row `state` stays under its budget. +- `state` (and `profiles --json` through it) reads the bucket only with a stored token: the Keychain item or the 600 file, or the test seam `SHARE_R2_TOKEN`; a profile with `api_token_cmd` or `r2_token_cmd`, or only an environment token, gets `cloud_error` and no command runs. It fetches the newest 25 non-local keys by `LastModified` and reports the rest as `cloud_more`. `SHARE_STATE_BRIEF` (the text `profiles`) reads no bucket. +- New `state` keys: `shares[].storage`, `.type`, `.by`; top-level `r2`, `storage_default` (R2-on origin), `cloud_error`, `cloud_more`. A member's gated pointer reports `access: "gated"`, its live pointer `kind: "live"`. +- Not built: `state` does not report `shadowed` ids (a round-1 warning). It would need a GET of every key the newest-25 rule skips; the origin's `ls` and `prune` already print the shadow line. +- Row 28 checks the rendered Caddyfile and `index.tsv`, not a running Caddy: the dry tenant harness does not serve. The suite's other sections cover the Caddyfile-to-running-config path. The key `m/../share.json` comes from a new dry seam, `SHARE_R2_DRY_LIST_EXTRA`, since a directory-backed bucket cannot hold it. +- Row 1 extends the existing byte-identity run with `profiles --json` and compares after removing the listing additions (the `ls` tag column, the new `state` keys). The named dry setup and the member leg of row 1 are covered by rows 15 and 17 rather than a second `origin/main` run. + +## TASK-7a: tar flag measurement (2026-10-01) + +Archives built byte by byte (Python `tarfile`, ustar) and by each tar, then listed and extracted on this Mac with bsdtar 3.5.3 (libarchive 3.7.4) and GNU tar 1.35 (Homebrew `gnu-tar`, installed for the measurement): + +| Member | `tar -tvf` (bsdtar / GNU) | `tar -xf` (bsdtar / GNU) | +|---|---|---| +| regular file, directory | `-rw-r--r--`, `drwxr-xr-x` on both; the name is the last field | extracted | +| symlink `doc/l -> /var/empty/target` | `l... ./doc/l -> /var/empty/target` on both | extracted as a symlink on both (exit 0): nothing stops it | +| hardlink to a sibling | `h... ./doc/b.txt link to ./doc/a.txt` on both, for an archive from either tar and for a hand-built one | extracted with a link count of 2 on both | +| hardlink to `/var/empty/target` | bsdtar: `link to /var/empty/target`; GNU: strips the `/` with a warning | both fail to link (exit 1 / 2) and leave the directory | +| absolute member `/tmp/x` | listed as `/tmp/x` on both (GNU warns on stderr) | both strip the leading `/` and extract `tmp/x` inside the target (exit 0) | +| `./doc/../../x` | listed as is on both | bsdtar refuses (`Path contains '..'`, exit 1); GNU refuses the member (exit 2) | +| `./doc/.env` | listed as is | extracted | +| FIFO | `p...` on both | not tried | +| an archive bsdtar makes on macOS | read by GNU tar, it holds `._*` AppleDouble members (and `LIBARCHIVE.xattr` headers) | `COPYFILE_DISABLE=1` with `--no-mac-metadata --no-xattrs` removes them | + +Build rules taken from it: + +- The pre-scan reads `tar -tf` (names) and `tar -tvf` (types) and refuses the whole import on any line whose mode does not start with `-` or `d`, any ` link to ` or ` -> `, an absolute name, a `..` segment, or a segment starting with `.`; a different line count between the two listings (a name with a newline) refuses too. The listing columns differ between the two tars, so only the first mode character and those two markers are read from `-tv`. +- Extraction flags: bsdtar `--no-same-owner --no-same-permissions --no-xattrs --no-acls --no-fflags --no-mac-metadata`; GNU tar `--no-same-owner --no-same-permissions --no-xattrs --no-acls --no-selinux` (it has no `--no-fflags`). Both sets extract the plain tree unchanged. The flavor comes from `tar --version`. +- After extraction, `find` refuses anything but files and directories and any file with a link count above 1 (the second line of defence for a hardlink the listing missed). +- For TASK-7b: `migrate` builds each tar with `COPYFILE_DISABLE=1` and `--no-mac-metadata --no-xattrs` on bsdtar, or the target's dotfile refusal trips on `._*` members. + +### TASK-7a (import, setup --token-stdin) + +- `import` takes an optional eighth argument, `:`, the round-1 manifest rule: the file count and a sha256 over each file's sha256 and path, sorted (`import_manifest`). A mismatch refuses the import. TASK-7b sends the same digest of `pub/`. +- `import` refuses on a profile that reads a bucket (an R2-on origin or an r2 profile): pointers are not written on that path, and `migrate` moves R2-off tenants only. +- The archive's root must hold exactly the row's name, as `pub//` does, so the row never points at a path the tar did not bring. +- The candidate row also passes `rows_in`, so an `access=` without its `access_rule=` (or a bad rule) is refused. A gated row keeps its app id; no Access call is made. +- The spool and the stage go through the EXIT trap. Caddy reloads only when the profile is serving; a stopped or `not_setup` profile renders at its next start. +- `refresh` of a row whose `by=` is another machine and whose `src` starts with `:` dies naming that machine. +- `setup --token-stdin` reads one line into the process's own `CLOUDFLARE_API_TOKEN` before any branch runs, so it works for the tunnel setup migrate calls and for `--r2`. An empty line dies before any call. The test drives `--r2` because the suite has no dry seam for the tunnel setup's API path. + +### TASK-7b (`migrate`) + +- The remote command is built exactly as spec'd (`/bin/bash -c "$(printf %s | base64 -D 2>/dev/null || printf %s | base64 -d)" share-migrate `), with the double-quote characters around `$(...)` kept as literal bytes in the argv element `migrate_remote_run` hands to `$sshcmd`. Dropping them (passing a bare, unquoted `$(...)`) is a live bug class, not a style nit: ssh (and the `SHARE_MIGRATE_SSH` seam) joins argv with spaces for the remote login shell to reparse, and an unquoted substitution gets word-split on the decoded script's own newlines by that shell, truncating the script to its first line. Caught by `tests/worker.mjs`-style direct reproduction, not by the suite's dry seam (which never joins-then-reparses locally); see the dry seam note below. +- A second, unrelated bug the live reproduction also surfaced: `migrate_remote_run`'s last statement was `mr_code=$?`, an assignment, which is always 0 regardless of what it captures. Every caller using the `cmd || rc=$?` idiom therefore never saw a real remote failure (the copy loop and the switch step both silently "succeeded" no matter what the remote returned). Fixed with an explicit `return "$mr_code"` as the function's last line. A `migrate_retire` ending on a bare `[[ -f $cert ]] && to_trash "$cert"` has the same shape of bug one level up: a false test makes that the function's (and so `cmd_migrate`'s) exit status under `set -e`, since a failing `&&`/`||` construct is only exempt from `errexit` when something runs after it in the same function; `migrate_retire` now ends on an explicit `return 0`. +- `cmd_migrate_probe`'s one-line reply is tab-separated, matching the index row convention, but `host_name` is empty on a `not_setup` target: bash `read` collapses a run of tab characters exactly like it collapses spaces (tab counts as "IFS whitespace" even when IFS is set to `$'\t'` alone), so an empty field between two tabs vanishes and every field after it shifts left. Fixed by sending `-` for an empty host (the same placeholder `profiles` already uses) rather than an empty field; no change needed on the read side since `rstate`/`rpempty`/... never go through the host slot. +- The same word-splitting hazard applies to every argument `migrate_remote_run` forwards over the ssh-joins-argv-then-reparses path, not just the decoded script: a zero-length argument (base64 of an empty `opts`) disappears when the remote shell re-tokenizes the joined command line, shifting the manifest argument into the opts position and failing `import`'s `b64dec` check. Fixed with a sentinel (`__share_empty_arg__`) substituted for `""` before the args are joined, and undone by a `for`-loop baked into the generated remote script (`migrate_script`) before `exec share "$@"`. +- A row already on this machine via an earlier import carries its true publisher in `opts`' `by=` token and its bare source embedded in the stored `src` column as `:`; sending `"$by:$src"` again (prepending the *current* machine's name) double-prefixes it. `migrate` now reads `by=` out of `opts` when present (propagating the original publisher unchanged and stripping `by=` before re-sending, since `import` appends its own and would otherwise see two), and only falls back to `r2_by` for a row that originated locally. +- `cmd_migrate`'s own `not_setup` preflight rule reads the spec literally ("an empty index and pub/ when not_setup") for a first attempt, but a *rerun* after a partial copy failure leaves exactly the ids this same move already copied sitting on a `not_setup` target. Row 21 needs that allowed. The check now diffs the target's `migrate-index` listing against this run's own candidate ids: any id not in that set is a stranger and still refuses; every id in it is this move resuming, not a stray. +- `svc_path()` had the same "wrong order" bug as the remote script: it iterates a fixed tool list (`caddy cloudflared jq curl pandoc gh trash security`) and appends each tool's directory as it is found, so a directory holding a *later*-iterated tool (here, `security`) lands after one holding an *earlier*-iterated tool (`curl`, `gh`, `trash`) in the baked `PATH=` string even when the caller's real `$PATH` has them in the opposite order. A live rerun of `tests/e2e-tenant.sh` hit this directly: the stub `security` the script relies on sat behind `/usr/bin` in the launchd plist's baked PATH, so the real Keychain answered instead and the tunnel token round trip failed. Fixed by walking the caller's own `$PATH` directories in order and keeping any that hold at least one needed tool, rather than walking the tool list and resolving each one independently. +- Dry seam: there is no bucket-backed dry mode for `migrate` (it refuses an R2-on tenant outright), so its local coverage is a `curl`/`security` shim pair plus `SHARE_MIGRATE_SSH` pointing at a wrapper that joins its own argv with spaces and replays it through `fish -c` (falling back to `sh -c`) on the same machine, exactly mirroring what a real ssh round trip does to the command line. This is also why the quoting bug above was invisible to every other dry seam in this suite: SPEC-007's R2 dry mode calls functions in-process, so an unquoted `$(...)` never gets a chance to be re-tokenized by a second shell. +- `cmd_migrate`'s `setup "$host_name" --force --token-stdin` switch step runs the real (non-dry) `cmd_setup` plain-tunnel path on whichever machine answers true to `host_ok` for that hostname, which for the local-only `SHARE_MIGRATE_SSH` seam is the same physical Mac both "ends" of the test share. On this machine that path reaches `svc_install`, i.e. a real `launchctl bootstrap`, unless the test's `HOME` carries an `hosts=` that does not match `this_host`; the suite's own `launchctl`/`systemctl` stubs (already global at the top of `tests/share.sh`) are what keep that call from doing anything, and the migrate rows additionally set `hosts=nobody` on every target fixture (and, for the row 31 rollback line run directly against the origin, flip the origin's own `hosts=` to `nobody` first) so the live-check branch of `cmd_setup` never runs at all in the common case. +||||||| d246452 + +### TASK-8 (Share Bar tenant rows) + +- Every new field decodes optional, so a pre-tenant CLI's `state` still renders: a missing `type` maps to `doc`/`file`, a missing `storage` shows no badge, and the `Storage` row hides when `r2` is absent or false. +- `RowGlyphs` in ShareBarCore holds the spec's symbol/word table as plain strings; `StatusItemController` resolves names with `NSImage(systemSymbolName:)` and drops any that fail. The lock moved out of `item.image` (now the type symbol) into the trailing attachment group: storage, link, lock. +- `state`'s `cloud_more` count rides the section's existing `more` line; `cloud_error` renders as one disabled line after the rows. +- The publish dialog's `Storage` row has three shapes the `sync` pass switches between: a picker (`On ` / `In the cloud`, preselected from `storage_default`) only when `r2 && serves_here`; a disabled `In the cloud` label on an r2 member; hidden otherwise. Only the picker adds `--cloud`/`--local` to the argv. A profile switch reloads the selection from the new profile's `storage_default`, since the picker is per-tenant. +- Not covered by tests: the visual layout of the attachment column and the dialog row switching (AppKit-only, manual). +||||||| d246452 + +### TASK-9a (`tests/e2e-tenant.sh`, L1 to L10, rehearsal R1) + +- The script covers legs L1 to L10 and rehearsal R1 in one file, modeled on `tests/e2e-r2.sh` (token minting through a 2h admin-minted publisher token, the throwaway-name refusal, trap-based cleanup with an outside-in leftover check) and `tests/e2e.sh` (a real tunnel setup, its local dev-server backend). Inputs: `SHARE_E2E_TENANT_HOST`, `SHARE_E2E_ALIAS_HOST`, `CLOUDFLARE_API_TOKEN`, `SHARE_E2E_R2_PUBLISHER_TOKEN` (or `SHARE_E2E_R2_TOKEN_ADMIN` to mint), `SHARE_E2E_ACCESS_EMAIL` or `SHARE_E2E_ACCESS_RULE`. L1 to L10 run once against `SHARE_E2E_TENANT_HOST`; R1 reuses the same two hostnames afterward, since L10 tears everything down first. +- A tiny stdlib Python HTTP+WebSocket echo server stands in for the live-share backend (L1's dev server, L3's POST and WebSocket echo asserts): no new dependency, a masked-frame decode and an unmasked reply for one text frame. +- A file-backed `security` stub replaces e2e-r2.sh's blanket "exit 44" stub: a tenant setup is a real `cloudflared` tunnel and needs its token to round-trip through `token_store`/`token_key` (store, then `cmd_start` reads it back) to ever connect, unlike the pure r2 backend e2e-r2.sh exercises, which never touches the Keychain. `docs/verification/access.md` hit the same real-Keychain-GUI-prompt problem and fixed it the same way (a file-backed round trip, never the real Keychain, never a literal token on the stub's own command line). + +**One live run** (admin token `op://Toolkit/cf-api-token/credential`, a run-scoped publisher token minted through `op://Toolkit/cf-tokens-admin/credential`, zone `d.foundation`, `SHARE_E2E_ACCESS_RULE=group:dwarves-ops`, hosts `share-e2e-79edd7.d.foundation` and `share-e2e-40ef61.d.foundation`): 20/70 checks passed. Two scripting bugs in `tests/e2e-tenant.sh` itself, found by this run and fixed in the same commit, explain nearly every failure: + +- Every `setup --r2`, `setup --no-r2`, and the R1 fold's `setup --r2 --alias` call was missing its hostname positional argument (`cmd_setup` requires it even when the profile's config already names a host); the dispatcher's own catch-all usage line is what L2 onward actually printed. Fixed: every call now reads `setup "$tenant" --r2 ...` etc. +- R1's C (a standalone r2 origin on the alias) tried a gated `add --access` with the admin `CLOUDFLARE_API_TOKEN` still in its environment; `share` refuses that for `--access` by design (an admin token stays admin-only for `setup`) and names the fix itself: store a narrower bucket-scoped token with `api-token` first, the same shape D2 in `### Dwarves` uses. Fixed: C now mints and stores a publisher token before its gated add, like B and D already did. +- A third issue, found by inspection after the run (not yet re-verified live): the original blanket `security` stub (see above) silently dropped A's real tunnel token, since `token_store` never checks `security -i`'s exit code; `cmd_start` then had nothing to hand `cloudflared`, so A's tunnel never connected and L1's own `live_check 60` timed out, which is the likely cause of L1's bare `exits 0: expected 0, got 1` with no printed `die` text reaching this run's log (unresolved: why the `die` line itself did not appear in the captured output is not root-caused). Fixed by the file-backed stub above; not re-run, per the one-live-run bound for this task. +- Every check from L2 through R1's second fold that depends on R2 being on (healthz, pointers, cloud links, the member joins, `ls`, the fold, the rollback) failed as a direct consequence of the two bugs above, not because of a CLI gap; `A's ls shows B's cloud row (TASK-6 merged list)` and `B's ls shows A's rows (TASK-6 merged list)` are the two checks that would still need TASK-6 (the merged list is not built yet) even with the script fixed. +- Cleanup verification: the trap's outside-in check (`leftovers()`, the e2e-r2.sh model: drain each bucket through the S3 listing, delete the bucket, the Worker, its routes, custom domains, DNS records, Access apps, and revoke every minted token, then re-GET each to confirm 404/empty) reported nothing left; a same-session manual audit against the live account (buckets, workers, DNS records, custom domains, tunnels, Access apps, routes named `share-e2e*`) independently confirmed zero rows in every category. One in-run assertion (`R1: alias bucket gone`) read `200` where `404` was expected moments before the trap's own drain-and-delete ran; the account-level audit after the trap finished found the bucket gone, consistent with R2's delete-then-GET having a brief read-after-delete lag rather than a leftover. +- Run log: `docs/verification/e2e-tenant-20261001T140035Z.log` (not committed; the script writes it fresh under `docs/verification/` on every run, keyed by UTC timestamp, per its own cleanup step). Static validation: `bash -n` and `shellcheck` both clean. +- Not exercised this run: the service-token fetch of a gated link (SPEC-007 TASK-1(l)'s method; no service-token credential is among this task's inputs, recorded as a skip rather than a failure); TASK-9b's `tests/e2e-migrate.sh` and rehearsal R2 (a separate task). +- Next real run should turn green through L10 on the fixes above; R1's alias-fold mechanics (the bucket match between the alias's existing Worker and the tenant's `--bucket` argument, the printed rollback list executed by hand) have not yet been proven live even once, since the single run never got past the hostname-argument bug to reach them in a working state. + diff --git a/docs/setup.md b/docs/setup.md index 6dd06f9..064ce26 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -220,6 +220,8 @@ For an external monitor (vps-mon or any uptime checker), watch `https://`; any publisher can remove any share. Only the install that added a share can `refresh` it, so a leaver's shares can be removed and added again, not refreshed. +## 4h. One hostname per tenant: R2 on the origin, the alias fold, moving the origin + +A tenant is one hostname. A tunnel profile (sections 1 to 3) serves every link from the origin machine's disk, and that is all a tenant needs. R2 is optional: the admin turns it on at the origin, and from then on each link picks local or cloud. A teammate can publish to a tenant from another machine only when its R2 is on; with R2 off the origin is the only publisher. + +| You want | Run | Where | +|---|---|---| +| Links that answer while the origin is off, and publishing from other machines | `setup --r2 --bucket ` | the origin, with the admin token | +| Cloud as the default for plain `add` | `setup --r2 --bucket --storage-default cloud` | the origin | +| Fold an older r2 hostname into the tenant | `setup --r2 --bucket --alias ` | the origin | +| Back to a plain tunnel (no alias set) | `setup --no-r2` | the origin | +| Move the tenant to another always-on machine (R2 off) | `migrate --to ` | the current origin | + +**Enable R2.** On the origin, with the admin token from 4f plus Workers Routes: Edit on the zone (the route on `/*`), and Access: Apps and Policies: Edit when an alias folds gated links: + +```sh +CLOUDFLARE_API_TOKEN= share --profile

setup s.example.com --r2 --bucket share-s [--storage-default local|cloud] +``` + +It refuses to run off the origin, on quick mode, or when a route on `/*` names another Worker, and every refusal comes before the first write. The tunnel and its DNS record stay as they are, so `setup s.example.com --no-r2` (which deletes the route) is the whole rollback. The origin keeps its tunnel config and gains `bucket=`, `r2_endpoint=`, and `storage_default=`. + +Then store a publisher token on the origin so local adds can write their pointer records: mint a token scoped to the share bucket as in 4g, plus Zone: Read, plus the Access scopes for gated adds, and store it with `share --profile

api-token` from a login session on that machine. Until a token is stored, a local `add` fails closed with a message naming R2; that is the cost of one namespace across both storages. `api-token --check` reports it. + +**Add a link.** + +```sh +share --profile

add ./report.pdf # storage_default decides +share --profile

add --cloud ./report.pdf # in the bucket: answers with the origin off +share --profile

add --local ./notes # on this machine +share --profile

add 3000 # live dev server: always local +``` + +On a member, `add` publishes to the bucket; `--local`, ports, and `--host` are refused. `share ls` marks each row `cloud`, `machine`, or `live` with its type and `by=`. + +**Fold an old r2 hostname.** `--alias f.example.com` moves a former r2 hostname into the tenant. The old hostname answers 301 to the same path on the tenant, the bucket is adopted in place, and each gated link keeps its Access app and AUD. If a step fails after the marker moved, the die prints the alias rollback list (rebind the old domain to its Worker, restore the marker host and app destinations, delete the pointer records, then `--no-r2`); `--no-r2` alone refuses while an alias is set. Rerunning the same command converges. + +**Move the origin.** `migrate` moves a tenant with R2 off to another machine you can ssh to. On the old origin, while it is online: + +```sh +CLOUDFLARE_API_TOKEN= share migrate --to user@new-origin [--remote-profile default] [--yes] +``` + +It copies every snapshot link with its id, name, dates, expiry, and gate, switches the hostname to a new tunnel on the new machine, checks every link through the public hostname, and only then moves the old rows aside (`~/share/index.migrated` and `~/share/migrated/`; nothing is deleted). Expect up to about a minute of 530 during the switch. Live links and `--host` links do not move: `migrate` lists live ones, refuses while a `--host` row exists, and refuses a tenant with R2 on. If the switch or the verify fails, it prints the rollback line, `share setup --tunnel-name --force`, to run on the old machine. + ## 5. Serve from a different machine Only machines listed in `hosts` serve. Two machines on one tunnel would split requests between two different `~/share` folders, so links would fail at random. -To move serving: install share on the new machine, run `share setup ` there (it reuses the tunnel and stores the token locally), then run `share service uninstall` on the old machine. Shares do not move with it; add them again on the new machine. +To move serving with its links, use `share migrate --to ` on the old machine (section 4h): it copies every snapshot link, switches the hostname to a new tunnel, verifies, and moves the old rows aside. To move without the links: install share on the new machine, run `share setup ` there (it reuses the tunnel and stores the token locally), then run `share service uninstall` on the old machine; shares do not move that way, so add them again on the new machine. ## 6. Teardown diff --git a/docs/specs/SPEC-008-one-host-per-tenant.md b/docs/specs/SPEC-008-one-host-per-tenant.md new file mode 100644 index 0000000..4ab1360 --- /dev/null +++ b/docs/specs/SPEC-008-one-host-per-tenant.md @@ -0,0 +1,600 @@ +# SPEC-008: one hostname per tenant, cloud storage optional per link + +Status: VALIDATED (three rounds; the round-3 critical folded without a fourth review per the cap; remaining warnings in the Decision Log, round 3) +Lane: full (an API and data contract between the CLI, the Worker, and Share Bar; a new public route in front of live links; a migration of two production hostnames) +Depth: research (outside: a route Worker passing requests through to a tunnel on its own hostname, Access order against a route Worker, custom-domain rebind, Access destination edits; TASK-1a and TASK-1b sample each before the task that consumes it) | blind-spot (failure: the Worker passes a request to the tunnel that a cloud record or a gate should have stopped, or the migration leaves a link dead) +References: SPEC-004 (Access gate), SPEC-005 (profiles), SPEC-006 (Share Bar across profiles), SPEC-007 (R2 backend), ADR-0002, ADR-0003, ADR-0005, ADR-0006, ADR-0007; `bin/share` `cmd_add`, `cmd_add_r2`, `rand_id`, `rows`, `share_url`, `cmd_ls`, `cmd_state`, `cmd_state_r2`, `cmd_profiles_json`, `cmd_setup`, `cmd_setup_r2`, `cmd_teardown`, `worker_js`, `r2_call`, `r2_snapshot`, `r2_healthz`; `mac/Sources/ShareBarCore/Snapshot.swift`, `MenuModel.swift`; `mac/Sources/ShareBar/StatusItemController.swift`. +Supersedes: SPEC-007 DEC-006 (a second hostname `f.d.foundation` beside `s.d.foundation`); SPEC-007 `## Out of scope` items "Share Bar reading an r2 default profile" (now in scope through the Keychain token only) and "migrating a tunnel profile's shares into a bucket" (now `share migrate` between machines; tunnel to bucket stays out). + +## Problem + +A tenant today can be two hostnames. Dwarves publishes machine-local links at `s.d.foundation` (the `dfoundation` tunnel profile on the Mac Mini) and cloud links at `f.d.foundation` (the `files` r2 profile, bucket `share-dfoundation`). People must know which hostname holds which link, and Share Bar shows two sections for one team. The personal tenant `s.han.ws` is served from the MacBook Air, so every personal link answers 530 while the Air sleeps. + +The operator direction (Han, firm): + +- A tenant is ONE hostname with ONE shared list. +- Storage is local by default: the origin machine's files through the tenant's tunnel, exactly as share works today. +- R2 is optional per tenant. It is off until the tenant admin enables it. Once on, each link picks local or cloud (`--cloud`), with a tenant default the admin sets. +- Exactly one always-on machine per tenant is the tunnel origin. The Mini is the origin for both tenants. +- Storage shows as a small indicator in the app, never as a separate subdomain. Share Bar rows also show the file type and the link type. + +## Context (research, 2026-10-01) + +Read-only probes from the Mini with the Toolkit API token (names, codes, and counts only; no id, token, or account id quoted). Commands: `## Grounding`. + +| Item | State | +|---|---| +| Mini profiles | `default` not set up; `dfoundation` serving `s.d.foundation` (tunnel `share-s-d-foundation`, `hosts=Mac-mini`, port 8791, one gated snapshot `a68960`); `files` serving `f.d.foundation` (`backend=r2`, bucket `share-dfoundation`, one gated cloud share `ba6377`) | +| `d.foundation` zone | `s.d.foundation` is a proxied CNAME to the tunnel; `f.d.foundation` is the Worker custom domain of `share-f-d-foundation` (an AAAA record the domain manages); one existing route Worker on the zone (`chat.d.foundation/dw-chat/*`), so route Workers work on this plan | +| Dwarves Access apps named `share ...` | one on `f.d.foundation/ba6377{,/*}`, one on `s.d.foundation/a68960{,/*}` | +| The guide link | `https://f.d.foundation/ba6377/support-ticket-guide/` answers 302 to the Dwarves Access login with `kid` equal to the app's AUD (gated cloud share) | +| `han.ws` zone (account "Han Ngo") | `s.han.ws` is a proxied CNAME to tunnel `air-share`, healthy with 4 connections (the Air is online now); R2 enabled, no share bucket; Analytics Engine answers; one route Worker exists (`radar.han.ws/*`); no Access app named `share ...`, so no personal gated share exists | +| Reachability | the Air reaches the Mini over ssh; the Mini cannot reach the Air | + +## Picture + +The request path for one tenant hostname: + +``` + visitor https://s.d.foundation// + https://f.d.foundation/ -- Worker answers 301 --> https://s.d.foundation/ + | + v + Cloudflare edge: TLS; per gated share an Access app on / and //* + | (one app per link, either storage, as today) + v + +-- R2 off (s.han.ws) -----------+ +-- R2 on (s.d.foundation) ---------------------------+ + | no Worker; the CNAME sends the | | Worker share- on route /* | + | request to the tunnel (today) | | 400 on %2F %5C %2E, low escapes, // | + +---------------+----------------+ | /healthz: own answer plus a probe of the tunnel | + | | GET m/ from the bucket | + | | cloud record -> R2 bytes (SPEC-007 checks, the | + | | Access JWT for a gated record) | + | | machine record, no record, R2 error | + | | -> fetch(request): same host, the route is | + | | skipped, the request goes to the origin | + | +--------------------------+--------------------------+ + v v + tunnel (one per tenant) --> cloudflared on the origin (the Mini) --> caddy --> pub/ or a live port +``` + +Who writes what: + +``` + origin (Mini) member (any other machine, R2 on only) + share add f -> pub/ share add f -> o/./ + m/ (cloud) + share add --cloud f -> o/ + m/ share add --local f, add 3000 -> refused: not the origin + share add 3000 -> live row + every local add, R2 on: m/ = {"v":2,"storage":"machine",...} (reserves the id, feeds the list) + \___________________ bucket share- ___________________/ + m/ records = the one shared list +``` + +## Design + +### Approaches considered + +| Question | Options | Chosen, with the reason | +|---|---|---| +| Default storage | R2 for snapshots, the tunnel for live and `--local`; the tunnel for everything, R2 per link | the tunnel (operator direction, 2026-10-01). A tenant with R2 off is today's share, byte for byte, with zero new infrastructure. R2 adds survival while the origin is off and publishing from other machines, and the tenant pays for it only when the admin turns it on. | +| How cloud and machine links share one hostname | two hostnames (today); a Worker in front that routes per id; a tunnel in front that proxies cloud ids to R2 | a Worker in front, only on tenants with R2 on. The edge already terminates TLS and runs Access there; a Worker can read the bucket through its binding with no token. A tunnel in front would need the origin up to serve cloud links, which defeats them. | +| How the Worker reaches the tunnel | (A) a route `/*` over the existing proxied CNAME, `fetch(request)` passes through to the origin; (B) a second, internal origin hostname behind an Access service token, the Worker fetching it with the token's secret; (C) the Worker as custom domain and the tunnel on another name | (A). The tunnel and its CNAME stay exactly as they are, so deleting the route is the whole rollback. No new hostname, DNS record, Access app, or service-token secret exists (the Worker keeps holding no credential, SPEC-007's rule). (B) costs a hostname, a secret with an expiry, and a second ingress rule, and its token would open every machine link to anyone holding it. (C) would change every existing link. TASK-1a(a) proves the pass-through. | +| Order inside the Worker | the tunnel first, R2 on 404; R2 first, the tunnel otherwise | R2 first (operator direction): one bucket read decides, the origin is never asked about a cloud link, and cloud links answer while the origin is off. | +| The shared list across machines | merge only on the origin (its index plus the bucket); a pointer record `m/` for every machine link | pointer records, written only when R2 is on. A member's `ls` and Share Bar see the origin's links too, and one namespace means a cloud add can never take an id the origin holds (`If-None-Match: *` on one key). With R2 off there is one publisher, so its index is the list. | +| The old hostname `f.d.foundation` | keep both; a Cloudflare redirect rule plus a placeholder DNS record; the tenant Worker answers 301 for an alias host | the tenant Worker takes the alias's custom domain and answers 301 to the same path on the tenant host. One Worker per tenant; the redirect lives in code the suite tests and the CLI deploys. | +| Moving the Air's shares to the Mini | publish through R2 and back; rsync plus a hand-edited index; `share migrate --to `, the receiver's own `share import` writing each share | `migrate` over ssh: the Air can reach the Mini, R2 is off on `s.han.ws`, and the receiving CLI validates and writes its own index (the only writer, as ADR-0003 keeps the app out of share's files). Ids and paths stay, so every link stays. | +| Which tunnel the new origin runs | reuse `air-share` on the Mini (`setup --tunnel-name air-share`, no DNS change); a new tunnel `share-` with the CNAME repointed (`setup --force`) | a new tunnel. A reused tunnel has two connectors until the Air stops, so setup's own three-in-a-row live check can land on the Air and fail; and the Air's config would keep the tunnel id the Mini now runs, so any later `teardown` on the Air deletes the Mini's tunnel. A new tunnel keeps each machine's teardown confined to its own tunnel (teardown deletes a DNS record only when it points at that tunnel). The cost is the DNS Edit scope and a short gap, bounded in `### Moving a tenant's origin`. | + +### Tenant, origin, member + +- A tenant is one hostname. Its origin is the one machine in the tunnel profile's `hosts=` (already a single machine: Cloudflare load-balances between connectors, so two serving machines give random 404s). +- R2 off: the origin is the only publisher. Other machines have no profile for the tenant; `share setup --quick` stays available to them as today. +- R2 on: the origin profile keeps its tunnel and gains `bucket=`, `r2_endpoint=`, and `storage_default=`. Any other machine is a member: a SPEC-007 r2 profile joined with a publisher token (`share --profile

setup --backend r2 --bucket `). A member publishes cloud links only. +- `docs/setup.md` onboarding says so plainly: a teammate can publish to a tenant only when its R2 is on. + +### Enabling R2 on a tenant (admin, on the origin) + +`CLOUDFLARE_API_TOKEN= share [--profile

] setup --r2 --bucket [--storage-default local|cloud] [--alias ]` + +It runs on the origin only (`host_ok`, else `run this on the origin, `), on a tunnel profile already set up for ``. SPEC-007's setup steps hold where this table does not replace them. Every read and every refusal comes before the first write. + +| Step | Call | Refuse or stop when | +|---|---|---| +| 1 | profile checks | not a named tunnel profile for ``; quick mode; `--alias` equal to `` or not a hostname | +| 2 | zone, account, role | the token cannot read `workers/scripts//settings` (only an admin enables R2) | +| 3 | SPEC-007 steps 4 and 5 on the bucket and the Worker `share-` | as SPEC-007, with one change: a marker naming `--alias` instead of `` is accepted (the fold) | +| 4 | with `--alias`: `GET workers/domains?hostname=` | the alias's custom domain names a service that is not `share-`, or that Worker's bindings are not `HOST == ` and `BUCKET == ` | +| 5 | `GET zones//workers/routes` | a route on `/*` or a wider pattern naming another script | +| 6 | the bucket snapshot (every `m/` record) and the local index | a local row's id holds a cloud record (a collision; named, nothing written) | +| 7 | with `--alias`: every gated cloud record's Access app (`GET access/apps/`) | an app whose name is not `share `, or no Apps Edit scope | +| 8 | deploy the Worker (bindings below), subdomain and previews off with the read-back (SPEC-007 step 10) | read-back not `false,false` | +| 9 | pointer records: `PUT m/` with `If-None-Match: *` for every local row | a 412 on an id step 6 saw free: die, the route is not attached yet | +| 10 | with `--alias`: each step-7 app gains `/` and `//*` (the PUT sends the full read-back with only `destinations` changed, which keeps every field and the AUD; an omitted field resets to its default; TASK-1b(g)); the gate probe on the new destinations passes three rounds (SPEC-004) | gate timeout: the added destinations are removed, die | +| 11 | write `bucket=`, `r2_endpoint=`, `storage_default=` (default `local`), and `aliases=` into the config, so every later die leaves a state `--no-r2` and a rerun can read | | +| 12 | marker `PUT share.json` with `If-Match`: `{"v":1,"host":"","aliases":[""]}` | 412 | +| 13 | `POST zones//workers/routes {pattern:"/*", script:, request_limit_fail_open:true}` (TASK-1b(k): both accounts are on Workers Paid, so the flag only matters after a drop to free) | an API error | +| 14 | with `--alias`: `PUT workers/domains {hostname:, service:, override_existing_origin:true}` (TASK-1b(f): rebinds in place with no failed request; the plain PUT answers `100116`) | an API error | +| 15 | live checks: `/healthz` three times in a row with this CLI's `X-Share-Worker` pair and `X-Share-Tunnel: 1`; a probe file through the tunnel leg; with `--alias`: `https:///healthz` answers 301 to `https:///healthz` | timeout: die naming the leg. Without `--alias` the named rollback is `--no-r2`. With `--alias` the die prints the alias rollback list (`## Migration`, Dwarves), never `--no-r2` alone | +| 16 | with `--alias`: each step-7 app drops its `` destinations; the local profile whose `hostname` is `` hands its `r2-own` lines to this profile | | + +A folded app keeps its name `share ` and its AUD. `access_delete` and the by-name purge accept an app named for this host or for any host in `aliases=`, so `rm` and expiry of a folded share delete its app. + +A rerun of the same command after a die at any step converges: each write is idempotent (deploy compares the SHA, pointers answer 412 for an id that already holds this machine's pointer and count as done, the app PUT and the marker PUT compare before writing, the route and domain reads show an existing binding to this Worker). + +Worker bindings on a tenant: SPEC-007's set plus `PASS="1"` (pass misses to the origin) and `ALIASES` (comma list). An r2-only profile (SPEC-007, custom domain) keeps `PASS=""` and today's answers. + +SPEC-007's `setup --backend r2 --bucket ` refuses a tenant host in its admin role, even with `--force`, before any write: when the deployed Worker binds `PASS="1"`, when the marker holds `aliases`, or when a route on `/*` exists. Its message names the tenant command, `run ' setup --r2 --bucket ' on the origin`. Without this, an admin token in the environment on a member (the join path's version hint points there) would redeploy the Worker without `PASS` and `ALIASES`, and replace the tunnel CNAME with a custom domain, taking every machine link and the alias down. The join role is unchanged; the join's version-mismatch hint names the tenant command when `X-Share-Tunnel` is present. + +`share [--profile

] setup --no-r2` (admin token, on the origin) is the rollback for a tenant without an alias: delete the route, then drop the four config keys. It refuses while the config's `aliases=` or the marker's `aliases` is non-empty (` redirects here; run the alias rollback first: ...` with the list), because an alias whose 301 lands on a plain tunnel turns every folded cloud link into a 404. The tunnel serves the hostname alone again. Pointer records, cloud records, the bucket, and the Worker stay; `teardown --yes --purge` on a member, or the purge path on the origin, removes them as SPEC-007 says. + +### Per-link storage on the origin + +| `add` form | R2 off | R2 on, `storage_default=local` | R2 on, `storage_default=cloud` | +|---|---|---|---| +| `add ` | local | local | cloud | +| `add --cloud ` | refused: `R2 is off for ; the tenant admin enables it with ' setup --r2 --bucket '` | cloud | cloud | +| `add --local ` | local | local | local | +| `add `, `--host` | local (live, own host) | local | local | +| `--cloud` with a port or `--host` | refused: `live links and --host stay on the origin's tunnel` | refused | refused | +| `--cloud` with `--local` | refused (usage) | refused | refused | + +A cloud add on the origin runs SPEC-007's `cmd_add_r2` path unchanged. A member runs it for every add; `--local` and `` on a member die with ` serves local links from its origin; this machine publishes cloud links only`. + +`rm`, `refresh`, and `hits` dispatch on the row's storage. On the origin a machine row takes today's tunnel path (plus its pointer delete) and a cloud row takes SPEC-007's r2 path. On a member, `rm` and `refresh` of a machine row die before any call with ` is served from the tenant's origin; remove it there`, because deleting the pointer would report success while the origin keeps serving the file (gated or public), and the origin's next reconcile would write the pointer back. + +With R2 on, a local add on the origin also writes the id's pointer record: + +1. `rand_id` checks the local index, `pub/`, and `access-pending` as today, then the bucket as SPEC-007's r2 branch does (`GET m/`, the `o/.` prefix scan). +2. After the stage is built and before `stage_publish` (and, for a gated add, before the Access app): `PUT m/` with `If-None-Match: *` and `{"v":2,"id","storage":"machine","name","by","added","expires","opts","type"}`. `opts` keeps only `noindex`, `live`, and `gated` (a flag, never the rule, so teammates' addresses stay out of objects every member reads). A 412 picks a fresh id; any other failure dies before anything is served, naming R2 (a tenant with R2 on keeps one namespace, so a local add fails closed; DEC-006 states the cost). +3. `rm` and `prune` of a local row delete its pointer after the local removal. Without a resolvable token (the login service's hourly prune runs with `SHARE_API_TOKEN_OFF=1`) the pointer stays; it is harmless (the Worker passes the id to the tunnel, which answers 404), and the next interactive `ls` or `prune` on the origin, or any member's `prune` once `expires` is past, deletes it. +4. Reconcile on the origin's interactive `ls` and `prune`, keyed on `storage == "machine"` alone (one origin per tenant, so every machine record is the origin's, migrated rows included): a machine record with no local row, whose S3 `LastModified` is over 10 minutes old, and whose id is neither in `access-pending` nor under a held add lock, is deleted (an add that died after step 2); a local row with no pointer gets one (`If-None-Match: *`; a 412 means a cloud record holds the id: printed as ` is shadowed by a cloud link; rm one of them`, never auto-deleted). + +Bucket rows are display data. A record read from the bucket, cloud or machine, never enters `rows()`, `index.tsv`, the Caddyfile, `refresh`, or a stage path, so a forged record cannot add a reverse proxy, a site block, or a source path on the origin. On a profile with a tunnel, the bucket reader writes its own snapshot file and never assigns `index` (SPEC-007's `r2_snapshot` rebinds `index` for the process, which on the origin would hand bucket rows to `write_caddyfile` in the same run). Every bucket key is checked against `^m/[0-9a-f]{6}$` (after URL decoding) before its record is read, displayed, built into a URL, or deleted. The same holds for every cloud verb on a profile with a tunnel: `cmd_rm_r2`, `r2_prune`, the cloud expiry, `hits`, and the purge read the reader's own file and never call `r2_snapshot`, never assign `index`, and never write `$index` (today `cmd_rm_r2` falls back to `r2_snapshot` and moves a file over `$index`), so one `prune` that expires a cloud row and then a local row renders only local rows. + +The origin's interactive bare `prune` and `ls` run SPEC-007's cloud expiry, its gated-expiry rule, and the orphan sweep through that reader, because after the fold the origin may be the only publisher left; the login service's hourly prune (token off) touches local rows only, as today. + +`by` is always `r2_by` (lowercase, `a-z0-9.-`): on local rows, pointers, `import`'s argument, and `refresh`'s `:` match, so `Mac-mini` and `mac-mini` never disagree. The `v:2` reader applies `r2_record_row`'s checks plus: `id` equal to the key's id; `expires` matching `^[0-9]{1,11}$` and `added` matching `^[0-9]{4}-[0-9]{2}-[0-9]{2}$` before any shell arithmetic sees them; `name` and `by` free of control bytes and tabs, `by` in the hostname set; `opts` tokens only from `noindex`, `live`, `gated`; `type` from the type table; any other field ignored. A record that fails is skipped like a forged row. + +Record versions: cloud records stay `v:1` and gain an optional `type` field that `v:1` readers ignore, so a v0.8.0 CLI and Worker keep reading every cloud record. Machine records are `v:2` (`WORKER_RECORD_V` becomes 2): a v0.8.0 CLI drops them from `ls` and its orphan sweep skips with a warning, and a v0.8.0 Worker answers 404 for them, which is correct for a Worker that cannot pass through. The new CLI's orphan sweep (`r2_record_raw`) accepts a `v:2` record whose `storage` is `machine` and which has no `prefix` as readable and referencing no upload, so pointers never block the sweep; only an unreadable record or one above `v:2` still does. + +### The Worker (WORKER_VERSION 4) + +Checks run in this order. Rows marked "as SPEC-007" are unchanged. + +| # | Request | Answer | +|---|---|---| +| 1 | Host is in `ALIASES` | GET or HEAD: 301 to `https://`; other methods: 405 | +| 2 | Host is not exactly `HOST` | 404 | +| 3 | raw path holds `%2F`, `%5C`, `%2E`, a low escape, or `//` | 400 (as SPEC-007; now also in front of the tunnel leg) | +| 4 | `/healthz`, `PASS` empty | as SPEC-007 | +| 5 | `/healthz`, `PASS` set | the Worker fetches `https:///healthz` (the route is skipped, TASK-1a(a)) with a 3 s timeout. A `200 ok` sets `X-Share-Tunnel: 1` and answers `200 ok`; anything else sets `X-Share-Tunnel: 0` and answers `503 tunnel down`. Both carry `X-Share-Worker` and `X-Share-Gate` | +| 6 | first segment is not 6 lowercase hex | `PASS` set: pass through (step 9); else 404 | +| 7 | `GET m/` returns a `v:1` record with no `storage` | SPEC-007 rows: version, id, prefix, and expiry checks (404 on any), the JWT for a gated record (404), then GET or HEAD served from R2 (other methods 405), hits counted | +| 8 | a `v:2` record whose `storage` is exactly `machine`; no record; or `BUCKET.get` itself throws or exceeds 2 s | `PASS` set: pass through; else 404 | +| 8b | any other record: not JSON, a `v` above 2, `v:2` with another `storage`, `v:1` with a `storage` key | 404, never passed through | +| 9 | pass through | `fetch(request)`, any method, the body streamed, the original headers (the Access JWT and cookie included). The origin's answer goes back rewrapped (`new Response(r.body, r)`, so `no-store` and `noindex` can be set on headers a `fetch` answer holds immutable), with status and body unchanged; a 101 WebSocket answer goes back untouched (TASK-1a(d)); except a failed fetch, a 503 without `cf-cache-status` (the origin's own answers carry it, edge errors do not), 520 to 527, or 530 (TASK-1a(e) pins the set; a 502 is not in it, DEC-013): `503`, `Retry-After: 60`, `Cache-Control: no-store`, `X-Robots-Tag: noindex, nofollow`, body `the machine serving this link is offline; try again later`. No hit is counted | + +Callers of `/healthz` read the cloud leg from the headers, never from the status alone: a `503` carrying a well-formed `X-Share-Worker` pair and `X-Share-Tunnel: 0` means the Worker is up and the origin is down. `r2_healthz` sets `hz_code=200` for it and a new `hz_tunnel=0`, so a member's join (`r2_wait_healthz join`), a gated `add` on a member, and an r2 `state` (`ready: true`) keep working while the origin is off, which is the case R2 exists for. Only the origin's own `status` and `state` report the tunnel leg, as they do today. + +A pass-through never serves R2 bytes, and a cloud record never reaches the tunnel. A gated machine link keeps its gate at the edge: Access runs before a route Worker (TASK-1a(b)) and the Worker forwards the visitor's credentials (TASK-1a(c)). An expired or malformed cloud record answers 404 and is never passed through, so a stale cloud id cannot fall back to a same-id file on the origin. Caddy keeps its own `@encsep` rule, so the origin refuses encoded separators even with the route deleted. + +Every s.d.foundation request now runs the Worker and one bucket read (two for a cloud hit). Dwarves is on the Workers paid plan; live dev-server links (many small requests) cost Worker requests in the included 10 million. + +### One shared list + +`ls`, `state`, and `profiles --json` read one row set per profile: + +| Profile | Rows | +|---|---| +| origin, R2 off | the local index, as today | +| origin, R2 on | the local index, plus every cloud record (machine records are the origin's own pointers; the local index is their truth, and an orphan pointer is the reconcile's, not a row) | +| member | every bucket record | + +Each row carries: + +| Field | Values | Source | +|---|---|---| +| `storage` | `machine` or `cloud` | the row's origin | +| `kind` | `snapshot` or `live` (the link type, unchanged) | the index opts, the record `opts` | +| `type` | `pdf`, `image`, `video`, `audio`, `folder`, `site`, `markdown`, `archive`, `text`, `other` | the table below | +| `by` | the machine that published it: the origin's own `this_host` for local rows, the import's `by=` for migrated rows, the record's `by` for bucket rows | | +| `access` | the rule or `null` (unchanged); a member sees `gated` for the origin's gated machine rows, because pointers carry no rule | | + +`share ls` prints one tag before each link: `cloud`, `machine`, or `live` (a live row is always machine), then the type and `by=`. + +File type, first match wins: + +| `type` | Rule | +|---|---| +| `site` | a live link; a folder holding `index.html` at its root; a single `.html` or `.htm` file | +| `folder` | any other folder | +| `markdown` | `.md`, `.markdown` | +| `pdf` | `.pdf` | +| `image` | `.png .jpg .jpeg .gif .webp .svg .heic .ico .avif` | +| `video` | `.mp4 .mov .webm .m4v .mkv` | +| `audio` | `.mp3 .wav .m4a .aac .flac .ogg` | +| `archive` | `.zip .tar .gz .tgz .bz2 .xz .7z .rar .dmg` | +| `text` | `.txt .csv .json .yaml .yml .toml .xml .log .sh .py .js .mjs .ts .go .rs .rb .swift .java .c .h .css .sql` | +| `other` | anything else | + +Local rows are typed at read time from `pub/` (`-d`, `-f`). New cloud records and pointers store `type` at add time. A `v:1` cloud record without `type` is typed by its name's extension, and a name with no extension counts as `folder`. + +`state` (schema stays 1; every addition is a new key): `shares[].storage`, `shares[].type`, `shares[].by`; top-level `r2` (`true` when the profile reads a bucket) and, on an origin with R2 on, `storage_default`. When bucket rows cannot be read, `state` still prints the local rows and adds `cloud_error` (the reason line), so the menu never loses machine links to an R2 problem. + +Token rule for listings: `state` and `profiles --json` read the bucket only with a token from the Keychain item or the 600 file. A profile whose token source is `api_token_cmd` (it may prompt) or the environment gets `cloud_error: "the menu reads cloud links only with a stored token: api-token"`. `ls` uses the full resolution, as SPEC-007. `profiles --json` already lists r2 rows (only the text `profiles` sets `SHARE_STATE_BRIEF`). To bound the poll's cost, `state` reads the `m/` listing (its `LastModified` per key), drops the keys whose id is in the local index (the origin's own pointers), and fetches only the newest 25 of the rest, the most any section shows; it reports the remaining listing count as `cloud_more`, which the section adds to its `more` line. `ls` still reads every record. At a 60 s poll that is about 1.1 million Class B reads and 43,000 listings a month per polling Mac, roughly $0.60 at list prices, whatever the record count. + +### Share Bar + +One section per profile, as SPEC-006. After the migration the Mini shows `default · s.han.ws` and `dfoundation · s.d.foundation`; the `files` section is gone. Each row: + +``` + [type icon] support-ticket-guide 30d left [storage] [link type] [lock] +``` + +- `NSMenuItem.image` is the file-type symbol (template). +- The trailing column, after the expiry text, holds three text attachments: the storage badge, the link-type marker, and `lock.fill` on a gated row (SPEC-006's lock moves from `image` into this group). +- The accessibility title is `, , , , ` plus `, login required` on a gated row. +- A section whose state has `cloud_error` gets one disabled line `Cloud links not listed: `. + +SF Symbols (every one ships on macOS 13, the package minimum; a unit test resolves each with `NSImage(systemSymbolName:)` and the app falls back to no image when one fails): + +| Role | Value | Symbol | Accessibility word | +|---|---|---|---| +| type | `pdf` | `doc.richtext` | PDF | +| type | `image` | `photo` | image | +| type | `video` | `film` | video | +| type | `audio` | `waveform` | audio | +| type | `folder` | `folder` | folder | +| type | `site` | `globe` | site | +| type | `markdown` | `doc.plaintext` | Markdown | +| type | `archive` | `archivebox` | archive | +| type | `text` | `doc.text` | text | +| type | `other` (or an unknown value) | `doc` | file | +| storage | `machine` | `desktopcomputer` | on this tenant's machine | +| storage | `cloud` | `cloud` | in the cloud | +| link | `snapshot` | `doc.on.doc` | snapshot | +| link | `live` | `dot.radiowaves.right` | live server | +| access | gated | `lock.fill` | login required | + +A row with no `storage` (an older CLI) shows no storage badge; a row with no `type` uses `doc`. `ShareBarCore` holds the mapping as plain strings (`RowGlyphs.type(_:)`, `.storage(_:)`, `.link(_:)`), so it is unit-tested without AppKit. + +The publish dialog gains a `Storage` popup only for a profile whose state has `r2: true` and is the origin (`serves_here`): `On ` or `In the cloud`, preselected from `storage_default`; the argv gains `--cloud` or `--local`. A member profile shows `In the cloud` as a disabled label. + +### Moving a tenant's origin: `share migrate` + +`share [--profile

] migrate --to [--remote-profile ] [--remote-bin ] [--yes]`, run on the current origin while it is online. It needs `CLOUDFLARE_API_TOKEN` (Tunnel Edit, DNS Edit, Zone Read on the tenant's zone) and an ssh login on the target as the user who will run share there. It moves a tenant with R2 off only: on a profile with `bucket=` it refuses (`R2 is on for ; moving its origin also moves the pointers and the publisher token, which this version does not do`). + +Every remote call is one fixed string that any login shell (fish on Han's machines) parses the same way, because ssh joins its argv and the remote shell parses it: `/bin/bash -c "$(printf %s | base64 -D 2>/dev/null || printf %s | base64 -d)" share-migrate ` built with no quote or backslash outside the base64, where every argument is a checked slug or base64 (`--remote-bin` and `--remote-profile` included); the script sets `PATH` from the `--remote-bin` directory plus `/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin`, and a target starting with `-` is refused. The switch token travels on ssh stdin to a new `setup --token-stdin` flag, which reads one line into the process's own `CLOUDFLARE_API_TOKEN`. + +``` + Air (old origin) Mini (new origin), over ssh + 1 preflight: named tunnel profile, this host in remote: import --probe -> "share-import 1"; + hosts, no --host row, token resolves --profile state: not_setup, or set up + for the same hostname; an empty index and pub/ + when not_setup; the profile's port pair free; + a Keychain write/read/delete round trip and + launchctl print gui/ answering + 2 per snapshot row: --profile import + tar -C pub/ -cf - . | ssh -------------------> + (stdin: the tar; validates, stages, publishes) + 3 the switch: the token on ssh stdin --------------> CLOUDFLARE_API_TOKEN from stdin; + --profile setup --force + --token-stdin + (new tunnel share-, CNAME repointed, + service, live check) + 4 verify through https:///: a nonce file that only the target holds answers 200 + three times in a row (so the Air cannot answer for it), then each moved link answers + 200, and each gated one 302 with kid equal to its app's AUD (a 200 there fails) + 5 retire: moved rows -> index.migrated, pub/ -> migrated/ (nothing deleted), + then the teardown path: service, tunnel air-share deleted, DNS left alone (it no longer + points at this tunnel), tokens forgotten +``` + +- `import` is a new verb (one usage line: `share import ... used by migrate over ssh`). It refuses: an `expires` outside `^[0-9]{1,11}$`, an `added` outside `^[0-9]{4}-[0-9]{2}-[0-9]{2}$`, a `src` with a control byte or a tab; an id that is not 6 lowercase hex, or that exists in the local index, `pub/`, or `access-pending`; a name that is empty, starts with `.`, holds `/` or a control byte; opts other than `noindex`, `access=`, `access_rule=` (a live or `host=` row never arrives); a `by` outside the hostname character set. It spools stdin to a temp file under the profile root (capped at `SHARE_IMPORT_MAX_BYTES`, 2 GiB) and lists it with `tar -tvf` first: any member that is not a regular file or a directory (symlinks and hardlinks included), an absolute path, a `..` segment, or a dotfile refuses the whole import before anything is extracted. Only then does it extract into a fresh stage, and it refuses again if `find` sees a link count above 1 or anything but files and directories (TASK-7a (its first step) pins the flags on bsdtar and GNU tar). It then publishes as `stage_publish` does and appends the row with the original added date and expiry, `by=` added to opts, and `src` set to `:`. It reloads caddy when the profile is serving and works on a `not_setup` profile (pub and index need no setup). It never creates an Access app; the gated row keeps its `access=`, because the app's destinations (`/`) do not change with the machine. +- `refresh` of a row whose `src` starts with `:` for another host dies with ` was moved from ; re-add it from a source on this machine`. +- Live rows and their ports are listed as `not moved: live link (port ); after the move run ' add ' on the new origin`. `migrate` refuses while a `--host` row exists (` has its own hostname on this tunnel; remove it first: rm `). +- Stop points: a failure in steps 1 and 2 changes nothing on this machine and leaves imported copies on the target, unserved unless its profile serves; a rerun skips ids the target already holds with the same name and added date. A failure in step 3 leaves this machine serving; the DNS record may already point at the new tunnel, and `migrate` then prints the rollback, ` setup --tunnel-name --force` on this machine. The tunnel name is explicit because `setup` derives `share-` when none is given, which is the name the target just created: without it the old origin would join the new tunnel as a second connector. With it, setup reuses `air-share` and repoints the CNAME there; the target's new tunnel is left with no DNS record and serves nothing. A failure in step 4 stops before step 5 and names each link that did not answer. +- `--yes` skips the one confirmation (`Move links of to and retire this machine as its origin? [y/N]`). +- Test seam: `SHARE_MIGRATE_SSH` replaces `ssh -o BatchMode=yes ` with a local command (for example `env HOME=

`), so the suite and the rehearsal run both sides on one machine. The seam joins the remote argv into one string and runs it through `fish -c` when fish is installed, else `sh -c`, so the suite parses the command as a real login shell does. + +What visitors see: before the run, the Air serves (530 whenever it sleeps, as today). During step 3, from the CNAME change until the Mini's connector is up (setup writes DNS before it stores the token, installs the service, and starts; its live check allows 60 s), up to about a minute of 530; R2 measures the real gap. After it, the Mini serves every moved link at the same URL. Live links from the Air end at step 5. + +### Tokens + +SPEC-007's model holds: an admin token from the environment for setup work, never stored; one bucket-scoped publisher token per machine, stored with `share api-token`, with an `expires_on`. + +| Role | Scopes | Source | +|---|---|---| +| admin, `setup --r2`, `--no-r2` | SPEC-007's minimal admin set (Workers Scripts Edit on the account, DNS Read and Zone Read on the zone, Workers R2 Storage Edit), plus Workers Routes Edit on the zone (the route), plus Access Apps and Policies Edit and Access Organizations Read (the alias fold's destination edits and the `TEAM` binding) | `op://Toolkit/cf-api-token/credential` (TASK-1b(h): it holds every scope on both accounts); a run-scoped token minted through `op://Toolkit/cf-tokens-admin` (`POST /user/tokens`, `expires_on` one day, the set in this row) also works and is the narrower choice | +| `migrate` switch | Cloudflare Tunnel Edit, DNS Edit, Zone Read on the tenant's zone | a run-scoped one-day token as above (the round-1 warning); `op://Toolkit/cf-api-token/credential` also edits tunnels and DNS on the personal account (TASK-1b(i)) | +| origin publisher (Mini, `dfoundation`) | Workers R2 Storage Bucket Item Write on `share-dfoundation` only, plus SPEC-004's Access scopes (gated adds), plus Zone Read, plus Account Analytics Read (`hits` of a cloud row) | minted once through `cf-tokens-admin`, `expires_on` set, stored with `share --profile dfoundation api-token`. `api-token` and every bucket call key the r2 branch on `bucket=` (today they key on `backend == r2`), so `r2_publisher_check` refuses an admin or account-wide R2 token on the origin too, when stored and when first used from the environment | +| member publisher | as SPEC-007 | as SPEC-007 (DEC-007 there: one per teammate) | + +No token reaches argv, a file, a Worker binding, or an object. `migrate` sends the switch token over ssh stdin, read on the target by `IFS= read -r` into the environment of the one `setup` process. + +## Migration + +Both runs follow a rehearsal on throwaway hostnames (`## Test plan`, R1 and R2). Each step names its rollback. + +### Dwarves (`s.d.foundation`, R2 on, `files` folded in) + +| # | Step | Check | Rollback | +|---|---|---|---| +| D1 | upgrade share on the Mini to the release | `share --profile dfoundation status` unchanged | `brew` pin of the previous version | +| D3 | `CLOUDFLARE_API_TOKEN= share --profile dfoundation setup s.d.foundation --r2 --bucket share-dfoundation --alias f.d.foundation` (storage default `local`) | the setup's live checks; `https://f.d.foundation/ba6377/support-ticket-guide/` answers 301 to `https://s.d.foundation/ba6377/support-ticket-guide/`, which answers 302 to the Access login with `kid` equal to the app's AUD; `a68960` still gated; `share --profile dfoundation ls` shows both rows, `ba6377` as `cloud`, `a68960` as `machine` | below | +| D2 | after D3 (its config gives `api-token` the bucket and endpoint): mint the origin publisher token (table above); `share --profile dfoundation api-token` stores it in the Keychain from a GUI login session on the Mini (the session Share Bar and Han's adds run in) | `api-token --check` passes the bucket and Access lines; a local `add` and `rm` from that session write and delete a pointer | revoke the token; local adds fail closed until a token is stored | +| D4 | `share --profile files teardown --yes` (r2 teardown is local: config, token, `r2-own` go; the bucket and every share stay) | `share profiles` lists `default` and `dfoundation`; Share Bar shows one Dwarves section | only after the D3 rollback (the join reads the marker host): `share --profile files setup f.d.foundation --backend r2 --bucket share-dfoundation`; revoke the `files` profile's publisher token once D6 is done | +| D5 | right after D3: move any external monitor of `f.d.foundation/healthz` to `s.d.foundation/healthz` | vps-mon shows the check green | revert the catalog line | +| D6 | after seven days with no rollback: delete the Worker script `share-f-d-foundation` (it has no domain since D3) | `GET workers/scripts/share-f-d-foundation` answers 404 | redeployable from the v0.8.0 `bin/share` | + +Rollback of D3, in order, all with the admin token, rehearsed in R1 (the step-15 die with `--alias` prints exactly this list): rebind `f.d.foundation` to `share-f-d-foundation` (`PUT workers/domains`); restore the marker host `f.d.foundation` (`If-Match`); restore the `f.d.foundation/ba6377{,/*}` destinations on its app; clear `aliases` in the marker and the config; delete every `v:2` machine record (a v0.8.0 sweep refuses to run while one exists); `share --profile dfoundation setup s.d.foundation --no-r2` (the route goes; `s.d.foundation` is the plain tunnel again). The old Worker still serves every `v:1` cloud record, and it answers 404 for the `v:2` pointers it cannot pass through, which is correct on `f.d.foundation`. Hits recorded before D3 stay in dataset `share_f_d_foundation`; `share hits` after D3 reads `share_s_d_foundation` only. + +### Personal (`s.han.ws`, R2 off, origin moves from the Air to the Mini) + +| # | Step | Check | Rollback | +|---|---|---|---| +| P1 | upgrade share on the Air and on the Mini | `share import --probe` on the Mini prints `share-import 1` | `brew` pin | +| P2 | on the Air: `CLOUDFLARE_API_TOKEN= share migrate --to tieubao@ --remote-profile default` | its step 4 (every moved link answers through `https://s.han.ws/`); on the Mini `share ls` shows them with `by=`; `share profiles` on the Mini lists `default · s.han.ws · serving` | `share setup s.han.ws --tunnel-name air-share --force` on the Air before step 5 (printed by `migrate`, run once in R2); after step 5 the Air's copies sit in `~/share/migrated/` and its rows in `~/share/index.migrated` | +| P3 | Share Bar on the Mini | two sections, `default · s.han.ws · Serving` and `dfoundation · s.d.foundation · Serving` | | + +The Mini's default profile is free today (`not_setup`), so `s.han.ws` lands there on port 8787 under the user that runs the `dfoundation` profile (`tieubao`). The new tunnel is `share-s-han-ws`; `air-share` is deleted in P2's step 5. R2 stays off; Han can enable it later with `setup --r2` on the Mini, which needs a bucket on the personal account (R2 is enabled there today, with no share bucket). After P2 the Air has no `s.han.ws` profile; it publishes personal links only after R2 is on (as a member) or by ssh to the Mini. + +## Byte identity + +A tunnel profile with no `bucket=` (every R2-off tenant) keeps every file, output line, and `state` key it has today, with three additions: the `ls` tag column (`machine` or `live`, the type, `by=`), and `state`'s new keys `storage`, `type`, `by`, `r2: false`. An r2 member profile keeps SPEC-007's behavior plus the same listing fields and its rows in `profiles --json`. Row 1 of the coverage matrix pins both. + +## Files + +- `bin/share`: `--cloud`, `--local`, `storage_default`; `setup --r2`, `--no-r2`, `--alias`; pointer records and the reconcile; the merged rows and the new fields in `ls`, `state`, `profiles --json`; `import`, `migrate`; Worker v3 in `worker_js`; usage lines and the skill rows. +- `tests/share.sh`: sections under `SHARE_R2_DRY=1` and `SHARE_MIGRATE_SSH`. +- `tests/worker.mjs`: pass-through, alias, and healthz cases with a stubbed origin `fetch`. +- `tests/e2e-tenant.sh` (new): live legs L1 to L10 and rehearsal R1. `tests/e2e-migrate.sh` (new): rehearsal R2. +- `mac/Sources/ShareBarCore/Snapshot.swift` (`Share.storage`, `.type`, `.by`; `Snapshot.r2`, `.storageDefault`, `.cloudError`), `RowGlyphs.swift` (new), `MenuModel.swift`, `Publish.swift`; `mac/Sources/ShareBar/StatusItemController.swift`; tests under `mac/Tests/ShareBarCoreTests/`. +- `README.md`, `docs/how-it-works.md`, `docs/setup.md` (tenant, origin, member, the onboarding rule), ADR-0008 (one hostname per tenant), `docs/verification/one-host-per-tenant.md`. + +## Failure modes + +| Failure | Detection | Behavior | +|---|---|---| +| the origin is off, R2 on | the Worker's pass-through gets 530 | cloud links serve; machine and live links answer the 503 offline page; `/healthz` answers 503 with `X-Share-Tunnel: 0` | +| the origin is off, R2 off | none at the edge | Cloudflare 530, as today | +| R2 or the bucket binding fails | the bucket read throws | the request passes through: machine links serve, cloud ids reach Caddy and answer 404, gated cloud ids are gated at the edge first; never served unchecked | +| a cloud record and a machine link share an id | `If-None-Match: *` on `m/`; the reconcile | a cloud add picks another id; a pre-existing collision (an origin on v0.8.0 while R2 was on) is printed by the origin's `ls` and setup step 6; the Worker serves the cloud record | +| a pointer write fails on a local add | `r2_call` code | the add dies before `stage_publish`; nothing is served | +| a pointer outlives its link | the reconcile, prune | deleted after 10 minutes by the origin, or after expiry by any member; until then the id passes through and answers 404 | +| a request the route Worker passes through re-enters the Worker | TASK-1a(a) | the spike blocks the build: the pass-through design changes to (B) before TASK-3 | +| Access does not run before a route Worker | TASK-1a(b) | the spike blocks the build; a gated machine link must never reach the origin unauthenticated | +| a forged cloud record over a machine id (a publisher with bucket write) | none; same trust as SPEC-007 DEC-011 | the cloud bytes answer, inside the edge's Access app for that path when one exists; publishers are named teammates | +| an alias rebind leaves `f.d.foundation` without a Worker | step 14's API answer; step 15's 301 check | die naming the rollback; the old Worker stays deployed | +| the gate probe on the added destinations times out | step 10 | the added destinations are removed; nothing else has changed | +| `migrate` loses ssh mid-copy | the pipe's exit code | stop; the target holds the copies already imported, unserved unless its profile serves; a rerun skips them | +| the switch fails after the CNAME moved | the remote setup's exit | `migrate` prints the one rollback command; the Air's tunnel and copies are intact until step 5 | +| a moved link does not answer | step 4 | stop before step 5, naming each link | +| a tar entry escapes the stage, or a symlink arrives | `import`'s stage scan | the import of that id dies; nothing is published for it | +| a member removes or refreshes a machine link | the row's `storage` | refused before any call; only the origin removes it | +| a forged bucket record names a live port, a host, or a shell expression | the `v:2` reader's checks; bucket rows never reach `rows()` | skipped or shown as text; nothing reaches the Caddyfile or arithmetic | +| the origin is off and a member joins or adds a gated link | `X-Share-Worker` and `X-Share-Tunnel` on a 503 | treated as Worker up; join and add proceed | +| an alias fold dies after the marker moved | step 11 wrote the config first | the die prints the alias rollback list; `--no-r2` refuses until it ran | +| an old share CLI on the origin while R2 is on | none | it serves and adds local links without pointers; the reconcile on the next upgraded `ls` writes them and names any collision | +| the token source is a command and the menu polls | the listing token rule | local rows plus `cloud_error`; no prompt during a poll | + +## Task Breakdown + +- [x] TASK-1a: edge spike on a throwaway hostname, one Worker, one tunnel, one Access app; answers in `docs/implementation-notes/one-host-per-tenant.md`. (a) A route Worker's `fetch(request)` and a `fetch` of `https:///healthz` reach the tunnel origin without running the Worker again. (b) Access answers an unauthenticated request to a gated path with its 302 before the route Worker runs; an admitted request reaches the Worker with `Cf-Access-Jwt-Assertion`. (c) The pass-through of an admitted request reaches the origin. (d) A POST and a WebSocket upgrade pass through to a live dev server. (e) The status codes a pass-through sees with the connector down, and a header that tells Caddy's own 502 apart. AC: each answered; (a) or (b) failing blocks TASK-3 and amends `### Approaches considered`. +- [x] TASK-1b: account spike with throwaway tokens and names. (f) Rebinding an existing custom domain to another Worker: in place, or DELETE then PUT, and the gap. (g) `PUT access/apps/` with added destinations keeps the AUD. (h) Whether `op://Toolkit/cf-api-token` holds Workers Routes Edit and Access Apps Edit on the Dwarves account. (i) Whether it edits tunnels and DNS on the personal account. (k) The Workers plan on each account and the route's failure mode at the request limit. AC: each answered, the token table amended to the source each role uses. +- [x] TASK-2a: per-link storage on the origin: config keys, `--cloud`, `--local`, their refusals, `rand_id` across both legs, the pointer write and its delete on `rm` and `prune`. Depends on TASK-1a. AC: rows 2, 4, 5. +- [x] TASK-2b: storage dispatch for `rm`, `refresh`, `hits` with the member refusals, the reconcile, the sweep rule for pointers, and the origin's cloud expiry through its own reader. Depends on TASK-2a. AC: rows 3, 6, 26, 27. +- [x] TASK-3: Worker v3 (`ALIASES`, `PASS`, two-leg `/healthz`, the record branches, the rewrap, the offline page, the method rule, `WORKER_RECORD_V` 2), `r2_healthz`'s header reading, and `tests/worker.mjs` with a stub whose responses hold immutable headers. Depends on TASK-1a. AC: rows 7 to 10, 29. +- [x] TASK-4: `setup --r2` and `--no-r2` without `--alias` (steps 1 to 3, 5, 6, 8, 9, 11, 13, 15) and the rerun convergence. Depends on TASK-2a, TASK-3, TASK-1b(h). AC: rows 11, 12, 13, 32. +- [x] TASK-5: `--alias` (steps 4, 7, 10, 12, 14, 16), the `--no-r2` alias refusal, the printed rollback list. Depends on TASK-4 and TASK-1b(f)(g). AC: rows 14, 30. +- [x] TASK-6: one shared list: the separate bucket reader, merged display rows, the `v:2` reader's checks, `storage`, `type`, `by`, `r2`, `storage_default`, `cloud_error`, `cloud_more`, the listing token rule, the newest-25 fetch in `state`. Depends on TASK-2b. AC: rows 1, 15, 16, 17, 28. +- [x] TASK-7a: `import` and `setup --token-stdin`. Depends on no other task; its first step measures the tar flags on bsdtar and GNU tar (absolute, `..`, symlink, and hardlink members) before the pre-scan is written. AC: row 18. +- [x] TASK-7b: `migrate` (preflight, copy, switch, verify, retire, the printed rollback). Depends on TASK-7a and TASK-1b(i). AC: rows 19, 20, 21, 31. +- [x] TASK-8: Share Bar: decode, `RowGlyphs`, row rendering, the `cloud_error` line, the `Storage` popup. Depends on TASK-6. AC: rows 22 to 25; `swift build` passes. +- [x] TASK-9a: `tests/e2e-tenant.sh` (L1 to L10) and rehearsal R1, with its run log. Depends on TASK-2a to TASK-6. AC: L1 to L10, R1, every cleanup assertion. +- [x] TASK-9b: `tests/e2e-migrate.sh` and rehearsal R2 (the one-machine leg, then the Air to the Mini on a throwaway name), with its run log. Depends on TASK-7b. AC: R2 and its cleanup assertions. +- [x] TASK-10: README, how-it-works, setup (onboarding), ADR-0008, the verification record with the negative controls. Depends on TASK-9a, TASK-9b. AC: each negative control has a red and a green run; each `docs/how-it-works.md` command, flag, and state key named in this spec is found by a `tests/share.sh` grep against `bin/share`. Done except the R2 rehearsal evidence, which lands with TASK-9b; one control (`bucket=` in a plain tunnel config) is not reachable in its spec form, see the verification record, Batch 6. +- [ ] TASK-11a: STOP for Han, then the Dwarves migration D1 to D5. Before D3 the lead posts the R1 log and the exact D3 command and waits for Han's explicit go; no loop may run D3 on its own. Each check is recorded in the verification record. Depends on TASK-10 and a release. AC: the Dwarves items of `## After state`. D6 runs seven days later, after a second go. +- [ ] TASK-11b: STOP for Han, then the personal move P1 to P3. Before P2 the lead posts the R2 log and the exact P2 command and waits for Han's explicit go; no loop may run P2 on its own (P2 deletes `air-share` at its step 5). Depends on TASK-10 and a release; independent of TASK-11a. AC: the personal items of `## After state`. + +## Test plan + +Local rows run in `tests/share.sh`: the r2 rows under `SHARE_R2_DRY=1` (SPEC-007's directory-backed bucket and call log), the migration rows with `SHARE_MIGRATE_SSH` pointing at a second `HOME` on the same machine. `tests/worker.mjs` drives the emitted Worker with an in-memory bucket and a stubbed global `fetch` that records each pass-through and answers from a fixture (200, 404, 530, a thrown error). Swift rows run in `swift test --package-path mac`. + +### Coverage matrix + +| # | Category | Case | Assert | +|---|---|---|---| +| 1 | compat | `origin/main`'s `bin/share` and the branch's, each under its own `HOME` with the same seams: a quick setup and a named dry setup with no `bucket=`, add file, add folder, add live, `--host`, `ls`, `state`, `profiles --json`; then an r2 member profile | every file byte-identical; stdout and stderr identical except the `ls` tag column; `state` equal after deleting the keys `storage`, `type`, `by`, `r2`; the member's `profiles --json` entry gains rows | +| 2 | storage | origin with R2 off: `add f`, `add --local f`, `add --cloud f`; R2 on with default `local`: `add f`, `add --cloud f`, `add 3000`, `add --cloud 3000`, `add --cloud --host x. f`, `add --cloud --local f`; default `cloud`: `add f`, `add --local f` | local, local, refused with the setup line; local plus pointer, cloud, live plus pointer, refused, refused, refused; cloud, local plus pointer; refusals log no PUT and write no row | +| 3 | member | an r2 member: `add f`, `add --local f`, `add 3000`; `rm` and `refresh` of a machine row; `rm` of a cloud row | cloud; refused; refused; both refused before any call (no `DELETE m/` logged); the cloud rm runs SPEC-007's path | +| 4 | pointer | local add with R2 on; the bucket answers 412 for the first candidate | `GET m/` and the prefix scan precede the pointer PUT; the pointer PUT carries `If-None-Match: *`, `v:2`, `storage:"machine"`, no `access=` and no `host=` in `opts`; the first candidate is skipped; the pointer PUT precedes the `pub/` rename (a watcher on `pub/`); a gated add's pointer PUT precedes `POST app` | +| 5 | pointer | the pointer PUT answers 500 three times | exit 1 naming R2; no `pub/`, no row, no app | +| 6 | reconcile | origin `ls` with: a machine record with no row, `LastModified` 11 min ago (one with `by` of another machine too); one 1 min ago; one 11 min ago whose id is in `access-pending`; a local row with no pointer; a local row whose id holds a cloud record; `rm` and `prune` of local rows with and without a token | both old orphan pointers deleted, the young one and the pending one kept; a pointer written for the bare row; the shadow line printed, nothing deleted; rm and prune delete the pointer with a token and leave it without one | +| 7 | Worker | `PASS` set: a cloud record; a machine record; no record; the bucket throws; a non-hex first segment; `/`; an expired cloud record; a cloud record whose prefix names another id | R2 bytes, no pass-through; pass-through x5 (machine, none, throw, non-hex, root); 404 with no pass-through x2 | +| 8 | Worker | `PASS` set, pass-through answers: 200 with a body; 404; 502 (with and without `cf-cache-status`); 530; an edge 503 with no `cf-cache-status`; a thrown fetch; a POST with a body; an `Upgrade: websocket` request | unchanged x4 (200, 404, both 502s; no Analytics Engine data point for any pass-through); 503 offline page with `Retry-After` x3 (530, the edge 503, the thrown fetch); the POST forwarded with its method and body; the upgrade forwarded and its 101 returned unchanged | +| 9 | Worker | `ALIASES=f.test`: GET `https://f.test/ba6377/g/?a=1`, HEAD, POST, `/healthz`; Host `x.test`; `PASS` empty: a miss | 301 to `https:///ba6377/g/?a=1`, 301, 405, 301; 404; 404 with no `fetch` call | +| 10 | Worker | `/healthz` with `PASS` set and the origin stub answering `200 ok`, then 530, then a timeout; with `PASS` empty | 200 `X-Share-Tunnel: 1`; 503 `X-Share-Tunnel: 0` x2; SPEC-007's answer with no stub call. Every row 7 to 10 answer carries `no-store` and `noindex`, and SPEC-007's rows 17 to 19 and 28 pass unchanged on the v3 source | +| 11 | setup | dry `setup --r2` on the origin | log order: script GET < bucket reads < marker GET < routes GET < `m/` list < script PUT < subdomain POST < subdomain GET < pointer PUTs < config write < route POST < healthz; config gains the four keys; the tunnel config and Keychain items unchanged | +| 12 | setup | refusals: on a non-origin; on quick mode; a publisher token (403 on the script read); a route on `/*` naming another script; a local id holding a cloud record; a marker naming a third host | each dies before any PUT, POST, or DELETE in the log | +| 13 | rollback | `--no-r2` after row 11 | route DELETE logged; the four keys gone; the next `add` writes no pointer; `rows()` output equal to before row 11 | +| 14 | alias | dry `--alias f.test` with a gated cloud record (then `rm` of it on the tenant); the alias domain naming another service; the gate probe failing | order: app GET < script PUT < pointer PUTs < app PUT (destinations added, AUD unchanged) < probes < config write < marker PUT `If-Match` < route POST < domain PUT < healthz < app PUT (alias destinations dropped); the other service: dies before any write; probe failure: the added destinations removed, no marker PUT, no route; the `rm` deletes the folded app | +| 15 | list | origin with R2 on: two local rows (one live, one gated), their pointers, two cloud records (one by another machine, one gated), a `v:1` cloud record without `type` named `guide` | `ls` shows six rows with tags `machine`, `live`, `cloud`, `cloud`, their types, and `by=`; no id twice; `state` rows carry `storage`, `type`, `by`; `guide` is `folder` | +| 16 | type | the type table, one fixture per row, plus a folder with `index.html`, a folder without, `Report.PDF`, `a.tar.gz`, no extension on a local file | each `type` as the table; the extension match ignores case | +| 17 | list | `state` with the bucket listing failing; with the token source `api_token_cmd`; `profiles --json` with an r2 member and a tunnel profile | local rows plus `cloud_error` (exit 0); no command run (a sentinel command that touches a file is never run), `cloud_error` names `api-token`; the member entry lists its rows | +| 18 | import | a tar with a regular tree; a symlink; a dotfile; a `../x` member; an absolute member; an id already in the index; opts `live`; opts `host=x`; a name with `/` | the first published with the original dates, `by=`, and `src` `:`; each other refused with no `pub/` and no row | +| 19 | migrate | two `HOME`s (A the old origin, B the target, `SHARE_MIGRATE_SSH` pointing at a wrapper that logs its own argv): A holds a snapshot, a folder, a gated snapshot, a live row | B's index holds the three snapshots with A's ids, names, expiries, and `access=`; B's `pub/` trees equal A's byte for byte; the live row is listed as not moved; B's dry setup ran with `--force --token-stdin` (the token is in no argv the wrapper or a `curl` shim logged); A's rows are in `index.migrated`, its trees in `migrated/`; A's teardown path ran and logged no DELETE of an Access app | +| 20 | migrate | a `--host` row on A; `bucket=` on A; B's profile set up for another hostname; B's `not_setup` profile holding an old row; the copy failing on the second id; the remote setup failing; a gated link answering 200 at verify | refused before any copy x4; stop with the first id on B and A unchanged; A unchanged and the rollback line printed with `--tunnel-name `; stop before retire, the gated link named | +| 21 | migrate | rerun after row 20's copy failure | the first id skipped as already moved; the rest copied | +| 22 | swift | decode `profiles --json` fixtures with and without the new fields; a share with an unknown `type` | `storage`, `type`, `by`, `r2`, `storageDefault`, `cloudError` decoded; absent fields are nil; unknown type maps to `doc` | +| 23 | swift | `RowGlyphs` over every value in the SF Symbols table | the symbol names and words match the table; every symbol resolves with `NSImage(systemSymbolName:)` on the test host | +| 24 | swift | `MenuModel` rows: a gated cloud PDF, a live machine row, an older-CLI row with no `storage` | accessibility titles `, , PDF, in the cloud, snapshot, login required`; `..., site, on this tenant's machine, live server`; no storage word for the old row | +| 25 | swift | `PublishForm` on an origin with `r2: true` and `storage_default: cloud`; on a member; on a tunnel profile with `r2: false` | the `Storage` popup preselects `In the cloud` and the argv carries `--cloud`; a disabled `In the cloud` label and no flag; no popup and no flag | +| 26 | sweep | R2 on: bare `prune` with machine pointers present and an unreferenced upload 25 h old; then a record that is not JSON | the old upload deleted, no warning; with the broken record, nothing deleted and the warning names it | +| 27 | dispatch | on the origin: `rm` of a cloud row, `rm` of a machine row, `refresh` of each, `hits` of each | the cloud ones take the r2 path (S3 calls, the SQL call); the machine ones take the tunnel path plus the pointer DELETE; `hits` of a machine row reads the access log | +| 28 | hostile | on an R2-on origin, and again on a member (`ls`, `state`, `profiles --json` only), forged records, plus a key `m/../share.json`, an expired valid cloud record whose id sorts before an expired local row (one `prune` expires both, cloud first), `rm` and `hits` of a forged cloud id, and one local `add` in the same process: `v:1` with `opts:"live host=x.test"` and `src:"x:22"`; `v:2` machine with `opts:"live"`, `src:"x:22"`, and `expires:"a[$(touch S)]"`; `v:2` with `name` holding a tab; then `ls`, `state`, `profiles --json`, `prune`, `refresh ` | the Caddyfile and the running Caddy config hold exactly the local rows (the expired one gone, the new add present, no bucket row); `index.tsv` holds no bucket row; the file `S` does not exist; the odd key is skipped and `share.json` survives; each forged record is skipped or shown as display text only; `refresh` of the forged ids dies | +| 29 | healthz | dry Worker answer `503` with this CLI's pair and `X-Share-Tunnel: 0`: a member joins, adds a gated cloud link, and runs `state` | the join exits 0; the gated add publishes; `state` has `ready: true`; a `503` with no `X-Share-Worker` header still fails all three | +| 30 | alias | dry `--alias f.test`, die injected at each of steps 12, 13, 14, 15; then `--no-r2`; then the printed rollback list; then a rerun of the fold | each die prints the alias rollback list, not `--no-r2` alone; `--no-r2` refuses while `aliases=` is set; after the list, the integration Worker for the alias serves the cloud record itself; the rerun converges to row 14's end state | +| 31 | rollback | after row 20's remote-setup failure, run the printed line on A in dry mode | the dry DNS PUT names A's own tunnel; A's config keeps its `tunnel_id`; no call touches B's tunnel | +| 32 | refuse | SPEC-007 `setup --backend r2 --bucket ` with the admin fixture against a tenant Worker (`PASS="1"`), with `--force`; against a marker holding `aliases`; against a route on `/*` | each dies before any PUT, POST, or DELETE, naming `setup --r2` on the origin | + +### Negative controls + +Each is a temporary one-line patch that must turn the named rows red, recorded with its red run in `docs/verification/one-host-per-tenant.md`. + +| Patch | Expected red | +|---|---| +| the Worker passes an expired cloud record through instead of 404 | row 7 | +| the Worker passes a miss through with `PASS` empty | row 9 | +| the Worker serves R2 for a `storage:"machine"` record | row 7 | +| drop the encoded-separator check before the pass-through | SPEC-007 row 18 on the v3 source | +| write the pointer after `stage_publish` | row 4 (the watcher sees `pub/` first) | +| let a pointer PUT failure warn instead of die | row 5 | +| take a member's `add --local` as cloud | row 3 | +| attach the route before the pointer backfill | row 11 | +| keep the alias destinations on the app after step 14 | row 14 | +| let `import` accept a symlink | row 18 | +| run the full teardown (with Access deletes) on the old origin before moving rows aside | row 19 | +| read the bucket in `state` through `api_token_cmd` | row 17 | +| write `bucket=` into a plain tunnel config at setup | row 1 | +| merge bucket rows through `rows()` | row 28 (the Caddyfile changes) | +| let SPEC-007's r2 setup take a tenant host with `--force` | row 32 | +| let the sweep count a machine pointer as unreadable | row 26 | +| let a member `rm` a machine row | row 3 | +| read the healthz status alone in `r2_healthz` | row 29 | +| print `--no-r2` as the step-15 rollback with an alias | row 30 | +| omit `--tunnel-name` from the printed migrate rollback | row 31 | + +### Live legs (`tests/e2e-tenant.sh`, by hand before the release) + +Inputs: `SHARE_E2E_TENANT_HOST` and `SHARE_E2E_ALIAS_HOST` (unused names on a zone the admin token edits), `CLOUDFLARE_API_TOKEN` (admin), `SHARE_E2E_R2_PUBLISHER_TOKEN`, `SHARE_E2E_ACCESS_EMAIL`. The bucket is `share-e2e-<6 hex>`, and the script refuses a hostname or bucket without the `share-e2e-` prefix, so `s.d.foundation`, `f.d.foundation`, `s.han.ws`, and every existing bucket are out of reach. Everything runs under mktemp `HOME`s with `SHARE_TEST_PORT_BASE` ports. + +| # | Step | Assert | +|---|---|---| +| L1 | origin A: tunnel setup of the tenant host; a local file, a gated local file, a live server on a test port | all answer as today (the gated one 302 with `kid == aud`) | +| L2 | `setup --r2` on A | `/healthz` 200 with `X-Share-Tunnel: 1` and this CLI's pair; pointers exist for L1's ids; `https://..workers.dev/healthz` does not answer 200 | +| L3 | through the Worker | L1's links answer as before; the gated one still 302s unauthenticated and serves with the service token (SPEC-007 TASK-1(l) method); the live server answers a POST and a WebSocket echo | +| L4 | A `add --cloud`; member B joins and adds | both cloud links 200 with `no-store`; `ls` on A and on B show the same five rows with tags | +| L5 | stop A's tunnel (`share stop`) | cloud links 200; machine and live links 503 offline page; `/healthz` 503 with `X-Share-Tunnel: 0`; a fresh member C joins and B adds a gated cloud link, both succeed, and B's `state` reads `ready: true`; start A, all back | +| L6 | encoded paths `/x/..%2F/f`, `///f`, `/%2F/f` on a machine id and a cloud id | 400 each, from the Worker | +| L7 | `--no-r2` | the route is gone; machine links serve straight from the tunnel; cloud links 404 at Caddy | +| L8 | `setup --r2` again | back to L4's state with no new pointer PUT for existing pointers | +| L9 | B's local teardown; A `rm` of each link | A's `ls` empty; every pointer and record gone; the gated apps 404 | +| L10 | admin purge and A teardown | cleanup below | + +Rehearsals (each before its production run): + +| # | Rehearsal | Assert | +|---|---|---| +| R1 | the Dwarves fold on throwaway names: A is a tunnel origin on ``, C an r2 profile on `` with a gated cloud share; then `setup --r2 --alias ` on A; then the rollback list in `### Dwarves`; then the fold again | after the fold: `https://///` 301 to `https://///`, which 302s with `kid` equal to the original AUD and serves with the service token; after the rollback: the alias serves the share itself and the tenant is a plain tunnel; after the second fold: as after the first | +| R2 | the personal move on throwaway names: A (stand-in for the Air) sets up `` with a snapshot, a folder, a gated snapshot, a live server; `migrate --to` B on the same machine (`SHARE_MIGRATE_SSH`), then once more from the real Air to the Mini over ssh (a second throwaway name) | every snapshot link answers from B (or the Mini) at the same URL; the gated one keeps its 302 and AUD; the live link is reported as not moved; A's tunnel is deleted, its DNS record points at B's tunnel; the measured 530 gap is logged. Before that, one run injects a remote-setup failure and runs the printed rollback: the hostname serves from A's tunnel again | + +Cleanup is asserted, then enforced, as SPEC-007's live leg does: after L10 and each rehearsal the script checks that no route, Worker, custom domain, DNS record, tunnel, bucket, Access app, or minted token with the throwaway names remains, and its EXIT trap runs the purges and a by-name fallback whatever happened before, printing each leftover. + +## Grounding + +Read-only probes, 2026-10-01, from the Mini, through a helper that sends the Toolkit token as `-H @<(printf ...)` and prints names, codes, and counts only: + +| Claim | Command (shape) | Excerpt | +|---|---|---| +| profiles on the Mini | `share profiles`; `share --profile

folder

" >"$WORK/src/folder/index.html" +echo "gated secret" >"$WORK/src/gated.txt" +mkdir -p "$WORK/live"; echo "live page" >"$WORK/live/index.html" +python3 -m http.server "$LPORT" --bind 127.0.0.1 --directory "$WORK/live" >/dev/null 2>&1 & PIDS+=("$!") +for _ in $(seq 1 50); do curl -s -o /dev/null "http://127.0.0.1:$LPORT/" && break; sleep 0.1; done +l1="$(mg_a add "$WORK/src/one.txt" 2>/dev/null | head -1)" +l2="$(mg_a add "$WORK/src/folder" 2>/dev/null | head -1)" +l4="$(mg_a add "$LPORT" 2>/dev/null | head -1)" +l3="" +if [[ -n $rule ]]; then + l3="$(mg_a add "$WORK/src/gated.txt" --access "$rule" 2>"$WORK/gate.err" | head -1)"; indent <"$WORK/gate.err" + g_id="$(cut -d/ -f4 <<<"$l3")" + g_aud="$(api "/accounts/$acct/access/apps?per_page=100" | jq -r --arg s "share $g_id $tenant " '[.result[]? | select(.name | startswith($s))] | first | .aud // empty')" +else + echo "gated leg: SKIP (no SHARE_E2E_ACCESS_RULE)" +fi +check "the snapshot link answers from A" "snapshot one" "$(fetch "$l1")" +check "the folder link answers from A" 1 "$(fetch "$l2" | grep -c '

folder

')" +check "the live link answers from A" 1 "$(fetch "$l4" | grep -c 'live page')" +[[ -z $l3 ]] || check "the gated link 302s to Access with kid == aud" 1 "$([[ "$(fetch -o /dev/null -w '%{http_code} %{redirect_url}' --path-as-is "$l3")" == 302*"kid=$g_aud"* ]] && echo 1 || echo 0)" + +verify_moved() { # verify_moved

state \| jq` | `default not_setup -`, `dfoundation serving s.d.foundation`, `files serving f.d.foundation`; one gated snapshot `a68960` on `dfoundation`, one gated cloud share `ba6377` on `files` | +| config keys | `grep '^(backend\|hostname\|hosts\|port\|bucket\|tunnel_name)='` | `files`: `backend=r2`, `bucket=share-dfoundation`, `port=r2`; `dfoundation`: `tunnel_name=share-s-d-foundation`, `hosts=Mac-mini`, `port=8791` | +| hostnames live | `curl -w %{http_code} https:///healthz` | `s.d.foundation 200`, `f.d.foundation 200`, `s.han.ws 200` | +| the guide link | `curl -w '%{http_code} %{redirect_url}'` | `302 https://dwarves.cloudflareaccess.com/cdn-cgi/access/login/f.d.foundation?kid=<64 hex>...` | +| DNS | `GET zones//dns_records?name=` | `s.d.foundation CNAME proxied tunnel`, `f.d.foundation AAAA proxied` (the Worker custom domain), `s.han.ws CNAME proxied tunnel` | +| Workers on the zones | `GET zones//workers/routes`, `GET accounts//workers/domains` | `chat.d.foundation/dw-chat/* -> df-chat-ui`, `radar.han.ws/* -> price-radar`; `f.d.foundation -> share-f-d-foundation` | +| tunnels | `GET accounts//cfd_tunnel?is_deleted=false` | `share-s-d-foundation healthy, 4 connections`; `air-share healthy, 4 connections` | +| Access apps | `GET accounts//access/apps` filtered on `share ` | Dwarves: `f.d.foundation/ba6377{,/*}`, `s.d.foundation/a68960{,/*}`; personal: none | +| personal account | `GET accounts//r2/buckets`, `POST .../analytics_engine/sql` | `200`, no bucket named `share*`; `200` | +| zone plans | `GET zones?name=` | both `Free Website`; Workers paid status not probed (SPEC-007 cost section assumed the Dwarves paid plan) | + +Not sampled (TASK-1a, TASK-1b, TASK-7a): the route pass-through, Access order against a route Worker, the subrequest under Access, WebSocket pass-through, the down-origin status codes, the custom-domain rebind, the AUD after a destination edit, the Toolkit token's route, Access, and tunnel scopes, tar member handling. + +Dry trace for one negative control: "write the pointer after `stage_publish`" moves the `r2_call PUT m/` line below the `mv` in the local add; row 4's background watcher records the first time `pub/` exists and compares it with the line number of `PUT m/` in `r2-calls.log` (the watcher appends a `SEEN pub/` line to the same log), so the order flips and the row fails mechanically. + +## Verification + +```sh +shellcheck bin/share install.sh tests/share.sh tests/e2e.sh tests/e2e-r2.sh tests/e2e-tenant.sh tests/e2e-migrate.sh demo/render.sh mac/*.sh +/bin/bash -n bin/share +bash tests/share.sh # includes tests/worker.mjs +swift test --package-path mac +bash mac/build.sh 0.0.0 +``` + +Then by hand: `tests/e2e-tenant.sh` (L1 to L10, R1) and `tests/e2e-migrate.sh` (R2); logs go to `docs/verification/one-host-per-tenant.md`. + +## After state + +- [ ] `https://f.d.foundation/ba6377/support-ticket-guide/` answers 301 to the same path on `s.d.foundation`, which keeps its Access gate. +- [ ] `share --profile dfoundation ls` on the Mini lists `a68960` (`machine`) and `ba6377` (`cloud`) in one list; the `files` profile is gone. +- [ ] `share --profile dfoundation add ./x` publishes from the Mini's disk; `add --cloud ./x` publishes to the bucket; both at `https://s.d.foundation//...`. +- [ ] With the Mini's `dfoundation` service stopped, the cloud link answers and the machine link answers the 503 offline page. +- [ ] `s.han.ws` serves from the Mini; every Air snapshot link answers at its old URL; no Worker or bucket exists for `han.ws` share. +- [ ] Share Bar on the Mini shows two sections, each row with its type icon, storage badge, and link-type marker, and the lock on gated rows. + +## Acceptance Criteria (global) + +1. A tenant with R2 off runs today's code paths: no Worker, no bucket call, files byte-identical (row 1). +2. With R2 on, a cloud record is never passed to the tunnel, a machine link is never served from R2, and an expired or malformed cloud record answers 404 (rows 7, 8). +3. A gated link stays gated on both legs: the edge app on every path, plus the Worker's JWT check for cloud records (rows 10, 14; L3; R1). +4. Every existing link keeps its URL or answers a 301 to it, through both migrations (R1, R2; `## After state`). +5. No token appears in argv, output, a file, a binding, or an object (rows 19, SPEC-007 row 16 on the new paths). +6. Every migration step has a rollback that the rehearsal ran. + +## Out of scope + +Moving a tunnel tenant's local shares into its bucket (a `--cloud` re-add covers one link); a member publishing a local link through the origin over ssh; more than one origin per tenant; `--host` and live links in R2; enabling R2 on `s.han.ws` now; a CSP sandbox on shared HTML (SPEC-007 Decisions item 6); per-link storage changes after publish. + +## Decisions for Han + +1. Personal SSH target and user for P2: the spec assumes `tieubao@` (the user that runs `dfoundation` today), so both tenants live under one account on the Mini. Proposed: accept. +2. The Mini's `s.han.ws` lands in its `default` profile (port 8787, Keychain item `share-tunnel:s.han.ws`). Proposed: accept; a named `personal` profile works the same if you prefer explicit names on the Mini. + +## Decision Log + +- DEC-001 (operator, 2026-10-01): one hostname and one shared list per tenant; storage is a per-link indicator, never a subdomain. +- DEC-002 (operator, 2026-10-01): local storage is the default; R2 is optional per tenant, off until the admin enables it, then per link (`--cloud`, `--local`) with an admin-set tenant default (`storage_default`). +- DEC-003 (operator): the Mini is the single origin for both tenants, because it is the only always-on machine; the Air's sleep is the cause of today's 530s on `s.han.ws`. +- DEC-004: the Worker reaches the tunnel by route pass-through on the same hostname; no internal hostname, no service token, and the rollback is one route delete. +- DEC-005: R2 first, then the tunnel; a cloud id never falls back to the origin. +- DEC-006: pointer records exist only with R2 on; they make the list shared on every member and the namespace collision-proof. A local add fails closed when R2 is on and the pointer cannot be written. The cost, stated plainly: on an R2-on tenant, local publishing depends on R2 answering and on a stored publisher token (Share Bar and the CLI alike); the gain over failing open is that no member's cloud add can take an id the origin holds, a 1 in 16.7 million chance per add, plus a list that is never missing a machine link. Accepted because the tenant admin turned R2 on deliberately; an R2-off tenant keeps today's independence. +- DEC-007: cloud records stay `v:1` (plus an optional `type`), pointers are `v:2`, so the old Worker remains a valid rollback for every cloud link. +- DEC-008: `f.d.foundation` becomes an alias the tenant Worker answers with 301; the bucket `share-dfoundation` is adopted in place, so `ba6377` keeps its record, bytes, and Access app AUD. +- DEC-009: `share migrate` moves an origin over ssh with the receiver's `import`, keeping ids, names, dates, expiries, and gates; nothing on the old origin is deleted, only moved aside. +- DEC-010: the new origin gets its own tunnel; reusing the old one would leave its id in the old machine's config, where a later teardown deletes the new origin's tunnel. +- DEC-011: bucket rows are display data only; the origin's index, Caddyfile, and stage paths never read a record another publisher could write. +- DEC-012 (lead, build batch 3): a member's `hits` keeps SPEC-007's rule of one account call and no bucket read, so it never refuses a machine row. It must not print a bare 0 for a link it cannot count: when the count is 0 and the id is not one of this install's own cloud adds (`r2-own`), it prints the count and then `this machine counts cloud links only; if is a machine link, its stats are on the tenant's origin: hits there`. This replaces the round-2 build rule that a member refuses a machine row. +- DEC-013 (lead, review fix pass): the Worker passes a 502 through unchanged instead of answering the offline page. TASK-1a(e) measured that a pass-through cannot tell a dead live port (Caddy answers 502 for the closed port, and Cloudflare swaps it for its own page) from Caddy itself being down: both arrive as the same 502. The offline page said "the machine serving this link is offline" for a machine that was up with one dead dev server. Cloudflare's signals for an unreachable origin (530, 520 to 527) still map to the offline page. The cost: with cloudflared up and Caddy itself dead, a visitor sees Cloudflare's 502 page, not the offline page. `WORKER_VERSION` is 4 for it. +- Round 1 (seven fresh-context reviewers, 2026-10-01): eleven criticals, all folded. Warnings that do not change the design went to `docs/implementation-notes/one-host-per-tenant.md` for the builder. + +| Change | Why (reviewer) | +|---|---| +| Bucket rows never enter `rows()`, the index, the Caddyfile, `refresh`, or a stage; the `v:2` reader checks every field before shell arithmetic; pointers carry `gated`, not the rule; row 28 | a forged record could add a reverse proxy to a local port or run a command through `expires` (1) | +| `setup --r2` writes its config before the marker; `--no-r2` refuses while an alias is set; the step-15 die prints the alias rollback list; reruns converge; row 30 | a die after the rebind plus the printed `--no-r2` would leave the guide link 404 for good (2) | +| A member's `rm` and `refresh` of a machine row refuse; `rm`, `refresh`, `hits` dispatch on storage; rows 3, 27 | a member saw "unpublished" while the origin kept serving the file (2) | +| The migrate rollback names `--tunnel-name `; row 31; R2 runs it | `setup` derives the new origin's tunnel name, so the printed line joined the old origin to the new tunnel (3) | +| `/healthz` 503 with a valid Worker pair counts as Worker up for joins, gated member adds, and r2 `state`; row 29; L5 | members could not join or publish while the origin was off, the case R2 exists for (3) | +| TASK-1 split into an edge spike and an account spike, the tar check moved into TASK-7a; TASK-7, TASK-9, TASK-11 split; TASK-11a and TASK-11b start with an explicit stop for Han | task size, and no human gate before the production migrations (4) | +| The orphan sweep reads a `v:2` machine pointer as readable and referencing nothing; row 26 | every pointer counted as unreadable, so the sweep would never run again on any machine (5) | +| Also fixed because the text was false or unsafe: `profiles --json` already lists r2 rows; the poll cost (newest 25 records, about $0.60 a month); reconcile keyed on `storage` and aged by `LastModified`; `migrate` refuses an R2-on tenant, runs a fixed `/bin/bash -c`, takes the token through `setup --token-stdin`, pre-scans the tar, validates `added`, `expires`, `src`, preflights the target, and verifies through a target-only nonce and the gated AUD; the real 530 bound; D2 after D3, D5 with D3, D4's rollback only after D3's; the Worker record branches, its 2 s read cap, and the response rewrap | reviewer warnings with a concrete wrong statement or unsafe default behind each (1, 2, 3, 5) | +| Not taken: reusing `air-share` (DEC-010); failing open on a pointer write (DEC-006); a redirect rule in place of `--alias` (the 301 stays in tested code; the alias surface is permanent but small, and it retires with `--no-r2` after the alias rollback) | (5) | +- Round 2 (five fresh-context reviewers re-ran the lenses that raised round-1 criticals, 2026-10-01): every round-1 critical cleared; two new criticals, both folded. + +| Change | Why (reviewer) | +|---|---| +| SPEC-007's r2 setup refuses a tenant host in its admin role, even with `--force`; row 32 | an admin token on a member would redeploy the Worker without `PASS` and swap the tunnel CNAME for a custom domain, and the new healthz reading would call that ready (3) | +| TASK-2 split into TASK-2a (storage and pointers) and TASK-2b (dispatch, reconcile, sweep, origin cloud expiry) | seven AC rows in one task (4) | +| Also folded: a bucket reader on the origin that never rebinds `index`; the 6-hex key check; row 28 with a same-process add and a member leg; `by` through `r2_by`; the origin prunes cloud rows; `api-token` and bucket calls keyed on `bucket=`; Analytics Read for the origin; the newest 25 skip local ids and report `cloud_more`; folded apps deletable by alias name; a base64 ssh script and a seam that parses like a login shell; no hit counted on a pass-through; the TASK-11b loop stop | warnings with a concrete wrong outcome behind each (1, 2, 3, 4, 5) | +- Round 3 (three fresh-context reviewers on the round-2 diff, the cap, 2026-10-01): every round-2 critical cleared. One new critical, folded without a fourth review because of the cap: the SPEC-007 cloud verbs on a tunnel profile (`cmd_rm_r2`, `r2_prune`) still rebound or overwrote `$index`, so a `prune` that expired a cloud row before a local row could render a forged `v:1` record into the Caddyfile. The reader rule now covers every cloud verb on a tunnel profile, and row 28 gains the cloud-then-local expiry, `rm`, and `hits` legs. Remaining warnings, binding on the build: + +| Warning (reviewer) | Build rule | Task | +|---|---|---| +| A v0.8.0 CLI with the admin token passes the `v:1` tenant marker and its version die suggests `--force` (3) | D1 checks that every machine holding the Dwarves admin token runs the release; the tenant marker gains a field v0.8.0's setup refuses only if TASK-4 finds one that does not break v0.8.0 joins | TASK-4, TASK-11a | +| `access_sweep` decides stale lines by `$backend`, so on the origin a crashed gated `--cloud` add's line is checked against the local index and the sweep deletes the app of a published gated cloud link (3) | the sweep decides per id: a line whose id holds a cloud record (fresh `GET m/`) takes SPEC-007's r2 rule | TASK-2b | +| Members have no `aliases=`, so a member's `rm` or expiry of a folded gated share cannot delete its app (1, 3) | a member stores `aliases=` at join, read once from the marker and confirmed against the Worker's `ALIASES` binding through the alias's 301; never read live from the marker (any publisher can write it); row 14 gains a member leg | TASK-5 | +| The SPEC-007 refusal's route read may answer 403 for SPEC-007's minimal admin scopes (1) | a non-200 on the Worker settings read or the marker read dies; a 403 on the routes read alone is tolerated because the `PASS` binding check already catches a tenant; row 32 gains that leg | TASK-4 | +| The ssh template: GNU `base64` wraps lines, `"$(...)"` needs fish 3.4 or later, the "no quote" wording contradicts the template, and a runner without fish exercises sh only (1, 3) | encode with `| tr -d '\n'`; the preflight names the minimum fish; the fixed template is the only quoting and arguments add none; CI installs fish for the seam | TASK-7b | +| `api-token` keyed on `bucket=` takes the r2 form and reports Access scopes as optional (3) | on a profile with a tunnel, the Access scopes stay required as SPEC-004 has them, and the form names both the bucket and Access permissions | TASK-2a | +| `cloud_more` and the newest 25 count orphan pointers and unreadable records (3) | display only; the count says "about" in the menu line | TASK-6, TASK-8 | +| TASK-1a and TASK-1b run long as lettered probes; TASK-4, TASK-8, TASK-10 sit at the five-file or five-row edge (4) | accepted: each letter is one probe; split TASK-8's tests if they exceed one file per source | all | diff --git a/docs/verification/e2e-migrate-20261005T093825Z.log b/docs/verification/e2e-migrate-20261005T093825Z.log new file mode 100644 index 0000000..a361c3b --- /dev/null +++ b/docs/verification/e2e-migrate-20261005T093825Z.log @@ -0,0 +1,118 @@ +=== before: nothing named for this run exists === + ok no DNS record for share-e2e-84ccc9.d.foundation + ok no tunnel for share-e2e-84ccc9.d.foundation + ok no DNS record for share-e2e-3abe33.d.foundation + ok no tunnel for share-e2e-3abe33.d.foundation +=== L0 origin A: tunnel setup; a snapshot, a folder, a gated snapshot, a live server === + | auth: API token (active) + | zone: d.foundation + | tunnel: created share-share-e2e-84ccc9-d-foundation + | route: share-e2e-84ccc9.d.foundation -> 127.0.0.1:20423 + | dns: created CNAME share-e2e-84ccc9.d.foundation + | token: stored in keychain (service share-tunnel.mga:share-e2e-84ccc9.d.foundation) + | service: runs at login (foundation.d.share.mga) + | ready: https://share-e2e-84ccc9.d.foundation/ is live. Next: share --profile mga add + ok A setup exits 0 + | A's tunnel 9bf68be8... + | share: checking the API token for profile mga (zone d.foundation) + | ok token found (CLOUDFLARE_API_TOKEN) + | ok Zone: Read GET /zones?name=d.foundation -> 1 zone + | ok Access: Organizations, Identity Providers, and Groups Read GET /accounts//access/organizations -> 200 + | ok Access: Apps and Policies Edit POST /accounts//access/apps {} -> 400 (12130, nothing created) + | share: waiting up to 900s for Cloudflare Access to enforce on share-e2e-84ccc9.d.foundation/8c22f0 (a new app can take minutes; Ctrl-C is safe: nothing is published, the next add or prune removes the app) + ok the snapshot link answers from A + ok the folder link answers from A + ok the live link answers from A + ok the gated link 302s to Access with kid == aud +=== M0 a remote-setup failure after the switch, then the printed rollback === + | auth: API token (active) + | preflight: ok (local-b, not_setup) + | copy: 1bed74 -> local-b (imported 1bed74) + | copy: 269bea -> local-b (imported 269bea) + | copy: 8c22f0 -> local-b (imported 8c22f0) + | not moved: live link 17a5c2 (port 20463); after the move run 'share --profile mga add 20463' on the new origin + | share: migrate: the switch on local-b failed; this machine may still be serving, or the DNS record may already point at the new tunnel. Rollback if share-e2e-84ccc9.d.foundation is down: share --profile mga setup share-e2e-84ccc9.d.foundation --tunnel-name share-share-e2e-84ccc9-d-foundation --force + ok M0: migrate exits 1 on the injected failure + ok M0: the printed rollback names A's own tunnel name + ok M0: the switch moved the DNS record off A's tunnel + ok M0: A kept its rows and its trees (nothing moved aside) + | running the printed rollback: share --profile mga setup share-e2e-84ccc9.d.foundation --tunnel-name share-share-e2e-84ccc9-d-foundation --force + | auth: API token (active) + | zone: d.foundation + | tunnel: reusing share-share-e2e-84ccc9-d-foundation + | route: share-e2e-84ccc9.d.foundation -> 127.0.0.1:20423 + | dns: replaced CNAME share-e2e-84ccc9.d.foundation (--force) + | token: stored in keychain (service share-tunnel.mga:share-e2e-84ccc9.d.foundation) + | service: runs at login (foundation.d.share.mga) + | ready: https://share-e2e-84ccc9.d.foundation/ is live. Next: share --profile mga add + ok M0: the printed rollback exits 0 + ok M0: the DNS record points at A's own tunnel again + ok M0 after the rollback: the snapshot link answers at the same URL + ok M0 after the rollback: the folder link answers at the same URL + ok M0 after the rollback: the gated link keeps its 302 and AUD +=== M1 migrate again: B already holds the copies; the switch, the verify, the retire === + | auth: API token (active) + | preflight: ok (local-b, serving) + | copy: 1bed74 already on local-b, skipping + | copy: 269bea already on local-b, skipping + | copy: 8c22f0 already on local-b, skipping + | not moved: live link 17a5c2 (port 20463); after the move run 'share --profile mga add 20463' on the new origin + | switch: local-b is the new origin for share-e2e-84ccc9.d.foundation + | verify: every moved link answers from local-b + | dns: share-e2e-84ccc9.d.foundation no longer points at this tunnel, left alone + | tunnel: deleted + | retire: share-e2e-84ccc9.d.foundation no longer serves from this machine; moved rows in index.migrated, trees in migrated/ + ok M1: migrate exits 0 + ok M1: a rerun skips the copies B already holds + ok M1: the live link is reported as not moved + ok M1: the snapshot link answers at the same URL + ok M1: the folder link answers at the same URL + ok M1: the gated link keeps its 302 and AUD + ok M1: the DNS record points at B's own tunnel, not A's + ok M1: A's tunnel is deleted + ok M1: A's rows moved aside, its trees under migrated/ + | measured gap on https://share-e2e-84ccc9.d.foundation/1bed74/one.txt: 0 probes (every 0.5 s) answered other than 200, spanning 0 s of 37 probes; codes: + | B's profile on this machine: serving null + ok M1 cleanup: the tenant's DNS and tunnels are gone after B's teardown +=== M2 a second throwaway name moved from air to this machine over ssh === + ok M2: the snapshot link answers from the stand-in before the move + | AIR setup: auth: API token (active) + | AIR setup: zone: d.foundation + | AIR setup: tunnel: created share-share-e2e-3abe33-d-foundation + | AIR setup: route: share-e2e-3abe33.d.foundation -> 127.0.0.1:20483 + | AIR setup: dns: created CNAME share-e2e-3abe33.d.foundation + | AIR setup: token: stored in keychain (service share-tunnel.mga:share-e2e-3abe33.d.foundation) + | AIR setup: service: runs at login (foundation.d.share.mga) + | AIR setup: ready: https://share-e2e-3abe33.d.foundation/ is live. Next: share --profile mga add + | LINK1 https://share-e2e-3abe33.d.foundation/3b45a3/one.txt + | LINK2 https://share-e2e-3abe33.d.foundation/83a818/folder/ + | LINK4 https://share-e2e-3abe33.d.foundation/3e7ae2/ + | LINK3 https://share-e2e-3abe33.d.foundation/a6f9af/gated.txt + | READY + | AIR migrate: auth: API token (active) + | AIR migrate: preflight: ok (mini-tieubao, not_setup) + | AIR migrate: copy: 3b45a3 -> mini-tieubao (imported 3b45a3) + | AIR migrate: copy: 83a818 -> mini-tieubao (imported 83a818) + | AIR migrate: copy: a6f9af -> mini-tieubao (imported a6f9af) + | AIR migrate: not moved: live link 3e7ae2 (port 20503); after the move run 'share --profile mga add 20503' on the new origin + | AIR migrate: switch: mini-tieubao is the new origin for share-e2e-3abe33.d.foundation + | AIR migrate: verify: every moved link answers from mini-tieubao + | AIR migrate: dns: share-e2e-3abe33.d.foundation no longer points at this tunnel, left alone + | AIR migrate: tunnel: deleted + | AIR migrate: retire: share-e2e-3abe33.d.foundation no longer serves from this machine; moved rows in index.migrated, trees in migrated/ + | MIGRATE-RC 0 + ok M2: the stand-in's script exits 0 + ok M2: migrate exits 0 on the stand-in + ok M2: the live link is reported as not moved + ok M2: the snapshot link answers from this machine at the same URL + ok M2: the folder link answers at the same URL + ok M2: the gated link keeps its 302 and AUD + ok M2: the DNS record points at a tunnel of this machine's profile + ok M2: only one tunnel remains for the name (the stand-in's is deleted) + | measured gap on https://share-e2e-3abe33.d.foundation/3b45a3/one.txt: 0 probes answered other than 200, spanning 0 s; codes: + ok M2 cleanup: the name's DNS is gone after B's teardown + +38/38 checks passed +PASS +tests/e2e-migrate.sh: line 151: 34136 Terminated: 15 python3 -m http.server "$LPORT" --bind 127.0.0.1 --directory "$WORK/live" > /dev/null 2>&1 +cleanup: nothing left for this run's names diff --git a/docs/verification/e2e-tenant-20261005T042854Z.log b/docs/verification/e2e-tenant-20261005T042854Z.log new file mode 100644 index 0000000..62f374e --- /dev/null +++ b/docs/verification/e2e-tenant-20261005T042854Z.log @@ -0,0 +1,190 @@ +=== before: nothing named for this run exists === + ok no Worker for share-e2e-95cdb6.d.foundation + ok no Worker for share-e2e-61fdd2.d.foundation + ok no DNS record for share-e2e-95cdb6.d.foundation + ok no DNS record for share-e2e-61fdd2.d.foundation +=== L1 origin A: tunnel setup; a local file, a gated local file, a live server === + | auth: API token (active) + | zone: d.foundation + | tunnel: created share-share-e2e-95cdb6-d-foundation + | route: share-e2e-95cdb6.d.foundation -> 127.0.0.1:8797 + | dns: created CNAME share-e2e-95cdb6.d.foundation + | token: stored in keychain (service share-tunnel:share-e2e-95cdb6.d.foundation) + | service: runs at login (foundation.d.share) + | ready: https://share-e2e-95cdb6.d.foundation/ is live. Next: share add + ok A setup exits 0 + | minted token a-origin (id 1ae828ec..., expires 2026-10-05T06:29:16Z) + ok machine link answers + ok dotfile not served + ok live link answers (plain HTTP through the tunnel) + | share: checking the API token for profile default (zone d.foundation) + | ok token found (api_token_cmd) + | ok Zone: Read GET /zones?name=d.foundation -> 1 zone + | ok Access: Organizations, Identity Providers, and Groups Read GET /accounts//access/organizations -> 200 + | ok Access: Apps and Policies Edit POST /accounts//access/apps {} -> 400 (12130, nothing created) + | share: waiting up to 900s for Cloudflare Access to enforce on share-e2e-95cdb6.d.foundation/80e57b (a new app can take minutes; Ctrl-C is safe: nothing is published, the next add or prune removes the app) + ok gated add printed a link + ok no 200 on the gated link during the wait + ok gated local link 302s to Access with kid == aud +=== L2 setup --r2 on A === + | deploying as admin + | worker: share-share-e2e-95cdb6-d-foundation 4 2209e2a60db0 deployed (passes misses to the tunnel) + | pointers: 3 written + | route: share-e2e-95cdb6.d.foundation/* -> share-share-e2e-95cdb6-d-foundation + | ready: R2 is on for https://share-e2e-95cdb6.d.foundation/ (bucket share-e2e-8f3cb6, storage default local). Store this origin's publisher token: share api-token + ok setup --r2 exits 0 + | token lists bucket share-e2e-8f3cb6 after 0s + ok /healthz carries X-Share-Tunnel: 1 + ok /healthz carries this CLI's pair + ok pointer m/3d7925 exists after setup --r2 + ok pointer m/58fd8c exists after setup --r2 + ok workers.dev does not answer 200 +=== L3 through the Worker === + ok machine link unchanged through the Worker + ok gated link still 302s unauthenticated + | service-token fetch (SPEC-007 TASK-1(l)) not exercised: no SHARE_E2E_ACCESS_SERVICE_TOKEN_ID/SECRET input + ok live link answers a POST + ok live link answers a WebSocket echo +=== L4 A add --cloud; member B joins and adds === + ok A's cloud link answers + ok A's cloud link carries no-store + | minted token b-member (id ba27aa9e..., expires 2026-10-05T06:31:02Z) + | token lists bucket share-e2e-8f3cb6 after 2s + | joining as publisher + | public-route check skipped (publisher token) + | ready: https://share-e2e-95cdb6.d.foundation/ serves from bucket share-e2e-8f3cb6. Store your publisher token: share api-token; then: share add + ok B join exits 0 + ok B join says joining as publisher + ok B's cloud link answers + | A ls: 10 line(s); B ls: 10 line(s) + ok A's ls shows B's cloud row (TASK-6 merged list) + ok B's ls shows A's rows (TASK-6 merged list) +=== L5 stop A's tunnel === + ok cloud link still answers while the tunnel is down + ok machine link answers the 503 offline page + ok live link answers the 503 offline page + ok /healthz answers X-Share-Tunnel: 0 with the tunnel down + | minted token d-fresh (id 167fa5db..., expires 2026-10-05T06:31:34Z) + | token lists bucket share-e2e-8f3cb6 after 2s + | joining as publisher + | public-route check skipped (publisher token) + | ready: https://share-e2e-95cdb6.d.foundation/ serves from bucket share-e2e-8f3cb6. Store your publisher token: share api-token; then: share add + ok fresh member D joins during the outage + | share: checking the API token for profile default (zone d.foundation) + | ok token found (api_token_cmd) + | ok Zone: Read GET /zones?name=d.foundation -> 1 zone + | ok Access: Organizations, Identity Providers, and Groups Read GET /accounts//access/organizations -> 200 + | ok Access: Apps and Policies Edit POST /accounts//access/apps {} -> 400 (12130, nothing created) + | share: waiting up to 900s for Cloudflare Access to enforce on share-e2e-95cdb6.d.foundation/e813a8 (a new app can take minutes; Ctrl-C is safe: nothing is published, the next add or prune removes the app) + ok D's gated cloud add succeeds during the outage + ok B's state reads ready: true during the outage + ok machine link back after start + ok live link back after start +=== L6 encoded paths (a machine id and a cloud id) === + ok /x/..%2F3d7925/doc/ answers 400 + ok /%2F3d7925/doc/ answers 400 + ok //3d7925/doc/ answers 400 + ok /x/..%2F04bcd3/doc/ answers 400 + ok /%2F04bcd3/doc/ answers 400 + ok //04bcd3/doc/ answers 400 +=== L7 --no-r2 === + | route: share-e2e-95cdb6.d.foundation/* removed + | ready: R2 is off for https://share-e2e-95cdb6.d.foundation/: the tunnel serves it alone. The bucket, its records, and Worker share-share-e2e-95cdb6-d-foundation stay + ok --no-r2 exits 0 + ok the route is gone + ok machine link serves straight from the tunnel (no X-Share-Worker) + ok cloud link 404s at Caddy (no route left to the Worker) +=== L8 setup --r2 again === + | deploying as admin + | worker: share-share-e2e-95cdb6-d-foundation 4 2209e2a60db0 already deployed + | pointers: 0 written + | route: share-e2e-95cdb6.d.foundation/* -> share-share-e2e-95cdb6-d-foundation + | ready: R2 is on for https://share-e2e-95cdb6.d.foundation/ (bucket share-e2e-8f3cb6, storage default local). Store this origin's publisher token: share api-token + ok rerun --r2 exits 0 + ok no new pointer PUT for an existing id (etag unchanged) + ok cloud link answers again +=== L9 B's local teardown; A rm of each link === + ok B's config is gone + ok the bucket is unaffected by B's local teardown + ok A's ls is empty + ok machine link gone + ok cloud link gone + ok gated app gone +=== L10 admin purge and A teardown === + | dns: deleted share-e2e-95cdb6.d.foundation + | tunnel: deleted + | token: removed + | service: removed + | config: moved to the Trash + ok A teardown exits 0 + ok tunnel DNS gone + ok bucket gone + ok Worker gone + +=== R1 Dwarves alias-fold rehearsal on throwaway names === + | deploying as admin + | worker: share-share-e2e-61fdd2-d-foundation 4 2209e2a60db0 deployed + | ready: https://share-e2e-61fdd2.d.foundation/ serves from bucket share-e2e-9bf46a. Next: share add + ok R1: C's standalone r2 origin on the alias exits 0 + | minted token c-admin (id 44a78c71..., expires 2026-10-05T06:34:11Z) + | token lists bucket share-e2e-9bf46a after 2s + | share: checking the API token for profile default (zone d.foundation) + | ok token found (api_token_cmd) + | ok Zone: Read GET /zones?name=d.foundation -> 1 zone + | ok Access: Organizations, Identity Providers, and Groups Read GET /accounts//access/organizations -> 200 + | ok Access: Apps and Policies Edit POST /accounts//access/apps {} -> 400 (12130, nothing created) + | share: waiting up to 900s for Cloudflare Access to enforce on share-e2e-61fdd2.d.foundation/9dc4fd (a new app can take minutes; Ctrl-C is safe: nothing is published, the next add or prune removes the app) + ok R1: C's gated cloud link printed + | auth: API token (active) + | zone: d.foundation + | tunnel: created share-share-e2e-95cdb6-d-foundation + | route: share-e2e-95cdb6.d.foundation -> 127.0.0.1:8798 + | dns: created CNAME share-e2e-95cdb6.d.foundation + | token: stored in keychain (service share-tunnel:share-e2e-95cdb6.d.foundation) + | service: runs at login (foundation.d.share) + | ready: https://share-e2e-95cdb6.d.foundation/ is live. Next: share add + ok R1: A2's plain tunnel on the tenant exits 0 + | deploying as admin + | worker: share-share-e2e-95cdb6-d-foundation 4 2209e2a60db0 deployed (passes misses to the tunnel) + | pointers: 0 written + | share: waiting up to 900s for Cloudflare Access to enforce on share-e2e-95cdb6.d.foundation/9dc4fd (a new app can take minutes; Ctrl-C is safe: nothing is published, the next add or prune removes the app) + | access: 1 gated link(s) also gated at share-e2e-95cdb6.d.foundation + | route: share-e2e-95cdb6.d.foundation/* -> share-share-e2e-95cdb6-d-foundation + | alias: share-e2e-61fdd2.d.foundation -> share-share-e2e-95cdb6-d-foundation (answers 301 to share-e2e-95cdb6.d.foundation) + | ready: R2 is on for https://share-e2e-95cdb6.d.foundation/ (bucket share-e2e-9bf46a, storage default local, share-e2e-61fdd2.d.foundation redirects here). Store this origin's publisher token: share api-token + | share: nothing was published; the Access app of 9dc4fd waits in access-pending, the next share add --access or share prune with a token removes it + ok R1: the fold (setup --r2 --alias) exits 0 + ok R1 after fold: alias 301s to the tenant + ok R1 after fold: the folded gated link 302s on the tenant with kid == aud +--- R1 rollback (### Dwarves order, the admin token) --- + 1. restored share-e2e-61fdd2.d.foundation destinations on app 8976d556-1c6a-4f7d-808c-c7989b6429a5 + 2. rebound share-e2e-61fdd2.d.foundation to share-share-e2e-61fdd2-d-foundation + 3. restored share.json host to share-e2e-61fdd2.d.foundation in bucket share-e2e-9bf46a + 4. removed the aliases= line from A2's config + 5. deleted every v:2 machine record + | route: share-e2e-95cdb6.d.foundation/* removed + | ready: R2 is off for https://share-e2e-95cdb6.d.foundation/: the tunnel serves it alone. The bucket, its records, and Worker share-share-e2e-95cdb6-d-foundation stay + 6. /Users/tieubao/workspace/dwarvesf/share/.claude/worktrees/one-host-per-tenant/bin/share setup share-e2e-95cdb6.d.foundation --no-r2 exited 0 + ok R1 rollback: the alias serves itself again (its own /healthz answers, no 301) + ok R1 rollback: the tenant is a plain tunnel (no route) +--- R1 the fold again --- + | deploying as admin + | worker: share-share-e2e-95cdb6-d-foundation 4 2209e2a60db0 already deployed + | pointers: 0 written + | access: 1 gated link(s) also gated at share-e2e-95cdb6.d.foundation + | route: share-e2e-95cdb6.d.foundation/* -> share-share-e2e-95cdb6-d-foundation + | alias: share-e2e-61fdd2.d.foundation -> share-share-e2e-95cdb6-d-foundation (answers 301 to share-e2e-95cdb6.d.foundation) + | ready: R2 is on for https://share-e2e-95cdb6.d.foundation/ (bucket share-e2e-9bf46a, storage default local, share-e2e-61fdd2.d.foundation redirects here). Store this origin's publisher token: share api-token + ok R1: the second fold exits 0 + ok R1 after the second fold: alias 301s to the tenant again + +=== R1 cleanup === + ok R1: tenant tunnel DNS gone + ok R1: alias bucket gone + ok R1: alias Worker gone + +73/73 checks passed +PASS +tests/e2e-tenant.sh: line 103: 47694 Terminated: 15 python3 "$WORK/echo.py" "$1" +tests/e2e-tenant.sh: line 103: 47694 Terminated: 15 python3 "$WORK/echo.py" "$1" +cleanup: nothing left for this run's names diff --git a/docs/verification/e2e-tenant-20261005T062235Z.log b/docs/verification/e2e-tenant-20261005T062235Z.log new file mode 100644 index 0000000..050fc89 --- /dev/null +++ b/docs/verification/e2e-tenant-20261005T062235Z.log @@ -0,0 +1,190 @@ +=== before: nothing named for this run exists === + ok no Worker for share-e2e-51eafb.d.foundation + ok no Worker for share-e2e-1021a1.d.foundation + ok no DNS record for share-e2e-51eafb.d.foundation + ok no DNS record for share-e2e-1021a1.d.foundation +=== L1 origin A: tunnel setup; a local file, a gated local file, a live server === + | auth: API token (active) + | zone: d.foundation + | tunnel: created share-share-e2e-51eafb-d-foundation + | route: share-e2e-51eafb.d.foundation -> 127.0.0.1:8797 + | dns: created CNAME share-e2e-51eafb.d.foundation + | token: stored in keychain (service share-tunnel:share-e2e-51eafb.d.foundation) + | service: runs at login (foundation.d.share) + | ready: https://share-e2e-51eafb.d.foundation/ is live. Next: share add + ok A setup exits 0 + | minted token a-origin (id 0bae8d38..., expires 2026-10-05T08:23:01Z) + ok machine link answers + ok dotfile not served + ok live link answers (plain HTTP through the tunnel) + | share: checking the API token for profile default (zone d.foundation) + | ok token found (api_token_cmd) + | ok Zone: Read GET /zones?name=d.foundation -> 1 zone + | ok Access: Organizations, Identity Providers, and Groups Read GET /accounts//access/organizations -> 200 + | ok Access: Apps and Policies Edit POST /accounts//access/apps {} -> 400 (12130, nothing created) + | share: waiting up to 900s for Cloudflare Access to enforce on share-e2e-51eafb.d.foundation/1cc85b (a new app can take minutes; Ctrl-C is safe: nothing is published, the next add or prune removes the app) + ok gated add printed a link + ok no 200 on the gated link during the wait + ok gated local link 302s to Access with kid == aud +=== L2 setup --r2 on A === + | deploying as admin + | worker: share-share-e2e-51eafb-d-foundation 4 2209e2a60db0 deployed (passes misses to the tunnel) + | pointers: 3 written + | route: share-e2e-51eafb.d.foundation/* -> share-share-e2e-51eafb-d-foundation + | ready: R2 is on for https://share-e2e-51eafb.d.foundation/ (bucket share-e2e-ff12c1, storage default local). Store this origin's publisher token: share api-token + ok setup --r2 exits 0 + | token lists bucket share-e2e-ff12c1 after 0s + ok /healthz carries X-Share-Tunnel: 1 + ok /healthz carries this CLI's pair + ok pointer m/0e8b45 exists after setup --r2 + ok pointer m/f6638b exists after setup --r2 + ok workers.dev does not answer 200 +=== L3 through the Worker === + ok machine link unchanged through the Worker + ok gated link still 302s unauthenticated + | service-token fetch (SPEC-007 TASK-1(l)) not exercised: no SHARE_E2E_ACCESS_SERVICE_TOKEN_ID/SECRET input + ok live link answers a POST + ok live link answers a WebSocket echo +=== L4 A add --cloud; member B joins and adds === + ok A's cloud link answers + ok A's cloud link carries no-store + | minted token b-member (id a59c4a9a..., expires 2026-10-05T08:24:32Z) + | token lists bucket share-e2e-ff12c1 after 4s + | joining as publisher + | public-route check skipped (publisher token) + | ready: https://share-e2e-51eafb.d.foundation/ serves from bucket share-e2e-ff12c1. Store your publisher token: share api-token; then: share add + ok B join exits 0 + ok B join says joining as publisher + ok B's cloud link answers + | A ls: 10 line(s); B ls: 10 line(s) + ok A's ls shows B's cloud row (TASK-6 merged list) + ok B's ls shows A's rows (TASK-6 merged list) +=== L5 stop A's tunnel === + ok cloud link still answers while the tunnel is down + ok machine link answers the 503 offline page + ok live link answers the 503 offline page + ok /healthz answers X-Share-Tunnel: 0 with the tunnel down + | minted token d-fresh (id ebc67505..., expires 2026-10-05T08:25:07Z) + | token lists bucket share-e2e-ff12c1 after 4s + | joining as publisher + | public-route check skipped (publisher token) + | ready: https://share-e2e-51eafb.d.foundation/ serves from bucket share-e2e-ff12c1. Store your publisher token: share api-token; then: share add + ok fresh member D joins during the outage + | share: checking the API token for profile default (zone d.foundation) + | ok token found (api_token_cmd) + | ok Zone: Read GET /zones?name=d.foundation -> 1 zone + | ok Access: Organizations, Identity Providers, and Groups Read GET /accounts//access/organizations -> 200 + | ok Access: Apps and Policies Edit POST /accounts//access/apps {} -> 400 (12130, nothing created) + | share: waiting up to 900s for Cloudflare Access to enforce on share-e2e-51eafb.d.foundation/d20f6f (a new app can take minutes; Ctrl-C is safe: nothing is published, the next add or prune removes the app) + ok D's gated cloud add succeeds during the outage + ok B's state reads ready: true during the outage + ok machine link back after start + ok live link back after start +=== L6 encoded paths (a machine id and a cloud id) === + ok /x/..%2F0e8b45/doc/ answers 400 + ok /%2F0e8b45/doc/ answers 400 + ok //0e8b45/doc/ answers 400 + ok /x/..%2F0ba664/doc/ answers 400 + ok /%2F0ba664/doc/ answers 400 + ok //0ba664/doc/ answers 400 +=== L7 --no-r2 === + | route: share-e2e-51eafb.d.foundation/* removed + | ready: R2 is off for https://share-e2e-51eafb.d.foundation/: the tunnel serves it alone. The bucket, its records, and Worker share-share-e2e-51eafb-d-foundation stay + ok --no-r2 exits 0 + ok the route is gone + ok machine link serves straight from the tunnel (no X-Share-Worker) + ok cloud link 404s at Caddy (no route left to the Worker) +=== L8 setup --r2 again === + | deploying as admin + | worker: share-share-e2e-51eafb-d-foundation 4 2209e2a60db0 already deployed + | pointers: 0 written + | route: share-e2e-51eafb.d.foundation/* -> share-share-e2e-51eafb-d-foundation + | ready: R2 is on for https://share-e2e-51eafb.d.foundation/ (bucket share-e2e-ff12c1, storage default local). Store this origin's publisher token: share api-token + ok rerun --r2 exits 0 + ok no new pointer PUT for an existing id (etag unchanged) + ok cloud link answers again +=== L9 B's local teardown; A rm of each link === + ok B's config is gone + ok the bucket is unaffected by B's local teardown + ok A's ls is empty + ok machine link gone + ok cloud link gone + ok gated app gone +=== L10 admin purge and A teardown === + | dns: deleted share-e2e-51eafb.d.foundation + | tunnel: deleted + | token: removed + | service: removed + | config: moved to the Trash + ok A teardown exits 0 + ok tunnel DNS gone + ok bucket gone + ok Worker gone + +=== R1 Dwarves alias-fold rehearsal on throwaway names === + | deploying as admin + | worker: share-share-e2e-1021a1-d-foundation 4 2209e2a60db0 deployed + | ready: https://share-e2e-1021a1.d.foundation/ serves from bucket share-e2e-62693d. Next: share add + ok R1: C's standalone r2 origin on the alias exits 0 + | minted token c-admin (id a3936f5f..., expires 2026-10-05T08:28:08Z) + | token lists bucket share-e2e-62693d after 2s + | share: checking the API token for profile default (zone d.foundation) + | ok token found (api_token_cmd) + | ok Zone: Read GET /zones?name=d.foundation -> 1 zone + | ok Access: Organizations, Identity Providers, and Groups Read GET /accounts//access/organizations -> 200 + | ok Access: Apps and Policies Edit POST /accounts//access/apps {} -> 400 (12130, nothing created) + | share: waiting up to 900s for Cloudflare Access to enforce on share-e2e-1021a1.d.foundation/60d392 (a new app can take minutes; Ctrl-C is safe: nothing is published, the next add or prune removes the app) + ok R1: C's gated cloud link printed + | auth: API token (active) + | zone: d.foundation + | tunnel: created share-share-e2e-51eafb-d-foundation + | route: share-e2e-51eafb.d.foundation -> 127.0.0.1:8798 + | dns: created CNAME share-e2e-51eafb.d.foundation + | token: stored in keychain (service share-tunnel:share-e2e-51eafb.d.foundation) + | service: runs at login (foundation.d.share) + | ready: https://share-e2e-51eafb.d.foundation/ is live. Next: share add + ok R1: A2's plain tunnel on the tenant exits 0 + | deploying as admin + | worker: share-share-e2e-51eafb-d-foundation 4 2209e2a60db0 deployed (passes misses to the tunnel) + | pointers: 0 written + | share: waiting up to 900s for Cloudflare Access to enforce on share-e2e-51eafb.d.foundation/60d392 (a new app can take minutes; Ctrl-C is safe: nothing is published, the next add or prune removes the app) + | access: 1 gated link(s) also gated at share-e2e-51eafb.d.foundation + | route: share-e2e-51eafb.d.foundation/* -> share-share-e2e-51eafb-d-foundation + | alias: share-e2e-1021a1.d.foundation -> share-share-e2e-51eafb-d-foundation (answers 301 to share-e2e-51eafb.d.foundation) + | ready: R2 is on for https://share-e2e-51eafb.d.foundation/ (bucket share-e2e-62693d, storage default local, share-e2e-1021a1.d.foundation redirects here). Store this origin's publisher token: share api-token + | share: nothing was published; the Access app of 60d392 waits in access-pending, the next share add --access or share prune with a token removes it + ok R1: the fold (setup --r2 --alias) exits 0 + ok R1 after fold: alias 301s to the tenant + ok R1 after fold: the folded gated link 302s on the tenant with kid == aud +--- R1 rollback (### Dwarves order, the admin token) --- + 1. restored share-e2e-1021a1.d.foundation destinations on app 5ced5278-12ad-4b96-b631-4df168ece616 + 2. rebound share-e2e-1021a1.d.foundation to share-share-e2e-1021a1-d-foundation + 3. restored share.json host to share-e2e-1021a1.d.foundation in bucket share-e2e-62693d + 4. removed the aliases= line from A2's config + 5. deleted every v:2 machine record + | route: share-e2e-51eafb.d.foundation/* removed + | ready: R2 is off for https://share-e2e-51eafb.d.foundation/: the tunnel serves it alone. The bucket, its records, and Worker share-share-e2e-51eafb-d-foundation stay + 6. /Users/tieubao/workspace/dwarvesf/share/.claude/worktrees/one-host-per-tenant/bin/share setup share-e2e-51eafb.d.foundation --no-r2 exited 0 + ok R1 rollback: the alias serves itself again (its own /healthz answers, no 301) + ok R1 rollback: the tenant is a plain tunnel (no route) +--- R1 the fold again --- + | deploying as admin + | worker: share-share-e2e-51eafb-d-foundation 4 2209e2a60db0 already deployed + | pointers: 0 written + | access: 1 gated link(s) also gated at share-e2e-51eafb.d.foundation + | route: share-e2e-51eafb.d.foundation/* -> share-share-e2e-51eafb-d-foundation + | alias: share-e2e-1021a1.d.foundation -> share-share-e2e-51eafb-d-foundation (answers 301 to share-e2e-51eafb.d.foundation) + | ready: R2 is on for https://share-e2e-51eafb.d.foundation/ (bucket share-e2e-62693d, storage default local, share-e2e-1021a1.d.foundation redirects here). Store this origin's publisher token: share api-token + ok R1: the second fold exits 0 + ok R1 after the second fold: alias 301s to the tenant again + +=== R1 cleanup === + ok R1: tenant tunnel DNS gone + ok R1: alias bucket gone + ok R1: alias Worker gone + +73/73 checks passed +PASS +tests/e2e-tenant.sh: line 103: 99817 Terminated: 15 python3 "$WORK/echo.py" "$1" +tests/e2e-tenant.sh: line 103: 99817 Terminated: 15 python3 "$WORK/echo.py" "$1" +cleanup: nothing left for this run's names diff --git a/docs/verification/one-host-per-tenant.md b/docs/verification/one-host-per-tenant.md new file mode 100644 index 0000000..45919d4 --- /dev/null +++ b/docs/verification/one-host-per-tenant.md @@ -0,0 +1,378 @@ +# Proof of done: one host per tenant, batch 1 + +Date: 2026-10-01 +Branch: docs/one-host-per-tenant +Spec: docs/specs/SPEC-008-one-host-per-tenant.md + +Batch 1 is TASK-1a and TASK-1b (live spikes; commands and answers in `docs/implementation-notes/one-host-per-tenant.md`) and TASK-2a (per-link storage on the origin, rows 2, 4, 5). + +## Green run + +``` +Command: SHARE_TEST_PORT_BASE=28787 gtimeout 900 bash tests/share.sh +Exit: 0 +Checks: 1112 ok, 0 FAIL, 0 SKIP (PASS), at 3ea7dfe +Section: === tenant: per-link storage on the origin (rows 2, 4, 5) ===, 35 checks +Tail: ok no real Keychain share item changed during the run + PASS +Verdict: PASS +``` + +``` +Command: node tests/worker.mjs +Exit: 0 +Tail: ok r28 malformed record aud + PASS +Verdict: PASS +``` + +`shellcheck bin/share install.sh tests/share.sh tests/e2e.sh tests/e2e-r2.sh demo/render.sh mac/*.sh` and `/bin/bash -n bin/share` are clean on the same tree. + +## Negative controls + +Each patch was applied to `bin/share` in the worktree, the tenant section was run on its own (the section's lines from `tests/share.sh` under a minimal `check` harness), then `git checkout -- bin/share` restored the file and the same run went green (35 ok, 0 FAIL). + +| Patch | Run | Red rows | +|---|---|---| +| write the pointer after `stage_publish` (the pointer line moved below the publish line in `cmd_add`) | exit with `fails=4` | row 4 "held at the pointer PUT, pub/ does not exist yet" (expected `1 0`, got `1 1`); row 5 "no pub/" (got `1 0 0 0`, twice) | +| let a pointer PUT failure warn instead of die (`r2_pointer_put`'s catch-all prints and returns 0) | exit with `fails=4` | row 5 "exits 1 naming R2" (got `0 0`, ungated and gated); row 5 "no pub/, no row, no app" (got `1 1 0 0`, and `1 1 1 0` gated: the app was created) | +| restore (`git checkout -- bin/share`) | `fails=0` | none | + +Dry trace for the first control: the row 4 run holds the add at `PUT m/c00001` through `SHARE_R2_DRY_PAUSE`, whose `.paused` marker tells the test it is inside the held call; with the patch, `stage_publish` has already moved the stage to `pub/c00001`, so the check reads `1` where it wants `0`. + +## Live spikes + +TASK-1a and TASK-1b ran against throwaway Cloudflare objects only (names `share-e2e-spk584462` and `share-e2e-spk13b60d`). Every object was deleted and checked from outside: the API listed none of them on either account afterwards, the minted token's id answered 404, and each zone's authoritative server answered NXDOMAIN for both names. + +# Batch 2 + +Batch 2 is TASK-2b, TASK-3, and TASK-4, each committed and checked on its own. + +## TASK-2b: dispatch, reconcile, sweep (rows 3, 6, 26, 27) + +``` +Command: SHARE_TEST_PORT_BASE=28787 gtimeout 900 bash tests/share.sh +Exit: 0 +Checks: 1151 ok, 0 FAIL, 0 SKIP (PASS), at 6df0318 +Section: === tenant: storage dispatch, member refusals, reconcile, sweep (rows 3, 6, 26, 27) ===, 39 checks +Verdict: PASS +``` + +`node tests/worker.mjs` PASS, and `shellcheck bin/share install.sh tests/share.sh tests/e2e.sh tests/e2e-r2.sh demo/render.sh mac/*.sh` clean, on the same tree. + +Negative controls: each patch went into a copy of tree `25dd9e9` (commit 6df0318), the tenant sections ran alone, then the unpatched copy ran green (0 FAIL). + +| Patch | Red rows | +|---|---| +| let a member `rm` a machine row (the refusal in `cmd_rm_r2` becomes a `DELETE m/`) | row 3 "member rm of a machine row is refused" (got `0 0`), "no DELETE m/, the pointer stays" (got `1`), "member refresh ... refused" (got `1 0 0`: the pointer was gone, so the message changed), row 27's machine rm (no pointer left to delete); `fails=4` | +| let the sweep count a machine pointer as unreadable (`r2_record_raw`'s machine branch raises) | row 26 origin and member sweeps (got `0 0 1`: skipped, warning printed), the not-JSON leg (got `0 0 0`: the warning named the pointer instead); `fails=3` | + +Dry trace for the first control: the member's `rm e10001` reads the snapshot, finds `e10001` in `r2_machine`, and with the patch logs `DELETE m/e10001`; the check reads one DELETE line in the member's `r2-calls.log` where it wants none. + +## TASK-3: Worker v3 and r2_healthz (rows 7 to 10, 29) + +``` +Command: SHARE_TEST_PORT_BASE=28787 gtimeout 900 bash tests/share.sh +Exit: 0 +Checks: 1211 ok, 0 FAIL, 0 SKIP (PASS), at 2fcbdb1 +Section: === tenant: the Worker is up while the origin is off (row 29) ===, 6 checks +Verdict: PASS +``` + +``` +Command: node tests/worker.mjs +Exit: 0 +Checks: 134 ok, 0 FAIL (PASS), at 2fcbdb1; the tenant rows 7 to 10 add 58 +Verdict: PASS +``` + +`shellcheck` and `/bin/bash -n bin/share` clean on the same tree. One earlier full run on this tree failed `500-row index answers under 3s` once while the machine was busy; the rerun took 0.84 s and passed. + +Negative controls on a copy of tree `a6ff281` (commit 2fcbdb1): + +| Patch | Runner | Red | Green (unpatched) | +|---|---|---|---| +| the Worker passes an expired cloud record through instead of 404 | `node tests/worker.mjs` | `r7 an expired cloud record is 404, never passed through` (got `200 1`), plus the WORKER_SHA self-check | PASS | +| the Worker passes a miss through with `PASS` empty | `node tests/worker.mjs` | `r9 PASS empty: a miss is 404 with no fetch call` (got `101 1`: the stub's last answer came back) and the machine-pointer leg (got `404 1`), plus the WORKER_SHA self-check | PASS | +| read the healthz status alone in `r2_healthz` (the header branch never fires) | full suite | row 29 tunnel-down: `state` (got `0 false`), the gated add (got `1`), the join (got `1 0`) | row 29 green; the copy has no `.git`, so the two `v0.5.1 CLI` checks that run `git show` fail in both runs (and `profiles --json wrote nothing under HOME` once in the red run, a flake); the worktree run above is 0 FAIL | + +Dry trace for the third control: the dry Worker writes `HTTP/2 503` with the pair and `x-share-tunnel: 0`; with the patch `hz_code` stays 503, so `state` reports `ready: false`, the gated add dies at its healthz check, and the join's three-in-a-row wait times out. + +## TASK-4: setup --r2 and --no-r2, no alias (rows 11, 12, 13, 32) + +``` +Command: SHARE_TEST_PORT_BASE=28787 gtimeout 900 bash tests/share.sh +Exit: 0 +Checks: 1243 ok, 0 FAIL, 0 SKIP (PASS), at 19926b1 +Section: === tenant: setup --r2 and --no-r2 on the origin (rows 11, 12, 13, 32) ===, 32 checks +Verdict: PASS +``` + +`node tests/worker.mjs` PASS, `shellcheck` and `/bin/bash -n bin/share` clean on the same tree. Every call ran against the dry seam (`SHARE_R2_DRY=1`, a `.test` hostname); nothing was created on a Cloudflare account. + +Negative controls on copies of tree `a2cd324` (commit 19926b1), the full suite each: + +| Patch | Red | Green (unpatched copy) | +|---|---|---| +| attach the route before the pointer backfill (a route POST above step 9) | row 11 call order (stops at `^API POST /zones/zone-dry/workers/routes$`), the rerun (a second route POST), row 13 (a route left after `--no-r2`), the fresh-bucket order; `fails=7` with the copy's two `git show` checks and one load flake | every tenant row green; the two `git show` checks and one load flake (`cleanup: nocred1 row gone`, an unrelated SPEC-004 row; both copies ran at once) | +| let SPEC-007's r2 setup take a tenant host with `--force` (the `PASS` refusal skipped when `--force` is set) | row 32 "a tenant Worker (PASS) is refused with --force, before any write" (got `0 0 1`: the setup ran on and wrote); `fails=3` with the two `git show` checks | row 32 green; only the two `git show` checks | + +The copies carry no `.git`, so the two `v0.5.1 CLI` checks that run `git show` fail in every copy run; the worktree run above is 0 FAIL. + +Dry trace for the first control: the patched setup logs `API POST /zones/zone-dry/workers/routes` before the first `PUT m/0a000`, so the order check finds the route POST's first line above the pointer PUTs and stops at that pattern; the rerun POSTs a second route because the patched line runs on every setup. + +# Batch 3 + +Batch 3 is the member `hits` wording (DEC-012), TASK-5, TASK-6, and TASK-7a, each committed and checked on its own. Every call ran against the dry seams (`SHARE_R2_DRY=1`, `SHARE_ACCESS_DRY=1`, `.test` hostnames); nothing was created on a Cloudflare account. + +## Green runs + +Each run: `SHARE_TEST_PORT_BASE=28787 gtimeout 900 bash tests/share.sh` in the worktree, then `node tests/worker.mjs`, `shellcheck bin/share install.sh tests/share.sh tests/e2e.sh tests/e2e-r2.sh demo/render.sh mac/*.sh`, and `/bin/bash -n bin/share`. + +| Task | Commit | `tests/share.sh` | Exit | `tests/worker.mjs` | shellcheck | +|---|---|---|---|---|---| +| member `hits` wording | fcd24bd | 1246 ok, 0 FAIL, PASS | 0 | PASS | clean | +| TASK-5 (rows 14, 30) | d246452 | 1281 ok, 0 FAIL, PASS | 0 | PASS | clean | +| TASK-6 (rows 1, 15, 16, 17, 28) | 1f9fb4b | 1312 ok, 0 FAIL, PASS | 0 | PASS | clean | +| TASK-7a (row 18, `--token-stdin`) | 32a502e | 1340 ok, 0 FAIL, PASS | 0 | PASS | clean | + +## Negative controls + +Each patch went into a `git clone --local` of the named commit in a mktemp directory (so the `git show` rows run as in the worktree), and the full suite ran there with `SHARE_TEST_PORT_BASE=40000`, one run at a time. An unpatched clone of the same commit ran green (`EXIT=0`) for d246452, 1f9fb4b, and 32a502e. + +| Patch | Commit | Red rows | Failed | +|---|---|---|---| +| member `hits` prints the bare count again (the note line becomes `:`) | fcd24bd | row 13 "no visits on an id another publisher added" and row 3 "member hits of a machine id never prints a bare 0" (both got `0 hits, 0 visitors` alone) | 2 | +| keep the alias destinations on the app after step 14 (`alias_dests drop` skipped) | d246452 | row 14 "dropped last, after the 301", row 14 end state, the rerun, and every row 30 convergence (each app still lists `f.example.test/{,/*}`) | 7 | +| print `--no-r2` as the step-15 rollback with an alias (`tenant_die` ignores the alias) | d246452 | row 30 at each of steps 12, 13, 14, 15 (got `1 0 0 0 1`: no list, `roll back with: share ...` printed) | 4 | +| read the bucket in `state` through `api_token_cmd` (the stored-token check skipped) | 1f9fb4b | row 17 "an api_token_cmd source" (got `0 1 6 0`: the sentinel command ran, the bucket rows listed, no `cloud_error`) | 3 | +| merge bucket rows through `rows()` (the origin's bucket read rebinds `index`, outside the prune's subshell) | 1f9fb4b | row 28 "the Caddyfile and index.tsv hold exactly the local rows" (the expired local row stayed, a bucket row counted in the index), plus rows 6, 11, 13, 14, 15, 30 | 45 | +| let `import` accept a symlink (the pre-scan takes an `l` mode and a `->` line, the post-scan a link) | 32a502e | row 18 "a symlink is refused" (got `0 1 1`: published, a row written) | 1 | + +Two unrelated lines failed in the red runs and are not counted as caught: row 29 "a member joins" in the `api_token_cmd` run (the healthz wait under load; green in every other run), and the real-launchd guard in both TASK-6 red runs. That guard saw a real `foundation.d.share` job appear on this machine during the first run and leave during the second. The suite stubs `launchctl` and labels its own jobs `share-selftest-`, so the job came from outside the suite; the worktree runs and the green clone saw no change. + +An earlier attempt ran two clones at once; the green one failed `prune still removes the payload` and a row 27 `rm` with exit 5 while the twin run trashed same-named files. Every control above ran alone. + +Dry trace for the TASK-5 rollback control: with the patch, the step-15 die at `ALIAS-301` calls `die "...; roll back with: share --profile ten setup ten.example.test --no-r2"`, so the output holds no `alias rollback for f.example.test` line and no ` 6. ...` step, and row 30's count reads `0` where it wants `1`. + +## Tar measurement + +TASK-7a's first step ran on bsdtar 3.5.3 and GNU tar 1.35; the table and the flags it fixed are in `docs/implementation-notes/one-host-per-tenant.md`. Nothing was extracted outside the scratch directories: both tars stripped the absolute member into the target and refused the `..` member, and the probe paths outside it did not exist afterwards. + +## Rollback + +Batch 3 changes only `bin/share`, its tests, and docs on the PR branch; no release, tap bump, or Cloudflare object exists for it. Rolling it back is reverting commits fcd24bd, d246452, 1f9fb4b, and 32a502e on the branch. + +# Batch 4 + +Batch 4 is TASK-7b (`migrate`, rows 19, 20, 21, 31), committed at f0dc1ac. No dry seam exists for `migrate` (it refuses an R2-on tenant), so its local coverage is a `curl`/`security` shim pair plus `SHARE_MIGRATE_SSH` pointing at a wrapper that joins its own argv with spaces and replays it through `fish -c` (or `sh -c`) on the same machine, exactly mirroring what a real ssh round trip does to the command line; nothing was created on a Cloudflare account. + +## Green run + +``` +Command: SHARE_TEST_PORT_BASE=30787 gtimeout 900 bash tests/share.sh +Exit: 1 (one pre-existing, unrelated flake; see below) +Checks: 1360 ok, 1 FAIL +Section: === migrate: moving a tenant's origin (rows 19, 20, 21, 31) ===, 20 checks, all ok +Tail: ok no real launchd or systemd share job appeared during the run + ok no real Keychain share item changed during the run +``` + +``` +Command: node tests/worker.mjs +Exit: 0 +Verdict: PASS +``` + +`shellcheck bin/share install.sh tests/share.sh tests/e2e.sh tests/e2e-r2.sh tests/e2e-tenant.sh demo/render.sh mac/*.sh` and `/bin/bash -n bin/share` are clean on the same tree (`tests/e2e-migrate.sh` does not exist yet; it is TASK-9b's deliverable). + +The one failing line, `profiles --json wrote nothing under HOME` (a `find -newer` marker check, pre-existing code untouched by this batch), is unrelated to migrate: it passed on a first full run of the same tree and failed on a second run of the identical code, the same intermittent-under-load pattern the TASK-3 and TASK-6 batches above already recorded for other lines. A rerun confined to the migrate section alone (the 20 checks above) is green every time. + +## Negative controls + +Each guarantee's patch ran as a one-off (a `sed`-patched copy of `bin/share`, the relevant migrate scenario run standalone against it, the stock file confirmed green immediately after). No patch was committed. + +| Guarantee | Patch | Red | Green (unpatched) | +|---|---|---|---| +| ids preserved | the remote `import` call for one id is rewritten to a different id before `bin/share` runs it | B's index never gains the original id (the renamed one lands instead) | row 19: B's index holds exactly A's own ids | +| source never deleted | `migrate_retire`'s `mv "$pub/$id" "$root/migrated/$id"` replaced with `rm -rf "$pub/$id"` | `migrated/` never exists after a run that otherwise completes | row 19: every moved id's tree lands in `migrated/`, byte for byte equal to B's copy | +| retire runs only after verify passes | the gated-link `access_probe_round ... || die` turned into `access_probe_round ...; true \|\| die` (the die never fires) | with a wrong Access `kid` forced on the gate check, `index.migrated` still gets written: retire ran past a verify that should have stopped it | row 20: the same wrong-`kid` scenario, unpatched, stops before retire and names the gated id | + +Dry trace for the first control: `cmd_import`'s own id argument comes from the sender's positional argv, so a wrapper script that rewrites `$2` before `exec`ing the real binary is enough to desync the id client-side of any base64 or checksum check; the manifest digest still matches (it is a digest of the tar's own bytes, not the id), so nothing else catches it except the id itself showing up wrong in B's index, which is exactly what the "ids preserved" guarantee asserts directly. + +## Rollback + +Batch 4 changes only `bin/share`, `tests/share.sh`, and docs on the PR branch; no release, tap bump, or Cloudflare object exists for it. Rolling it back is reverting commit f0dc1ac on the branch. + +# Batch 5 + +Batch 5 is the `kit:battery` review fix pass on this PR (five findings plus two items raised from the live rehearsal), commits a045013..2a0fb65. Each finding carries its own commit with a failing test shown red before the fix, then green after; the fast per-finding reruns below are scratch slices of the real `tests/share.sh` (same fixtures and helper functions, the unrelated middle of the file cut out) used purely to get a faster red/green loop during the fix; no scratch file was committed. + +## Findings and their red/green + +| # | Finding | Fix | Red (before) | Green (after) | +|---|---|---|---|---| +| 1 | migrate handed the target no `--tunnel-name`; a source on the default name collided, the target reused the source's own tunnel, and `migrate_retire` then deleted it out from under the new origin | always pass a deterministic `--tunnel-name` distinct from the source's own (`-m` suffix); `migrate_retire` also refuses the tunnel/DNS deletes if the target's own tunnel id is ever reported equal to this machine's | row 19m: target's tunnel_id equals the source's | row 19m: target's tunnel_id never equals the source's; rows 19/20/21/31 unaffected | +| 2 | `tests/share.sh` printed the same "PASS"/"N FAILED" summary whether the run finished or crashed partway | a `reached_end` flag set only at the true end; the EXIT trap reports ABORTED and exits 2 when it is missing; the summary prints the total check count | isolated trap test: a truncated run exits 0 with no ABORTED text | isolated trap test, then the embedded self-test: a truncated run exits 2 and prints ABORTED | +| 3 | `cmd_setup`'s old_host guard only fires when old_host is set; import into a not_setup profile then `setup ` served its gated rows ungated | refuse when old_host is empty and the profile holds an `access=` row; migrate's own switch (`--token-stdin`) is exempt, since it never renames the hostname | setup on a not_setup profile with a gated row: no refusal, dies on the live Cloudflare call instead (with the nocurl stub, before any call assertion fails) | refused before any Cloudflare call, naming the gated row | +| 4 | the Worker's `pass()` treated any 502 as the whole machine offline, same as Cloudflare's 520-527 edge codes | drop the explicit 502 check; only 530, 520-527, and a 503 with no `cf-cache-status` mean offline | `node tests/worker.mjs`: a dead-port 502 rewritten to the 503 offline page | the 502 passes through unchanged, with or without `cf-cache-status` | +| 5 | implementation notes claimed the migrate preflight reads `tunnel_name` from the API by `tunnel_id` when the config key is missing; the code never did | `cmd_migrate` now reads it back from the API when the local key is absent, and dies naming the manual fix if neither is known; note corrected to say where this runs | row 31b: rollback line printed `--tunnel-name --force` (empty) | row 31b: names the tunnel read back by id, never empty | +| 6 | `alias_dests drop` answered HTTP 400 live (3/3 rehearsals), the same PUT shape that worked for `add` two calls earlier | the PUT body drops the legacy `self_hosted_domains`/`domain` mirror fields so the API recomputes them from the new `destinations`, instead of echoing back a stale value that no longer names a surviving destination once drop removes it | row 14 ("the fold exits 0") against a genuinely reverted `alias_dests`, cf_dry extended to reject the same self_hosted_domains/destinations inconsistency a real PUT does, fixture carrying a realistic `self_hosted_domains`: fold dies, destinations keep the alias's own entries. Confirmed again with an isolated jq trace of the exact body construction (same expressions, no harness): pre-fix body retains `self_hosted_domains` naming a URI absent from the new `destinations` | row 14/30 (40+ checks) all pass end to end; the same isolated jq trace shows the stripped body consistent | +| 7 | `tests/e2e-tenant.sh` had five script bugs found across three live rehearsals (a cold R2 token, a split TCP read in the echo origin, a missing-file check, two Access-app lookups assuming a folded app renames to the tenant), plus L8's `etag_before` reading through A's own profile right after `--no-r2` removed its `bucket=` | applied the run1→run3 fix patch; L8 now reads the pointer's ETag by a direct S3 HEAD on the bucket name, independent of A's profile config | (live-only; no dry seam exercises the R2/Access live path) | `bash -n tests/e2e-tenant.sh` and shellcheck clean; the lead reruns R1/L8 live | + +## Green run + +``` +Command: SHARE_TEST_PORT_BASE=32787 gtimeout 900 bash tests/share.sh +Exit: 0 +Checks: 1238 checks, PASS +Tail: ok a run truncated before the end exits 2 + ok a run truncated before the end reports ABORTED + ok no real launchd or systemd share job appeared during the run + ok no real Keychain share item changed during the run +``` + +A first run of the same tree, same command, showed one failing line unrelated to any of the seven items above (`row 6: pub/ absent while a PROBE fail was logged`, a `sleep 0.05` filesystem-polling race against a background watcher, pre-existing and untouched by this batch): the same intermittent-under-load pattern Batch 4 already recorded for a different line. The rerun above is clean. + +``` +Command: node tests/worker.mjs +Exit: 0 +Verdict: PASS +``` + +``` +Command: swift test --package-path mac +Exit: 0 +Verdict: Executed 176 tests, with 0 failures (0 unexpected) +``` + +`shellcheck bin/share install.sh tests/share.sh tests/e2e.sh tests/e2e-r2.sh tests/e2e-tenant.sh demo/render.sh mac/*.sh` and `/bin/bash -n bin/share tests/share.sh tests/e2e-tenant.sh` are clean on the same tree. + +## Rollback + +Batch 5 changes `bin/share`, `tests/share.sh`, `tests/worker.mjs`, `tests/e2e-tenant.sh`, and `docs/implementation-notes/one-host-per-tenant.md` on the PR branch; no release, tap bump, or Cloudflare object exists for it. Rolling it back is reverting commits a045013..2a0fb65 on the branch. + +# Batch 6 + +Batch 6 is the live rerun of `tests/e2e-tenant.sh` on the fix-pass head (1440259), the fixes that rerun forced, the TASK-10 docs, and the negative controls the earlier batches had not covered. Final live result: 73/73, run log `docs/verification/e2e-tenant-20261005T042854Z.log`. TASK-9b (`tests/e2e-migrate.sh`, rehearsal R2) is not run here: it needs one command from Han on the Air. + +## Live e2e: five runs, one green + +Every run used the PR worktree's `bin/share` (`SHARE_BIN`), zone `d.foundation`, an admin token from `op://Toolkit/cf-api-token/credential`, run-scoped 2h publisher tokens minted through `op://Toolkit/cf-tokens-admin/credential`, and `SHARE_E2E_ACCESS_RULE=group:dwarves-ops`. Hostnames were `share-e2e-<6 hex>` pairs chosen per run. The earlier logs are not committed (two of them print the account and zone ids in a rollback line). + +| Run | Head | Result | What failed | Cause and action | +|---|---|---|---|---| +| 1 | 1440259 | 69/73 | L3 WebSocket echo; R1 rollback x2; R1 "alias bucket gone" | R1's fold passed for the first time (alias 301, folded gated link 302 with `kid == aud`, second fold), so the live 400 is fixed. The rest: the WebSocket client used the local resolver, which answered "No route to host" for a name made seconds earlier (script bug, now DoH). `setup --no-r2` exited 1 with no output (a real bug in `alias_rollback`, below). The cleanup deleted the alias bucket with an admin env token on a profile with no stored publisher token (script bug) | +| 2 | f6a6f41 | 71/73 | R1 rollback x2 | the script parsed `r2-call`'s `code= etag=` line as `etag=` alone, so the marker host restore went out with an empty `If-Match` and the marker kept its `aliases` (script bug) | +| 3 | dac8541 | 70/72 | R1 standalone origin and gated link | Cloudflare answered HTTP 500 adding the Worker custom domain for the alias host; the gated R1 legs did not run. Transient; the die names the call | +| 4 | dac8541 | 72/73 | L1 "gated local link 302s to Access with kid == aud": got 200 | the first fetch after `add` returned reached an edge that did not yet enforce the app. Not reproduced in run 5; see the note below | +| 5 | dac8541 | 73/73 PASS | none | R1 (fold, rollback, second fold, cleanup) green end to end | + +R1 passes: the fold exits 0; `https:///healthz` answers 301 to the tenant; the folded gated link 302s on the tenant with `kid == aud`; the rollback list runs in its printed order and `--no-r2` leaves a plain tunnel (no route); the second fold converges and the alias 301s again; the cleanup deletes the bucket, the Worker, and the tenant DNS. + +Gate note from run 4: share's own check passed three rounds of `kid == aud` before the link was published, and a fetch through DoH one second later still answered 200 once. Access enforcement is per edge and eventual (ADR-0006 measured seconds to minutes). The e2e keeps the strict check, because "no ungated window after `add` returns" is the guarantee the leg tests. A lead decision, outside SPEC-008: whether `add --access` should probe through a second resolver before it publishes. + +## Cleanup proof, from outside the run + +A snapshot script listed every resource class the e2e touches, before and after each run, from the Cloudflare API and `launchctl`. Names and truncated ids only. The baseline and every later snapshot were identical (`diff` empty) after runs 1, 2, 3, 4, and 5. + +| Class | Baseline (and after every run) | +|---|---| +| Workers named `share-*` | `share-f-d-foundation` | +| Worker custom domains, routes, DNS records matching `share-e2e*` | none | +| Tunnels named `share-*` (not deleted) | `share-s-d-foundation` | +| R2 buckets named `share-*` | `share-dfoundation` | +| Access apps named `share ...` | `s.d.foundation`, `f.d.foundation` (the two live Dwarves apps) | +| Minted user tokens (`a-origin`, `b-member`, `d-fresh`, `c-admin`) | none | +| launchd labels and LaunchAgents plists matching `share` | `foundation.d.share.dfoundation`, the Share Bar app job | + +The script's own `leftovers` sweep printed `cleanup: nothing left for this run's names` on all five runs. No process named `echo.py` or `share-e2e` remained. Nothing the run did not create was touched: the two live Access apps, `share-f-d-foundation`, `share-s-d-foundation`, and `share-dfoundation` are in the baseline and in every later snapshot. + +## Findings the rerun forced + +| # | Finding | Fix | Red | Green | +|---|---|---|---|---| +| 1 | `alias_rollback` assigned `ptrs` from a pipeline ending in `grep`; with no local rows and no machine records the grep matches nothing, and under `set -e` the run died with exit 1 and no output, so `--no-r2` and every `tenant_die` printed nothing for a tenant with an alias and no local rows (e6b2a13) | `{ grep ... \|\| true; }` in the pipeline | the new row 30 check, run against the unfixed `bin/share` in a clone: `row 30: with no local rows and no pointers, --no-r2 still refuses and prints the list: expected '1 1 1 1', got '1 0 0 0'`, 1 FAIL of 1243 | the same check on the fixed tree: 1243 checks, PASS | +| 2 | five `tests/e2e-tenant.sh` script bugs (743b229, dac8541): DoH for the WebSocket client; the R1 rollback removes `aliases=` from the profile config (step 4 of the printed list); the marker ETag parsed from `code= etag=`; the alias liveness check reads `/healthz` (a standalone r2 Worker answers 404 at the site root by design); the cleanup's delete-prefix uses the `SHARE_R2_TOKEN` seam | script edits | runs 1 and 2 | run 5, 73/73 | + +## Negative controls + +The spec lists 20. Batches 1 to 5 recorded a red run for 14 of them (the expired-record, `PASS`-empty, pointer-order, pointer-failure, route-order, alias-destination, symlink, `api_token_cmd`, `rows()`-merge, SPEC-007 `--force`, sweep, member-`rm`, healthz-status, and `--no-r2`-rollback controls). Batch 6 adds 5 and records one whose spec form is not reachable, with a substitute. Each patch went into a `git clone --local` of dac8541 in a scratch directory and the suite (or `node tests/worker.mjs` for a Worker patch) ran there, one run at a time with its own `SHARE_TEST_PORT_BASE`. The green run for every row is the unpatched clone of the same commit. + +| Green (unpatched dac8541) | Result | +|---|---| +| `SHARE_TEST_PORT_BASE=41100 gtimeout 900 bash tests/share.sh` | exit 0, 1243 checks, PASS | +| `node tests/worker.mjs` | exit 0, PASS | + +| Spec control | Patch | Runner | Red | +|---|---|---|---| +| the Worker serves R2 for a `storage:"machine"` record (row 7), tested in the inverse form | the machine-pointer branch in `worker_js` made unreachable (`&& false`), so a pointer no longer passes through | `node tests/worker.mjs` | `r7 a machine record passes through` (got 404, no fetch) and the r8 pass-through rows (13 FAIL, plus `WORKER_SHA` drift, which any Worker edit trips) | +| drop the encoded-separator check before the pass-through (SPEC-007 row 18 on the v3 source) | the `%2f \|%5c \|%2e \|low escape \|//` test replaced with `if (false)` | `node tests/worker.mjs` | `r18` for `..%2F`, `%2f`, `..%5C`, `%2E`, `//`, and a low escape (all 404, expected 400), and `r7 /x/..%2Fb1c2d3/f.txt is 400 in front of the tunnel` (12 FAIL plus `WORKER_SHA`) | +| take a member's `add --local` as cloud (row 3) | the member `--local` die replaced with `storage=cloud` | suite | `row 3: member add --local f is refused` (got `0 0`), 1 FAIL | +| run the full teardown (with Access deletes) on the old origin before moving rows aside (row 19) | `cmd_rm` of every moved id at the top of `migrate_retire` | suite | `row 19` migrate exits 0, B's trees equal A's, the rows move to `index.migrated`, the live row stays, migrate still exits 0; `row 21` rerun skips the moved id (9 FAIL, one of them the harness's idler check after the abort) | +| omit `--tunnel-name` from the printed migrate rollback (row 31) | `--tunnel-name $old_tunnel_name` dropped from the switch-failure rollback line | suite | `row 20: a remote-setup failure ... printing the rollback with A's own tunnel-name` and `row 31b` (3 FAIL, one the idler check) | +| write `bucket=` into a plain tunnel config at setup (row 1) | `echo "bucket=leak"` added to `write_config` | suite | no red: 1243 checks, PASS. Not reachable: row 1's compat run does a quick setup only, and no suite row runs a named tunnel `setup` through `write_config`. The live e2e covers it (L1 setup, L7 `--no-r2`) | +| replacement for the row above: `--no-r2` keeps `bucket=` (row 13) | `tenant_config` stops dropping `bucket=` | suite | `rerun: the config is the same`, `row 13: the config is the one before setup --r2`, `row 13: the next add writes no pointer` (3 FAIL) | + +One gap is open: the `write_config` row-1 control above. Closing it needs a named-setup fixture with a Cloudflare stub, which no suite row has; it is the one place where only the live e2e proves "a tenant with R2 off writes no `bucket=`". + +## Docs check (TASK-10) + +`tests/share.sh` now checks that every verb, flag, config key, and state key the tenant docs name is in `docs/how-it-works.md` and in `bin/share` (the four generic state keys are checked in the doc only, since `storage`, `type`, `by`, and `r2` are too common to grep in the source; rows 15 and 1 pin them in `state` output). It has three negative controls inside the suite: a doc missing `--storage-default`, a `bin/share` missing `--remote-bin`, and a doc missing the `storage` state key each fail the check. All pass in the green run above. + +## Rollback + +Batch 6 changes `bin/share` (one line in `alias_rollback`), `tests/share.sh`, `tests/e2e-tenant.sh`, README, `docs/how-it-works.md`, `docs/setup.md`, ADR-0008, the spec (DEC-013, the 502 rows, Worker version 4), the implementation notes, and this record, on the PR branch. No release, tap bump, or standing Cloudflare object exists for it. Rolling it back is reverting commits e6b2a13..dac8541 and the docs commit that follows. + +# Batch 7 + +Batch 7 is the lead's three safety decisions and the CI fix, on top of Batch 6. + +| # | Change | Red (unfixed 476f4f8, new tests) | Green | +|---|---|---|---| +| 1 | `migrate_retire` refuses the DNS and tunnel delete when `migrate-tunnel-id` returns nothing, and names `teardown --yes` for later | row 19e: `no DNS record or tunnel is deleted` got `0 2`; the refusal text absent | row 19e passes: migrate exits 0, rows moved aside, no DELETE of a DNS record or tunnel | +| 2 | the custom-domain add (standalone r2 setup and the alias bind) retries a Cloudflare 5xx, three tries with doubling backoff, and dies loud after (`after up to 3 tries`); the dry seam gains `SHARE_R2_DRY_FAIL_TIMES` | four checks: two 500s then success got `1 1`, a persistent 500 got one PUT, not three | both paths succeed after two stubbed 500s (three PUTs) and die after three on a persistent 500; the alias die still prints the rollback list | +| 3 | `add --access` polls the published link until it answers 302 or 403 (`SHARE_ACCESS_POST_WAIT`, 30 s) before printing it; a link still answering anything else exits 1 with the status, the edge risk, and `share rm `; the link stays published | four checks: the link printed with no probes, a 200 never failed the add | `200,200,302` prints after three probes; `403` counts; a persistent `200` exits 1 with no link on stdout, and the warning names the rm command | + +Red run: `1256` checks, 11 FAIL (the 10 above plus one cascade from leftover gated rows, fixed in the test). Green runs on the final tree: `1256 checks, PASS`, one earlier run showed a single row 29 timing failure (`SHARE_R2_WAIT=1` join under load) that did not repeat. `node tests/worker.mjs`: PASS. Live: `tests/e2e-tenant.sh` on 1a76f2e (the first commit with the real link check against Cloudflare), 73/73, log `docs/verification/e2e-tenant-20261005T062235Z.log`; the before and after snapshots of every resource class are identical. + +## CI + +The CI lint job failed on every push since the tenant work began, and the test jobs have failed since the migrate rows landed. Three causes, all fixed: + +| Job | Cause | Fix | +|---|---|---| +| lint | CI's shellcheck reports SC2015 twice in `bin/share` and SC2002 once in `tests/share.sh`; the local shellcheck 0.11 is quieter | restructured the three lines, same behavior; a later SC2001 in a new debug line fixed too | +| test (ubuntu, macos) | the migrate rows run `setup` on the target, which dies with `missing: caddy cloudflared` on a runner with no cloudflared; every dev machine has one | the suite appends stand-in `caddy` and `cloudflared` last on PATH, so a real binary still wins. Reproduced locally with a PATH that hides cloudflared and fish: red before, `1256 checks, PASS` after | + +The macOS runner also showed one `tty: the prompt appeared and the token did not echo` failure on 1440259 that did not recur; it is the known flaky row. + +# Batch 8 + +Batch 8 is TASK-9b: `tests/e2e-migrate.sh` and the live rehearsal R2 of the personal move, on throwaway names. Final result: 38/38, run log `docs/verification/e2e-migrate-20261005T093825Z.log`. The second machine was the real Air, driven from the Mini with `mini-run --host air`, migrating over real ssh to the Mini. + +| Leg | What it ran | Result | +|---|---|---| +| M0 | origin A (a snapshot, a folder, a gated snapshot, a live server), `migrate` to B on the same machine with a remote `setup` that does the real switch and then reports failure; the printed rollback run on A | exit 1 with the rollback line naming A's own tunnel; the DNS record moved to B's tunnel; the printed rollback exits 0 and the record points at A's own tunnel again; every link answers, the gated one with `kid == aud` | +| M1 | the same name migrated again, B already holding the copies | exit 0; the copies are skipped; the live link is reported as not moved; every snapshot link answers at the same URL, the gated one keeps its 302 and AUD; the DNS record points at B's tunnel; A's tunnel is deleted; A's rows are in `index.migrated` and its trees under `migrated/` | +| M2 | a second throwaway name set up on the Air, five links published, `migrate --to` the Mini over ssh | the same assertions as M1, from the real Air to the Mini | + +Measured gap: probes every 0.5 s through DoH on a snapshot link logged no non-200 answer across the switch on either run (37 probes in M1; the M2 gap logger likewise). The spec expected up to about a minute of 530; the DNS record flips and the new tunnel connects inside a probe interval at this scale, so the number to plan with is "under a few seconds", not a minute. + +## Findings the rehearsal forced + +| # | Finding | Fix | Red | Green | +|---|---|---|---|---| +| 1 | `migrate` archived only ``, but a bare-file snapshot keeps its generated `index.html` beside the file at the id root. The receiver counted one file where the sender's manifest counted two and refused the share as a truncated copy: every single-file link would have failed the personal move (9ae6350) | the archive is the whole `pub/` tree; `import` checks that `` is in it | the suite fixture gains the generated `index.html` it lacked: 17 FAIL in rows 19 to 31 | 1256 checks, PASS | +| 2 | after a failed switch the target profile is serving this hostname, so the rerun's preflight died with `port for this profile is already in use`; the printed rollback led to a dead end (f37e816) | the preflight accepts a target already serving the same hostname | new row 19s: got `1 1` | 1257 checks, PASS | +| 3 | script bugs in `tests/e2e-migrate.sh` itself: profile names must be lowercase; the Air-side run needs the real `HOME` for ssh while share keeps a throwaway one; a new name needs a moment to resolve | script edits | runs 1 to 5 | run 6, 38/38 | + +Both product bugs live only on the real-account path: no fixture in the suite had a generated index beside a bare file or a target that was already serving. + +## Cleanup proof + +Snapshots of every Cloudflare resource class (Workers, domains, routes, DNS, tunnels, buckets, Access apps, minted tokens) and of `launchctl` and LaunchAgents on the Mini were identical before and after the runs. On the Air, the count of `foundation.d.share` launchd jobs and the LaunchAgents listing were the same before and after (its real `s.han.ws` service was never touched: the legs used profiles `mga` and `mgb` under throwaway HOMEs). The script's own sweep printed `cleanup: nothing left for this run's names` on every run. One leftover surfaced: the M2 target's data root under the real HOME (`~/share/profiles/mgb`), because the script's cleanup only looked for the profile config. The script now moves that root aside too. + +## Rollback + +Batch 8 adds `tests/e2e-migrate.sh` and changes `bin/share` (`migrate_tar`, the import root check, the migrate preflight) and `tests/share.sh`. No release, tap bump, or standing Cloudflare object exists for it. diff --git a/mac/Package.swift b/mac/Package.swift index 5fbd731..316de67 100644 --- a/mac/Package.swift +++ b/mac/Package.swift @@ -10,7 +10,11 @@ let package = Package( .testTarget( name: "ShareBarCoreTests", dependencies: ["ShareBarCore"], - resources: [.copy("Fixtures/state.json"), .copy("Fixtures/profiles.json")] + resources: [ + .copy("Fixtures/state.json"), + .copy("Fixtures/profiles.json"), + .copy("Fixtures/profiles-tenant.json"), + ] ), ] ) diff --git a/mac/Sources/ShareBar/StatusItemController.swift b/mac/Sources/ShareBar/StatusItemController.swift index cf4a8e5..172eeb8 100644 --- a/mac/Sources/ShareBar/StatusItemController.swift +++ b/mac/Sources/ShareBar/StatusItemController.swift @@ -239,20 +239,27 @@ final class StatusItemController: NSObject, @unchecked Sendable { let item = NSMenuItem(title: row.title, action: nil, keyEquivalent: "") item.attributedTitle = rowAttributedTitle(row, tabLocation: tabLocation, font: font) item.setAccessibilityTitle(row.accessibilityTitle) - if row.access != nil, let lock = NSImage(systemSymbolName: "lock.fill", accessibilityDescription: "Login required") { - lock.isTemplate = true - item.image = lock + if let typeImage = NSImage(systemSymbolName: row.typeGlyph.symbol, accessibilityDescription: row.typeGlyph.word) { + typeImage.isTemplate = true + item.image = typeImage } item.submenu = submenu(for: row) menu.addItem(item) } if section.more > 0 { - let moreItem = NSMenuItem(title: "\(section.more) more (\(section.command) ls)", action: nil, keyEquivalent: "") + let prefix = section.cloudMore ? "about " : "" + let moreItem = NSMenuItem(title: "\(prefix)\(section.more) more (\(section.command) ls)", action: nil, keyEquivalent: "") moreItem.isEnabled = false menu.addItem(moreItem) } + if let cloudError = section.cloudError { + let errorItem = NSMenuItem(title: "Cloud links not listed: \(cloudError)", action: nil, keyEquivalent: "") + errorItem.isEnabled = false + menu.addItem(errorItem) + } + if section.showStop { menu.addItem(sectionItem("Stop Sharing", action: #selector(stopSharing), profile: section.profile)) } else if section.showStart { @@ -277,8 +284,11 @@ final class StatusItemController: NSObject, @unchecked Sendable { } /// The name in the plain font/color, a tab, then the trailing text right-aligned at - /// `tabLocation` in `secondaryLabelColor` (macOS's shortcut-hint grey). `item.title` is - /// still set to the plain name (see `addSection`) so accessibility reads the name alone. + /// `tabLocation` in `secondaryLabelColor` (macOS's shortcut-hint grey), and after it + /// the marker column: the storage badge, the link-type marker, and the lock on a + /// gated row. `item.title` is still set to the plain name (see `addSection`) so + /// accessibility reads the name alone. A symbol name that fails to resolve falls + /// back to no image rather than a blank box. private func rowAttributedTitle(_ row: Row, tabLocation: CGFloat, font: NSFont) -> NSAttributedString { let paragraphStyle = NSMutableParagraphStyle() paragraphStyle.tabStops = [NSTextTab(textAlignment: .right, location: tabLocation, options: [:])] @@ -296,6 +306,29 @@ final class StatusItemController: NSObject, @unchecked Sendable { .paragraphStyle: paragraphStyle, ] )) + + var markers = [row.linkGlyph] + if let storage = row.storageGlyph { + markers.insert(storage, at: 0) + } + if row.access != nil { + markers.append(RowGlyphs.lock) + } + let symbolConfig = NSImage.SymbolConfiguration(pointSize: font.pointSize, weight: .regular) + for marker in markers { + guard let image = NSImage(systemSymbolName: marker.symbol, accessibilityDescription: marker.word)? + .withSymbolConfiguration(symbolConfig) else { continue } + image.isTemplate = true + result.append(NSAttributedString( + string: " ", + attributes: [.font: font, .paragraphStyle: paragraphStyle] + )) + let attachment = NSTextAttachment() + attachment.image = image + // Baseline-to-descender alignment: the marker bottoms out where the text does. + attachment.bounds = CGRect(origin: CGPoint(x: 0, y: font.descender), size: image.size) + result.append(NSAttributedString(attachment: attachment)) + } return result } @@ -501,6 +534,16 @@ final class StatusItemController: NSObject, @unchecked Sendable { audiencePopup.addItems(withTitles: ["Anyone with the link", "Only people who log in"]) audiencePopup.selectItem(at: form.audience == .login ? 1 : 0) + // One `Storage:` row holds two possible controls: the picker an r2-on origin + // gets, and the disabled label a member gets. `sync` picks between them (and + // hides the row entirely for a plain tunnel profile) as the profile changes. + let storagePopup = NSPopUpButton(frame: .zero, pullsDown: false) + let storageLabel = NSTextField(labelWithString: "In the cloud") + storageLabel.isEnabled = false + let storageControls = NSStackView(views: [storagePopup, storageLabel]) + storageControls.orientation = .horizontal + let storageRow = labeledRow("Storage:", storageControls) + let ruleField = NSTextField(frame: .zero) ruleField.placeholderString = "group:, email:a@x.io,b@y.io, or domain:" ruleField.stringValue = form.rule @@ -512,6 +555,7 @@ final class StatusItemController: NSObject, @unchecked Sendable { let accessory = NSStackView(views: [ labeledRow("Profile:", profilePopup), + storageRow, labeledRow("Who can open:", audiencePopup), labeledRow("Rule:", ruleField), loginNote, @@ -525,13 +569,34 @@ final class StatusItemController: NSObject, @unchecked Sendable { ruleField.widthAnchor.constraint(greaterThanOrEqualToConstant: 220).isActive = true alert.accessoryView = accessory - let sync: (PublishForm) -> Void = { [ruleField, loginNote, alert] form in + let sync: (PublishForm) -> Void = { [ruleField, loginNote, storageRow, storagePopup, storageLabel, alert] form in // Only on a real change: rewriting the text mid-edit would move the caret. if ruleField.stringValue != form.rule { ruleField.stringValue = form.rule } ruleField.isEnabled = form.audience == .login loginNote.isHidden = !(form.audience == .login && !form.loginAvailable) + switch form.storageField { + case .none: + storageRow.isHidden = true + case .member: + storageRow.isHidden = false + storagePopup.isHidden = true + storageLabel.isHidden = false + case .picker(let machine): + storageRow.isHidden = false + storageLabel.isHidden = true + storagePopup.isHidden = false + let titles = ["On \(machine)", "In the cloud"] + if storagePopup.itemTitles != titles { + storagePopup.removeAllItems() + storagePopup.addItems(withTitles: titles) + } + let index = form.storage == .cloud ? 1 : 0 + if storagePopup.indexOfSelectedItem != index { + storagePopup.selectItem(at: index) + } + } alert.buttons[0].title = form.buttonTitle alert.buttons[0].isEnabled = form.canPublish } @@ -542,6 +607,8 @@ final class StatusItemController: NSObject, @unchecked Sendable { profilePopup.action = #selector(AlertForm.profileChanged(_:)) audiencePopup.target = alertForm audiencePopup.action = #selector(AlertForm.audienceChanged(_:)) + storagePopup.target = alertForm + storagePopup.action = #selector(AlertForm.storageChanged(_:)) ruleField.target = alertForm ruleField.action = #selector(AlertForm.ruleEdited(_:)) ruleField.delegate = alertForm @@ -555,7 +622,7 @@ final class StatusItemController: NSObject, @unchecked Sendable { defaults.set(profile, forKey: "publish.lastProfile") defaults.set(rule ?? "", forKey: PublishForm.ruleKey(for: profile)) Task { [weak self] in - await self?.publishBatch(paths: paths, profile: profile, rule: rule) + await self?.publishBatch(paths: paths, profile: profile, rule: rule, storage: form.storageFlag) } } @@ -576,9 +643,9 @@ final class StatusItemController: NSObject, @unchecked Sendable { /// failed re-read; the alert for that event adds `Not published: ` for the /// paths that never ran. @MainActor - private func publishBatch(paths: [String], profile: String, rule: String?) async { + private func publishBatch(paths: [String], profile: String, rule: String?, storage: PublishStorage? = nil) async { for (index, path) in paths.enumerated() { - guard let argv = PublishChoice.args(profile: profile, rule: rule, path: path) else { continue } + guard let argv = PublishChoice.args(profile: profile, rule: rule, path: path, storage: storage) else { continue } actionLogger.log("add profile=\(profile, privacy: .public) path=\(path, privacy: .private)") let unpublished = Array(paths.dropFirst(index + 1)) let ok = await performMutation( @@ -601,7 +668,7 @@ final class StatusItemController: NSObject, @unchecked Sendable { #if DEBUG func debugAddPaths(_ paths: [String]) { guard let profiles = currentProfiles, let profile = PublishChoice.eligible(profiles).first else { return } - Task { [weak self] in await self?.publishBatch(paths: paths, profile: profile, rule: nil) } + Task { [weak self] in await self?.publishBatch(paths: paths, profile: profile, rule: nil, storage: nil) } } #endif @@ -1016,6 +1083,11 @@ private final class AlertForm: NSObject, NSTextFieldDelegate, @unchecked Sendabl resync() } + @objc func storageChanged(_ sender: NSPopUpButton) { + form.choose(storage: sender.indexOfSelectedItem == 1 ? .cloud : .local) + resync() + } + @objc func ruleEdited(_ sender: NSTextField) { form.rule = sender.stringValue resync() diff --git a/mac/Sources/ShareBarCore/MenuModel.swift b/mac/Sources/ShareBarCore/MenuModel.swift index 8a73919..cef0a0c 100644 --- a/mac/Sources/ShareBarCore/MenuModel.swift +++ b/mac/Sources/ShareBarCore/MenuModel.swift @@ -74,7 +74,14 @@ public struct Row: Sendable, Equatable { public let canRefresh: Bool public let canCopy: Bool public let removeText: String - /// What VoiceOver reads for the row: name, status, and the login gate when present. + /// The file-type marker (`NSMenuItem.image`); `doc`/`file` on a row with no `type`. + public let typeGlyph: RowGlyph + /// The storage badge; nil on a row with no `storage` (an older CLI), which shows none. + public let storageGlyph: RowGlyph? + /// The link-type marker (`snapshot` or `live server`). + public let linkGlyph: RowGlyph + /// What VoiceOver reads for the row: name, status, the type/storage/link words, and + /// the login gate when present. public let accessibilityTitle: String /// The composite key the hits cache uses. public var key: String { "\(profile)|\(id)" } @@ -88,9 +95,14 @@ public struct Row: Sendable, Equatable { access = share.access canRefresh = share.kind == "snapshot" canCopy = !(url.contains("") || url.contains("")) - accessibilityTitle = access == nil - ? "\(title), \(trailing)" - : "\(title), \(trailing), login required" + typeGlyph = RowGlyphs.type(share.type) + storageGlyph = RowGlyphs.storage(share.storage) + linkGlyph = RowGlyphs.link(share.kind) + var voice = "\(title), \(trailing), \(typeGlyph.word)" + if let storageGlyph { voice += ", \(storageGlyph.word)" } + voice += ", \(linkGlyph.word)" + if access != nil { voice += ", \(RowGlyphs.lock.word)" } + accessibilityTitle = voice if let ownHost = share.ownHost { removeText = access == nil ? "Remove \(share.name)? This also deletes the DNS record for \(ownHost)." @@ -124,10 +136,17 @@ public struct Section: Sendable, Equatable { public let health: Health public let rows: [Row] public let more: Int + /// True when `more` includes `cloud_more` (orphan pointers and unreadable records the + /// origin could not confirm): the menu line hedges with "about" rather than claiming + /// an exact count. + public let cloudMore: Bool public let showStart: Bool public let showStop: Bool public let showSetUp: Bool public let accessPending: Int + /// The `cloud_error` reason `state` reported, when the bucket listing failed but local + /// rows still printed; the section shows it as a disabled line. + public let cloudError: String? /// `share` for the default profile, `share --profile ` otherwise; the command /// the `more` and `access_pending` hint lines name (` ls`, ` prune`). public let command: String @@ -145,13 +164,19 @@ public struct Section: Sendable, Equatable { if health == .error { rows = [] more = 0 + cloudMore = false showStart = false showStop = false showSetUp = false + cloudError = nil } else { let allRows = (entry.state?.shares ?? []).map { Row(share: $0, profile: entry.name, now: now) } rows = Array(allRows.prefix(rowCap)) - more = max(0, allRows.count - rowCap) + // `cloud_more` counts rows `state` never fetched; they exist but are not + // shown, so they ride the same "N more" hint as capped local rows. + more = max(0, allRows.count - rowCap) + (entry.state?.cloudMore ?? 0) + cloudMore = (entry.state?.cloudMore ?? 0) > 0 + cloudError = entry.state?.cloudError // An r2 profile serves from Cloudflare; the CLI refuses start and stop there. let runsHere = entry.state?.backend != "r2" showStart = runsHere && entry.state?.state != "serving" && entry.state?.servesHere == true diff --git a/mac/Sources/ShareBarCore/Publish.swift b/mac/Sources/ShareBarCore/Publish.swift index e3cced7..8e9f0e4 100644 --- a/mac/Sources/ShareBarCore/Publish.swift +++ b/mac/Sources/ShareBarCore/Publish.swift @@ -22,6 +22,24 @@ public enum Audience: Sendable, Equatable { case login } +/// Where one published link lives; the argv flag it maps to. +public enum PublishStorage: Sendable, Equatable { + case local // --local: this tenant's origin machine + case cloud // --cloud: the tenant's bucket +} + +/// What the dialog's `Storage` row shows for the selected profile. +public enum StorageField: Sendable, Equatable { + /// No control at all: a tunnel profile with R2 off, or an older CLI with no `r2` key. + case none + /// An r2 member profile publishes cloud links only: a disabled `In the cloud` label. + case member + /// An origin with R2 on (the tunnel runs here): an `On ` / `In the cloud` + /// popup, `machine` being this Mac's name out of the state `hosts` line (or its first + /// entry when this Mac is not named there). + case picker(machine: String) +} + /// Which profiles can take an `add`, and the argv each add runs. public enum PublishChoice { /// Eligible profiles: `stopped` or `serving` with no `error`, in listing order. `add` @@ -38,13 +56,19 @@ public enum PublishChoice { } /// argv for one add; nil unless `path` is absolute, so a bare `8080` can never reach - /// `add` and become a live port share by accident. - public static func args(profile: String, rule: String?, path: String) -> [String]? { + /// `add` and become a live port share by accident. `storage` adds `--cloud` or + /// `--local`; nil leaves the choice to the CLI's `storage_default`. + public static func args(profile: String, rule: String?, path: String, storage: PublishStorage? = nil) -> [String]? { guard path.hasPrefix("/") else { return nil } + var argv = ["--profile", profile, "add"] if let rule, !rule.isEmpty { - return ["--profile", profile, "add", "--access", rule, path] + argv += ["--access", rule] } - return ["--profile", profile, "add", path] + if let storage { + argv.append(storage == .cloud ? "--cloud" : "--local") + } + argv.append(path) + return argv } } @@ -57,6 +81,17 @@ public struct PublishForm: Sendable, Equatable { public private(set) var audience: Audience /// The rule field's text; meaningful only while `audience == .login`. public var rule: String + /// The `Storage` popup's selection; read through `storageField`/`storageFlag`, which + /// decide whether the choice applies at all. + public private(set) var storage: PublishStorage + + /// This Mac's name, as the `hosts=` config key would spell it (the CLI's `this_host`: + /// `uname -n` before the first dot). Overridable in tests. + public static let defaultCurrentMachineName: () -> String = { + ProcessInfo.processInfo.hostName.split(separator: ".", maxSplits: 1).first.map(String.init) + ?? ProcessInfo.processInfo.hostName + } + public static var currentMachineName: () -> String = defaultCurrentMachineName private let profiles: ProfilesSnapshot private let lastRules: [String: String] @@ -75,13 +110,16 @@ public struct PublishForm: Sendable, Equatable { let stored = lastRules[profile ?? ""] ?? "" rule = stored audience = stored.isEmpty ? .anyone : .login + storage = PublishForm.defaultStorage(of: profile, in: profiles) } /// Under `login` the audience and the typed rule survive a profile switch, even when /// the new profile's stored choice is `anyone`. Under `anyone` the new profile's own - /// stored choice loads. + /// stored choice loads. Storage reloads from the new profile's `storage_default` + /// either way: the picker is per-tenant, not a remembered preference. public mutating func select(profile name: String) { profile = name + storage = PublishForm.defaultStorage(of: name, in: profiles) if audience == .anyone { let stored = lastRules[name] ?? "" rule = stored @@ -93,6 +131,38 @@ public struct PublishForm: Sendable, Equatable { audience = newAudience } + public mutating func choose(storage newStorage: PublishStorage) { + storage = newStorage + } + + /// The `Storage` row for the selected profile: a picker on an R2-on origin serving + /// here, a disabled `In the cloud` label on an r2 member, nothing otherwise. + public var storageField: StorageField { + guard let state = profiles.profiles.first(where: { $0.name == profile })?.state else { return .none } + if state.backend == "r2" { return .member } + if state.r2 == true && state.servesHere { + guard !state.hosts.isEmpty else { return .picker(machine: "this machine") } + let names = state.hosts.split(separator: " ").map(String.init) + let mine = PublishForm.currentMachineName() + let machine = names.contains(mine) ? mine : (names.first ?? state.hosts) + return .picker(machine: machine) + } + return .none + } + + /// The `--cloud`/`--local` flag the add argv gains; nil unless the picker is showing + /// (a member's adds are cloud already, and a plain tunnel profile takes no flag). + public var storageFlag: PublishStorage? { + if case .picker = storageField { return storage } + return nil + } + + /// A picker's preselection: the profile's `storage_default`, local when unset. + private static func defaultStorage(of profile: String?, in profiles: ProfilesSnapshot) -> PublishStorage { + let state = profile.flatMap { name in profiles.profiles.first { $0.name == name }?.state } + return state?.storageDefault == "cloud" ? .cloud : .local + } + /// `login` needs a named setup; a quick-mode profile cannot take `--access`. public var loginAvailable: Bool { profiles.profiles.first { $0.name == profile }?.state?.mode != "quick" diff --git a/mac/Sources/ShareBarCore/RowGlyphs.swift b/mac/Sources/ShareBarCore/RowGlyphs.swift new file mode 100644 index 0000000..2dbd8d7 --- /dev/null +++ b/mac/Sources/ShareBarCore/RowGlyphs.swift @@ -0,0 +1,57 @@ +import Foundation + +/// One row marker: the SF Symbol name the app resolves, plus the word accessibility reads +/// for it. +public struct RowGlyph: Sendable, Equatable { + public let symbol: String + public let word: String + + public init(symbol: String, word: String) { + self.symbol = symbol + self.word = word + } +} + +/// The mapping from a share's `type`, `storage`, and `kind` to row markers, held as plain +/// strings so every rule is unit-testable without AppKit. The app resolves the names with +/// `NSImage(systemSymbolName:)` and falls back to no image when one fails. +public enum RowGlyphs { + /// The file-type marker. `other`, an unknown value, and a missing `type` (a row from + /// an older CLI) all take the plain `doc` file marker. + public static func type(_ value: String?) -> RowGlyph { + switch value { + case "pdf": return RowGlyph(symbol: "doc.richtext", word: "PDF") + case "image": return RowGlyph(symbol: "photo", word: "image") + case "video": return RowGlyph(symbol: "film", word: "video") + case "audio": return RowGlyph(symbol: "waveform", word: "audio") + case "folder": return RowGlyph(symbol: "folder", word: "folder") + case "site": return RowGlyph(symbol: "globe", word: "site") + case "markdown": return RowGlyph(symbol: "doc.plaintext", word: "Markdown") + case "archive": return RowGlyph(symbol: "archivebox", word: "archive") + case "text": return RowGlyph(symbol: "doc.text", word: "text") + default: return RowGlyph(symbol: "doc", word: "file") + } + } + + /// The storage badge; nil when the row carries no `storage` or an unknown one, so the + /// row simply shows no badge. + public static func storage(_ value: String?) -> RowGlyph? { + switch value { + case "machine": return RowGlyph(symbol: "desktopcomputer", word: "on this tenant's machine") + case "cloud": return RowGlyph(symbol: "cloud", word: "in the cloud") + default: return nil + } + } + + /// The link-type marker. Anything that is not `live` reads as a snapshot. + public static func link(_ kind: String) -> RowGlyph { + switch kind { + case "live": return RowGlyph(symbol: "dot.radiowaves.right", word: "live server") + default: return RowGlyph(symbol: "doc.on.doc", word: "snapshot") + } + } + + /// The gate marker on a gated row (`access` set); it sits in the same trailing group + /// as the badges. + public static let lock = RowGlyph(symbol: "lock.fill", word: "login required") +} diff --git a/mac/Sources/ShareBarCore/Snapshot.swift b/mac/Sources/ShareBarCore/Snapshot.swift index 5c7780a..f8346f8 100644 --- a/mac/Sources/ShareBarCore/Snapshot.swift +++ b/mac/Sources/ShareBarCore/Snapshot.swift @@ -10,13 +10,19 @@ public struct Share: Decodable, Sendable, Equatable { public let expires: Int // epoch seconds; 0 means never /// The `--access` rule on a gated share; nil means public. public let access: String? + /// `machine` or `cloud`; nil on a CLI that predates per-link storage. + public let storage: String? + /// The file type (`pdf`, `image`, `folder`, ...); nil on an older CLI. + public let type: String? + /// The machine that published the link; nil on an older CLI. + public let by: String? enum CodingKeys: String, CodingKey { - case id, name, url, kind, expires, access + case id, name, url, kind, expires, access, storage, type, by case ownHost = "own_host" } - public init(id: String, name: String, url: String, kind: String, ownHost: String?, expires: Int, access: String? = nil) { + public init(id: String, name: String, url: String, kind: String, ownHost: String?, expires: Int, access: String? = nil, storage: String? = nil, type: String? = nil, by: String? = nil) { self.id = id self.name = name self.url = url @@ -24,6 +30,9 @@ public struct Share: Decodable, Sendable, Equatable { self.ownHost = ownHost self.expires = expires self.access = access + self.storage = storage + self.type = type + self.by = by } } @@ -45,11 +54,24 @@ public struct Snapshot: Decodable, Sendable, Equatable { /// `"r2"` on a profile that serves from R2 through a Worker (nothing runs on this Mac, /// so start and stop do not apply); nil on a tunnel profile or an older CLI. public let backend: String? + /// True when the profile reads a bucket; nil on an older CLI. + public let r2: Bool? + /// `local` or `cloud` on an origin with R2 on; nil elsewhere and on an older CLI. + public let storageDefault: String? + /// Why cloud rows are missing from `shares` when they are (the reason line `state` + /// prints instead of failing); nil when the bucket read worked or there is none. + public let cloudError: String? + /// Cloud rows `state` did not fetch (it caps its bucket reads at 25); the section + /// adds it to the `more` line. + public let cloudMore: Int? enum CodingKeys: String, CodingKey { - case schema, state, ready, mode, host, hosts, service, shares, skipped, backend + case schema, state, ready, mode, host, hosts, service, shares, skipped, backend, r2 case servesHere = "serves_here" case accessPending = "access_pending" + case storageDefault = "storage_default" + case cloudError = "cloud_error" + case cloudMore = "cloud_more" } public init( @@ -64,7 +86,11 @@ public struct Snapshot: Decodable, Sendable, Equatable { shares: [Share], skipped: Int?, accessPending: Int? = nil, - backend: String? = nil + backend: String? = nil, + r2: Bool? = nil, + storageDefault: String? = nil, + cloudError: String? = nil, + cloudMore: Int? = nil ) { self.schema = schema self.state = state @@ -78,6 +104,10 @@ public struct Snapshot: Decodable, Sendable, Equatable { self.skipped = skipped self.accessPending = accessPending self.backend = backend + self.r2 = r2 + self.storageDefault = storageDefault + self.cloudError = cloudError + self.cloudMore = cloudMore } } diff --git a/mac/Tests/ShareBarCoreTests/Fixtures/profiles-tenant.json b/mac/Tests/ShareBarCoreTests/Fixtures/profiles-tenant.json new file mode 100644 index 0000000..f669095 --- /dev/null +++ b/mac/Tests/ShareBarCoreTests/Fixtures/profiles-tenant.json @@ -0,0 +1,100 @@ +{ + "schema": 1, + "profiles": [ + { + "name": "dfoundation", + "state": { + "schema": 1, + "state": "serving", + "ready": true, + "mode": "named", + "host": "s.d.foundation", + "hosts": "Mac-mini", + "serves_here": true, + "service": true, + "access_pending": 0, + "r2": true, + "storage_default": "local", + "cloud_more": 3, + "shares": [ + { + "id": "ba6377", + "name": "support-ticket-guide", + "url": "https://s.d.foundation/ba6377/support-ticket-guide/", + "kind": "snapshot", + "own_host": null, + "expires": 0, + "storage": "cloud", + "type": "site", + "by": "han-air" + }, + { + "id": "a68960", + "name": "quarterly-report.pdf", + "url": "https://s.d.foundation/a68960/quarterly-report.pdf", + "kind": "snapshot", + "own_host": null, + "expires": 1793087100, + "access": "group:dwarves-ops", + "storage": "machine", + "type": "pdf", + "by": "mac-mini" + }, + { + "id": "b3100a", + "name": "localhost:3000", + "url": "https://s.d.foundation/b3100a/", + "kind": "live", + "own_host": null, + "expires": 0, + "storage": "machine", + "type": "site", + "by": "mac-mini" + } + ] + } + }, + { + "name": "files", + "state": { + "schema": 1, + "state": "serving", + "ready": true, + "mode": "named", + "host": "s.d.foundation", + "hosts": "", + "serves_here": false, + "service": false, + "access_pending": 0, + "backend": "r2", + "r2": true, + "cloud_error": "the menu reads cloud links only with a stored token: share --profile files api-token", + "shares": [] + } + }, + { + "name": "personal", + "state": { + "schema": 1, + "state": "serving", + "ready": true, + "mode": "named", + "host": "s.han.ws", + "hosts": "hans-air-m4", + "serves_here": false, + "service": true, + "access_pending": 0, + "shares": [ + { + "id": "4f9ff2", + "name": "notes.txt", + "url": "https://s.han.ws/4f9ff2/notes.txt", + "kind": "snapshot", + "own_host": null, + "expires": 1793087100 + } + ] + } + } + ] +} diff --git a/mac/Tests/ShareBarCoreTests/PublishFormTests.swift b/mac/Tests/ShareBarCoreTests/PublishFormTests.swift index e19ff8e..6c7e0f6 100644 --- a/mac/Tests/ShareBarCoreTests/PublishFormTests.swift +++ b/mac/Tests/ShareBarCoreTests/PublishFormTests.swift @@ -151,6 +151,100 @@ final class PublishFormTests: XCTestCase { XCTAssertNil(PublishChoice.args(profile: "p", rule: nil, path: "report.pdf"), "no bare filename reaches add") } + // MARK: - storage field + + private func origin(r2: Bool, storageDefault: String? = nil, servesHere: Bool = true, name: String = "dfoundation", hosts: String = "Mac-mini") -> ProfileEntry { + makeEntry(name: name, state: makeSnapshot( + state: "serving", hosts: hosts, servesHere: servesHere, r2: r2, storageDefault: storageDefault + )) + } + + func testAnOriginWithR2ShowsThePickerPreselectedFromStorageDefault() { + var snapshot = makeProfiles([origin(r2: true, storageDefault: "cloud")]) + var form = PublishForm(profiles: snapshot, lastProfile: nil, lastRules: [:]) + XCTAssertEqual(form.storageField, .picker(machine: "Mac-mini")) + XCTAssertEqual(form.storage, .cloud) + XCTAssertEqual(form.storageFlag, .cloud) + + snapshot = makeProfiles([origin(r2: true, storageDefault: "local")]) + form = PublishForm(profiles: snapshot, lastProfile: nil, lastRules: [:]) + XCTAssertEqual(form.storage, .local) + XCTAssertEqual(form.storageFlag, .local) + + // A chosen value reaches the argv as --cloud / --local. + form.choose(storage: .cloud) + XCTAssertEqual( + PublishChoice.args(profile: "dfoundation", rule: nil, path: "/tmp/a", storage: form.storageFlag), + ["--profile", "dfoundation", "add", "--cloud", "/tmp/a"] + ) + form.choose(storage: .local) + XCTAssertEqual( + PublishChoice.args(profile: "dfoundation", rule: nil, path: "/tmp/a", storage: form.storageFlag), + ["--profile", "dfoundation", "add", "--local", "/tmp/a"] + ) + } + + func testPickerNamesTheCurrentMachineWhenHostsListsMoreThanOne() { + PublishForm.currentMachineName = { "air" } + defer { PublishForm.currentMachineName = PublishForm.defaultCurrentMachineName } + + let snapshot = makeProfiles([origin(r2: true, storageDefault: "cloud", hosts: "mini air")]) + let form = PublishForm(profiles: snapshot, lastProfile: nil, lastRules: [:]) + XCTAssertEqual(form.storageField, .picker(machine: "air"), "picks this Mac out of a multi-host hosts= value") + } + + func testPickerFallsBackToTheFirstHostWhenThisMachineIsNotInHosts() { + PublishForm.currentMachineName = { "neither-of-these" } + defer { PublishForm.currentMachineName = PublishForm.defaultCurrentMachineName } + + let snapshot = makeProfiles([origin(r2: true, storageDefault: "cloud", hosts: "mini air")]) + let form = PublishForm(profiles: snapshot, lastProfile: nil, lastRules: [:]) + XCTAssertEqual(form.storageField, .picker(machine: "mini"), "falls back to the first host, never the whole list") + } + + func testAMemberProfileShowsADisabledCloudLabelAndNoFlag() { + let member = makeEntry(name: "files", state: makeSnapshot( + state: "serving", host: "s.d.foundation", hosts: "", servesHere: false, backend: "r2", r2: true + )) + let form = PublishForm(profiles: makeProfiles([member]), lastProfile: nil, lastRules: [:]) + + XCTAssertEqual(form.storageField, .member) + XCTAssertNil(form.storageFlag, "a member adds cloud links with no flag") + } + + func testATunnelProfileWithR2OffShowsNoStorageControl() { + let form = PublishForm(profiles: makeProfiles([serving("a")]), lastProfile: nil, lastRules: [:]) + XCTAssertEqual(form.storageField, .none) + XCTAssertNil(form.storageFlag) + + // An older CLI emits no r2 key at all: same shape, no control. + let legacy = makeEntry(name: "old", state: makeSnapshot(state: "serving")) + XCTAssertEqual(PublishForm(profiles: makeProfiles([legacy]), lastProfile: nil, lastRules: [:]).storageField, .none) + } + + func testAnOriginNotServingHereShowsNoStorageControl() { + let form = PublishForm(profiles: makeProfiles([origin(r2: true, storageDefault: "cloud", servesHere: false)]), + lastProfile: nil, lastRules: [:]) + XCTAssertEqual(form.storageField, .none, "the popup belongs to the origin only") + XCTAssertNil(form.storageFlag) + } + + func testAProfileSwitchReloadsStorageFromTheNewProfilesDefault() { + let snapshot = makeProfiles([ + origin(r2: true, storageDefault: "cloud"), + serving("plain"), + ]) + var form = PublishForm(profiles: snapshot, lastProfile: nil, lastRules: [:]) + XCTAssertEqual(form.storage, .cloud) + + form.select(profile: "plain") + XCTAssertEqual(form.storageField, .none) + XCTAssertNil(form.storageFlag) + + form.select(profile: "dfoundation") + XCTAssertEqual(form.storage, .cloud, "the picker re-preselects from storage_default") + } + // MARK: - MenuModel.canPublish func testCanPublishIsFalseOnlyWhenNoProfileIsEligible() { diff --git a/mac/Tests/ShareBarCoreTests/RowAndModelRulesTests.swift b/mac/Tests/ShareBarCoreTests/RowAndModelRulesTests.swift index ef038c8..ef8d1c1 100644 --- a/mac/Tests/ShareBarCoreTests/RowAndModelRulesTests.swift +++ b/mac/Tests/ShareBarCoreTests/RowAndModelRulesTests.swift @@ -135,7 +135,7 @@ final class RowAndModelRulesTests: XCTestCase { func testGatedRowGetsTheGateTextAndMarker() { let gated = rowFor(makeShare(name: "ops-report.pdf", access: "group:dwarves-ops")) XCTAssertEqual(gated.access, "group:dwarves-ops") - XCTAssertEqual(gated.accessibilityTitle, "ops-report.pdf, never, login required") + XCTAssertEqual(gated.accessibilityTitle, "ops-report.pdf, never, file, snapshot, login required") XCTAssertEqual(gated.removeText, "Remove ops-report.pdf? The copy goes to the Trash and its login gate is deleted.") let gatedOwnHost = rowFor(makeShare(name: "docs", ownHost: "docs.example.com", access: "email:a@x.io")) @@ -148,7 +148,35 @@ final class RowAndModelRulesTests: XCTestCase { func testPublicRowHasNoGateText() { let row = rowFor(makeShare(name: "notes.txt")) XCTAssertNil(row.access) - XCTAssertEqual(row.accessibilityTitle, "notes.txt, never") + XCTAssertEqual(row.accessibilityTitle, "notes.txt, never, file, snapshot") + } + + // MARK: - row glyphs + + func testAGatedCloudPdfReadsItsTypeStorageLinkAndGate() { + let row = rowFor(makeShare( + name: "ops-report.pdf", access: "group:dwarves-ops", storage: "cloud", type: "pdf" + )) + XCTAssertEqual(row.typeGlyph, RowGlyph(symbol: "doc.richtext", word: "PDF")) + XCTAssertEqual(row.storageGlyph, RowGlyph(symbol: "cloud", word: "in the cloud")) + XCTAssertEqual(row.linkGlyph, RowGlyph(symbol: "doc.on.doc", word: "snapshot")) + XCTAssertEqual(row.accessibilityTitle, "ops-report.pdf, never, PDF, in the cloud, snapshot, login required") + } + + func testALiveMachineRowReadsSiteOnThisTenantsMachineLiveServer() { + let row = rowFor(makeShare( + name: "localhost:3000", kind: "live", storage: "machine", type: "site" + )) + XCTAssertEqual(row.trailing, "live") + XCTAssertEqual(row.typeGlyph.symbol, "globe") + XCTAssertEqual(row.accessibilityTitle, "localhost:3000, live, site, on this tenant's machine, live server") + } + + func testAnOlderCLIRowHasNoStorageBadge() { + let row = rowFor(makeShare(name: "notes.txt")) + XCTAssertNil(row.storageGlyph) + XCTAssertEqual(row.typeGlyph, RowGlyph(symbol: "doc", word: "file")) + XCTAssertEqual(row.accessibilityTitle, "notes.txt, never, file, snapshot") } // MARK: - per-profile section rules @@ -287,6 +315,47 @@ final class RowAndModelRulesTests: XCTestCase { XCTAssertEqual(model.sections[0].command, "share --profile a") } + // MARK: - cloud_error and cloud_more + + func testCloudErrorLandsOnTheSection() { + let profiles = makeProfiles([ + makeEntry(name: "a", state: makeSnapshot( + state: "serving", + cloudError: "the menu reads cloud links only with a stored token: share --profile a api-token" + )), + ]) + let model = MenuModel(profiles: profiles, failure: nil, now: Date()) + + XCTAssertEqual(model.sections[0].cloudError, "the menu reads cloud links only with a stored token: share --profile a api-token") + XCTAssertTrue(model.sections[0].health == .ok, "a cloud_error section still lists its local rows") + } + + func testCloudMoreRidesTheMoreLine() { + let shares = (1...12).map { makeShare(id: "id\($0)", name: "file\($0).txt") } + let profiles = makeProfiles([ + makeEntry(name: "a", state: makeSnapshot(state: "serving", shares: shares, cloudMore: 3)), + makeEntry(name: "b", state: makeSnapshot(state: "serving")), + ]) + let model = MenuModel(profiles: profiles, failure: nil, now: Date()) + + XCTAssertEqual(model.sections[0].rows.count, 10) + XCTAssertEqual(model.sections[0].more, 5, "2 over the cap plus 3 unfetched cloud rows") + XCTAssertNil(model.sections[1].cloudError) + XCTAssertEqual(model.sections[1].more, 0) + } + + func testCloudMoreMarksTheSectionForTheAboutWording() { + let shares = (1...30).map { makeShare(id: "id\($0)", name: "file\($0).txt") } + let profiles = makeProfiles([ + makeEntry(name: "a", state: makeSnapshot(state: "serving", cloudMore: 3)), + makeEntry(name: "b", state: makeSnapshot(state: "serving", shares: shares)), + ]) + let model = MenuModel(profiles: profiles, failure: nil, now: Date()) + + XCTAssertTrue(model.sections[0].cloudMore, "a cloud_more count needs the \"about\" wording") + XCTAssertFalse(model.sections[1].cloudMore, "a locally-capped more count is exact, no \"about\"") + } + // MARK: - helpers private func rowFor(_ share: Share, now: Date = Date()) -> Row { diff --git a/mac/Tests/ShareBarCoreTests/RowGlyphsTests.swift b/mac/Tests/ShareBarCoreTests/RowGlyphsTests.swift new file mode 100644 index 0000000..67b56d2 --- /dev/null +++ b/mac/Tests/ShareBarCoreTests/RowGlyphsTests.swift @@ -0,0 +1,67 @@ +import AppKit +import XCTest +@testable import ShareBarCore + +/// The type/storage/link symbol table: every value maps to the spec's symbol name and +/// accessibility word, and every named symbol resolves on the test host. +final class RowGlyphsTests: XCTestCase { + func testTypeGlyphsMatchTheTable() { + let cases: [(String, RowGlyph)] = [ + ("pdf", RowGlyph(symbol: "doc.richtext", word: "PDF")), + ("image", RowGlyph(symbol: "photo", word: "image")), + ("video", RowGlyph(symbol: "film", word: "video")), + ("audio", RowGlyph(symbol: "waveform", word: "audio")), + ("folder", RowGlyph(symbol: "folder", word: "folder")), + ("site", RowGlyph(symbol: "globe", word: "site")), + ("markdown", RowGlyph(symbol: "doc.plaintext", word: "Markdown")), + ("archive", RowGlyph(symbol: "archivebox", word: "archive")), + ("text", RowGlyph(symbol: "doc.text", word: "text")), + ("other", RowGlyph(symbol: "doc", word: "file")), + ] + for (value, glyph) in cases { + XCTAssertEqual(RowGlyphs.type(value), glyph, value) + } + } + + func testUnknownAndMissingTypesFallBackToDoc() { + for value in [nil, "other", "spreadsheet", "PDF", ""] { + XCTAssertEqual(RowGlyphs.type(value), RowGlyph(symbol: "doc", word: "file"), value ?? "nil") + } + } + + func testStorageGlyphsMatchTheTable() { + XCTAssertEqual(RowGlyphs.storage("machine"), RowGlyph(symbol: "desktopcomputer", word: "on this tenant's machine")) + XCTAssertEqual(RowGlyphs.storage("cloud"), RowGlyph(symbol: "cloud", word: "in the cloud")) + } + + func testMissingAndUnknownStorageShowNoBadge() { + for value in [nil, "local", "bucket", ""] { + XCTAssertNil(RowGlyphs.storage(value), value ?? "nil") + } + } + + func testLinkGlyphsMatchTheTable() { + XCTAssertEqual(RowGlyphs.link("live"), RowGlyph(symbol: "dot.radiowaves.right", word: "live server")) + XCTAssertEqual(RowGlyphs.link("snapshot"), RowGlyph(symbol: "doc.on.doc", word: "snapshot")) + XCTAssertEqual(RowGlyphs.link("whatever"), RowGlyph(symbol: "doc.on.doc", word: "snapshot")) + } + + func testLockGlyph() { + XCTAssertEqual(RowGlyphs.lock, RowGlyph(symbol: "lock.fill", word: "login required")) + } + + func testEverySymbolResolvesAsAnSFSymbol() { + let names = [ + "doc.richtext", "photo", "film", "waveform", "folder", "globe", + "doc.plaintext", "archivebox", "doc.text", "doc", + "desktopcomputer", "cloud", "doc.on.doc", "dot.radiowaves.right", + "lock.fill", + ] + for name in names { + XCTAssertNotNil( + NSImage(systemSymbolName: name, accessibilityDescription: "test"), + "\(name) must resolve on the package minimum (macOS 13)" + ) + } + } +} diff --git a/mac/Tests/ShareBarCoreTests/SnapshotDecodeTests.swift b/mac/Tests/ShareBarCoreTests/SnapshotDecodeTests.swift index ff0235a..b1eff20 100644 --- a/mac/Tests/ShareBarCoreTests/SnapshotDecodeTests.swift +++ b/mac/Tests/ShareBarCoreTests/SnapshotDecodeTests.swift @@ -110,6 +110,66 @@ final class SnapshotDecodeTests: XCTestCase { XCTAssertEqual(snapshot.shares.count, 1) } + func testDecodesTheTenantFields() throws { + let snapshot = try loadFixtureTenantProfiles() + + let origin = try XCTUnwrap(snapshot.profiles.first { $0.name == "dfoundation" }?.state) + XCTAssertEqual(origin.r2, true) + XCTAssertEqual(origin.storageDefault, "local") + XCTAssertEqual(origin.cloudMore, 3) + XCTAssertNil(origin.cloudError) + + let cloudRow = origin.shares[0] + XCTAssertEqual(cloudRow.storage, "cloud") + XCTAssertEqual(cloudRow.type, "site") + XCTAssertEqual(cloudRow.by, "han-air") + XCTAssertNil(cloudRow.access) + + let gatedMachineRow = origin.shares[1] + XCTAssertEqual(gatedMachineRow.storage, "machine") + XCTAssertEqual(gatedMachineRow.type, "pdf") + XCTAssertEqual(gatedMachineRow.by, "mac-mini") + XCTAssertEqual(gatedMachineRow.access, "group:dwarves-ops") + + let liveRow = origin.shares[2] + XCTAssertEqual(liveRow.storage, "machine") + XCTAssertEqual(liveRow.kind, "live") + + let member = try XCTUnwrap(snapshot.profiles.first { $0.name == "files" }?.state) + XCTAssertEqual(member.backend, "r2") + XCTAssertEqual(member.r2, true) + XCTAssertEqual(member.cloudError, "the menu reads cloud links only with a stored token: share --profile files api-token") + } + + func testAbsentTenantFieldsDecodeAsNil() throws { + let snapshot = try loadFixtureProfiles() + let state = try XCTUnwrap(snapshot.profiles[1].state) + XCTAssertNil(state.r2) + XCTAssertNil(state.storageDefault) + XCTAssertNil(state.cloudError) + XCTAssertNil(state.cloudMore) + XCTAssertNil(state.shares[0].storage) + XCTAssertNil(state.shares[0].type) + XCTAssertNil(state.shares[0].by) + + // The tenant fixture's third profile carries no new keys either, so one decode + // run proves both fixture shapes keep working. + let legacy = try XCTUnwrap(try loadFixtureTenantProfiles().profiles.first { $0.name == "personal" }?.state) + XCTAssertNil(legacy.r2) + XCTAssertNil(legacy.shares[0].storage) + XCTAssertNil(legacy.shares[0].type) + XCTAssertNil(legacy.shares[0].by) + } + + func testAShareWithAnUnknownTypeStillDecodes() throws { + let share = try JSONDecoder().decode( + Share.self, + from: Data(#"{"id":"a1","name":"n","url":"u","kind":"snapshot","own_host":null,"expires":0,"type":"spreadsheet","storage":"cloud","by":"mini"}"#.utf8) + ) + XCTAssertEqual(share.type, "spreadsheet", "the raw value survives decode") + XCTAssertEqual(RowGlyphs.type(share.type), RowGlyph(symbol: "doc", word: "file"), "the row maps it to doc") + } + func testFixtureBuildsAWorkingMenuModel() throws { // Not a MenuModel-rules test (see MenuModelHeaderTests / SectionTests for those); // just confirms the decoded fixture flows end to end into a model. diff --git a/mac/Tests/ShareBarCoreTests/TestSupport.swift b/mac/Tests/ShareBarCoreTests/TestSupport.swift index 7027e98..2273e61 100644 --- a/mac/Tests/ShareBarCoreTests/TestSupport.swift +++ b/mac/Tests/ShareBarCoreTests/TestSupport.swift @@ -32,12 +32,18 @@ func makeSnapshot( service: Bool = true, shares: [Share] = [], skipped: Int? = nil, - accessPending: Int? = nil + accessPending: Int? = nil, + backend: String? = nil, + r2: Bool? = nil, + storageDefault: String? = nil, + cloudError: String? = nil, + cloudMore: Int? = nil ) -> Snapshot { Snapshot( schema: schema, state: state, ready: ready, mode: mode, host: host, hosts: hosts, servesHere: servesHere, service: service, shares: shares, skipped: skipped, - accessPending: accessPending + accessPending: accessPending, backend: backend, r2: r2, + storageDefault: storageDefault, cloudError: cloudError, cloudMore: cloudMore ) } @@ -50,9 +56,12 @@ func makeShare( kind: String = "snapshot", ownHost: String? = nil, expires: Int = 0, - access: String? = nil + access: String? = nil, + storage: String? = nil, + type: String? = nil, + by: String? = nil ) -> Share { - Share(id: id, name: name, url: url, kind: kind, ownHost: ownHost, expires: expires, access: access) + Share(id: id, name: name, url: url, kind: kind, ownHost: ownHost, expires: expires, access: access, storage: storage, type: type, by: by) } /// One profile entry: its `state`, or its `error` line (the JSON shape the CLI emits is @@ -91,6 +100,16 @@ func loadFixtureProfiles() throws -> ProfilesSnapshot { return try JSONDecoder().decode(ProfilesSnapshot.self, from: data) } +/// Loads the tenant-era `share profiles --json` fixture: an R2-on origin, an r2 member +/// carrying `cloud_error`, and an older-CLI-shaped tunnel profile with no new fields. +func loadFixtureTenantProfiles() throws -> ProfilesSnapshot { + guard let url = Bundle.module.url(forResource: "profiles-tenant", withExtension: "json") else { + throw NSError(domain: "TestSupport", code: 1, userInfo: [NSLocalizedDescriptionKey: "fixture not found"]) + } + let data = try Data(contentsOf: url) + return try JSONDecoder().decode(ProfilesSnapshot.self, from: data) +} + /// A plain (non-async) thread-safe string accumulator for tests that collect output streamed /// from a background queue via `onOutput`. Kept as ordinary synchronous methods, not an actor, /// so the lock calls are never made directly from an async context. diff --git a/tests/e2e-migrate.sh b/tests/e2e-migrate.sh new file mode 100755 index 0000000..faca2ed --- /dev/null +++ b/tests/e2e-migrate.sh @@ -0,0 +1,337 @@ +#!/bin/bash +# Live rehearsal R2 of SPEC-008: moving a tenant's origin with `share migrate`, on throwaway names, against a +# real Cloudflare account. Three legs, each with proof through the API that nothing is left: +# M0 a remote-setup failure after the switch, then the printed rollback: the hostname serves from A's tunnel again +# M1 the same name migrated again (B already holds the copies): every snapshot link answers from B at the same +# URL, the gated one keeps its 302 and AUD, the live link is reported as not moved, A's tunnel is gone and the +# DNS record points at B's tunnel; the 530 gap is measured and logged +# M2 (optional) a second throwaway name moved from a real second machine (SHARE_E2E_AIR_SSH) to this one over ssh +# Run it by hand before the personal move (P2). CI has no account or token. TASK-9b of SPEC-008. +# +# SHARE_E2E_TENANT_HOST=share-e2e-m1.example.com SHARE_E2E_AIR_HOST=share-e2e-m2.example.com \ +# CLOUDFLARE_API_TOKEN= SHARE_E2E_ACCESS_RULE=group: SHARE_E2E_AIR_SSH=air \ +# SHARE_E2E_MINI_TARGET=mini-tieubao SHARE_BIN=bin/share tests/e2e-migrate.sh +# +# Inputs: +# SHARE_E2E_TENANT_HOST an unused hostname whose first label starts with share-e2e, on a zone the admin token edits +# SHARE_E2E_AIR_HOST a second such name in the same zone; with SHARE_E2E_AIR_SSH unset the M2 leg is skipped +# CLOUDFLARE_API_TOKEN the admin token: Tunnel Edit, DNS Edit, Zone Read, Access Apps Edit, Access Organizations Read +# SHARE_E2E_ACCESS_RULE group:, email:, or domain:: the gated snapshot (without it the gated checks SKIP) +# SHARE_E2E_AIR_SSH the ssh alias of the machine that stands in for the Air; it must reach SHARE_E2E_MINI_TARGET +# SHARE_E2E_MINI_TARGET how that machine reaches THIS one over ssh (a Host alias or user@host) +# The script refuses a hostname without the share-e2e prefix, so no real tenant is reachable. Every leg runs under +# mktemp HOMEs and its own share profiles (mga, mgb), so a real login service on either machine is never touched. +# A stub `security` first on PATH keeps the real Keychain out of it. Tokens reach share through the environment and +# stdin, never argv. It tests the share it is pointed at: SHARE_BIN (default: share on PATH). +set -uo pipefail + +tenant="${SHARE_E2E_TENANT_HOST:?set SHARE_E2E_TENANT_HOST to an unused share-e2e* hostname}" +air_host="${SHARE_E2E_AIR_HOST:-}" +admin="${CLOUDFLARE_API_TOKEN:?set CLOUDFLARE_API_TOKEN to the admin token}" +rule="${SHARE_E2E_ACCESS_RULE:-}" +air_ssh="${SHARE_E2E_AIR_SSH:-}" +mini_target="${SHARE_E2E_MINI_TARGET:-}" +share="${SHARE_BIN:-share}" +for h in "$tenant" ${air_host:+"$air_host"}; do + [[ ${h%%.*} == share-e2e* ]] || { echo "refused: hostname '$h' must have a first label starting with share-e2e"; exit 2; } +done +[[ -z $air_host || ${tenant#*.} == "${air_host#*.}" ]] || { echo "refused: $tenant and $air_host must share one zone"; exit 2; } +[[ -z $air_ssh || ( -n $air_host && -n $mini_target ) ]] || { echo "refused: SHARE_E2E_AIR_SSH needs SHARE_E2E_AIR_HOST and SHARE_E2E_MINI_TARGET"; exit 2; } +for c in jq curl python3 cloudflared caddy; do command -v "$c" >/dev/null || { echo "missing: $c"; exit 2; }; done +# the M2 leg's target side runs under this account's own HOME (ssh gives it no other), so its profile must not exist yet +[[ -z $air_ssh || ! -e $HOME/.config/share/profiles/mgb ]] || { echo "refused: $HOME/.config/share/profiles/mgb already exists; the M2 leg needs that profile name free"; exit 2; } +command -v "$share" >/dev/null || { echo "missing: $share on PATH (or set SHARE_BIN)"; exit 2; } +share="$(command -v "$share")"; share="$(cd "$(dirname "$share")" && pwd)/$(basename "$share")" +unset SHARE_CONFIG_DIR SHARE_ROOT SHARE_PROFILE XDG_CONFIG_HOME SHARE_HOSTNAME SHARE_BACKEND SHARE_R2_TOKEN SHARE_R2_DRY SHARE_R2_KEY_ID SHARE_PORT +export SHARE_CLIPBOARD=0 + +WORK=$(mktemp -d) +mkdir -p "$WORK/bin" "$WORK/kc" "$WORK/failbin" +# A file-backed stand-in for the Keychain: a real tunnel setup needs a real round trip (store the tunnel token, read it +# back to start cloudflared), and the real Keychain prompts on a first write from a script with no one to click it, and +# is locked in a plain ssh session. `security -i` batch-adds, `add-generic-password` is migrate's own probe, `find-` reads. +cat >"$WORK/bin/security" <<'PY' +#!/usr/bin/env python3 +import sys, os, re +store = os.environ.get("SHARE_E2E_KC_STORE") or os.path.join(os.path.dirname(os.path.realpath(__file__)), "..", "kc") +def path(key): return os.path.join(store, re.sub(r"[^A-Za-z0-9_.-]", "_", key)) +args = sys.argv[1:] +def opt(flag): return args[args.index(flag) + 1] if flag in args and args.index(flag) + 1 < len(args) else "" +if args[:1] == ["-i"]: + line = sys.stdin.readline() + m_s = re.search(r'-s\s+"([^"]*)"', line); m_w = re.search(r'-w\s+"([^"]*)"', line) + if m_s and m_w: + with open(path(m_s.group(1)), "w") as f: f.write(m_w.group(1)) + sys.exit(0) +elif args[:1] == ["add-generic-password"]: + with open(path(opt("-s")), "w") as f: f.write(opt("-w")) + sys.exit(0) +elif args[:1] == ["find-generic-password"]: + p = path(opt("-s")) + if os.path.isfile(p): + with open(p) as f: sys.stdout.write(f.read()) + sys.exit(0) + sys.exit(44) +elif args[:1] == ["delete-generic-password"]: + p = path(opt("-s")) + if os.path.isfile(p): os.remove(p); sys.exit(0) + sys.exit(44) +sys.exit(44) +PY +chmod +x "$WORK/bin/security" +export PATH="$WORK/bin:$PATH" SHARE_E2E_KC_STORE="$WORK/kc" +# the remote side runs this share through a wrapper, so the one remote bin dir holds the stub and the binary under test +printf '#!/bin/bash\nexec bash "%s" "$@"\n' "$share" >"$WORK/bin/share"; chmod +x "$WORK/bin/share" +# M0's remote bin: the real setup runs (the switch happens), then the wrapper reports a failure +cp "$WORK/bin/security" "$WORK/failbin/security" +# shellcheck disable=SC2016 # the wrapper's own $@ and $rc stay literal +printf '#!/bin/bash\nbash "%s" "$@"; rc=$?\nfor a in "$@"; do [[ $a == setup ]] && exit 1; done\nexit $rc\n' "$share" >"$WORK/failbin/share" +chmod +x "$WORK/failbin/share" + +RUNLOG="$WORK/run.log" +VERIF_DIR="$(cd "$(dirname "$0")/.." && pwd)/docs/verification" +mkdir -p "$VERIF_DIR" +RUNLOG_KEEP="$VERIF_DIR/e2e-migrate-$(date -u +%Y%m%dT%H%M%SZ).log" +exec > >(tee "$RUNLOG") 2>&1 + +fails=0 total=0 +check() { # check