Sign service proxy hostnames so they can't be enumerated

docs/ServiceProxy.md

@@ -1,6 +1,12 @@
 # HTTPS Proxy for App Services
 
-Running app services (`type: service`) bind a port on a compute node and publish `http://<node>:<port>/...`. Set `apps.service_proxy_domain` to republish them over HTTPS at a per-job subdomain instead.
+Running app services (`type: service`) bind a port on a compute node and publish `http://<node>:<port>/...`. Set `apps.service_proxy_domain` to republish them over HTTPS at a per-job subdomain instead:
+
+```
+https://job-12-k7m2qhxr.services.example.org/lab?token=8f2c...
+```
+
+The `job-<id>-<mac>` label is signed with `session_secret_key`. Job ids are a small global sequence, so an unsigned `job-12` label would let anyone who can reach the proxy sweep `job-1`..`job-500` and find — and, for any service that does not enforce its own token, reach — every running service on the instance. The MAC is 8 base32 characters (40 bits); guessing it is online-only against a reverse proxy that answers 403, so a sweep of that space takes decades at 10k requests/second.
 
 Design rationale: `docs/superpowers/specs/2026-08-28-app-service-https-proxy-design.md`.
 
@@ -21,22 +27,25 @@ Design rationale: `docs/superpowers/specs/2026-08-28-app-service-https-proxy-des
 ## Fileglancer configuration
 
 ```yaml
+session_secret_key: "<a long random string>"
 apps:
   service_proxy_domain: "services.example.org"
 ```
 
-Leave it empty to disable; the direct `http://<node>:<port>` URL is then published unchanged.
+Leave `service_proxy_domain` empty to disable; the direct `http://<node>:<port>` URL is then published unchanged.
+
+`session_secret_key` is required when the proxy domain is set, and the server refuses to start without it. An unset key is generated at random per process, so under `uvicorn --workers N` each worker would sign hostnames with a different key and most proxied requests would be refused. Rotating it invalidates live service URLs, on top of the session revocation rotation already causes.
 
 ## Reverse proxy configuration
 
-Fileglancer does not proxy the traffic itself. It exposes `GET /api/apps/resolve`, which reads the `Host` header and answers `204` with `X-Fg-Upstream: <host>:<port>`, or `403`. The reverse proxy resolves each request through it and connects to the upstream directly, so no proxied bytes pass through the application server.
+Fileglancer does not proxy the traffic itself. It exposes `GET /api/apps/resolve`, which reads the `Host` header and answers `204` with `X-Fg-Upstream: <host>:<port>` and `X-Fg-Upstream-Scheme: http|https`, or `403`. The reverse proxy resolves each request through it and connects to the upstream directly, so no proxied bytes pass through the application server.
 
 Add a server block for the wildcard zone. This assumes a `map $http_upgrade $connection_upgrade` block already exists at the http level:
 
 ```nginx
 server {
   listen 443 ssl http2;
-  server_name ~^job-\d+\.services\.example\.org$;
+  server_name ~^job-\d+-[a-z2-7]+\.services\.example\.org$;
 
   ssl_certificate     /etc/nginx/certs/services-wildcard.crt;
   ssl_certificate_key /etc/nginx/certs/services-wildcard.key;
@@ -52,11 +61,18 @@ server {
   location / {
     auth_request     /_fg_resolve;
     auth_request_set $upstream $upstream_http_x_fg_upstream;
+    auth_request_set $upscheme $upstream_http_x_fg_upstream_scheme;
 
     # Required because proxy_pass targets a variable. Use whatever resolver the
     # host actually runs; 127.0.0.53 is systemd-resolved's stub.
     resolver 127.0.0.53 valid=30s;
-    proxy_pass http://$upstream;
+    proxy_pass $upscheme://$upstream;
+
+    # An app that terminates TLS does so with a certificate it generated on the
+    # compute node, so there is no trust anchor to verify against. See "TLS to
+    # the app host" below. Leave proxy_ssl_name at its default.
+    proxy_ssl_verify      off;
+    proxy_ssl_server_name on;
 
     proxy_set_header Host              $host;
     proxy_set_header X-Real-IP         $remote_addr;
@@ -70,6 +86,17 @@ server {
     proxy_buffering    off;
     proxy_read_timeout 3600s;
   }
+
+  # A refused resolution means the service is gone, not that the user did
+  # something wrong, so serve Fileglancer's explanation instead of nginx's
+  # stock 403 page. The `=` makes the response carry the page's own 503.
+  error_page 403 = /_fg_unavailable;
+
+  location = /_fg_unavailable {
+    internal;
+    proxy_pass       http://127.0.0.1:8989/api/apps/service-unavailable;
+    proxy_set_header Host $host;
+  }
 }
 ```
 
@@ -79,15 +106,28 @@ Also add this to the **main** server block, so the resolve endpoint is not reach
   location = /api/apps/resolve { return 404; }
 ```
 
-Four details are load-bearing:
+Seven details are load-bearing:
 
 - **`internal;`** on the `/_fg_resolve` location makes it reachable only from nginx's own `auth_request` subrequest, never from a client. Together with the `return 404` in the main server block, it is what keeps the unauthenticated resolve endpoint off the network. Do not remove either.
 - **`proxy_set_header Host $host`** passes the app subdomain through unchanged, so the app sees `Host` and `Origin` as the same value. This is what makes JupyterLab's WebSocket origin check pass without per-app configuration.
 - **`resolver`** is mandatory. Without it nginx refuses to start when `proxy_pass` targets a variable.
 - **`proxy_buffering off`** and the long `proxy_read_timeout` suit long-lived WebSocket and streaming sessions, such as the remote desktop app.
+- **`proxy_pass $upscheme://$upstream`**, rather than a hardcoded `http://`, is what lets an app that serves HTTPS be reached at all. Sending a plaintext request to a TLS listener does not silently lose encryption — the app rejects it — so hardcoding the scheme breaks those apps outright.
+- **`proxy_intercept_errors` must stay off** (its default) for the `error_page 403` above to mean what it says. The 403 it catches is the one nginx generates when `auth_request` is denied; turning interception on would also catch a 403 from the app itself — a JupyterLab token rejection, say — and replace it with the "503 Service Unavailable" page.
+- **`/_fg_unavailable` is a prefix location, not a named one**, because nginx refuses a `proxy_pass` with a URI part inside a named location (`proxy_pass cannot have URI part in location given by regular expression, or inside named location`). `internal;` is what keeps it out of the URL space the app sees, so a request for that path gets a 404 rather than the error page.
 
 The existing HTTP-to-HTTPS redirect block is typically `default_server` with `server_name _`, in which case it already covers the new subdomains.
 
+## TLS to the app host
+
+The hop from the browser to the reverse proxy is always HTTPS. The hop from the proxy to the compute node is whatever the service itself published: Fileglancer reports the scheme from the service's own URL file and the proxy dials it with that. An app that fronts itself with a TLS terminator — Caddy, stunnel, its own `--certfile` — is reached over HTTPS; an app that serves plain HTTP is reached over HTTP, and nothing is required of it.
+
+**This is opportunistic encryption, and it authenticates nothing.** `proxy_ssl_verify` is `off` because the certificates in question are generated on the compute node at launch and signed by nobody: there is no trust anchor to check them against, and the node's name and port change every launch, so there is nothing stable to pin either. What it buys is that the service's token stops crossing the node network in cleartext. What it does not buy is any assurance that the thing answering on that host and port is the service — the upstream is read from a file the user's own job wrote, so the proxy could not make that claim regardless of certificates.
+
+Leave `proxy_ssl_name` at its default, which is the host from `proxy_pass`. Apps that generate their own certificate name it after the compute node, so overriding this to `$host` (the `job-<id>` subdomain) would send an SNI value no app's certificate carries.
+
+To see how much traffic is still cleartext, read the `plaintext` figure in the aggregate log line described below. It counts alongside `hit` and `miss` rather than instead of them, so it reads as a fraction of the total.
+
 ## Verification
 
 Once DNS and the certificate are in place, launch each service app and confirm it loads and stays connected. WebSocket behavior is the thing to watch:
@@ -103,7 +143,7 @@ If an app rejects the proxied origin, fix it in that app's manifest (most server
 ## Residual risks
 
 - Set `apps.service_proxy_upstream_zone` to the DNS zone your compute nodes live in, e.g. `nodes.example.org`. Without it the proxy will dial any host the Fileglancer host can reach, because the upstream comes from a file the user's job wrote. Loopback, the unspecified address, link-local (including cloud instance metadata), multicast and reserved addresses are always refused, since those are what reach the Fileglancer host itself. Private and public node addresses are allowed — an address on a routable interface is already reachable directly by any cluster user. Matching is on whole DNS labels, so a leading dot is optional and a sibling zone like `evil-nodes.example.org` does not qualify. The zone applies to hostnames only: a service that publishes the node's IP instead of its name is still accepted, so setting a zone will not break one. To confine those, set `apps.service_proxy_upstream_networks` to the CIDR networks your nodes occupy, e.g. `10.20.0.0/16`. The two settings divide the space between them — the zone governs upstreams published as names, the networks govern upstreams published as addresses — and each is empty by default, meaning no restriction on that form. Bad CIDR entries are refused at startup rather than at request time, since a typo there would otherwise reject every address upstream while looking configured.
-- Per-job subdomains make a running service much cheaper to *find* — `https://job-<small number>.<zone>/` instead of port-scanning compute nodes — and the proxy vhost is reachable from wherever the Fileglancer HTTPS host is. The service's own token is still the only credential, but weigh this before exposing the vhost on a wide network.
+- The signed hostname is not a substitute for a service enforcing its own token. It is unguessable, but a hostname leaks where a query string does not: plaintext SNI on the wire, DNS resolvers, and the proxy's own access log. Treat it as what makes enumeration infeasible, and `${FG_SERVICE_TOKEN}` as the credential. An app with no authentication of its own (TensorBoard, for one) is protected only by the label.
 - The resolve endpoint is called once per proxied HTTP request, so a single page load of an app like JupyterLab generates dozens. Successful resolutions are cached in-process for 10 seconds, which collapses that burst to roughly one database read per service per 10 seconds per worker. Refusals are deliberately not cached, so a service starts resolving the moment it publishes its URL. The endpoint is excluded from the per-request access log for the same reason and reports running totals once a minute instead — grep for `service proxy resolve totals` to see hits, misses and refusals by reason.
 - That 10-second cache is also the window in which a job that has just stopped can still be proxied. Compute-node ports get recycled, so the window is kept short deliberately; if a port is reused within it, a client can briefly reach the new occupant, which will reject it for lack of that service's own token.
-- A service that manages its own URL (`auto_url` unset) should write its URL file exactly once. The cached upstream is refreshed only while someone has the job's detail page open, so a URL that changes mid-run can go stale.
+- A service that manages its own URL (`auto_url` unset) should write its URL file exactly once, and should write the scheme it actually serves. Both the upstream and its scheme are taken from that file: publishing `http://` for a listener that speaks TLS, or the reverse, produces a hop that fails rather than one that merely works unencrypted. The cached upstream is refreshed only while someone has the job's detail page open, so a URL that changes mid-run can go stale.

docs/config.yaml.template

@@ -197,10 +197,11 @@ session_cookie_secure: true
 #                               # Wildcard DNS zone serving per-job HTTPS
 #                               # subdomains for running services. When set, a
 #                               # service is published at
-#                               # https://job-<id>.<zone>/ instead of its own
-#                               # http://<node>:<port>/ URL. Needs a wildcard
-#                               # certificate for the zone and a reverse proxy
-#                               # resolving upstreams via /api/apps/resolve;
+#                               # https://job-<id>-<mac>.<zone>/ instead of its
+#                               # own http://<node>:<port>/ URL. Needs a wildcard
+#                               # certificate for the zone, a reverse proxy
+#                               # resolving upstreams via /api/apps/resolve, and
+#                               # session_secret_key (which signs the hostname);
 #                               # see docs/ServiceProxy.md. Empty = direct URL.
 #   service_proxy_upstream_zone: nodes.example.org
 #                               # DNS zone the proxy is allowed to dial, i.e.