[krokicki]

Publishes each running app service at a per-job HTTPS subdomain — https://job-123.services.int.janelia.org/lab?token=abc — which nginx proxies to the compute node. Stacked on #437.

Deployment half: JaneliaSciComp/fileglancer-hub#15 (the nginx vhost). Both must land; each is inert without the other.

This ships dark. With apps.service_proxy_domain unset (the default), the published URL is byte-for-byte what it is today. Nothing changes until the DNS and certificate work below lands.

Why

A service job publishes http://<node>:<port>/...?tkn=<token> and we render that as the "Open Service" link. Two problems: Fileglancer is served over HTTPS, so the link is mixed content with the token in cleartext, and the hostname and port change on every launch, so the link is neither memorable nor shareable.

Reachability is explicitly not a problem being solved — users can already reach compute nodes directly, and still can.

How

nginx terminates TLS for a wildcard zone and asks Fileglancer which upstream a hostname maps to, via auth_request. Fileglancer's whole contribution is one endpoint, so no proxied bytes pass through Python — which matters because one of the shipped apps is a noVNC remote desktop, i.e. a video stream.

Subdomains rather than a path prefix because a prefix would need every app told its base path, and openvscode-server has no documented subpath configuration. That trades a one-time DNS and certificate request for an open-ended per-app debugging tail.

Design doc: docs/superpowers/specs/2026-08-28-app-service-https-proxy-design.md. Operator guide: docs/ServiceProxy.md.

What's in it

• JobDB.service_url caches the URL a service publishes, written through where the job detail endpoint already reads it. Migration c3e9b7f41a28, one nullable column.
• GET /api/apps/resolve maps a job-<id>.<zone> hostname to a validated host:port, or 403s. Unauthenticated by design: the session cookie is scoped to the Fileglancer hostname and never reaches an app subdomain, so there is no user to check. nginx marks it internal.
• fileglancer/apps/serviceproxy.py holds the URL translation and the upstream validation.
• Successful resolutions sit in a bounded TTL cache and the endpoint is excluded from per-request access logging, reporting aggregate totals once a minute instead.

Frontend: no changes. JobDetail.tsx renders whatever service_url contains.

New settings, all empty by default

apps:
  service_proxy_domain: services.int.janelia.org    # public zone browsers hit
  service_proxy_upstream_zone: int.janelia.org      # confines name upstreams
  service_proxy_upstream_networks:                  # confines address upstreams
    - 10.20.0.0/16

The two upstream settings divide the space rather than overlapping: a hostname has no address to check without a DNS lookup (which this path must never do — it runs once per proxied request), and a bare address has no zone to match.

Security notes worth a reviewer's attention

The upstream is user-controlled. service_url is a file the user's own job writes, and an auto_url: false app writes it deliberately. Review found that a user could plant http://127.0.0.1:8989/ and aim nginx at the app server's own loopback, or 169.254.169.254 for instance metadata. Fixed by refusing hosts that reach the Fileglancer host itself — loopback, unspecified, link-local, multicast, reserved — plus the two optional allowlists above.

Private and public node addresses are deliberately allowed: an address bound to a routable interface is already reachable directly by any cluster user, so proxying to it grants nothing new. Blocking RFC 1918 would break real deployments to prevent nothing.

The always-refused set is not an allowlist opt-out. Naming 127.0.0.0/8 in service_proxy_upstream_networks still will not make loopback dialable.

internal; and the main-block return 404 are load-bearing. The resolve endpoint discloses a running service's host:port for any job id, and job ids are sequential. Both nginx directives keep it off the network; docs/ServiceProxy.md says not to remove either.

Per-job subdomains make a running service cheaper to find — https://job-<small number>.<zone>/ instead of port-scanning nodes — and the vhost is reachable from wherever the Fileglancer HTTPS host is. The service's own token is still the only credential, unchanged from today, but worth weighing before exposing the vhost widely.

Turning it on

Nothing changes until apps.service_proxy_domain is set, and that needs a wildcard DNS record, a *.services.int.janelia.org certificate and the nginx vhost first — none of which live in this repo.

The deployment checklist lives in JaneliaSciComp/fileglancer-hub#15, kept in one place so it does not drift, and because that PR stays open until the deploy is finished while this one merges as soon as review passes.

Two ordering constraints worth knowing here, since they are about this PR rather than about nginx:

• A fileglancer release containing this PR has to be cut and pinned in the hub's pixi.toml (currently ==3.1.0, which has neither the endpoint nor the settings). The nginx block 502s until then.
• The existing *.int.janelia.org certificate does not cover job-1.services.int.janelia.org: a wildcard matches exactly one label, and browsers enforce that strictly. A new certificate is required, not just a DNS record.

Testing

1011 backend tests pass. The suite covers the resolve endpoint's refusal paths, the raw-vs-proxied URL invariant end to end, adversarial upstream inputs (CRLF, userinfo, bracketed IPv6, out-of-range ports, homograph hosts, loopback and metadata targets), the cache's staleness window, and the access-log exclusion.

Known limitations, recorded rather than hidden

• An IPv6 literal can never be an upstream: the authority pattern forbids colons in the host, so IPv6 entries in service_proxy_upstream_networks parse but match nothing.
• A service that manages its own URL (auto_url unset) should write its URL file exactly once. The cached upstream refreshes only while someone has the job detail page open, so a URL that changes mid-run can go stale.
• The upstream cache's TTL is the window in which a just-stopped job can still be proxied. Kept to seconds deliberately, since compute-node ports get recycled.

@StephanPreibisch @JaneliaSciComp/fileglancer

[krokicki]

Deployment half: JaneliaSciComp/fileglancer-hub#15 — the nginx vhost, with the same deployment checklist. Neither PR does anything without the other.

[mkitti]

The last hop is still cleartext — and two existing apps break on it

This encrypts browser→nginx. The nginx→node hop stays plain HTTP, so ?tkn= still crosses the cluster network in the clear, and docs/ServiceProxy.md reads as though the whole path is TLS.

This is a prerequisite for setting apps.service_proxy_domain, not a follow-up. Two Fileglancer apps already serve TLS with self-signed certs behind Caddy, and both already write https:// to $SERVICE_URL_PATH:

• JaneliaSciComp/marimo_ai_sandbox — container/caddy-lib.sh generates an openssl self-signed leaf (SANs: node FQDN, short hostname, localhost, host IPs); container/https-wrap.sh:262 publishes https://<host>:<port>/?access_token=…. It bypasses auto_url specifically so Fileglancer doesn't publish the backend's plain-HTTP port instead of Caddy's TLS one.
• JaneliaSciComp/mhs-spec-review#1 — Caddy tls internal on a fixed port 8443, publishes https://$(hostname -f):8443/.

Since upstream_from_service_url keeps only host:port, enabling the proxy makes nginx do proxy_pass http://<node>:8443 at a TLS listener, and Caddy answers a plaintext request to an HTTPS port with a 400. Both apps break the day the proxy is turned on. The auto_url-unset bullet in the residual-risks section treats self-managed URLs as a staleness footnote; the dropped scheme is the more immediate consequence and should be named there.

Certificates are a separate discussion (issue to follow). The scheme plumbing itself is small and needs no PKI decisions:

1. Stop discarding the scheme. upstream_from_service_url returns bare netloc. Apps already declare it (AuthoringApps.md:602, and the two above), so it's a free per-job signal. Return it too, validated against a hard {http, https} enumeration. Per-job matters: TensorBoard has no TLS option, so no global switch works.

2. A second header, not a wider regex. X-Fg-Upstream-Scheme: https alongside the existing header. _UPSTREAM_RE is the injection gate for proxy_pass interpolation — an enumerated scheme in its own header keeps it as strict as it is today.

   Watch the cache: cached_upstream stores a bare str and the hit path returns it directly. Add the scheme on the miss path only and every cached hit downgrades to http, i.e. an HTTPS service flaps between encrypted and cleartext on a 10s cycle. The cached value has to carry both.

3. Count plaintext upstreams; don't add a policy setting. No new config — the behavior should simply be "use the scheme the app published." An operator still wants to know how much of the traffic is cleartext, and record_resolve already aggregates outcomes into one log line a minute, so splitting hits into hit / hit_plaintext answers that at zero risk of breaking a launch.

   I floated an apps.service_proxy_upstream_tls: allow | require setting here earlier. Withdrawn: require would 403 every non-HTTPS upstream, which is exactly what we don't want given TensorBoard has no TLS option at all, and with require gone the setting has one legal value and stops earning its place in a config surface this PR already grows by three.

nginx half: JaneliaSciComp/fileglancer-hub#15. Its per-app verification checklist lists only the five shipped service apps — both Caddy apps should be on it, since they're the only ones that exercise the TLS path at all.

One gap this exposes beyond TLS: mhs-spec-review pins its OAuth callback to a fixed host and port, so republishing it at a job subdomain likely breaks the callback regardless of the scheme. There's currently no way for a manifest to say "don't proxy me." Probably its own issue.

[krokicki]

@mkitti Per our discussion, we'll treat support for self-signed certs as a separate nice to have.