[krokicki]

Deployment half of the app-service HTTPS proxy. The application half is JaneliaSciComp/fileglancer#440 — both must land, and this one is inert on its own.

What it does

Adds a vhost for job-<id>.services.int.janelia.org. A browser opens a running app service (JupyterLab, marimo, OpenVSCode, TensorBoard, the noVNC desktop) at that name over HTTPS; nginx asks Fileglancer which compute node and port the job is serving on, then proxies straight to the node.

The upstream is dynamic — a new node and port on every launch — so it is resolved per request through auth_request against Fileglancer's /api/apps/resolve, which answers 204 with X-Fg-Upstream: <host>:<port> or 403. Fileglancer never carries the proxied traffic itself, which matters because the desktop app is effectively a video stream.

Why the browser can't just keep using the direct link

Today a service publishes http://<node>:<port>/...?tkn=<token>. Fileglancer is served over HTTPS, so that link is mixed content with the token in cleartext, and the host and port change on every launch. Reachability is not the problem being solved — nodes stay directly reachable.

Two directives are load-bearing, please don't drop them in review

• internal; on location = /_fg_resolve. The resolve endpoint is unauthenticated by design (the session cookie is scoped to the Fileglancer hostname and never reaches an app subdomain, so there is no user to check). It will resolve any running service's host:port for a guessable, sequential job id. internal is what makes it reachable only from nginx's own subrequest.
• location = /api/apps/resolve { return 404; } in the primary server block, so the same endpoint is not exposed on fileglancer.int.janelia.org either.

Also worth knowing:

• resolver is mandatory, not decorative. nginx refuses to start when proxy_pass targets a variable and no resolver is configured. It is set to 127.0.0.53 (systemd-resolved's stub) — change it if this host runs something else.
• proxy_set_header Host $host is what makes JupyterLab's WebSocket origin check pass: the app then sees Host and Origin as the same value, so no per-app configuration is needed.
• proxy_buffering off and the 3600s read timeout are for long-lived WebSocket sessions; notebooks and the desktop hold connections open for hours.

Verified

nginx -t passes with this block in place. Because nginx.conf here is an http{} fragment rather than a standalone config, I validated it by including it in a wrapper config with throwaway self-signed certs and unprivileged ports — so the auth_request, internal and resolver directives are confirmed valid in their contexts, not merely plausible.

Not verified, and not verifiable until DNS and the certificate exist: whether each app actually behaves behind the proxy. That is step 5 below.

Deployment checklist

• @neomorphic Add the wildcard DNS record. No zone delegation needed — a wildcard may sit at any level (RFC 4592), so one record in the existing int.janelia.org zone is enough:
  *.services IN A <fileglancer host IP>
  A wildcard never matches its own parent, so this does not make services.int.janelia.org itself resolve. Add a plain services IN A record if we want the bare alias; the proxy does not need it.
• @neomorphic Request a *.services.int.janelia.org certificate. The existing *.int.janelia.org wildcard does not cover job-1.services.int.janelia.org — a wildcard matches exactly one label and browsers enforce that strictly. Confirm with whoever issues certs that a nested wildcard is acceptable under our PKI policy; public CAs issue them routinely, internal policies are sometimes fussier about depth.
• @neomorphic Install the certificate at /etc/nginx/certs/services-wildcard.{crt,key} on the dev server, or change the paths in this block to match wherever it lands.
• Confirm the resolver address matches what the host actually runs.
• Release a fileglancer version containing #440 and bump the pin in pixi.toml (currently ==3.1.0, which has neither the endpoint nor the settings). This block 502s until that is done, since /api/apps/resolve will not exist.
• Set apps.service_proxy_domain: services.int.janelia.org in the Fileglancer config.
• Confine where the proxy may dial, using apps.service_proxy_upstream_zone and/or apps.service_proxy_upstream_networks. Worth thinking about: our nodes are <host>.int.janelia.org, so the zone setting would have to be int.janelia.org, which permits every internal host rather than just compute nodes. The CIDR setting is probably the more useful control here. This matters because a service publishes its own upstream from a file its job writes — see the security notes on #440.
• nginx -t, then reload.
• Test each service app end to end. WebSockets are where a proxied origin usually breaks:
  • JupyterLab — kernel connects, a cell executes
  • marimo — notebook is interactive, not stuck "connecting"
  • OpenVSCode — editor loads, a terminal opens
  • Remote Desktop — noVNC canvas renders and accepts input
  • TensorBoard — plots load
• Check the aggregate log line for refusals: grep 'service proxy resolve totals'. A misconfigured zone or CIDR shows up there as refused_no_upstream rather than as user complaints.

If an app rejects the proxied origin, fix it in that app's manifest (most servers have an allowed-origin or base-URL option) rather than weakening this configuration.

Related

• Application code, settings and operator guide: HTTPS proxy for app services fileglancer#440
• Full design rationale: docs/superpowers/specs/2026-08-28-app-service-https-proxy-design.md in that PR
• Operator reference, which this block is copied from and must stay in sync with: docs/ServiceProxy.md in that PR

@StephanPreibisch @JaneliaSciComp/fileglancer

[krokicki]

Application half: JaneliaSciComp/fileglancer#440 — the resolve endpoint, settings and operator guide this block depends on. This vhost 502s until a fileglancer release containing it is pinned here.

[neomorphic]

Looks good to me. There are a few minor quibbles that I noted.

[mkitti]

Follow-up: proxy_pass http://$upstream leaves the last hop cleartext

Not a blocker — but browser→nginx is TLS while nginx→node is not, so the service token still crosses the cluster network in the clear. Application half of this is JaneliaSciComp/fileglancer#440 (a X-Fg-Upstream-Scheme header); certificates are a separate issue. The nginx change:

auth_request_set $upstream  $upstream_http_x_fg_upstream;
auth_request_set $upscheme  $upstream_http_x_fg_upstream_scheme;
proxy_pass $upscheme://$upstream;

nginx re-parses the proxy_pass URL per request when it contains variables and honors an https:// prefix there — but that's the one load-bearing assumption, so confirm it on this build rather than trusting it. Portable fallback if not: two locations dispatched via error_page 418 = @tls_upstream on the scheme value.

Initially pair with proxy_ssl_verify off and proxy_ssl_server_name on: opportunistic encryption, no upstream authentication. That's not a regression — nothing authenticates the upstream today either, and it's already a string the user's job wrote — but it has to be documented as exactly that, not as verified TLS.

Also worth a line in the deployment checklist: which hop is encrypted at each stage, since the per-app verification steps will be run against a mixed fleet (Jupyter can serve TLS, TensorBoard can't).