Skip to content

feat(netproxy): HTTP CONNECT proxy dialer + registry client WithDialer - #50

Merged
TeoSlayer merged 4 commits into
mainfrom
feat/netproxy-connect-dialer
Sep 23, 2026
Merged

TeoSlayer merged 4 commits into
mainfrom
feat/netproxy-connect-dialer

Conversation

@TeoSlayer

@TeoSlayer TeoSlayer commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

What

Two additions so pilot components can run where the only way out is an authenticating HTTP CONNECT proxy:

  1. New package netproxy: proxy resolution plus an HTTP CONNECT dialer.
  2. registry/client WithDialer option: every registry connection (primary, pool members, reconnects, BinaryClient) can be opened through a caller-supplied dial function, for example a netproxy.Dialer.

Why

In Meta Muse agent VMs the only egress is an authenticating HTTPS_PROXY that allows CONNECT host:443 and nothing else. UDP is blocked, direct TCP is killed, and DNS for *.pilotprotocol.network is poisoned to 198.18.x.x. The released v1.13.9 daemon installs fine there, but it never registers: it dials the registry directly. This PR gives web4 the pieces to send the registry (TLS over TCP) and HTTP/WSS traffic through the proxy. The target is kept as a host name all the way to the proxy and is never resolved locally, and TLS stays end-to-end with the real server.

API

package netproxy

const ModeAuto = "auto"; const ModeOff = "off"; const ModeExplicit = "explicit"
const DefaultTimeout = 30 * time.Second

type Resolver struct{ /* opaque, immutable; nil never proxies */ }
func Parse(spec string) (*Resolver, error)        // "" / "auto" -> FromEnvironment, "off" -> Off, else Explicit
func FromEnvironment() (*Resolver, error)         // HTTPS_PROXY/https_proxy/ALL_PROXY/all_proxy (+HTTP_PROXY/http_proxy for http://), NO_PROXY
func Explicit(proxyURL string) (*Resolver, error) // http(s)://[user:pass@]host[:port]; proxies every target
func Off() *Resolver
func (r *Resolver) Mode() string
func (r *Resolver) Enabled() bool
func (r *Resolver) String() string                // log-safe: "auto: http://***@proxy:3128 (NO_PROXY=...)"
func (r *Resolver) Warnings() []error             // env vars skipped as unusable, one *EnvError each
func (r *Resolver) ProxyForAddr(addr string) (*url.URL, error)
func (r *Resolver) ProxyForRequest(req *http.Request) (*url.URL, error) // http.Transport.Proxy
func Redact(u *url.URL) string                    // "http://***@host:port"
type EnvError struct{ Var string; Err error }

type Dialer struct {
    Resolver  *Resolver
    Timeout   time.Duration // 0 = DefaultTimeout
    Forward   func(ctx context.Context, network, addr string) (net.Conn, error)
    TLSConfig *tls.Config   // for https:// proxies only
}
func NewDialer(r *Resolver) *Dialer
func (d *Dialer) DialContext(ctx context.Context, network, addr string) (net.Conn, error)
func (d *Dialer) Dial(network, addr string) (net.Conn, error)
type ConnectError struct{ Target string; StatusCode int }
package client // registry/client

type DialContextFunc func(ctx context.Context, network, addr string) (net.Conn, error)
type DialOption func(*dialOptions)
func WithDialer(dial DialContextFunc) DialOption // nil = default direct dial

// Existing constructors gain a trailing variadic option (existing call sites compile unchanged):
func Dial(addr string, opts ...DialOption) (*Client, error)
func DialPool(addr string, size int, opts ...DialOption) (*Client, error)
func DialTLS(addr string, tlsConfig *tls.Config, opts ...DialOption) (*Client, error)
func DialTLSPool(addr string, tlsConfig *tls.Config, size int, opts ...DialOption) (*Client, error)
func DialTLSPinned(addr, fingerprint string, opts ...DialOption) (*Client, error)
func DialTLSPinnedPool(addr, fingerprint string, size int, opts ...DialOption) (*Client, error) // new
func DialBinary(addr string, opts ...DialOption) (*BinaryClient, error)

Behaviour notes:

  • Credentials never leak. Proxy userinfo is only written to the proxy in Proxy-Authorization. Errors, String(), Redact and ConnectError never include it, and ConnectError also leaves out the proxy's reason phrase. The userinfo is split at the last @, so tokens with unescaped /, ?, # or @ (for example base64) decode intact. They are never parsed as the proxy host, so they never reach logs, errors or DNS.
  • Env variables are independent. An unusable HTTP_PROXY, http_proxy, ALL_PROXY or all_proxy (malformed, or socks5://) is skipped and reported through Warnings(). It never disables a valid HTTPS_PROXY. Only an unusable HTTPS_PROXY/https_proxy is an error.
  • In auto mode, localhost and loopback are always exempt, and NO_PROXY matching follows x/net/http/httpproxy. With no HTTP_PROXY, plain http:// and ws:// requests use the TLS proxy, because direct egress is killed in the target sandbox. Explicit mode proxies everything.
  • In the registry client, TLS runs over the dialer's conn with ServerName defaulting to the address host. SNI, CA verification and pinned-fingerprint checks are unchanged, and the caller's tls.Config is never mutated. The existing 5s per-attempt dialTimeout now covers proxy connect + CONNECT + TLS handshake.
  • No new dependencies.

Also: go 1.25.13

The required govulncheck check has failed on every PR since mid-August. The go1.25.12 standard library has GO-2026-6218 (net/url), GO-2026-6090 (crypto/tls), GO-2026-5972 (encoding/asn1) and GO-2026-5026 (net/http idna). All are fixed in go1.25.13 and are reachable from code already on main. This PR bumps the go directive to 1.25.13, which pilotprotocol already requires. govulncheck: no vulnerabilities.

Tests

  • netproxy: an in-process CONNECT proxy maps .invalid host names to loopback, which proves targets are sent by name and never resolved locally. Coverage includes Basic auth, 407/403/5xx, an unreachable proxy, https:// proxies, context deadline/cancel, read-ahead bytes after the 200, NO_PROXY and loopback bypass, and an http.Transport driven by ProxyForRequest. Every failure path asserts that no credential appears.
  • Regression tests for the review findings:
    • TestFromEnvVariablesAreIndependent and TestFromEnvUnusableTLSProxyIsAnError: a valid HTTPS_PROXY next to a socks or garbage HTTP_PROXY/http_proxy still proxies registry.pilotprotocol.network:443. A socks ALL_PROXY alone is only a warning.
    • TestUnescapedDelimitersInUserinfo, TestUnusableUserinfoNeverLeaks, TestRedactMisparsedURL: credentials with /, ?, # or @ are handled.
    • TestDialerUnescapedCredentialsBesideUnusableHTTPProxy: end to end. Checks the Proxy-Authorization the proxy receives, that the only address dialed is the real proxy (so no DNS lookup of a secret), and that the error text is clean.
    • Mutation-checked: reverting either fix makes these tests fail.
  • registry/client: all constructors and reconnect paths go through WithDialer. This covers pinned TLS through a CONNECT proxy, SNI and CA verification over the tunnel, and that the tls.Config is not mutated.
  • GOWORK=off go test -race ./... -count=1, go vet ./..., govulncheck@v1.1.4 ./... (go1.25.13): pass.

🤖 Generated with Claude Code

teovl and others added 3 commits September 24, 2026 00:09
Add package netproxy so Pilot components can reach their :443 endpoints
from hosts whose only egress is an authenticating HTTP proxy that allows
CONNECT host:443 (hosted agent sandboxes: UDP blocked, direct TCP killed,
local DNS for the Pilot hostnames poisoned).

- Resolver picks the proxy per target: Explicit(url), FromEnvironment()
  (HTTPS_PROXY/https_proxy, fallback ALL_PROXY/all_proxy; HTTP_PROXY for
  plain http:// requests; NO_PROXY/no_proxy; loopback never proxied) or
  Off(). Parse("auto"|"off"|URL) maps a -proxy style setting. The same
  decision is exposed as ProxyForAddr (raw TCP) and ProxyForRequest
  (http.Transport.Proxy), so TCP dials and HTTP clients agree.
- Dialer.DialContext tunnels through the chosen proxy (plain TCP for
  http://, TLS to the proxy for https://), sends CONNECT host:port with
  Host and Basic Proxy-Authorization, parses the reply with
  http.ReadResponse, keeps bytes read ahead of the tunnel, honours ctx
  cancellation/deadline plus a dial timeout, and closes the conn on every
  error path. The target is never resolved locally.
- Credentials never reach errors or logs: non-2xx replies become
  ConnectError ("proxy CONNECT host:port: 407 Proxy Authentication
  Required", status text from net/http, never the proxy's reason phrase);
  Redact and Resolver.String render http://***@host:port.
- NO_PROXY matching follows x/net/http/httpproxy, reimplemented to keep
  the module dependency-free.

Tests run an in-process CONNECT proxy that only knows made-up .invalid
host names: success, Basic auth header, 407 with credential-echoing
reason phrase, NO_PROXY bypass, https:// proxy, ctx cancel/deadline and
Timeout, read-ahead bytes preserved, hostname-only CONNECT, and an
http.Transport driven by ProxyForRequest.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add DialOption / WithDialer(DialContextFunc) as a trailing variadic
option on Dial, DialPool, DialTLS, DialTLSPool, DialTLSPinned and
DialBinary, plus DialTLSPinnedPool (already referenced by the
DialTLSPool error message). When set, the dialer opens every
connection: the primary, each pool conn, Client.reconnect,
reconnectEntry and BinaryClient.reconnect. The registry address reaches
the dialer unresolved, so a netproxy.Dialer asks the proxy to CONNECT by
host name.

For TLS the handshake runs on top of the dialer's conn via tls.Client;
ServerName defaults to the address host (as crypto/tls.Dial does) on a
clone, so SNI, CA verification and pinned-fingerprint verification are
unchanged, and the caller's tls.Config is never mutated. The dial and
handshake are bounded by the existing 5s dialTimeout.

Without WithDialer every path performs exactly the direct dials it did
before (same timeouts and ctx handling); existing call sites compile
unchanged.

Tests reach a TLS registry stand-in only through a CONNECT proxy under
registry.pilot.invalid (never resolvable locally): DialTLS (SNI and CA
check), untrusted cert, DialTLSPinned (match and mismatch),
DialTLSPool / DialTLSPinnedPool with concurrent sends, reconnect on the
primary, pinned and pool paths, plain Dial / DialPool, DialBinary with
reconnect, and the WithDialer contract (unresolved addr, bounded ctx,
error propagation, conn closed on handshake failure, nil keeps direct).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
FromEnvironment parsed every proxy variable and failed as a whole when any
of them was unusable, so an http_proxy Go cannot use (socks5h://, stray
whitespace) or a socks ALL_PROXY threw away a valid HTTPS_PROXY. Callers
then fell back to direct dials, which a proxy-only sandbox (Meta Muse)
kills. Each variable is now parsed on its own: an unusable HTTP_PROXY,
http_proxy, ALL_PROXY or all_proxy is skipped (the next variable in its
list applies) and reported per variable by the new Resolver.Warnings as
*EnvError; String() names the skipped variables. Only an unusable
HTTPS_PROXY / https_proxy, which explicitly names the TLS proxy, is still
an error (an *EnvError naming it).

parseProxyURL used url.Parse, which ends the authority at the first '/',
'?' or '#'. Credentials holding those characters unescaped (base64 tokens)
had part of the secret parsed as the proxy host, which then appeared in
String()/Redact, in dial errors, and in a DNS lookup. The userinfo is now
split off at the LAST '@' and percent-decoded by hand; only the part after
it goes through url.Parse. Unusable values are rejected with errors that
never echo credentials (including a "scheme" that may be a user name).
Redact also withholds everything before the last '@' when handed a URL
that url.Parse misparsed this way.

Tests cover both findings at the resolver level and end to end through the
in-process CONNECT proxy (Proxy-Authorization, dialed addresses, errors).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.41964% with 25 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
netproxy/netproxy.go 94.97% 7 Missing and 4 partials ⚠️
netproxy/dialer.go 92.15% 4 Missing and 4 partials ⚠️
netproxy/noproxy.go 92.98% 2 Missing and 2 partials ⚠️
registry/client/dialer.go 93.93% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

The required govulncheck check has failed on every PR since mid-August:
go1.25.12's standard library carries GO-2026-6218 (net/url), GO-2026-6090
(crypto/tls), GO-2026-5972 (encoding/asn1) and GO-2026-5026 (net/http
idna), all fixed in go1.25.13 and reachable from code already on main
(registry/wire, decision, driver). CI takes its Go version from go.mod.
pilotprotocol already requires go 1.25.13, so this adds no new
requirement for it. With 1.25.13: govulncheck reports no vulnerabilities.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@TeoSlayer
TeoSlayer merged commit 47086f3 into main Sep 23, 2026
11 checks passed
@TeoSlayer
TeoSlayer deleted the feat/netproxy-connect-dialer branch September 23, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants