Any way to run socks5 or https proxy behind reverse proxy with SSL?

[enchained]

My home ISP is blocking custom port connections to my remote server, but 443 works fine, so I have been switching everything to the reverse proxy so it would work again.

Example of my working setup for sonarr web ui:
Domain Names: sonarr.mydomain.com
Scheme http Forward Hostname / IP mydomain.com Forward Port 33333 Block Common Exploits
SSL certificate: *.mydomain.com Force SSL HTTP/2 Support

While most services started being accessible after I put them under reverse proxy, I still can't get 3proxy working. I tried both its socks5 proxy port and https proxy port in a config similar to my sonarr one.

Is there an additional config that could possibly make it work, or it is simply impossible to make socks5 or https proxy work under the reverse proxy? My goal is to make proxy accessible under 443 instead of custom port so I could use it from home again.
I know nothing about manual nginx configs so I decided to ask here.

[tigran-sargsyan-w]

The important difference is that Sonarr speaks HTTP, while SOCKS5 does not.

A normal NPM Proxy Host uses Nginx's HTTP layer:

client -> HTTP/HTTPS -> Nginx -> HTTP upstream

That works because Nginx can parse the HTTP request and route it using the Host header, path, etc.

SOCKS5 is a different raw TCP protocol. If SOCKS5 bytes arrive at an HTTP reverse-proxy listener, there is no HTTP request for Nginx to route.

For a SOCKS5 endpoint you need TCP/stream proxying:

client -> TCP -> Nginx stream -> 3proxy SOCKS5 port

Nginx Proxy Manager exposes raw TCP forwarding through Streams, not Proxy Hosts.

There is another problem with putting everything on the same port 443: a normal SOCKS5 connection does not contain an HTTP Host header, so Nginx cannot choose a backend based on a domain name in the same way it does for HTTPS websites.

So the practical options are:

• expose SOCKS5 through an NPM Stream on another reachable port;
• use a separate IP for the TCP service;
• or use a VPN/tunnel/proxy protocol designed to operate over HTTPS/WebSocket if your ISP only permits port 443.

The Proxy Host configuration that works for Sonarr cannot directly proxy a normal SOCKS5 endpoint because they are different application protocols.