docs(troubleshooting): SSH publickey denial, and why guessing the username bans your IP - #226
Merged
Merged
Conversation
…our IP Two failures that present as one, and the second is self-inflicted. An ~/.ssh/config pinning only HostName makes OpenSSH fall back to the LOCAL account name, so a correctly installed key still gets "Permission denied (publickey)" — the key is only half the credential and the login name is not derivable from the key comment or an email address. Probing plausible names to find it then trips the host's brute-force protection, which black-holes port 22 while https keeps returning 200. That combination reads as "the server is down", which is the wrong conclusion and leads to more retries, which with fail2ban refresh the ban timer and keep you locked out. Includes the diagnostic that distinguishes the two (web fine + port 22 dead = banned, not an outage), the config fix, how to get unbanned, and a single-attempt verification rather than a loop. Three things fixed while writing it up, all repo conventions: - No angle-bracket placeholders. CLAUDE.md forbids them in PowerShell snippets because `<` is a redirection operator; this file already uses the `$var = "..."` form elsewhere ($repo, $PORT, $mac) and now does here too. - No hardcoded username. An earlier draft used the author's Windows login as the example; it now says $env:USERNAME, matching the de-hardcoding applied to this file's ruff entry in #196. - MaxAuthTries and a ban are no longer conflated. MaxAuthTries closes one connection and you still get a prompt next time; a ban drops packets. That distinction is exactly what "connection timed out" vs "Permission denied" tells you, so it is load-bearing for the diagnosis. Which mechanism this host runs was NOT confirmed from outside, and the doc says so rather than asserting fail2ban. Verified: every PowerShell snippet parsed with [System.Management.Automation.Language.Parser]::ParseInput (6 of 6 clean), and the here-string append was executed against a temp file — it expands $sshUser correctly and writes ASCII with no BOM (first bytes 72,111,115), which matters because PowerShell's default redirect would emit UTF-16 LE with a BOM into .ssh/config. Added a Select-String guard first, since a blind append duplicates an existing Host block and OpenSSH silently honours the FIRST match. The production host was deliberately not probed over SSH while writing this. Co-Authored-By: WOZCODE <contact@withwoz.com>
Review of the first draft found that I had validated the wrong thing. I verified every snippet PARSED and executed the here-string against a temp file I created myself -- the one input shape that cannot fail. Every real defect lived in runtime behaviour against a hostile starting state. All reproduced before fixing: - P0: `Out-File -Append` into an existing ~/.ssh/config with no trailing newline welds the new stanza onto the last line. Measured output: " User olduserHost highfive". The stanza never registers, and HostName now carries three trailing tokens, which OpenSSH treats as a fatal parse error for EVERY ssh invocation -- so the documented remedy escalates "I can't reach one server" into "ssh doesn't run at all", for someone already locked out. Now mirrors the safe-write pattern this same file already uses for .wslconfig: Test-Path, and if the file exists print + open notepad rather than appending. Only auto-writes when absent. - P0: "with fail2ban every further attempt refreshes the ban timer" is false. Once banned, the firewall drops packets before sshd, so no new auth failure is ever logged and nothing re-triggers the filter; bantime.increment lengthens a SUBSEQUENT ban after the current one expires. The advice (stop retrying) is right, the reason was wrong -- and it contradicted my own blockquote three paragraphs up that declines to assert which mechanism this host even runs. - `-ErrorAction SilentlyContinue` does NOT suppress Select-String's ObjectNotFound on a missing file in PS 5.1 (reproduced: full red ItemNotFoundException), and Out-File on a path whose .ssh dir does not exist throws DirectoryNotFoundException. So the fresh-machine case -- the likeliest state for someone who has never logged in -- produced a scary error followed by a hard failure. Now New-Item -Force the directory first. - `-o BatchMode=yes` was the wrong verification: it suppresses passphrase and host-key prompts, so an unloaded agent identity fails with "Permission denied (publickey)", byte-identical to the error being diagnosed. Replaced with `ssh -v`, whose "debug1: Authenticating to host:22 as 'name'" line IS the diagnosis. Confirmed against a real SSH endpoint rather than assumed. - Test-NetConnection | Select-Object emits a table plus a WARNING, not the bare "False" the comment implied; now -InformationLevel Quiet. Also: the ipify hop can report a proxy's egress while ssh goes out from another address, so the owner would unban a stranger -- replaced with `fail2ban-client status sshd`, read on the host. A sentinel throw stops the unedited placeholder from being written as a real User. IdentityFile is now conditional (pinning it overrides the defaults, so naming a missing key burns another auth attempt). "byte-for-byte" corrected -- sshd ignores the trailing comment. Structure: moved out of "## Server stack" (whose other entries are all dev stack) into its own "## Production host access (SSH)" section, and linked from both places a reader is told to ssh: production-runbook.md and production-deployment.md. Added the chapter-11 lesson, since "probing prod with guessed usernames gets you banned, and the ban looks like an outage" is an incident, not a symptom->fix. Verified this time at runtime, not by parsing: the safe-write block exercised against all three starting states (existing-config-without-trailing-newline is left byte-identical; fresh machine creates the file, registers the stanza, and writes ASCII with no BOM; the sentinel throws), plus all 5 fenced snippets parse and the ssh -v line shape confirmed live. Repo gates and prettier clean. Co-Authored-By: WOZCODE <contact@withwoz.com>
…f throwing
Round-2 review. Both findings were verified by running them, which is the
habit the previous two rounds kept missing.
1. The section led with `ssh -v`, which is the wrong tool twice over: it sends
an auth attempt to production — in a document whose entire point is "stop
sending auth attempts to production" — and once you ARE banned (part 2 of
the same section) it hangs at "Connecting to" and never reaches the line
being quoted, so the "one command that shows you this" is unavailable
exactly when it is needed. `ssh -G highfive` prints the resolved config
with ZERO packets; run here it printed `user wienh`, which is the entire
bug, offline and free. It now leads the section, and `ssh -v` is demoted to
the final single-attempt verification where a real connection is the point.
2. The placeholder sentinel did not hold. `if (...) { throw }` is a complete
statement, so pasted into a console the throw prints red and execution
CONTINUES into the next statement, which wrote:
User the-name-the-owner-gave-you
— a config that looks fixed while sending a name indistinguishable from a
guess, i.e. feeding the exact ban this page exists to prevent. Reproduced in
a real runspace, statement by statement. The check is now the first arm of
the if/elseif/else that does the writing, so it cannot be stepped past: the
whole chain is one statement, which is precisely why it works.
Also, verified with `ssh -G -F`: an earlier `Host *` / `User git` block beats a
later `Host highfive` / `User realname`. Appending at the end — which the
notepad branch effectively told the reader to do — is silently wrong. That
branch now prints the literal three lines and says to put them ABOVE any
`Host *`.
And the central diagnostic over-committed: "web fine + port 22 dead = banned"
is equally consistent with your own network blocking outbound tcp/22 (corp LAN,
VPN, hotel), which sends the operator to ask the owner to unban an address that
was never banned. Added a third-host control probe and a three-row table that
separates ban / egress-block / real outage.
Smaller corrections: dropped-vs-rejected is now hedged (fail2ban's default
iptables action REJECTs, so "connection refused" is as likely as a timeout);
`bantime.increment` is flagged as off-by-default rather than asserted; the
sample output no longer contains the maintainer's real Windows account and home
path; production-deployment.md now carries a real markdown link rather than a
comment inside a bash fence; and there is an out-of-band path for the case
where the OWNER is the one locked out (fail2ban-client needs a shell on the
host, so it is useless to them).
Verified at the outcome level, not the mechanism level: the fix block was
executed in a real runspace with correct paste semantics against all three
starting states — unedited placeholder writes NOTHING, a real name on a fresh
machine produces a config that `ssh -G` resolves to `user realname` in ASCII
with no BOM, and an existing config with no trailing newline is left
byte-identical. The maintainer's real ~/.ssh/config was confirmed untouched
throughout. Repo gates and prettier clean.
Co-Authored-By: WOZCODE <contact@withwoz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Captures an access failure that cost a working session, plus the incident lesson behind it.
The failure
ssh highfivereturnsPermission denied (publickey)even with the key correctly installed, because~/.ssh/configpins onlyHostName— so OpenSSH sends the local account name as the login. The key is only half the credential; the login name is the other half, and it is not derivable from the key comment, your email, or the repo.The trap is what comes next. Guessing the username (
root,ubuntu,admin, …) trips the host's brute-force protection, and port 22 goes dead whilehttps://highfive.schutera.com/keeps returning200. That asymmetry reads as "the SSH daemon is down", which invites more retries — the one action that cannot help.This is not hypothetical: the repo owner's own
~/.ssh/configis currently in exactly this state, with# User ?commented out awaiting the login name.What the section gives you
ssh -G highfiveas the opening move — prints the resolved config, sends zero packets, so it costs no auth attempt and still works while banned. It printeduser wienhhere, which is the whole bug in one line.fail2ban-clientneeds a shell on the host, so it's useless to them).Plus a chapter-11 lessons entry — "probing prod with guessed usernames gets you banned, and the ban looks like an outage" is an incident, not a symptom→fix — and links from both places a reader is told to
ssh(production-runbook.md,production-deployment.md).Three defects review caught in my own draft
Worth listing, because they're all the same mistake: I verified the mechanism and not the outcome.
1. The config-writing block corrupted existing configs.
Out-File -Appendinto a~/.ssh/configwith no trailing newline welds the stanza onto the last line:The stanza never registers, and
HostNamenow carries three trailing tokens — a fatal parse error for everysshinvocation, not just this host. So the documented remedy escalated "I can't reach one server" into "sshdoesn't run at all", for someone already locked out. Now mirrors theTest-Path→ notepad safe-write precedent this file already uses for.wslconfig.2. The placeholder guard didn't hold.
if (...) { throw }is a complete statement, so pasted into a console the throw prints red and execution continues into the write — producingUser the-name-the-owner-gave-you, a config that looks fixed while sending a name indistinguishable from a guess. Reproduced in a real runspace. The check is now the first arm of theif/elseif/elsethat does the writing, so it cannot be stepped past.3. "Retrying refreshes the ban timer" was false. Once banned, the firewall blocks the packet before sshd, so no auth failure is logged and nothing re-triggers the filter. The advice (stop retrying) was right; the reason was invented — and it contradicted my own blockquote three paragraphs above that declines to assert which mechanism this host even runs.
Also fixed:
-o BatchMode=yesmasked passphrase/host-key failures asPermission denied (publickey), byte-identical to the error being diagnosed;-ErrorAction SilentlyContinuedoes not suppressSelect-String'sPathNotFoundin PS 5.1;Out-Filethrows on a missing.sshdir; and an earlierHost *block silently beats a laterHost highfive(verified withssh -G -F: resolved touser git, notuser realname).Verification
Outcome-level, in a real runspace with correct paste semantics, against all three starting states:
throwversion wrote it)ssh -Gresolvesuser realname; ASCII, no BOMssh -Gand theHost *precedence were confirmed against real OpenSSH, and theAuthenticating to … as '<user>'line shape against a live SSH endpoint. The maintainer's real~/.ssh/configwas confirmed untouched throughout. All five repo gate scripts and prettier are clean.Docs-only — no code, no wire shapes, no UI, so ADR-014 / the Playwright rule don't apply.
🤖 Reviewed across two rounds with the repo's
senior-reviewergate.