Skip to content

docs(troubleshooting): SSH publickey denial, and why guessing the username bans your IP - #226

Merged
cofade merged 3 commits into
mainfrom
docs/ssh-access-troubleshooting
Aug 18, 2026
Merged

docs(troubleshooting): SSH publickey denial, and why guessing the username bans your IP#226
cofade merged 3 commits into
mainfrom
docs/ssh-access-troubleshooting

Conversation

@cofade

@cofade cofade commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Captures an access failure that cost a working session, plus the incident lesson behind it.

The failure

ssh highfive returns Permission denied (publickey) even with the key correctly installed, because ~/.ssh/config pins only HostName — so OpenSSH sends the local account name as the login. The key is only half the credential; the login name is the other half, and it is not derivable from the key comment, your email, or the repo.

The trap is what comes next. Guessing the username (root, ubuntu, admin, …) trips the host's brute-force protection, and port 22 goes dead while https://highfive.schutera.com/ keeps returning 200. That asymmetry reads as "the SSH daemon is down", which invites more retries — the one action that cannot help.

This is not hypothetical: the repo owner's own ~/.ssh/config is currently in exactly this state, with # User ? commented out awaiting the login name.

What the section gives you

  • ssh -G highfive as the opening move — prints the resolved config, sends zero packets, so it costs no auth attempt and still works while banned. It printed user wienh here, which is the whole bug in one line.
  • A three-row diagnostic table with a control probe against a third host, because "web fine + port 22 dead" is equally consistent with your own network blocking outbound tcp/22 (corp LAN, VPN, hotel). Without the control you ask the owner to unban an address that was never banned.
  • A safe config-writing block, and the placement rule that makes it work.
  • The unban path, including what to do when the owner is the one locked out (fail2ban-client needs a shell on the host, so it's useless to them).

Plus a chapter-11 lessons entry — "probing prod with guessed usernames gets you banned, and the ban looks like an outage" is an incident, not a symptom→fix — and links from both places a reader is told to ssh (production-runbook.md, production-deployment.md).

Three defects review caught in my own draft

Worth listing, because they're all the same mistake: I verified the mechanism and not the outcome.

1. The config-writing block corrupted existing configs. Out-File -Append into a ~/.ssh/config with no trailing newline welds the stanza onto the last line:

    User olduserHost highfive

The stanza never registers, and HostName now carries three trailing tokens — a fatal parse error for every ssh invocation, not just this host. So the documented remedy escalated "I can't reach one server" into "ssh doesn't run at all", for someone already locked out. Now mirrors the Test-Path → notepad safe-write precedent this file already uses for .wslconfig.

2. The placeholder guard didn't hold. if (...) { throw } is a complete statement, so pasted into a console the throw prints red and execution continues into the write — producing User the-name-the-owner-gave-you, a config that looks fixed while sending a name indistinguishable from a guess. Reproduced in a real runspace. The check is now the first arm of the if/elseif/else that does the writing, so it cannot be stepped past.

3. "Retrying refreshes the ban timer" was false. Once banned, the firewall blocks the packet before sshd, so no auth failure is logged and nothing re-triggers the filter. The advice (stop retrying) was right; the reason was invented — and it contradicted my own blockquote three paragraphs above that declines to assert which mechanism this host even runs.

Also fixed: -o BatchMode=yes masked passphrase/host-key failures as Permission denied (publickey), byte-identical to the error being diagnosed; -ErrorAction SilentlyContinue does not suppress Select-String's PathNotFound in PS 5.1; Out-File throws on a missing .ssh dir; and an earlier Host * block silently beats a later Host highfive (verified with ssh -G -F: resolved to user git, not user realname).

Verification

Outcome-level, in a real runspace with correct paste semantics, against all three starting states:

Starting state Result
Unedited placeholder writes nothing (the throw version wrote it)
Fresh machine creates config; ssh -G resolves user realname; ASCII, no BOM
Existing config, no trailing newline left byte-identical

ssh -G and the Host * precedence were confirmed against real OpenSSH, and the Authenticating to … as '<user>' line shape against a live SSH endpoint. The maintainer's real ~/.ssh/config was confirmed untouched throughout. All five repo gate scripts and prettier are clean.

Docs-only — no code, no wire shapes, no UI, so ADR-014 / the Playwright rule don't apply.

🤖 Reviewed across two rounds with the repo's senior-reviewer gate.

cofade and others added 3 commits August 18, 2026 10:47
…our IP

Two failures that present as one, and the second is self-inflicted.

An ~/.ssh/config pinning only HostName makes OpenSSH fall back to the LOCAL
account name, so a correctly installed key still gets "Permission denied
(publickey)" — the key is only half the credential and the login name is not
derivable from the key comment or an email address. Probing plausible names to
find it then trips the host's brute-force protection, which black-holes port 22
while https keeps returning 200. That combination reads as "the server is
down", which is the wrong conclusion and leads to more retries, which with
fail2ban refresh the ban timer and keep you locked out.

Includes the diagnostic that distinguishes the two (web fine + port 22 dead =
banned, not an outage), the config fix, how to get unbanned, and a
single-attempt verification rather than a loop.

Three things fixed while writing it up, all repo conventions:

- No angle-bracket placeholders. CLAUDE.md forbids them in PowerShell snippets
  because `<` is a redirection operator; this file already uses the
  `$var = "..."` form elsewhere ($repo, $PORT, $mac) and now does here too.
- No hardcoded username. An earlier draft used the author's Windows login as
  the example; it now says $env:USERNAME, matching the de-hardcoding applied
  to this file's ruff entry in #196.
- MaxAuthTries and a ban are no longer conflated. MaxAuthTries closes one
  connection and you still get a prompt next time; a ban drops packets. That
  distinction is exactly what "connection timed out" vs "Permission denied"
  tells you, so it is load-bearing for the diagnosis. Which mechanism this host
  runs was NOT confirmed from outside, and the doc says so rather than
  asserting fail2ban.

Verified: every PowerShell snippet parsed with
[System.Management.Automation.Language.Parser]::ParseInput (6 of 6 clean), and
the here-string append was executed against a temp file — it expands $sshUser
correctly and writes ASCII with no BOM (first bytes 72,111,115), which matters
because PowerShell's default redirect would emit UTF-16 LE with a BOM into
.ssh/config. Added a Select-String guard first, since a blind append duplicates
an existing Host block and OpenSSH silently honours the FIRST match. The
production host was deliberately not probed over SSH while writing this.

Co-Authored-By: WOZCODE <contact@withwoz.com>
Review of the first draft found that I had validated the wrong thing. I
verified every snippet PARSED and executed the here-string against a temp file
I created myself -- the one input shape that cannot fail. Every real defect
lived in runtime behaviour against a hostile starting state. All reproduced
before fixing:

- P0: `Out-File -Append` into an existing ~/.ssh/config with no trailing
  newline welds the new stanza onto the last line. Measured output:
  "    User olduserHost highfive". The stanza never registers, and HostName
  now carries three trailing tokens, which OpenSSH treats as a fatal parse
  error for EVERY ssh invocation -- so the documented remedy escalates "I
  can't reach one server" into "ssh doesn't run at all", for someone already
  locked out. Now mirrors the safe-write pattern this same file already uses
  for .wslconfig: Test-Path, and if the file exists print + open notepad
  rather than appending. Only auto-writes when absent.

- P0: "with fail2ban every further attempt refreshes the ban timer" is false.
  Once banned, the firewall drops packets before sshd, so no new auth failure
  is ever logged and nothing re-triggers the filter; bantime.increment
  lengthens a SUBSEQUENT ban after the current one expires. The advice (stop
  retrying) is right, the reason was wrong -- and it contradicted my own
  blockquote three paragraphs up that declines to assert which mechanism this
  host even runs.

- `-ErrorAction SilentlyContinue` does NOT suppress Select-String's
  ObjectNotFound on a missing file in PS 5.1 (reproduced: full red
  ItemNotFoundException), and Out-File on a path whose .ssh dir does not exist
  throws DirectoryNotFoundException. So the fresh-machine case -- the likeliest
  state for someone who has never logged in -- produced a scary error followed
  by a hard failure. Now New-Item -Force the directory first.

- `-o BatchMode=yes` was the wrong verification: it suppresses passphrase and
  host-key prompts, so an unloaded agent identity fails with "Permission denied
  (publickey)", byte-identical to the error being diagnosed. Replaced with
  `ssh -v`, whose "debug1: Authenticating to host:22 as 'name'" line IS the
  diagnosis. Confirmed against a real SSH endpoint rather than assumed.

- Test-NetConnection | Select-Object emits a table plus a WARNING, not the bare
  "False" the comment implied; now -InformationLevel Quiet.

Also: the ipify hop can report a proxy's egress while ssh goes out from another
address, so the owner would unban a stranger -- replaced with
`fail2ban-client status sshd`, read on the host. A sentinel throw stops the
unedited placeholder from being written as a real User. IdentityFile is now
conditional (pinning it overrides the defaults, so naming a missing key burns
another auth attempt). "byte-for-byte" corrected -- sshd ignores the trailing
comment.

Structure: moved out of "## Server stack" (whose other entries are all dev
stack) into its own "## Production host access (SSH)" section, and linked from
both places a reader is told to ssh: production-runbook.md and
production-deployment.md. Added the chapter-11 lesson, since "probing prod with
guessed usernames gets you banned, and the ban looks like an outage" is an
incident, not a symptom->fix.

Verified this time at runtime, not by parsing: the safe-write block exercised
against all three starting states (existing-config-without-trailing-newline is
left byte-identical; fresh machine creates the file, registers the stanza, and
writes ASCII with no BOM; the sentinel throws), plus all 5 fenced snippets
parse and the ssh -v line shape confirmed live. Repo gates and prettier clean.

Co-Authored-By: WOZCODE <contact@withwoz.com>
…f throwing

Round-2 review. Both findings were verified by running them, which is the
habit the previous two rounds kept missing.

1. The section led with `ssh -v`, which is the wrong tool twice over: it sends
   an auth attempt to production — in a document whose entire point is "stop
   sending auth attempts to production" — and once you ARE banned (part 2 of
   the same section) it hangs at "Connecting to" and never reaches the line
   being quoted, so the "one command that shows you this" is unavailable
   exactly when it is needed. `ssh -G highfive` prints the resolved config
   with ZERO packets; run here it printed `user wienh`, which is the entire
   bug, offline and free. It now leads the section, and `ssh -v` is demoted to
   the final single-attempt verification where a real connection is the point.

2. The placeholder sentinel did not hold. `if (...) { throw }` is a complete
   statement, so pasted into a console the throw prints red and execution
   CONTINUES into the next statement, which wrote:
       User the-name-the-owner-gave-you
   — a config that looks fixed while sending a name indistinguishable from a
   guess, i.e. feeding the exact ban this page exists to prevent. Reproduced in
   a real runspace, statement by statement. The check is now the first arm of
   the if/elseif/else that does the writing, so it cannot be stepped past: the
   whole chain is one statement, which is precisely why it works.

Also, verified with `ssh -G -F`: an earlier `Host *` / `User git` block beats a
later `Host highfive` / `User realname`. Appending at the end — which the
notepad branch effectively told the reader to do — is silently wrong. That
branch now prints the literal three lines and says to put them ABOVE any
`Host *`.

And the central diagnostic over-committed: "web fine + port 22 dead = banned"
is equally consistent with your own network blocking outbound tcp/22 (corp LAN,
VPN, hotel), which sends the operator to ask the owner to unban an address that
was never banned. Added a third-host control probe and a three-row table that
separates ban / egress-block / real outage.

Smaller corrections: dropped-vs-rejected is now hedged (fail2ban's default
iptables action REJECTs, so "connection refused" is as likely as a timeout);
`bantime.increment` is flagged as off-by-default rather than asserted; the
sample output no longer contains the maintainer's real Windows account and home
path; production-deployment.md now carries a real markdown link rather than a
comment inside a bash fence; and there is an out-of-band path for the case
where the OWNER is the one locked out (fail2ban-client needs a shell on the
host, so it is useless to them).

Verified at the outcome level, not the mechanism level: the fix block was
executed in a real runspace with correct paste semantics against all three
starting states — unedited placeholder writes NOTHING, a real name on a fresh
machine produces a config that `ssh -G` resolves to `user realname` in ASCII
with no BOM, and an existing config with no trailing newline is left
byte-identical. The maintainer's real ~/.ssh/config was confirmed untouched
throughout. Repo gates and prettier clean.

Co-Authored-By: WOZCODE <contact@withwoz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant