Cut startup cost, keep command arguments whole, tighten up conf and creds

CLAUDE.md

@@ -35,14 +35,16 @@ Things that bite in this suite:
 
 `cli.main` wires the three halves together: `load_conf` produces the configuration, `get_env` turns it into an environment, and `run_cmd`/`run_shell` hand that environment to a shell. Errors reach the user through `logs.error`, which exits with status 1; `logs.vlog` output only appears under `--verbose` and is the first thing to reach for when debugging a configuration.
 
-**`conf.py` — where the configuration comes from.** Later sources win, merged deeply by `merge_conf`: defaults, then `~/.keycmd`, then every `.keycmd` found walking up from the working directory (outermost first), then the first `pyproject.toml` found walking up, whose `[tool.keycmd]` table is used. `find_file` stops at a `.git` directory, at the home folder, and at the root of the file system, so the walk never escapes a repository. `USERPROFILE` is a module attribute so tests can point the user config elsewhere. The merged result is `cast` to `Conf` rather than validated: it is user authored, and `get_env` reports violations as user errors.
+**`conf.py` — where the configuration comes from.** Later sources win, merged deeply by `merge_conf`: defaults, then `~/.keycmd`, then every `.keycmd` found walking up from the working directory (outermost first), then the first `pyproject.toml` found walking up, whose `[tool.keycmd]` table is used. Both searches cover the same ground, so `load_conf` collects them in a single pass over `walk_up`, which stops at a `.git` directory, at the home folder, and at the root of the file system, so the walk never escapes a repository. `USERPROFILE` is a module attribute so tests can point the user config elsewhere. The merged result is `cast` to `Conf` rather than validated: it is user authored, and `get_env` reports violations as user errors.
 
 **`creds.py` — configuration to environment.** `get_env` copies `os.environ` and adds a variable per entry of `[keys]`, looking each credential up in the keyring; `[aliases]` re-expose an existing key under another name with different `b64`/`format` options, without a second keyring lookup. `expose` applies `format` first and `b64` second, which is what makes `{username}:{password}` basic auth work.
 
 **`wsl.py` — the boundary between WSL and Windows.** WSL users install keycmd on Windows, which leaves it a Windows process with a Windows idea of a shell. `from_wsl` decides whether it was called from a distro — a `wsl.exe`/`wslhost.exe` ancestor decides it, a Windows shell found first decides against it, and a UNC working directory settles the rest — after which `run_shell`/`run_cmd` hand the work to `wsl.exe` rather than to a Windows shell. A UNC working directory also names the distro, which `wsl_argv` passes as `--distribution` so that a second distro does not send the command to the default one. `KEYCMD_WSL` overrides that decision in either direction. Neither side of the boundary inherits the other's environment, so `share_env` lists the exposed variables in `WSLENV`, which is the only thing that crosses.
 
-**`shell.py` — where the platform differences live.** `get_shell` asks shellingham which shell invoked the process and falls back to `$SHELL` or `%COMSPEC%`. `cmd` takes `/C` and keeps the command's arguments separate; every other shell takes `-c` and a single joined string. `exec` replaces the process with `execvpe` on posix, but runs a subprocess on Windows, which has no equivalent; `USE_SUBPROCESS` and the `IS_WINDOWS`/`IS_POSIX` flags are module attributes so tests can drive both paths on either platform.
+**`shell.py` — where the platform differences live.** `get_shell` asks shellingham which shell invoked the process and falls back to `$SHELL` or `%COMSPEC%`. `cmd` takes `/C` and keeps the command's arguments separate; every other shell takes `-c` and the single string `join_cmd` builds. One argument is already a command line — the form the README recommends — and is handed over as typed, so the shell interprets it; several arguments are an argv vector, and `quote` protects each so that the shell does not split them into words a second time. `quote` reaches for `shlex` for posix shells and doubles the quote for powershell, which also needs the call operator once its command name ends up quoted. `exec` replaces the process with `execvpe` on posix, but runs a subprocess on Windows, which has no equivalent; `USE_SUBPROCESS` and the `IS_WINDOWS`/`IS_POSIX` flags are module attributes so tests can drive both paths on either platform.
 
 ## Conventions
 
 The package is fully annotated and ships a `py.typed` marker, so `ANN` rules apply to `keycmd/` while the test suite is exempt. `ty` is configured with `python-version = "3.13"`, the oldest supported release, so it catches typing features that are newer than `requires-python` allows. CI matches that split: the latest Python on all three platforms, plus a single job on the oldest.
+
+keycmd sits in front of every command a user runs through it, so its own startup is latency the user pays each time. A handful of imports cost more than everything else in the package together, and none of them is needed on every run: `keyring` (and the backend it goes on to discover) only once a credential is looked up, `pprint` only under `--verbose`, and `subprocess` only on the Windows path, which cannot replace its own process. They are imported inside the function that needs them, which keeps `import keycmd.cli` at roughly a third of what it would otherwise cost, and `test_cli_import_stays_lean` fails if one of them wanders back up to module level. Reach for a lazy import when adding a dependency that most runs will not touch, and leave the rest at the top of the file where they belong.

README.md

@@ -176,6 +176,17 @@ There are two main ways to use the CLI:
 
 The first is the most preferred method, since your secrets will only be exposed as environment variables during a one-off command. The latter is less preferable, but can be convenient if you are debugging some process that depends on the credentials you are exposing.
 
+Quoting the whole command as one argument is what lets you use your shell's syntax inside it, as in `keycmd 'echo $SECRET | tr a-z A-Z'`: keycmd hands that line to your shell exactly as you typed it, and your shell does the rest.
+
+You can also write the command out as separate arguments, and then keycmd keeps them separate:
+
+```bash
+# arrives as a single argument, spaces and all
+keycmd mytool --message 'hello world'
+```
+
+Since each argument is passed on as the word it was, your shell's syntax is *not* interpreted a second time in this form. If you want `$SECRET` expanded, either let your own shell expand it, or use the single argument form above.
+
 ## Configuration
 
 > **Note**
@@ -390,8 +401,9 @@ keycmd: merged config:
           'ARTIFACTS_TOKEN_B64': {'b64': True,
                                   'credential': 'korijn@poetry-repository-main',
                                   'username': 'korijn'}}}
-keycmd: exposing credential korijn@poetry-repository-main belonging to user korijn as environment variable ARTIFACTS_TOKEN (b64: False)
-keycmd: exposing credential korijn@poetry-repository-main belonging to user korijn as environment variable ARTIFACTS_TOKEN_B64 (b64: True)
+keycmd: keyring backend: <keyring.backends.Windows.WinVaultKeyring object at 0x000001F8C2A1B4D0>
+keycmd: exposing credential korijn@poetry-repository-main with user korijn as environment variable ARTIFACTS_TOKEN (b64: False, format: None)
+keycmd: exposing credential korijn@poetry-repository-main with user korijn as environment variable ARTIFACTS_TOKEN_B64 (b64: True, format: None)
 keycmd: detected shell: C:\Windows\System32\cmd.exe
 keycmd: running command: ['C:\\Windows\\System32\\cmd.exe', '/C', 'echo', '%ARTIFACTS_TOKEN_B64%']
 aSdtIG5vdCB0aGF0IHN0dXBpZCA6KQ==
@@ -403,6 +415,19 @@ Since keycmd uses keyring as its backend, you're not limited to just working wit
 
 See the [third party backends](https://github.com/jaraco/keyring/#third-party-backends) list for all options.
 
+### Startup time
+
+Left to itself, keyring works out which backend to use by loading every backend registered by every installed package and picking the most suitable one. That search runs on each `keycmd` invocation and, on a machine with a few packages installed, costs more time than the whole of the rest of a `keycmd` run put together.
+
+If that shows up in your shell, name the backend you already know you want, and keyring will load that one instead of going looking:
+
+```bash
+# in your shell profile; use the backend your platform actually uses
+export PYTHON_KEYRING_BACKEND=keyring.backends.SecretService.Keyring
+```
+
+`keyring --list-backends` prints the names to choose from, and `keycmd --verbose` will tell you which one ends up being used. The setting is keyring's own, so it applies to everything else using keyring too.
+
 ## Development
 
 This project uses [uv](https://docs.astral.sh/uv/) for dependency management, [ruff](https://docs.astral.sh/ruff/) for linting and formatting, and [ty](https://docs.astral.sh/ty/) for type checking.
@@ -425,7 +450,7 @@ uv run pytest tests
 
 ### Testing
 
-CI runs the test suite on Windows, macOS and Linux on the latest Python, plus one job on the oldest supported Python to catch anything newer than it allows. The suite adapts to the platform it runs on: it exercises every shell of the platform that is installed (`sh`, `bash` and `zsh` on posix, `cmd`, `powershell` and `pwsh` on Windows), and it skips the process replacement tests on Windows, which has no `execvpe`.
+CI runs the test suite on Windows, macOS and Linux on the latest Python, plus one job on the oldest supported Python to catch anything newer than it allows. The suite adapts to the platform it runs on: it exercises every shell of the platform that is installed (`sh`, `bash` and `zsh` on posix, `cmd` and `powershell` on Windows, and `pwsh` on either, since it installs everywhere and quotes its own way), and it skips the process replacement tests on Windows, which has no `execvpe`.
 
 The tests that read and write credentials need a real OS keyring that can be unlocked without user interaction. They are skipped with a message if there is no such keyring, so the rest of the suite still runs. Set `KEYCMD_REQUIRE_OS_KEYRING=1` to turn those skips into failures instead; CI sets it so that a broken keyring setup can't quietly reduce the coverage of a run.
 

keycmd/conf.py

@@ -1,9 +1,9 @@
 import tomllib
+from collections.abc import Iterator
 from pathlib import Path
-from pprint import pformat
-from typing import Any, Literal, NotRequired, TypedDict, cast, overload
+from typing import Any, NotRequired, TypedDict, cast
 
-from .logs import vlog
+from .logs import vlog, vlog_pretty
 
 
 class KeyConf(TypedDict):
@@ -54,27 +54,16 @@ def load_pyproj(path: Path) -> dict[str, Any]:
     return data.get("tool", {}).get("keycmd", {})
 
 
-@overload
-def find_file(fname: str, first_only: Literal[True] = True) -> Path | None: ...
+def walk_up() -> Iterator[Path]:
+    """Yield the working directory and its parents, nearest first
 
-
-@overload
-def find_file(fname: str, first_only: Literal[False]) -> list[Path]: ...
-
-
-def find_file(fname: str, first_only: bool = True) -> Path | list[Path] | None:
-    """Find a file by walking up the filesystem, starting at cwd"""
+    The walk stops at a git repository, so that it never leaves one, and
+    otherwise just below the home folder or at the root of the file system.
+    """
     cur = Path.cwd()
     home = Path.home()
-    results: list[Path] = []
     while True:
-        candidate = cur / fname
-        if candidate.is_file():
-            hit = candidate.resolve()
-            if first_only:
-                return hit
-            else:
-                results.append(hit)
+        yield cur
         # don't search outside git repositories
         if (cur / ".git").is_dir():
             break
@@ -85,12 +74,6 @@ def find_file(fname: str, first_only: bool = True) -> Path | list[Path] | None:
         if cur.parent == cur:
             break
         cur = cur.parent
-    if not first_only:
-        # return .keycmd files in order in which they should
-        # be loaded and merged
-        results.reverse()
-        return results
-    return None
 
 
 def defaults() -> dict[str, Any]:
@@ -129,22 +112,32 @@ def load_conf() -> Conf:
         vlog(f"loading config file {user_keyconf}")
         conf = merge_conf(conf, load_toml(user_keyconf))
 
-    # .keycmd
-    local_keycmds = find_file(".keycmd", first_only=False)
-    for local_keycmd in local_keycmds:
+    # both searches cover the same ground, so they share a single walk
+    local_keycmds: list[Path] = []
+    pyproj: Path | None = None
+    for directory in walk_up():
+        candidate = directory / ".keycmd"
+        if candidate.is_file():
+            local_keycmds.append(candidate.resolve())
+        if pyproj is None:
+            candidate = directory / "pyproject.toml"
+            if candidate.is_file():
+                pyproj = candidate.resolve()
+
+    # .keycmd, outermost first, so that the nearest one wins
+    for local_keycmd in reversed(local_keycmds):
         if local_keycmd == user_keyconf:
             vlog(f"skipping config file {local_keycmd} (already loaded)")
             continue
         vlog(f"loading config file {local_keycmd}")
         conf = merge_conf(conf, load_toml(local_keycmd))
 
     # pyproject.toml
-    pyproj = find_file("pyproject.toml")
     if pyproj is not None:
         vlog(f"loading config file {pyproj}")
         conf = merge_conf(conf, load_pyproj(pyproj))
 
-    vlog(f"merged config:\n{pformat(conf)}")
+    vlog_pretty("merged config:\n", conf)
 
     # the config is user authored, so this is a statement of the shape keycmd
     # expects rather than a guarantee; get_env reports violations as user errors

keycmd/creds.py

@@ -1,14 +1,12 @@
 import base64
 from os import environ
 
-import keyring
-
-from .conf import Conf
+from .conf import AliasConf, Conf, KeyConf
 from .logs import error, vlog
 from .wsl import share_env
 
-# credential, username, password, apply_b64, format string
-KeyData = tuple[str, str, str, bool, str | None]
+# credential, username, password
+KeyData = tuple[str, str, str]
 
 
 def b64(value: str) -> str:
@@ -36,52 +34,72 @@ def expose(
     env[key] = password
 
 
+def expose_conf(
+    env: dict[str, str],
+    name: str,
+    data: KeyData,
+    src: KeyConf | AliasConf,
+    what: str,
+) -> None:
+    """Expose a credential under the b64 and format options of one config entry
+
+    A key and an alias differ in where the credential comes from, not in
+    what happens to it on the way into the environment.
+    """
+    apply_b64 = src.get("b64", False)
+    format_string = src.get("format")
+    expose(env, name, *data, apply_b64, format_string)
+    vlog(
+        f"{what} as environment variable {name}"
+        f" (b64: {apply_b64}, format: {format_string})"
+    )
+
+
 def get_env(conf: Conf) -> dict[str, str]:
     """Load credentials from the OS keyring according to user configuration"""
     env = environ.copy()
 
     key_data: dict[str, KeyData] = {}
-    for key, src in conf["keys"].items():
-        password = keyring.get_password(src["credential"], src["username"])
-        if password is None:
-            error(
-                f"MISSING credential {src['credential']}"
-                f" with user {src['username']}"
-                f" as it does not exist"
+    keys = conf["keys"]
+    if keys:
+        # keyring, and the backend it goes on to discover, together cost
+        # more time and memory than everything else keycmd does; a run that
+        # looks up no credential should not have to pay for either
+        import keyring
+
+        # which backend keyring settles on decides where the credentials
+        # come from, so it is the first thing to check when they are not
+        # the ones that were expected
+        vlog(f"keyring backend: {keyring.get_keyring()}")
+
+        for key, src in keys.items():
+            credential = src["credential"]
+            username = src["username"]
+            password = keyring.get_password(credential, username)
+            if password is None:
+                error(
+                    f"MISSING credential {credential}"
+                    f" with user {username}"
+                    f" as it does not exist"
+                )
+            key_data[key] = (credential, username, password)
+            expose_conf(
+                env,
+                key,
+                key_data[key],
+                src,
+                f"exposing credential {credential} with user {username}",
             )
-        apply_b64 = src.get("b64", False)
-        format_string = src.get("format")
-        key_data[key] = (
-            src["credential"],
-            src["username"],
-            password,
-            apply_b64,
-            format_string,
-        )
-        expose(env, key, *key_data[key])
-        vlog(
-            f"exposing credential {src['credential']}"
-            f" with user {src['username']}"
-            f" as environment variable {key}"
-            f" (b64: {apply_b64}, format: {format_string})"
-        )
 
-    for alias, alias_src in conf.get("aliases", {}).items():
+    aliases = conf.get("aliases", {})
+    for alias, alias_src in aliases.items():
+        # the credential is already in hand, only the options differ
         data = key_data.get(alias_src["key"])
         if data is None:
             error(f"MISSING alias key {alias_src['key']}")
-        # re-use base data but replace apply_b64 and format_string
-        credential, username, password, _, _ = data
-        apply_b64 = alias_src.get("b64", False)
-        format_string = alias_src.get("format")
-        expose(env, alias, credential, username, password, apply_b64, format_string)
-        vlog(
-            f"aliasing {alias_src['key']}"
-            f" as environment variable {alias}"
-            f" (b64: {apply_b64}, format: {format_string})"
-        )
+        expose_conf(env, alias, data, alias_src, f"aliasing {alias_src['key']}")
 
     # an environment does not cross the boundary between WSL and windows
     # by itself, whichever side of it the command ends up running on
-    share_env(env, [*conf["keys"], *conf.get("aliases", {})])
+    share_env(env, [*keys, *aliases])
     return env

keycmd/logs.py

@@ -19,7 +19,20 @@ def log(msg: object, err: bool = False) -> None:
 
 def vlog(msg: object) -> None:
     if _verbose:
-        print(f"keycmd: {msg}")
+        log(msg)
+
+
+def vlog_pretty(prefix: str, value: object) -> None:
+    """Log a value under --verbose, pretty printed over as many lines as it takes
+
+    pprint costs more to import than everything else keycmd reaches for in
+    the standard library, and only this function ever needs it, so it stays
+    out of the import path until a verbose run asks for it.
+    """
+    if _verbose:
+        from pprint import pformat
+
+        log(f"{prefix}{pformat(value)}")
 
 
 def error(msg: object) -> NoReturn: