feat(ENG-15367): let commands choose what telemetry they send

AGENTS.md

@@ -72,7 +72,8 @@ Tests live alongside code: `cloudsmith_cli/cli/tests/` and `cloudsmith_cli/core/
 ## Usage telemetry
 
 - **What it is for:** one pseudonymous `cli command executed` event per run, sent to PostHog so the team can see which commands, flags and setups are used; every property states the question it answers.
-- **Rules:** only send names defined in code (commands, parameters, declared option spellings, env var names), enums, booleans and numbers, never a value from the user. Never send for credential helper invocations, `mcp start`, the `telemetry` group, shell completion, non-Cloudsmith API hosts, SSL verification off, development builds without the key, or any opt-out (`DO_NOT_TRACK`, `CLOUDSMITH_NO_TELEMETRY`, `telemetry = false`).
+- **Rules:** only send names defined in code (commands, parameters, declared option spellings, env var names), enums, booleans and numbers, never a value from the user. Never send for shell completion, non-Cloudsmith API hosts, SSL verification off, development builds without the key, or any opt-out (`DO_NOT_TRACK`, `CLOUDSMITH_NO_TELEMETRY`, `telemetry = false`).
+- **Per-command policy:** a command (or a group and everything under it) that should send less than everything declares it with `@telemetry.send(telemetry.NONE | ERRORS | USAGE)` from `cloudsmith_cli/cli/telemetry.py`, put directly under its `@<group>.command()` line; `ALL` (Sentry error reports and usage events) is the default and the nearest policy wins. Credential helpers, `mcp start` and the `telemetry` group use `ERRORS`. A policy only narrows what is sent: the opt-outs always win. `test_every_command_that_sends_less_than_all_is_listed_here` in `cli/tests/test_usage_capture.py` lists every such command, so update it in the same change.
 - **Adding a property:** declare it in `cloudsmith_cli/core/usage/event.py` (`PROPERTIES`, with the question it answers), fill it in `_values`, and extend `cloudsmith_cli/core/tests/test_usage_event.py`; new commands and flags need no telemetry change because capture reads the Click context.
 - **Performance:** `test_startup_imports.py` fails if importing the CLI loads heavy modules (HTTP clients, `cloudsmith_api`, `rich`) or any command module, so the usage code imports its HTTP session inside functions that run at the end of a run; the wait is capped by the shared 1 s budget and failures stay silent.
-- **Testing locally:** `CLOUDSMITH_TELEMETRY=log` prints the event instead of sending it, and a test project needs `CLOUDSMITH_TELEMETRY_POSTHOG_KEY` (plus `CLOUDSMITH_TELEMETRY_POSTHOG_HOST` for a non-default host), because development builds (source checkouts and editable installs) send nothing without that key. Use the `usage_events` pytest fixture in tests.
+- **Testing locally:** `CLOUDSMITH_TELEMETRY=log` prints the usage event and any Sentry error report (scrubbed) instead of sending them, and a test project needs `CLOUDSMITH_TELEMETRY_POSTHOG_KEY` (plus `CLOUDSMITH_TELEMETRY_POSTHOG_HOST` for a non-default host), because development builds (source checkouts and editable installs) send nothing without that key. Use the `usage_events` pytest fixture in tests.

CHANGELOG.md

@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
   - `cloudsmith telemetry id` shows the install ID that the CLI stores in `telemetry.json` in its config directory, or `none`.
   - `cloudsmith telemetry reset` deletes the stored install ID. The CLI creates a new, unrelated ID on the next command.
   - All three commands support `-F json` and `-F pretty_json`.
+- Added `CLOUDSMITH_TELEMETRY=log` to see what the CLI would send. When a command fails with an unexpected error, the CLI writes the error report to stderr after it removes the private data, and it does not send the report. The CLI also writes the usage event that it would send. The opt-outs still apply: with `DO_NOT_TRACK`, `CLOUDSMITH_NO_TELEMETRY`, or `telemetry = false`, the CLI writes nothing.
 
 ### Changed
 

cloudsmith_cli/cli/command.py

@@ -278,7 +278,8 @@ def main(self, *args, **kwargs):
             from ..core import telemetry
 
             # Reporting is silent: the user sees the same line either way.
-            telemetry.report_exception(e)
+            if _errors_allowed():
+                telemetry.report_exception(e)
             click.echo(f"Error: {telemetry.format_exception_summary(e)}", err=True)
             _echo_certificate_hint(e)
             sys.exit(1)
@@ -287,6 +288,22 @@ def main(self, *args, **kwargs):
             usage_capture.exit_run()
 
 
+def _errors_allowed():
+    """Whether the command's ``@telemetry.send`` policy allows error reports.
+
+    The contexts are still recorded here (they are cleared in ``finally``).
+    """
+    try:
+        from ..core.telemetry_common import Send
+        from . import usage_capture
+
+        return Send.ERRORS in usage_capture.current_policy()
+    except Exception:  # pylint: disable=broad-exception-caught
+        # A policy narrows reporting; if it cannot be read, fall back to the
+        # default (the user's opt-outs are still checked by the reporter).
+        return True
+
+
 def _finish_usage(usage, started, *, nested):
     """Hand the run to usage telemetry; never raises, never prints."""
     try:

cloudsmith_cli/cli/commands/credential_helper/cargo.py

@@ -12,11 +12,14 @@
 import click
 
 from ....credential_helpers.cargo import execute
+from ... import telemetry
 from ...decorators import common_api_auth_options, resolve_credentials
 from ...usage_capture import TrackedCommand
 
 
+# Run by package managers on every request: report crashes, not usage.
 @click.command(cls=TrackedCommand, context_settings={"ignore_unknown_options": True})
+@telemetry.send(telemetry.ERRORS)
 @click.option(
     "--cargo-plugin",
     is_flag=True,

cloudsmith_cli/cli/commands/credential_helper/docker.py

@@ -12,11 +12,14 @@
 import click
 
 from ....credential_helpers.docker import execute
+from ... import telemetry
 from ...decorators import common_api_auth_options, resolve_credentials
 from ...usage_capture import TrackedCommand
 
 
+# Run by package managers on every request: report crashes, not usage.
 @click.command(cls=TrackedCommand)
+@telemetry.send(telemetry.ERRORS)
 @click.argument("operation", required=False, default="get")
 @common_api_auth_options
 @resolve_credentials

cloudsmith_cli/cli/commands/credential_helper/generic.py

@@ -10,6 +10,7 @@
 import click
 
 from ....credential_helpers.generic import execute
+from ... import telemetry
 from ...decorators import (
     common_api_auth_options,
     common_cli_config_options,
@@ -18,7 +19,9 @@
 from ...usage_capture import TrackedCommand
 
 
+# Run by package managers on every request: report crashes, not usage.
 @click.command(cls=TrackedCommand)
+@telemetry.send(telemetry.ERRORS)
 @common_cli_config_options
 @common_api_auth_options
 @resolve_credentials

cloudsmith_cli/cli/commands/credential_helper/pnpm.py

@@ -11,12 +11,15 @@
 import click
 
 from ....credential_helpers.pnpm import execute
+from ... import telemetry
 from ...config import Options
 from ...decorators import common_api_auth_options, resolve_credentials
 from ...usage_capture import TrackedCommand
 
 
+# Run by package managers on every request: report crashes, not usage.
 @click.command(cls=TrackedCommand)
+@telemetry.send(telemetry.ERRORS)
 @click.argument("repo", required=False, default="npm.cloudsmith.io")
 @common_api_auth_options
 @resolve_credentials
@@ -65,7 +68,7 @@
 
     exit_code, stdout, stderr = execute(
         repo,
         credential=opts.credential,
         api_host=opts.api_host,
         org=opts.org,
     )

cloudsmith_cli/cli/commands/credential_helper/terraform.py

@@ -14,6 +14,7 @@
 import click
 
 from ....credential_helpers.terraform import execute
+from ... import telemetry
 from ...decorators import (
     common_api_auth_options,
     common_cli_config_options,
@@ -22,7 +23,9 @@
 from ...usage_capture import TrackedCommand
 
 
+# Run by package managers on every request: report crashes, not usage.
 @click.command(cls=TrackedCommand, context_settings={"ignore_unknown_options": True})
+@telemetry.send(telemetry.ERRORS)
 @click.option(
     "-r",
     "--repo",

cloudsmith_cli/cli/commands/mcp.py

@@ -18,7 +18,7 @@
     from ...core.mcp.data import OpenAPITool
     from ..config import Options
 
-from .. import command, decorators, utils
+from .. import command, decorators, telemetry, utils
 from ..exceptions import handle_api_exceptions
 from .main import main
 
@@ -48,6 +48,8 @@ def mcp_(ctx, opts):  # pylint: disable=unused-argument
 
 
 @mcp_.command(name="start")
+# Long-running: report crashes, not usage.
+@telemetry.send(telemetry.ERRORS)
 @decorators.initialise_api
 @decorators.initialise_mcp
 @click.pass_context

cloudsmith_cli/cli/commands/telemetry.py

@@ -8,6 +8,9 @@
 from ...core import telemetry_common
 from ...core.usage import identity, settings, state
 from .. import command, decorators, usage_capture, utils
+
+# Not `from .. import telemetry`: this module defines the `telemetry` group.
+from ..telemetry import ERRORS, send
 from .main import main
 
 
@@ -41,6 +44,8 @@ def _reason(opts, typed=()):
 
 
 @main.group(cls=command.AliasGroup)
+# Manages telemetry itself: report crashes, never usage.
+@send(ERRORS)
 def telemetry():
     """See and control pseudonymous usage telemetry."""
 

cloudsmith_cli/cli/telemetry.py

@@ -0,0 +1,23 @@
+# Copyright 2026 Cloudsmith Ltd
+"""What a command may send: ``@telemetry.send(telemetry.ERRORS)``.
+
+``ALL`` (error reports and usage events) is the default, so only commands
+that send less need the decorator:
+
+* ``NONE``: nothing.
+* ``ERRORS``: Sentry reports of unexpected exceptions only.
+* ``USAGE``: the usage event only.
+
+A policy on a group applies to every command under it unless a command sets
+its own. The user's opt-outs always win. Imported by command modules, so it
+must stay light: only the stdlib-only ``core.telemetry_common``.
+"""
+
+from ..core.telemetry_common import Send, send
+
+NONE = Send.NONE
+ERRORS = Send.ERRORS
+USAGE = Send.USAGE
+ALL = Send.ALL
+
+__all__ = ["ALL", "ERRORS", "NONE", "USAGE", "Send", "send"]

cloudsmith_cli/cli/tests/test_unexpected_errors.py

@@ -493,3 +493,78 @@ def test_python_m_exits_1_on_unexpected_exception(tmp_path, monkeypatch, capsys)
 
     assert exc_info.value.code == 1
     assert "Error: RuntimeError: boom" in capsys.readouterr().err
+
+
+# --- Per-command policy (@telemetry.send) and log mode ----------------------
+
+
+def _crashing_app():
+    from cloudsmith_cli.cli import telemetry as policy
+    from cloudsmith_cli.cli.command import AliasGroup
+
+    @click.group(cls=AliasGroup)
+    def app():
+        pass
+
+    def crash():
+        raise RuntimeError("boom")
+
+    for name, level in [
+        ("default", None),
+        ("errors", policy.ERRORS),
+        ("usage", policy.USAGE),
+        ("none", policy.NONE),
+    ]:
+        callback = crash if level is None else policy.send(level)(lambda: crash())
+        app.command(name=name)(callback)
+    return app
+
+
+@pytest.mark.parametrize(
+    "command, reported", [("default", 1), ("errors", 1), ("usage", 0), ("none", 0)]
+)
+def test_only_commands_whose_policy_has_errors_report(
+    runner, sentry_events, command, reported
+):
+    result = runner.invoke(_crashing_app(), [command])
+    # The user sees the same thing either way.
+    assert result.exit_code == 1
+    assert result.stderr.strip().endswith("Error: RuntimeError: boom")
+    assert len(sentry_events) == reported
+
+
+def test_log_mode_prints_the_scrubbed_report_and_sends_nothing(
+    runner, tmp_path, sentry_events, monkeypatch
+):
+    monkeypatch.setenv("CLOUDSMITH_TELEMETRY", "log")
+    with whoami_raises(RuntimeError("boom")):
+        result = runner.invoke(main, whoami_args(tmp_path))
+
+    assert result.exit_code == 1
+    assert sentry_events == []
+    assert "would send error report (log mode, nothing sent)" in result.stderr
+    printed = result.stderr.split("nothing sent):\n", 1)[1]
+    logged, _ = json.JSONDecoder().raw_decode(printed)
+    assert logged["tags"]["command"] == "whoami"
+    assert logged["exception"]["values"][-1]["type"] == "RuntimeError"
+    # Scrubbed exactly as a sent report would be.
+    assert API_KEY not in result.stderr
+
+
+def test_log_mode_prints_without_a_dsn(runner, tmp_path, sentry_events, monkeypatch):
+    monkeypatch.setenv("CLOUDSMITH_TELEMETRY", "log")
+    monkeypatch.setenv(telemetry.DSN_ENV, "")
+    with whoami_raises(RuntimeError("boom")):
+        result = runner.invoke(main, whoami_args(tmp_path))
+    assert "would send error report" in result.stderr
+
+
+def test_log_mode_still_honours_the_opt_out(
+    runner, tmp_path, sentry_events, monkeypatch
+):
+    monkeypatch.setenv("CLOUDSMITH_TELEMETRY", "log")
+    monkeypatch.setenv("DO_NOT_TRACK", "1")
+    with whoami_raises(RuntimeError("boom")):
+        result = runner.invoke(main, whoami_args(tmp_path))
+    assert "would send error report" not in result.stderr
+    assert sentry_events == []

cloudsmith_cli/cli/tests/test_usage_capture.py

@@ -4,6 +4,7 @@
 import pytest
 from click.testing import CliRunner
 
+from cloudsmith_cli.cli import telemetry as cli_telemetry
 from cloudsmith_cli.cli import usage_capture
 from cloudsmith_cli.cli.command import AliasGroup
 
@@ -304,3 +305,90 @@ def test_record_credential_keeps_the_last_resolved_source():
         store_ctx, SimpleNamespace(source_name="credentials_file")
     )
     assert store_ctx.meta[usage_capture.META_KEY]["auth_source"] == "credentials_file"
+
+
+def _every_command():
+    """Yield (path, command, parent path) for the whole real CLI."""
+    import importlib
+    import pkgutil
+
+    from cloudsmith_cli.cli import commands
+    from cloudsmith_cli.cli.commands.main import main
+
+    for info in pkgutil.walk_packages(
+        commands.__path__, prefix=commands.__name__ + "."
+    ):
+        importlib.import_module(info.name)
+
+    def walk(cmd, path):
+        yield path, cmd
+        for name, sub in getattr(cmd, "commands", {}).items():
+            yield from walk(sub, (*path, name))
+
+    yield from walk(main, ())
+
+
+def test_every_command_that_sends_less_than_all_is_listed_here():
+    from cloudsmith_cli.cli.telemetry import ALL, ERRORS
+    from cloudsmith_cli.core.telemetry_common import declared_policy
+
+    # Change this only on purpose: it is the reviewed list of what does not
+    # send everything.
+    expected = {
+        "credential-helper cargo": ERRORS,
+        "credential-helper docker": ERRORS,
+        "credential-helper generic": ERRORS,
+        "credential-helper pnpm": ERRORS,
+        "credential-helper terraform": ERRORS,
+        "mcp start": ERRORS,
+        "telemetry": ERRORS,
+        "telemetry id": ERRORS,
+        "telemetry reset": ERRORS,
+        "telemetry status": ERRORS,
+    }
+    effective = {}
+    for path, cmd in _every_command():
+        inherited = effective.get(" ".join(path[:-1]), ALL) if path else ALL
+        effective[" ".join(path)] = declared_policy(cmd) or inherited
+    assert {p: v for p, v in effective.items() if v is not ALL} == expected
+
+
+@click.group(cls=AliasGroup)
+def policy_app():
+    pass
+
+
+@policy_app.group(cls=AliasGroup)
+@cli_telemetry.send(cli_telemetry.ERRORS)
+def quiet():
+    pass
+
+
+@quiet.command()
+@click.pass_context
+def inherits(ctx):
+    CAPTURED["facts"] = usage_capture.collect_facts(
+        ctx.meta, outcome="success", exit_code=0, duration_ms=1, nested=False
+    )
+
+
+@quiet.command()
+@cli_telemetry.send(cli_telemetry.ALL)
+@click.pass_context
+def overrides(ctx):
+    CAPTURED["facts"] = usage_capture.collect_facts(
+        ctx.meta, outcome="success", exit_code=0, duration_ms=1, nested=False
+    )
+
+
+@pytest.mark.parametrize(
+    "args, allowed",
+    [(["quiet", "inherits"], False), (["quiet", "overrides"], True)],
+)
+def test_the_command_policy_reaches_the_run_facts(args, allowed):
+    CliRunner().invoke(policy_app, args, catch_exceptions=False)
+    assert CAPTURED["facts"].usage_allowed is allowed
+
+
+def test_a_command_without_a_policy_sends_usage():
+    assert _run(["listing"]).usage_allowed is True