Skip to content

Track CLI invocation source and full command#1293

Open
ItIsUday wants to merge 7 commits intomainfrom
feature/track-cli-invokation
Open

Track CLI invocation source and full command#1293
ItIsUday wants to merge 7 commits intomainfrom
feature/track-cli-invokation

Conversation

@ItIsUday
Copy link
Copy Markdown
Contributor

@ItIsUday ItIsUday commented Apr 28, 2026

Summary

  • Sends a COMMAND_INVOCATION tracking event on every CLI invocation with the full command string (rawCommand), exit code, caller, and CI provider
  • Auto-detects the CLI command from argv (maps to Command enum, falls back to UNKNOWN for typos)
  • SMART_TESTS_CALLER env var lets wrappers (e.g. GitHub Action) identify themselves
  • CI provider auto-detected from standard env vars (GITHUB_ACTIONS, JENKINS_URL, CIRCLECI, CODEBUILD_BUILD_ID)
  • caller, ciProvider, exitCode sent inside metadata JSONB — no DB migration needed
  • Existing error/performance tracking unchanged

Test plan

  • Unit tests for _detect_command() across all commands + typos + global options
  • Unit tests for send_command_tracking() payload structure
  • Unit tests for caller/ciProvider in metadata for all event types
  • Full test suite passes (376 tests, 0 new failures)

ItIsUday and others added 2 commits April 28, 2026 14:57
Addresses LCHUX-315. The backend can now identify whether CLI
invocations come from direct usage, our GitHub Action wrapper, or
another integration by reading the User-Agent header on every request.

- SMART_TESTS_CALLER env var (defaults to "cli") lets wrappers
  identify themselves (e.g. github-action, jenkins-plugin)
- detect_ci_provider() auto-detects the CI environment from standard
  env vars (GITHUB_ACTIONS, JENKINS_URL, CIRCLECI, CODEBUILD_BUILD_ID)
- Both values are appended to User-Agent as Caller/{value} CI/{value}
- Both values are also included in the cli_tracking payload

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add test_tracking.py covering detect_ci_provider() and caller/
  ciProvider fields in tracking payloads
- Add User-Agent tests for caller and CI provider segments
- Replace clear=True with surgical env cleanup in test_http_client.py
  to avoid wiping SMART_TESTS_BASE_URL (same issue as PR #1279)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ItIsUday ItIsUday self-assigned this Apr 28, 2026
ItIsUday and others added 4 commits May 4, 2026 14:45
- Add send_command_tracking() to send a COMMAND_INVOCATION event with
  the full CLI argv, exit code, caller, and CI provider in metadata
- Add _detect_command() to map argv to Command enum values; falls back
  to UNKNOWN for typos or unrecognized commands
- Refactor TrackingClient: extract construct_payload() from _post_payload()
  so both send_command_tracking and existing methods share payload construction
- Move caller/ciProvider into metadata JSONB (not top-level fields)
- Add UNKNOWN and COMMAND_INVOCATION to respective enums

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Capture sys.argv and exit code, send COMMAND_INVOCATION tracking
event after every CLI run (including typos and failures).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Caller and CI provider are now sent via cli_tracking metadata only.
Remove Caller/ and CI/ segments from User-Agent header.

Update test_tracking with tests for _detect_command, send_command_tracking,
and caller/ciProvider in metadata. Revert test_http_client to original
User-Agent assertions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ItIsUday ItIsUday changed the title Track CLI invocation source via User-Agent Track CLI invocation source and full command May 4, 2026
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ItIsUday
Copy link
Copy Markdown
Contributor Author

ItIsUday commented May 5, 2026

There is two pending items though, would like to have an opinion on how to proceed

  • There's no input sanitization yet
  • The full raw command is captured, so if a command goes something like this, smart-tests record tests file --session 10 ./test-results/*.xml, then the full list of /test-results/ would be expanded in the raw string, which could be huge. Maybe we can just truncate the raw command?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant