feat(python-setup): report manual-setup opt-out telemetry - #2170
Open
rugpanov wants to merge 3 commits into
Open
feat(python-setup): report manual-setup opt-out telemetry#2170rugpanov wants to merge 3 commits into
rugpanov wants to merge 3 commits into
Conversation
*Why* The manual opt-out added in #2158 (databricks.python.environmentSetup: manual, plus the "Use manual setup" button on the E_FETCH failure) is only observable today via the generic commandExecution event, and opting out by editing the setting directly is invisible. We can't measure how often users fall back to a manually managed environment, or from where. *What* - New python_env.manual_setup.optout event, recorded when the opt-out command write succeeds, with scope (workspace|global) and source (error_popup| command_palette). The E_FETCH popup tags its invocation as error_popup; palette invocations read as command_palette. - Add setupMode (uv | pip | fallback-pip) to the once-per-session python_env.setup.detected event so opt-out prevalence is measurable per session. Derived in one place (resolveSetupMode, beside isUvSetupSuitable): manual => fallback-pip; auto splits by uv-suitability into uv / pip. The exact manager stays in primaryManager. Categorical data only; no paths, package names, or PII. *Verification* - yarn build (tsc) and yarn test:lint clean. - yarn test:unit green (1061 passing). Added unit tests: resolveSetupMode cases; recordManualSetupOptout scope/source; emitDetection setupMode for uv / pip / fallback-pip; and the popup passing {source: "error_popup"}. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
🤖 Integration tests ❌ 1 of 41 test jobs failed for |
*Why* Multi-source review of the opt-out telemetry raised: the event over-counted when the setting was already manual; the command-handler glue was untested; and two type names collided / broke the repo's OptOut casing. *What* - Record the opt-out only on a genuine auto->manual transition (not on every successful write), so repeat invocations while already manual don't inflate the count. Extract the handler into optOutOfAutomatedPythonSetup with injected deps, covered by unit tests (transition, already-manual, no-folder, write failure). - Rename the reported-flow type PythonEnvSetupMode -> ReportedSetupMode to avoid collision with PythonEnvironmentSetupMode / PythonSetupMode. - Rename Optout -> OptOut in identifiers to match the repo's optOutOfInstallPrompt. - Clarify the setupMode field comment: emitted per (trigger, project); measure prevalence by distinct user, not raw event share. - Use import type for the now type-only telemetry imports. *Verification* - yarn build (tsc) and yarn test:lint clean; yarn test:unit green (1065 passing). Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
🤖 Integration tests triggered for |
The helper injects out the VS Code UI globals (window/workspace/commands) but still imports the ConfigurationTarget enum as a runtime value, so its tests run under the extension-host harness. Reword the doc-comment that claimed 'without VS Code globals'. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
Contributor
Author
|
🤖 Integration tests ✅ passed for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up telemetry deferred from #2158 (manual opt-out for uv-native Python setup). Makes the opt-out measurable — both the deliberate action and the standing prevalence across sessions.
What & why
Today the only signal for a manual opt-out is the generic
commandExecutionevent that fires for the "Use manual setup" button, and opting out by editingdatabricks.python.environmentSetupdirectly in settings is invisible. This adds:python_env.manual_setup.optout— recorded when the opt-out command write succeeds (not on a failed write). Fields:scope:workspace(a folder is open) |global(no-folder fallback).source:error_popup(the E_FETCH "Use manual setup" button) |command_palette.setupModeonpython_env.setup.detected(once per session):uv(uv-native flow),pip(legacy/non-uv — the exact manager stays inprimaryManager), orfallback-pip(opted out). Theauto|manualsetting is recoverable from it (fallback-pip⇔manual). Derived byresolveSetupMode, kept besideisUvSetupSuitableso "which flow is in effect" has one source of truth.Categorical data only — no paths, package names, or PII (enforced by the allowlist discipline in
telemetry/).Out of scope
The dashboard panel that visualises these events is a separate follow-up — the events need a released build to reach users first (the same "not released yet" gap the existing
reportOfferedcounters have).Verification
yarn build(tsc) +yarn test:lintclean.yarn test:unitgreen (1061 passing), including new unit tests forresolveSetupMode,recordManualSetupOptout,emitDetection'ssetupMode, and the popup passing{source: "error_popup"}.This pull request and its description were written by Isaac.