Skip to content

Record fetched URL path in doc fetch analytics - #8554

Closed
brandonreid wants to merge 2 commits into
mainfrom
doc-fetch-url-path-telemetry
Closed

brandonreid wants to merge 2 commits into
mainfrom
doc-fetch-url-path-telemetry

Conversation

@brandonreid

Copy link
Copy Markdown
Contributor

WHY are these changes introduced?

shopify doc fetch is what the shopify-app-store-review skill uses to pull the App Store AI self-review requirements page. Since #7771 the command event already carries api_key/project_type when run inside an app, but nothing in the public payload says which document was fetched — the URL only exists in args, which is privacy: TEXT and redacted in the warehouse. So today we can see "this app ran doc fetch" but not "this app ran a self-review", which blocks correlating skill usage with review outcomes.

WHAT is this pull request doing?

  • Adds cmd_doc_fetch_url_path to the app_cli3_command public schema and bumps MONORAIL_COMMAND_TOPIC to 1.29.
  • docFetchService records URL.pathname only — no query string or fragment — after the host allowlist check passes, so the value is always a public shopify.dev docs path.
  • Extends CmdFieldsFromMonorail with the cmd_doc_ prefix so the field type-checks through addPublicMetadata.

Depends on the schema landing first: Shopify/monorail#24489.

How to test your changes?

pnpm vitest run packages/cli/src/cli/services/commands/doc/fetch.test.ts

Or with SHOPIFY_CLI_ANALYTICS_DEBUG-style verbose output, run shopify doc fetch --url https://shopify.dev/docs/apps/launch/app-store-review/app-store-ai-self-review-requirements --verbose and confirm the analytics payload contains cmd_doc_fetch_url_path.

Measuring impact

  • Existing analytics will cater for this addition — shopify-dw.base.base__monorail_app_cli3_command picks up new public fields on the 1.x topic.

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes

Adds cmd_doc_fetch_url_path to the app_cli3_command public payload
(schema 1.29) and populates it from doc fetch with the pathname only,
so fetches of specific documents (e.g. the App Store self-review
requirements page) can be attributed alongside the app context the
global metadata hook already provides.

Assisted-By: devx/56e14e24-e249-4bda-810f-42156f6e7afb
@github-actions github-actions Bot added the Area: @shopify/cli @shopify/cli package issues label Sep 15, 2026
Addresses review on Shopify/monorail#24489: the field can now only hold
the path of a shopify.dev document that actually resolved, never a
typo'd or made-up path from a failed request.

Assisted-By: devx/56e14e24-e249-4bda-810f-42156f6e7afb
@github-actions

Copy link
Copy Markdown
Contributor

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

  • Some seemingly private modules might be re-exported through public modules.
  • If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/node/metadata.d.ts
@@ -34,7 +34,7 @@ export type SensitiveSchema<T> = T extends RuntimeMetadataManager<infer _TPublic
  * @returns A container for the metadata.
  */
 export declare function createRuntimeMetadataContainer<TPublic extends AnyJson, TSensitive extends AnyJson = Record<string, never>>(defaultPublicMetadata?: Partial<TPublic>): RuntimeMetadataManager<TPublic, TSensitive>;
-type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
+type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'cmd_doc_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
 declare const coreData: RuntimeMetadataManager<CmdFieldsFromMonorail, {
     commandStartOptions: {
         startTime: number;
packages/cli-kit/dist/public/node/monorail.d.ts
@@ -2,7 +2,7 @@ import { JsonMap } from '../../private/common/json.js';
 import { DeepRequired } from '../common/ts/deep-required.js';
 export { DeepRequired };
 type Optional<T> = T | null;
-export declare const MONORAIL_COMMAND_TOPIC = "app_cli3_command/1.28";
+export declare const MONORAIL_COMMAND_TOPIC = "app_cli3_command/1.29";
 export interface Schemas {
     [MONORAIL_COMMAND_TOPIC]: {
         sensitive: {
@@ -49,6 +49,7 @@ export interface Schemas {
             cmd_all_timing_network_ms?: Optional<number>;
             cmd_all_timing_prompts_ms?: Optional<number>;
             cmd_all_timing_active_ms?: Optional<number>;
+            cmd_doc_fetch_url_path?: Optional<string>;
             env_auto_upgrade_enabled?: Optional<boolean>;
             env_auto_upgrade_accepted?: Optional<boolean>;
             env_auto_upgrade_skipped_reason?: Optional<string>;

@brandonreid

Copy link
Copy Markdown
Contributor Author

Opting to close. Will rely on the private args URL for now instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: @shopify/cli @shopify/cli package issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant