Skip to content

feat(searchconsole): add url inspection command - #1094

Open
laihenyi wants to merge 2 commits into
openclaw:mainfrom
laihenyi:feat/searchconsole-url-inspect
Open

feat(searchconsole): add url inspection command#1094
laihenyi wants to merge 2 commits into
openclaw:mainfrom
laihenyi:feat/searchconsole-url-inspect

Conversation

@laihenyi

@laihenyi laihenyi commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Scope

Adds gog searchconsole inspect <siteUrl> <url> — a thin wrapper around the Search Console URL Inspection API (urlInspection.index:inspect) that exposes per-page indexing detail not currently available through searchconsole sites / searchanalytics / query / sitemaps:

  • coverage state (the per-URL indexing reason shown in the Search Console UI)
  • indexing / page-fetch / robots.txt state, last crawl time, crawled-as
  • google vs. user canonical, sitemaps, count of referring URLs

Implementation notes:

  • Pure additive subcommand under the existing searchconsole group; no new OAuth scope — uses the already-granted webmasters scope via the existing service wrapper, so no re-authorization is needed.
  • --language (BCP-47, default en-US) for translated issue messages; --json output for scripting.
  • Text output follows the existing sanitize/table conventions of the neighboring Search Console commands.

Changes

  • internal/cmd/searchconsole.go — new SearchConsoleInspectCmd + registration in the command group
  • internal/cmd/searchconsole_more_test.go — httptest coverage: JSON + text output (asserts request payload incl. default language), empty-arg usage errors before any service call, and service-error propagation
  • internal/googleauth/service.go — regenerate README auth-services table source of truth (Search Console note now includes URL Inspection)
  • Regenerated command reference (docs/commands/gog-searchconsole-inspect.md, index/generated pages), Search Console skill command table, README product table
  • Unreleased changelog note with PR reference and contributor credit

Testing performed

  • go test ./internal/cmd/ (incl. all SearchConsole tests), ./internal/googleauth, ./scripts (README freshness) pass
  • make lint (0 issues), make deadcode, make docs-check (767 command pages), make agent-skills-check all pass
  • go build ./cmd/gog passes

Live verification (real Search Console property, 2026-09-07)

Ran against a verified property owned by the test account (siteOwner level). Account alias, property, and inspected URLs are redacted below for privacy; the output fields themselves are verbatim. Details can be shared privately with a maintainer on request.

gog --json --account <redacted-account> searchconsole inspect sc-domain:<redacted-property> https://<redacted-host>/

{
  "inspectionResult": {
    "indexStatusResult": {
      "coverageState": "Submitted and indexed",
      "crawledAs": "MOBILE",
      "googleCanonical": "https://<redacted-host>/",
      "indexingState": "INDEXING_ALLOWED",
      "lastCrawlTime": "2026-09-05T23:25:28Z",
      "pageFetchState": "SUCCESSFUL",
      "referringUrls": ["<redacted>", "…4 total"],
      "robotsTxtState": "ALLOWED",
      "userCanonical": "https://<redacted-host>/",
      "verdict": "PASS"
    },
    "inspectionResultLink": "https://search.google.com/search-console/inspect?resource_id=…&id=<redacted>"
  },
  "inspection_url": "https://<redacted-host>/",
  "site_url": "sc-domain:<redacted-property>"
}

Text output (same invocation, no --json):

FIELD             VALUE
url               https://<redacted-host>/
verdict           PASS
coverage_state    Submitted and indexed
indexing_state    INDEXING_ALLOWED
page_fetch_state  SUCCESSFUL
robots_txt_state  ALLOWED
crawled_as        MOBILE
last_crawl_time   2026-09-05T23:25:28Z
canonical         https://<redacted-host>/
referring_urls    4

Add `gog searchconsole inspect <siteUrl> <url>` wrapping the URL
Inspection API (searchconsole/v1 urlInspection.index:inspect), which
exposes per-page indexing detail (coverage state, page fetch state,
robots.txt state, indexing state, last crawl time, canonical vs user
canonical, sitemaps, referring URLs) not available through the existing
sites/searchanalytics/sitemaps commands. Uses the already-granted
`webmasters` OAuth scope, so no re-authorization is needed.

Supports `--language` (BCP-47) for translated issue messages and `--json`
for scripting.

- httptest coverage for JSON and text output, empty-arg usage errors
  before service creation, and service errors.
- Regenerate command reference (docs/commands), Search Console skill
  command table, and README product table; add Unreleased changelog note.

Live-tested against a real Search Console property: not yet; regression
coverage and account/scope expectations verified locally.
@laihenyi
laihenyi requested a review from a team as a code owner September 7, 2026 02:46
@clawsweeper

clawsweeper Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 7, 2026
@clawsweeper

clawsweeper Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 6, 2026, 11:15 PM ET / September 7, 2026, 03:15 UTC.

ClawSweeper review

What this changes

Adds a Search Console URL inspection command with translated issue messages, JSON and table output, focused tests, and updated command documentation.

Merge readiness

Ready for maintainer review

This PR remains useful: current main lacks the inspection command, the supplied real-property output demonstrates the added behavior, and no blocking defect was found. The earlier release-note finding is resolved.

Priority: P2
Reviewed head: 617bce86933f92de1513f55aad79540b4f7456b7

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused implementation with relevant real-world output, supplemental tests, and no remaining actionable findings.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The supplied real-property run exercises the new inspection handler through the authenticated CLI and shows Google's index status in both JSON and text output.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): The supplied real-property run exercises the new inspection handler through the authenticated CLI and shows Google's index status in both JSON and text output.
Evidence reviewed 8 items Policy and patch scope: Read the complete root AGENTS.md; filesystem discovery found no nested AGENTS.md or maintainer notes. Applied its read-only review, output, and unreleased changelog guidance. The pinned introduced delta contains 10 files, with 260 additions and 3 deletions.
Distinct capability absent from main: The main command group contains sites, searchanalytics, query, and sitemaps, with no inspect command. A main-tree search found URL Inspection only in the existing read-only transport allowlist and its test, not an exposed inspection implementation. No merged replacement is established by the supplied context. Inspection of the release tree could not complete because a required local Git object was unavailable.
Production request and output: The added handler validates empty arguments before service creation, uses the existing account-selected Search Console client, sends the property, inspection URL, and language, and renders the API result through existing JSON/table helpers.
Findings None None.
Security None None.

How this fits together

gogcli's Search Console commands send authenticated requests to Google's API. The new command accepts a property and page URL, then returns Google's indexed-page status for scripts or terminal users.

flowchart LR
  A[Property and page URL] --> B[Inspection command]
  C[Selected Google account] --> D[Existing authenticated client]
  B --> D
  D --> E[Google URL Inspection API]
  E --> F[JSON or terminal table]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test growth Production +82/-1; tests +122/-0 Production growth implements one bounded API wrapper and its service description, supported by focused command tests.

Technical review

Best possible solution:

Expose indexed-page diagnostics through the existing Search Console command group while preserving established authentication, read-only enforcement, and output conventions.

Do we have a high-confidence way to reproduce the issue?

Not applicable to a new capability; the supplied real-property transcript demonstrates the added command's successful JSON and text behavior.

Is this the best way to solve the issue?

Yes. A thin wrapper around the existing authenticated client is consistent with neighboring commands and avoids new credentials, dependencies, or persistent state.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 981ca4a163e8.

Labels

Label justifications:

  • P2: This is a useful, bounded addition to an existing service integration without evidence of an urgent user-facing regression.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The supplied real-property run exercises the new inspection handler through the authenticated CLI and shows Google's index status in both JSON and text output.
  • proof: sufficient: Contributor real behavior proof is sufficient. The supplied real-property run exercises the new inspection handler through the authenticated CLI and shows Google's index status in both JSON and text output.

Evidence

What I checked:

  • Policy and patch scope: Read the complete root AGENTS.md; filesystem discovery found no nested AGENTS.md or maintainer notes. Applied its read-only review, output, and unreleased changelog guidance. The pinned introduced delta contains 10 files, with 260 additions and 3 deletions. (AGENTS.md, 617bce86933f)
  • Distinct capability absent from main: The main command group contains sites, searchanalytics, query, and sitemaps, with no inspect command. A main-tree search found URL Inspection only in the existing read-only transport allowlist and its test, not an exposed inspection implementation. No merged replacement is established by the supplied context. Inspection of the release tree could not complete because a required local Git object was unavailable. (internal/cmd/searchconsole.go:18, 981ca4a163e8)
  • Production request and output: The added handler validates empty arguments before service creation, uses the existing account-selected Search Console client, sends the property, inspection URL, and language, and renders the API result through existing JSON/table helpers. (internal/cmd/searchconsole.go:519, 617bce86933f)
  • Existing safety integration: The unchanged read-only transport already permits the URL Inspection POST endpoint. The branch changes only the Search Console service description, preserving its OAuth scopes and existing account/client selection. (internal/googleapi/read_only.go:103, 981ca4a163e8)
  • Google API contract: The production call at internal/cmd/searchconsole.go:544 directly depends on Google's URL Inspection contract. The official reference confirms the request fields, indexed-version semantics, and acceptance of both existing webmasters scopes: URL Inspection API reference.
  • Real behavior proof: The complete supplied PR body, captured under sourceRevision 639d5440c75fd4951b81256b72f875a16f1e4a851fc8702c8fe51f0d45dfb3c9, records a 2026-09-07 run against an account-owned Search Console property. Redacted JSON and terminal output show PASS, indexed coverage, canonical URL, crawl time, and referring-URL count through the new command. This is direct runtime evidence beyond the supplied httptest coverage and successful checks. (617bce86933f)

Likely related people:

  • Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • haresh seenivasagan: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)
  • reviewed 2026-09-07T02:48:21.950Z sha 589b456 :: needs real behavior proof before merge. :: [P3] Add the required release-note reference and contributor credit
  • reviewed 2026-09-07T03:03:39.036Z sha 617bce8 :: needs maintainer review before merge. :: none

Regenerate the README auth services block from its source of truth
(internal/googleauth service note) so the freshness test stays green,
and add the PR reference and contributor credit to the Unreleased entry
per repository AGENTS.md.
@clawsweeper

clawsweeper Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant