Skip to content

feat: import GCP Cloud Logging into a workspace - #35

Merged
STRRL merged 2 commits into
masterfrom
conductor/lapp-gcp-import-cli
Jul 25, 2026
Merged

feat: import GCP Cloud Logging into a workspace#35
STRRL merged 2 commits into
masterfrom
conductor/lapp-gcp-import-cli

Conversation

@STRRL

@STRRL STRRL commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Problem

Logs had to already exist as local files before LAPP could help: to
investigate an incident on GCP you had to remember the right gcloud
invocation, pipe its output around, and lose track of what query produced
which file. ADR 0004 decided imports should be recorded snapshots.

Solution

workspace import gcp pulls a query plus time range from GCP Cloud Logging
through the machine's ADC credentials, lands it as one enveloped NDJSON file
in logs/, and records provenance under import-runs/<run-id>/record.json.
The imported file flows through the existing discovery pipeline like any
other log file; the command never starts discovery (ADR 0005).

Major Changes

  • pkg/gcplog (new)
    • pure ConvertEntry mapping a GCP entry to the ADR 0006 envelope
      (jsonPayload → payload as-is, textPayload → payload.message, missing
      severity → DEFAULT, timestamps normalized to RFC3339 UTC)
    • fixture pairs include sanitized real GCP entries (Cloud Tasks, IAM
      audit, embedded text log) alongside synthetic edge cases
    • logadmin fetcher with ADC, precise RFC3339Nano query bounds, and a
      credential-failure hint pointing at gcloud auth application-default login
  • pkg/workspace
    • RunImport orchestration behind an injectable fetch seam: ImportRun
      record lifecycle (import-runs//record.json, symmetric to
      discovery-runs), zero-entries success without a file, orchestration-level
      limit cap, duplicate run id rejection, no stuck RUNNING records
  • cmd/lapp
    • workspace import gcp --topic --project [--filter] (--since | --from/--to) [--limit]; validation errors create no record, run failures leave a
      FAILED record; success output points at workspace discover

Note

Medium Risk
New external GCP dependency and credential path; import writes workspace files and records but is isolated from discovery/analyze until the user runs discover.

Overview
Adds workspace import gcp so logs can be pulled from GCP Cloud Logging (ADC) into a workspace as a single enveloped NDJSON file in logs/, with query/time-range provenance in import-runs/<run-id>/record.json. Import does not start discovery; users still run workspace discover afterward, same as add-log.

New pkg/gcplog maps GCP entries to the ADR 0006 envelope (ConvertEntry), builds time-bounded Cloud Logging filters, and fetches via logadmin with limit/truncation and ADC error hints. pkg/workspace adds ImportRun persistence and RunImport orchestration behind an injectable fetcher (success with zero entries skips writing a file; failures persist FAILED records; duplicate run IDs rejected).

workspace create now creates import-runs/; docs and go.mod add GCP logging/API dependencies and note gcloud auth application-default login.

Reviewed by Cursor Bugbot for commit 661b50f. Bugbot is set up for automated code reviews on this repo. Configure here.

STRRL added 2 commits July 25, 2026 14:30
New workspace import gcp command pulls a query plus time
range through ADC and lands enveloped NDJSON plus an
import-runs provenance record, per ADR 0004. Validation
errors leave no record; credential failures leave a FAILED
record. Conversion is a pure function with fixture pairs
including sanitized real GCP entries.
@STRRL
STRRL marked this pull request as ready for review July 25, 2026 21:36
@STRRL
STRRL merged commit 1a22d7c into master Jul 25, 2026
2 checks passed
@STRRL
STRRL deleted the conductor/lapp-gcp-import-cli branch July 25, 2026 21:36

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 661b50f. Configure here.

Comment thread pkg/gcplog/fetch.go
if err != nil {
return FetchResult{}, err
}
result.Lines = append(result.Lines, line)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Limit keeps oldest log entries

High Severity

FetchLines lists Cloud Logging entries without requesting newest-first order, then stops after --limit rows from the iterator. The Logging API returns entries in ascending timestamp order by default, so a capped import keeps the earliest part of the window—not the recent logs typical for --since incident pulls.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 661b50f. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant