feat: import GCP Cloud Logging into a workspace

CLAUDE.md

@@ -1,92 +1,94 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Project Overview
-
-LAPP (Log Auto Pattern Pipeline) discovers log templates from log streams using the Drain algorithm, labels them with semantic IDs via LLM, and builds structured file-based workspaces for AI-assisted log investigation. It also includes an agentic analyzer that uses LLMs to investigate logs.
-
-## Commands
-
-```bash
-make build              # Build embedded frontend assets, then output/lapp
-make clean              # Remove generated build artifacts
-make dev                # Clean, build, and start lapp web on 127.0.0.1:8080
-make proto-gen          # Generate protobuf/Connect code
-make test               # Run unit and integration tests
-make check              # Run formatting, linting, type checks, build, and unit tests
-
-# Run a single test
-go test -v -run TestFunctionName ./pkg/pattern/
-```
-
-## CLI Usage
-
-```bash
-go run ./cmd/lapp/ workspace create <topic>
-go run ./cmd/lapp/ workspace add-log --topic <topic> <logfile>
-go run ./cmd/lapp/ workspace add-log --topic <topic> --stdin
-go run ./cmd/lapp/ workspace discover --topic <topic> [--model <model>]
-go run ./cmd/lapp/ workspace analyze --topic <topic> [question] [--model <model>]
-go run ./cmd/lapp/ web [--addr 127.0.0.1:0]
-```
-
-Topic names are sanitized to lower-kebab-case. Workspaces live under `~/.lapp/workspaces/<topic>/`.
-
-## Architecture
-
-```
-cmd/lapp/                CLI entrypoint (cobra commands: workspace create/add-log/discover/analyze)
-pkg/logsource/           Read log files → channel of LogLine
-pkg/multiline/           Detect log entry boundaries, merge continuation lines
-pkg/pattern/             Drain-based log pattern discovery and template matching
-pkg/semantic/            LLM-based semantic labeling of Drain patterns
-pkg/workspace/           DiscoveryRun execution and run-scoped file writer
-pkg/store/               DuckDB storage primitives (not yet on the CLI discovery path)
-pkg/config/              Model resolution (flag → $MODEL_NAME → default)
-pkg/analyzer/            Agentic log analysis via eino ADK + ACP providers
-integration_test/        Integration tests against Loghub-2.0 datasets
-```
-
-### DiscoveryRun (discover)
-
-`add-log` is a pure copy into `logs/` and never triggers discovery. Each `workspace discover` starts a DiscoveryRun: reads ALL files in `logs/`, runs fresh Drain + semantic labeling, and writes run-scoped `patterns/` and `notes/`.
-When `lapp web` starts, it marks any previous `QUEUED` or `RUNNING` DiscoveryRuns as failed because those local workers no longer exist.
-DiscoveryRun records persist structured `progress` and `error` fields; frontend code renders those facts into user-facing text.
-
-```
-workspace.Discover(ctx, cfg)
-  → Read all logs/ files → multiline.MergeSlice() per file → tagged lines
-  → pattern.DrainParser.Feed(all content) → Templates() → filter Count > 1
-  → semantic.Label(ctx, cfg, patterns)  ← LLM batches with per-batch retry
-  → workspace.NewBuilder(...).BuildAll()
-    → discovery-runs/<run-id>/patterns/<semantic-id>/pattern.md + samples.log
-    → discovery-runs/<run-id>/patterns/unmatched/samples.log
-    → discovery-runs/<run-id>/notes/summary.md + errors.md
-    → discovery-runs/<run-id>/AGENTS.md
-```
-
-### Multiline Detection
-
-Uses a token graph trained on 70+ timestamp formats. Lines are tokenized (first 60 bytes), matched against a directed graph of valid token transitions, and scored 0.0-1.0. Score > 0.5 means "new log entry". If no timestamps are ever detected, falls back to line-by-line.
-
-### Analyzer
-
-Runs an eino ADK agent (15 max iterations) with filesystem tools (grep, read_file, execute) against a structured workspace directory.
-
-## Environment Variables
-
-- `OPENROUTER_API_KEY`: Required for semantic labeling in `workspace discover`
-- `MODEL_NAME`: Override default LLM model (default: `google/gemini-3-flash-preview`)
-- ACP provider credentials/login: Required for `workspace analyze` through the selected provider
-- `.env` file is auto-loaded via godotenv
-
-## Tech Stack
-
-- Go, cobra CLI, go-drain3, DuckDB (duckdb-go/v2), cloudwego/eino ADK, OpenRouter semantic labeling, ACP providers
-
-## Code Style
-
-- `nolint` directives go on the line above the target, not as end-of-line comments
-- Compile-time interface guards: `var _ MyInterface = (*MyImpl)(nil)`
-- Always use `make build` to verify compilation, never bare `go build` (it drops a binary in the project root)
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+LAPP (Log Auto Pattern Pipeline) discovers log templates from log streams using the Drain algorithm, labels them with semantic IDs via LLM, and builds structured file-based workspaces for AI-assisted log investigation. It also includes an agentic analyzer that uses LLMs to investigate logs.
+
+## Commands
+
+```bash
+make build              # Build embedded frontend assets, then output/lapp
+make clean              # Remove generated build artifacts
+make dev                # Clean, build, and start lapp web on 127.0.0.1:8080
+make proto-gen          # Generate protobuf/Connect code
+make test               # Run unit and integration tests
+make check              # Run formatting, linting, type checks, build, and unit tests
+
+# Run a single test
+go test -v -run TestFunctionName ./pkg/pattern/
+```
+
+## CLI Usage
+
+```bash
+go run ./cmd/lapp/ workspace create <topic>
+go run ./cmd/lapp/ workspace add-log --topic <topic> <logfile>
+go run ./cmd/lapp/ workspace add-log --topic <topic> --stdin
+go run ./cmd/lapp/ workspace import gcp --topic <topic> --project <project> [--filter <filter>] (--since <duration> | --from <ts> --to <ts>) [--limit <n>]
+go run ./cmd/lapp/ workspace discover --topic <topic> [--model <model>]
+go run ./cmd/lapp/ workspace analyze --topic <topic> [question] [--model <model>]
+go run ./cmd/lapp/ web [--addr 127.0.0.1:0]
+```
+
+Topic names are sanitized to lower-kebab-case. Workspaces live under `~/.lapp/workspaces/<topic>/`.
+
+## Architecture
+
+```
+cmd/lapp/                CLI entrypoint (cobra commands: workspace create/add-log/import/discover/analyze)
+pkg/logsource/           Read log files → channel of LogLine
+pkg/multiline/           Detect log entry boundaries, merge continuation lines
+pkg/pattern/             Drain-based log pattern discovery and template matching
+pkg/semantic/            LLM-based semantic labeling of Drain patterns
+pkg/workspace/           DiscoveryRun execution and run-scoped file writer
+pkg/store/               DuckDB storage primitives (not yet on the CLI discovery path)
+pkg/config/              Model resolution (flag → $MODEL_NAME → default)
+pkg/analyzer/            Agentic log analysis via eino ADK + ACP providers
+integration_test/        Integration tests against Loghub-2.0 datasets
+```
+
+### DiscoveryRun (discover)
+
+`add-log` is a pure copy into `logs/` and never triggers discovery. `workspace import gcp` pulls a snapshot from GCP Cloud Logging through ADC credentials, lands it as enveloped NDJSON in `logs/`, and records provenance under `import-runs/<run-id>/record.json`; it never triggers discovery either. Each `workspace discover` starts a DiscoveryRun: reads ALL files in `logs/`, runs fresh Drain + semantic labeling, and writes run-scoped `patterns/` and `notes/`.
+When `lapp web` starts, it marks any previous `QUEUED` or `RUNNING` DiscoveryRuns as failed because those local workers no longer exist.
+DiscoveryRun records persist structured `progress` and `error` fields; frontend code renders those facts into user-facing text.
+
+```
+workspace.Discover(ctx, cfg)
+  → Read all logs/ files → multiline.MergeSlice() per file → tagged lines
+  → pattern.DrainParser.Feed(all content) → Templates() → filter Count > 1
+  → semantic.Label(ctx, cfg, patterns)  ← LLM batches with per-batch retry
+  → workspace.NewBuilder(...).BuildAll()
+    → discovery-runs/<run-id>/patterns/<semantic-id>/pattern.md + samples.log
+    → discovery-runs/<run-id>/patterns/unmatched/samples.log
+    → discovery-runs/<run-id>/notes/summary.md + errors.md
+    → discovery-runs/<run-id>/AGENTS.md
+```
+
+### Multiline Detection
+
+Uses a token graph trained on 70+ timestamp formats. Lines are tokenized (first 60 bytes), matched against a directed graph of valid token transitions, and scored 0.0-1.0. Score > 0.5 means "new log entry". If no timestamps are ever detected, falls back to line-by-line.
+
+### Analyzer
+
+Runs an eino ADK agent (15 max iterations) with filesystem tools (grep, read_file, execute) against a structured workspace directory.
+
+## Environment Variables
+
+- `OPENROUTER_API_KEY`: Required for semantic labeling in `workspace discover`
+- GCP ADC (`gcloud auth application-default login`): Required for `workspace import gcp`
+- `MODEL_NAME`: Override default LLM model (default: `google/gemini-3-flash-preview`)
+- ACP provider credentials/login: Required for `workspace analyze` through the selected provider
+- `.env` file is auto-loaded via godotenv
+
+## Tech Stack
+
+- Go, cobra CLI, go-drain3, DuckDB (duckdb-go/v2), cloudwego/eino ADK, OpenRouter semantic labeling, ACP providers
+
+## Code Style
+
+- `nolint` directives go on the line above the target, not as end-of-line comments
+- Compile-time interface guards: `var _ MyInterface = (*MyImpl)(nil)`
+- Always use `make build` to verify compilation, never bare `go build` (it drops a binary in the project root)

README.md

@@ -81,6 +81,7 @@ DiscoveryRuns are one-time local tasks. If `lapp web` starts and finds a previou
 | `workspace create <topic>` | Create a workspace under `~/.lapp/workspaces/` |
 | `workspace list` | List all workspace topics |
 | `workspace add-log --topic <topic> <file>` | Copy a log file into the workspace |
+| `workspace import gcp --topic <topic> --project <p> --since 1h` | Import a snapshot from GCP Cloud Logging |
 | `workspace discover --topic <topic>` | Run pattern discovery over all log files |
 | `workspace analyze --topic <topic> [question]` | Run AI analysis (`--acp claude|codex|gemini`) |
 

cmd/lapp/workspace.go

@@ -46,6 +46,7 @@ func workspaceCmd() *cobra.Command {
 	cmd.AddCommand(workspaceCreateCmd())
 	cmd.AddCommand(workspaceListCmd())
 	cmd.AddCommand(workspaceAddLogCmd())
+	cmd.AddCommand(workspaceImportCmd())
 	cmd.AddCommand(workspaceDiscoverCmd())
 	cmd.AddCommand(workspaceAnalyzeCmd())
 	return cmd
@@ -126,7 +127,7 @@ func runWorkspaceCreate(_ *cobra.Command, args []string) error {
 		return err
 	}
 
-	for _, sub := range []string{"logs", workspace.DiscoveryRunsDirName} {
+	for _, sub := range []string{"logs", workspace.DiscoveryRunsDirName, workspace.ImportRunsDirName} {
 		if err := os.MkdirAll(filepath.Join(dir, sub), 0o755); err != nil {
 			return errors.Errorf("create %s: %w", sub, err)
 		}

cmd/lapp/workspace_import.go

@@ -0,0 +1,177 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"github.com/go-errors/errors"
+	"github.com/spf13/cobra"
+	"github.com/strrl/lapp/pkg/gcplog"
+	"github.com/strrl/lapp/pkg/workspace"
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/codes"
+)
+
+var importGCPTopic string
+var importGCPProject string
+var importGCPFilter string
+var importGCPSince string
+var importGCPFrom string
+var importGCPTo string
+var importGCPLimit int
+
+func workspaceImportCmd() *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "import",
+		Short: "Import logs from an external provider into the workspace",
+	}
+	cmd.AddCommand(workspaceImportGCPCmd())
+	return cmd
+}
+
+func workspaceImportGCPCmd() *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "gcp",
+		Short: "Import logs from GCP Cloud Logging",
+		Long: `Pull matching entries from GCP Cloud Logging into the workspace as one
+NDJSON log file under logs/, and record the ImportRun under import-runs/.
+
+Uses Application Default Credentials; LAPP never manages provider auth.
+Exactly one time range form is required: --since, or --from together with --to.
+This never starts discovery. Run 'lapp workspace discover --topic <topic>'
+afterwards.`,
+		Args: cobra.NoArgs,
+		RunE: runWorkspaceImportGCP,
+	}
+	cmd.Flags().StringVar(&importGCPTopic, "topic", "", "workspace topic (required)")
+	cmd.Flags().StringVar(&importGCPProject, "project", "", "GCP project id (required)")
+	cmd.Flags().StringVar(&importGCPFilter, "filter", "", "Cloud Logging filter, combined with the time range")
+	cmd.Flags().StringVar(&importGCPSince, "since", "", "import entries from the last duration, e.g. 1h or 30m")
+	cmd.Flags().StringVar(&importGCPFrom, "from", "", "start of time range, RFC3339, e.g. 2026-07-25T00:00:00Z")
+	cmd.Flags().StringVar(&importGCPTo, "to", "", "end of time range, RFC3339")
+	cmd.Flags().IntVar(&importGCPLimit, "limit", 100000, "maximum number of entries to import")
+	_ = cmd.MarkFlagRequired("topic")
+	_ = cmd.MarkFlagRequired("project")
+	return cmd
+}
+
+// resolveImportTimeRange validates the mutually exclusive time range flags
+// and resolves them to a concrete UTC from/to pair.
+func resolveImportTimeRange(since, from, to string, now time.Time) (fromTime, toTime time.Time, err error) {
+	hasSince := since != ""
+	hasFromTo := from != "" || to != ""
+	if hasSince && hasFromTo {
+		return time.Time{}, time.Time{}, errors.New("--since and --from/--to are mutually exclusive; use one time range form")
+	}
+	if !hasSince && !hasFromTo {
+		return time.Time{}, time.Time{}, errors.New("a time range is required: use --since <duration>, or --from and --to (RFC3339)")
+	}
+
+	if hasSince {
+		duration, parseErr := time.ParseDuration(since)
+		if parseErr != nil {
+			return time.Time{}, time.Time{}, errors.Errorf("invalid --since duration %q: %w", since, parseErr)
+		}
+		if duration <= 0 {
+			return time.Time{}, time.Time{}, errors.Errorf("--since must be a positive duration, got %q", since)
+		}
+		toTime = now.UTC()
+		return toTime.Add(-duration), toTime, nil
+	}
+
+	if from == "" || to == "" {
+		return time.Time{}, time.Time{}, errors.New("--from and --to must be provided together")
+	}
+	fromTime, err = time.Parse(time.RFC3339, from)
+	if err != nil {
+		return time.Time{}, time.Time{}, errors.Errorf("invalid --from timestamp %q, expected RFC3339: %w", from, err)
+	}
+	toTime, err = time.Parse(time.RFC3339, to)
+	if err != nil {
+		return time.Time{}, time.Time{}, errors.Errorf("invalid --to timestamp %q, expected RFC3339: %w", to, err)
+	}
+	if fromTime.After(toTime) {
+		return time.Time{}, time.Time{}, errors.Errorf("--from %s is after --to %s", from, to)
+	}
+	return fromTime.UTC(), toTime.UTC(), nil
+}
+
+func runWorkspaceImportGCP(cmd *cobra.Command, _ []string) error {
+	dir, err := topicToDir(importGCPTopic)
+	if err != nil {
+		return err
+	}
+
+	// Validate workspace exists
+	if _, err := os.Stat(filepath.Join(dir, "logs")); os.IsNotExist(err) {
+		hint := availableWorkspacesHint()
+		return errors.Errorf("not a workspace: %s (no logs/ directory)%s", dir, hint)
+	}
+
+	if strings.TrimSpace(importGCPProject) == "" {
+		return errors.New("--project must not be blank")
+	}
+
+	from, to, err := resolveImportTimeRange(importGCPSince, importGCPFrom, importGCPTo, time.Now())
+	if err != nil {
+		return err
+	}
+	if importGCPLimit <= 0 {
+		return errors.Errorf("--limit must be positive, got %d", importGCPLimit)
+	}
+
+	// Validation is done; anything past this point is a run failure, so the
+	// usage help would only be noise.
+	cmd.SilenceUsage = true
+
+	ctx, span := otel.Tracer("lapp/cmd").Start(cmd.Context(), "cmd.WorkspaceImportGCP")
+	defer span.End()
+
+	result, err := workspace.RunImport(ctx, workspace.ImportConfig{
+		Dir:      dir,
+		Provider: "gcp",
+		Project:  importGCPProject,
+		Filter:   importGCPFilter,
+		From:     from,
+		To:       to,
+		Limit:    importGCPLimit,
+		Fetcher: func(fetchCtx context.Context, req workspace.ImportRequest) (workspace.ImportFetchResult, error) {
+			fetched, err := gcplog.FetchLines(fetchCtx, gcplog.FetchRequest{
+				Project: req.Project,
+				Filter:  req.Filter,
+				From:    req.From,
+				To:      req.To,
+				Limit:   req.Limit,
+			})
+			if err != nil {
+				return workspace.ImportFetchResult{}, err
+			}
+			return workspace.ImportFetchResult{
+				Lines:     fetched.Lines,
+				Truncated: fetched.Truncated,
+			}, nil
+		},
+	})
+	if err != nil {
+		span.RecordError(err)
+		span.SetStatus(codes.Error, err.Error())
+		return err
+	}
+
+	topic := filepath.Base(dir)
+	if result.EntryCount == 0 {
+		fmt.Printf("No entries matched. ImportRun %s succeeded with 0 entries; no log file was written.\n", result.RunID)
+		span.SetStatus(codes.Ok, "")
+		return nil
+	}
+	if result.Truncated {
+		fmt.Printf("Warning: result truncated at --limit %d; narrow the time range or filter to import everything.\n", importGCPLimit)
+	}
+	fmt.Printf("Imported %d entries into logs/%s.\nDiscovery has not run yet. Run it with:\n\n  lapp workspace discover --topic %s\n", result.EntryCount, result.LogFileName, topic)
+	span.SetStatus(codes.Ok, "")
+	return nil
+}