[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-10-29

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-10-29

Executive Summary

• Sessions Analyzed: 50 runs (3 with full events.jsonl agent transcripts; 47 are CI gates, command dispatchers, and skipped Squad runs)
• Analysis Period: 2026-10-29 20:42 → 21:08 UTC (all on 4 copilot/*model-routing* / route-agent-* branches; transcripts are all for PR Route Claude Code User Documentation Review with model routing (cost / economy) #73032)
• Completion Rate: 46.0% (23/50), above the 30-day mean of 41.1%
• Average Duration: 1.7 min per run (median 0.1 min); agent time in the 3 transcribed sessions: 5.8 min, 32 s, 13 s
• Experimental Strategy: none (roll = 91, standard analysis)

This is the second snapshot recorded for 2026-10-29. The morning run covered the 2026-10-28 10:22–16:54 UTC window and had 0/50 transcripts. This run covers the evening window, and 3/50 transcripts are available again.

Key Metrics

Metric	Value	Trend (vs morning snapshot)
Total Sessions	50	→
Successful Completions	23 (46.0%)	↑ (from 15 / 30.0%)
Failed/Abandoned	2 (4.0%): 1 failure, 1 cancelled	↓ (from 18)
Gated / skipped (not executed)	25 (50.0%): 10 action_required, 15 skipped	—
Average Duration	1.7 min	↓ (from 90.3 min)
Loop Detection Rate	1 of 3 transcripts (33%)	— (not recorded before)
Context Issues	0 of 3 transcripts	→

📈 Session Trends Analysis

Completion Patterns

Completion swings widely from day to day (4%–80%). That tracks which workflow population lands in the 50-run sample more than any change in agent quality. Today's 46% sits just above the 30-day mean. Only 2 runs actually failed (PR Data Prefetch failure, one cancelled Stale Lock Files). The rest of the non-successes are Squad runs that skipped or are waiting for approval (action_required).

Duration & Efficiency

Today has one of the shortest average run durations on record (1.7 min). That's because 27/50 runs were near-zero-length Squad/Agentic Commands dispatches. Breaks in the lines are days with no cache snapshot (10-04 → 10-13). Loop counts are recorded from today onward.

Success Factors ✅

All three transcribed sessions published their safe output, and every one of the 5 tool errors was recovered (n = 3, so treat the rates as indicative).

1. Pre-fetched PR context files: all 3 sessions started from /tmp/gh-aw/agent/pr-diff.patch / pr-meta.json and didn't call GitHub APIs for diffs.
   • Success rate: 3/3
   • Example: Test Quality Sentinel read the changed-files list, saw no _test.go / .test.cjs changes, and finished in 5 turns.
2. Explicit act-vs-noop rubric: Test Quality Sentinel finished with task_complete + noop in 13 s of agent time.
   • Success rate: 1/1
3. Delegating to a cheap specialist sub-agent: PR Code Quality Reviewer (gpt-5.6-luna) ran a grumpy-coder sub-agent on claude-haiku-5.5 in the background (3 tool calls, 14 s) and submitted one unified review in 11 turns at ~1.2 AIU.
   • Success rate: 1/1
4. Stdin fallback for safe-output payloads: when inline JSON broke shell quoting, the agent wrote the payload to a file and used safeoutputs <tool> . < file, which succeeded.
   • Success rate: 2/2 recoveries

Failure Signals ⚠️

1. Shell failures masked as success: tool.execution_complete reported success=true for 2 bash calls that exited 1: a shell syntax error from inline JSON quoting, and a safeoutputs schema error. The exit code appears only in the result text.
   • Effect: reported tool success is 95.9% (70/73); actual success is 93.2% (68/73)
   • Example: Matt Pocock Skills Reviewer, submit_pull_request_review → unknown parameter 'requestReview'
2. Guessed safe-output parameters: the agent added a requestReview field that isn't in the tool schema, then had to strip it with a python3 edit and resubmit.
   • Rate: 1/3 sessions
3. Over-exploration / search loop: Matt Pocock Skills Reviewer ran 50 turns and 2.46M input tokens (97% cache reads). That included 9 near-duplicate grep passes for claude-haiku-4.5. It also ran gh aw compile / go run ./cmd/gh-aw compile on the workspace, then reverted the change with git checkout --. 36 of its 45 bash calls began with a redundant cd /home/runner/work/gh-aw/gh-aw &&.
   • Rate: 1/3 sessions
4. Permission denials on instructed commands: Test Quality Sentinel's command -v safeoutputs was denied by the unattended permission fallback, even though the system prompt tells agents to run exactly that. A grumpy-coder python3 -c call was also denied by workflow tool permissions. Both sessions recovered.
   • Rate: 2/3 sessions (also seen 10-16 and 10-27)
5. ID transcription error: PR Code Quality Reviewer passed read_agent an agent ID with one character dropped → Agent not found. It recovered via list_agents.
   • Rate: 1/3 sessions

Prompt Quality Analysis 📝

Per-Prompt Breakdown

High-Quality Prompt Characteristics

• Names exact pre-fetched input files and forbids redundant API reads: 3/3 successful sessions
• Gives an explicit terminal rule (noop rubric / single unified review): 3/3 successful sessions
• Bounds exploration ("open at most 5 additional source files", static review only): 1/3 (PR Code Quality Reviewer). This session used the fewest tokens per finding.

Example High-Quality Prompt (paraphrased):

Read the pre-fetched diff and metadata in one parallel turn. Start the grumpy-coder
sub-agent in the background. Open at most 5 extra source files. This is a static
review: do not execute repository code or build commands. Submit one unified review.

Low-Quality Prompt Characteristics

• No explicit exploration/turn budget or static-only boundary: the one session without these (Matt Pocock Skills Reviewer) ran 50 turns and invoked the compiler
• The safe-output call shape isn't spelled out, so the agent guessed a field (requestReview)

Example Low-Quality Prompt (paraphrased):

Review the PR against the skills catalog and submit a review.

All 3 transcribed prompts rated high quality overall. The other 47 runs have no prompt data.

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 14 open PRs (0.0%). None of the 14 PR branches had in-progress runs; all 3 in-progress runs are on main.
• Historical Baseline: 0.0% orphaned rate (30-day mean over 30 recorded days)
• Status: NORMAL

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today. 13/14 open PRs have Copilot assigned. The one that doesn't (#72543) has no active gate runs.

CI Waste Estimate

• Orphaned gate-hours today: 0 gates × 0 h ≈ 0 CI-minutes wasted
• Recoverable capacity: none needed

Data note: the baseline query in the workflow prompt reads .orphan_rate, but every earlier history entry stored orphan_rate_pct, so the query as written returns null (it would report "insufficient history"). The baseline above uses orphan_rate_pct. Today's entry writes both fields.

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: 1 of 3 (33%)
• Average loop count: 9 repeated searches in the looping session
• Common loop patterns: re-grepping the same model identifier with small pattern variations across pkg/workflow/*.go, docs, and workflow files

Tool Usage

• Most used tools: bash 54, view 9, task 2, grep 2, read_agent 2, glob 1, list_agents 1, read_bash 1, task_complete 1
• Tool success rates: 95.9% as reported; 93.2% after counting non-zero exit codes
• Missing tools: none requested

Token & Cost Efficiency

Session	Model(s)	Turns	Input tokens	Output tokens	AIU
Matt Pocock Skills Reviewer	claude-sonnet-5 + haiku-4.5 sub-agent	50	2,459,149	24,640	~88.9
Test Quality Sentinel	claude-sonnet-5	5	199,971	977	~13.3
PR Code Quality Reviewer	gpt-5.6-luna + haiku-5.5 sub-agent	11	216,142	3,999	~1.2

The Test Quality Sentinel no-op cost ~11× more than the full 11-turn code-quality review.

The pr-triage sub-agent in Matt Pocock Skills Reviewer is set by its agent definition to default to claude-haiku-4.5. Today's Squad replies on the routing PRs say the router never ranks that model first. The sub-agent default bypasses the router.

Context Issues

• Sessions with confusion: 0 of 3
• Common confusion points: none. The sessions are non-interactive with prompts that name their files.
• Clarification requests: 0

Experimental Analysis

Standard analysis only. No experimental strategy this run (roll = 91 ≥ 30).

Actionable Recommendations

For Users Writing Task Descriptions

1. State a static-only boundary and an exploration budget: copy PR Code Quality Reviewer's "open at most N files; do not execute repository code" into other review prompts.
   • Example: Before: "Review the PR against the skills catalog." → After: "Static review only. Read at most 5 extra files, then adjudicate."
2. Spell out the exact safe-output payload: list the allowed fields (body, event, pull_request_number), or tell the agent to run safeoutputs <tool> --help first.
   • Example: Before: "submit a review" → After: "submit_pull_request_review with only event and body"
3. Prefer stdin for multi-line review bodies: bodies with apostrophes or parentheses break inline single-quoted JSON.
   • Example: Before: safeoutputs tool '{...it's...}' → After: printf '%s' '<json>' | safeoutputs tool ., or escape apostrophes as \u0027

For System Improvements

1. Exit-code-aware tool success: report success=false when a bash tool exits non-zero, so session metrics and agents both see the failure.
   • Potential impact: High
2. Deterministic pre-gate for Test Quality Sentinel: skip the agent job when the PR touches no test files. This saves ~13 AIU per no-op PR (a recurring no-op, also seen 10-21, 10-26, and 10-27).
   • Potential impact: High
3. Align permissions with prompt instructions: allow command -v safeoutputs, or drop that instruction for engines that deny it.
   • Potential impact: Medium
4. Route the Matt Pocock reviewer through an economy goal or a turn cap: 50 turns and ~89 AIU for a two-file workflow PR.
   • Potential impact: Medium

For Tool Development

1. Client-side schema validation in the safeoutputs CLI: reject unknown fields with the list of allowed fields before the MCP round-trip.
   • Frequency of need: 1 session today
   • Use case: avoids guess-and-retry on review payloads
2. Data pipeline: write {run_id}-conversation.txt or update the prompt: the fetch module now writes {run_id}-events.jsonl, but the analysis prompt still describes -conversation.txt files.
   • Frequency of need: every run
   • Use case: keeps the analysis instructions aligned with the data actually fetched

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: 46.0% vs a 30-day mean of 41.1% (range 4–80%). Still volatile and driven by which workflows are sampled.
• Average duration trend: 1.7 min, far below the morning snapshot's 90.3 min (long-running gate outliers then vs. short dispatcher runs now)
• Quality improvement: transcripts are available again (3/50) after the morning's 0/50. The same 3 review-bot workflows show up whenever transcripts are available (10-21, 10-26, 10-27, 10-29). Matt Pocock Skills Reviewer turn counts stay high (60 on 10-27, 50 today).

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:      23 (46.0%)
Failed Sessions:              2 (4.0%)
Abandoned (skipped/gated):   25 (50.0%)
In-Progress Sessions:         0 (0.0%)

Average Session Duration:   1.7 min
Median Session Duration:    0.1 min
Longest Session:           12.4 min (Matt Pocock Skills Reviewer)
Shortest Session:           0.0 min

Loop Detection:            1 of 3 transcribed sessions (33%)
Context Issues:            0 of 3 transcribed sessions (0%)
Tool Failures:             5 occurrences (3 reported + 2 hidden behind success=true)

High-Quality Prompts:      3 (100% of transcribed)
Medium-Quality Prompts:    0 (0%)
Low-Quality Prompts:       0 (0%)

Next Steps

• Review recommendations with team
• Make bash tool success reflect exit codes (Copilot CLI telemetry)
• Add a no-test-changes pre-gate to Test Quality Sentinel
• Fix the orphan-baseline field name (orphan_rate vs orphan_rate_pct) and the conversation-log filename in the workflow prompt
• Schedule follow-up analysis in 24 hours

---

Analysis generated automatically on 2026-10-29 at 21:40 UTC
Run ID: 40955410097
Workflow: Copilot Session Insights

Generated by 📊 Copilot Session Insights · claude · routed: opus55 high · ~161 AIC · ⌖ ~2 AIC · ⊞ 14.6K · ◷

• expires on Oct 30, 2026, 1:40 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #73144.