Plan for issue #10

plans/10.md

@@ -0,0 +1,209 @@
+# Implementation Plan for Issue #10
+
+## Issue Summary
+
+The model may fail to post replies to review comments, resulting in wasted tokens and poor UX. This happens because:
+
+1. The `gh` CLI doesn't provide a straightforward way to reply to inline review comments
+2. When the model doesn't follow instructions correctly, no comment is posted at all
+3. Users have no visibility into what happened or why there's no response
+
+From the failed workflow run (https://github.com/TomzxCode/llm-augmented-workflows/actions/runs/20909952268), the permission_denials log shows that the `Skill` tool was denied, but the workflow still completed without posting any reply.
+
+## Feasibility Assessment
+
+- **Status**: ✅ Feasible
+- **Complexity**: Low
+- **Estimated Effort**: Low
+
+The solution is straightforward. The GitHub REST API provides an endpoint for replying to inline review comments:
+```
+POST /repos/{owner}/{repo}/pulls/comments/{comment_id}/replies
+```
+
+This endpoint requires:
+- `comment_id`: The database ID of the parent comment (not the node_id)
+- `body`: The reply content
+
+This approach has been validated and works correctly.
+
+## Proposed Solution
+
+### Overview
+
+Instead of using `gh pr comment` which only creates general PR-level comments, we need to use the GitHub REST API to post replies to inline review comments. The solution involves:
+
+1. **Pass the inline comment ID through the workflow**: Update the workflow to capture and pass the inline comment's database ID
+2. **Update the skill to use the correct API**: Modify the skill to detect inline comments and use the appropriate API endpoint
+3. **Add fallback visibility**: Ensure users are informed even when the model can't post a proper reply
+
+### Implementation Steps
+
+1. **Update `claude-review.yml` to pass inline comment ID**
+
+   - Files to modify: `.github/workflows/claude-review.yml`
+   - Approach: Add a new output variable to capture the inline comment's database ID when the event is `pull_request_review_comment`
+   - Testing considerations: Verify the ID is correctly extracted for both inline and general comments
+
+   ```yaml
+   # In the "Check if this is a plan PR comment" step, add:
+   core.setOutput('comment_id', context.payload.comment.id || '');
+   ```
+
+   Then pass it as an environment variable:
+   ```yaml
+   COMMENT_ID=${{ github.event.comment.id }}
+   ```
+
+2. **Update `review-plan-comment.md` to handle inline replies**
+
+   - Files to modify: `.claude/commands/review-plan-comment.md`
+   - Approach: Add logic to detect `COMMENT_TYPE=inline` and use the REST API for replies
+   - Testing considerations: Test both inline and general comment scenarios
+
+   The skill should:
+   - When `COMMENT_TYPE=inline`: Use `gh api repos/{owner}/{repo}/pulls/comments/{comment_id}/replies -f body="..."`
+   - When `COMMENT_TYPE=general`: Use `gh pr comment $PR_NUMBER --body "..."`
+
+3. **Add fallback comment posting**
+
+   - Files to modify: `.claude/commands/review-plan-comment.md`
+   - Approach: If posting a reply fails (e.g., API error), fall back to posting a general PR comment with context
+   - Testing considerations: Test error handling by temporarily using invalid comment IDs
+
+   Example fallback:
+   ```bash
+   # If inline reply fails, post a general comment referencing the original
+   gh pr comment $PR_NUMBER --body "Reply to ${COMMENT_TYPE} comment on ${path}:${line}: [response]"
+   ```
+
+4. **Add response posting verification**
+
+   - Files to modify: `.claude/commands/review-plan-comment.md`
+   - Approach: After posting, verify the comment was created successfully and log any failures
+   - Testing considerations: Ensure the skill exits with appropriate status
+
+### Dependencies
+
+- External dependencies: None (using existing `gh` CLI)
+- Internal dependencies:
+  - `.github/workflows/claude-review.yml` - provides the context
+  - `.claude/commands/review-plan-comment.md` - consumes the context
+
+### Risk Assessment
+
+- Potential risks:
+  - **Risk**: The comment ID from GitHub event may be the `node_id` instead of `database_id`
+    - **Mitigation**: The `github.event.comment.id` in GitHub Actions should already be the database ID. Test to confirm.
+  - **Risk**: Model may still fail to call the `gh` command
+    - **Mitigation**: This is a model behavior issue, but having clearer instructions in the skill helps
+  - **Risk**: API permissions may be insufficient
+    - **Mitigation**: The workflow already has `pull-requests: write` permission, which should be sufficient
+
+## Alternatives Considered
+
+1. **Use GraphQL API for replies**: Initially explored using `addPullRequestReviewThreadReply` mutation, but the REST API endpoint is simpler and works reliably.
+
+2. **Always post general comments regardless of comment type**: This would ensure visibility but loses the threading context for inline comments.
+
+3. **Modify the Claude Code action**: Could add built-in support for posting replies, but this would require upstream changes and increases dependency on external tooling.
+
+## Implementation Details
+
+### Modified `.github/workflows/claude-review.yml` (Step 1)
+
+The `Check if this is a plan PR comment` step needs to output the comment ID:
+
+```yaml
+- name: Check if this is a plan PR comment
+  id: check-plan
+  uses: actions/github-script@v8
+  with:
+    script: |
+      // ... existing code ...
+      let commentId = '';
+
+      if (eventName === 'pull_request_review_comment') {
+        // For inline comments, get the database ID
+        commentId = context.payload.comment.id;
+        // ... rest of existing code ...
+      } else if (eventName === 'issue_comment') {
+        // ... existing code ...
+      }
+
+      core.setOutput('comment_id', commentId);
+```
+
+Then pass it to the skill:
+
+```yaml
+- name: Respond to review comment
+  # ... existing config ...
+  with:
+    prompt: |
+      /review-plan-comment
+      REPO=${{ github.repository }}
+      PR_NUMBER=${{ github.event.pull_request.number }}
+      COMMENT_AUTHOR=${{ github.event.comment.user.login }}
+      COMMENT_BODY=${{ github.event.comment.body }}
+      COMMENT_TYPE=${{ github.event_name == 'pull_request_review_comment' && 'inline' || 'general' }}
+      COMMENT_ID=${{ steps.check-plan.outputs.comment_id }}
+```
+
+### Modified `.claude/commands/review-plan-comment.md` (Step 2)
+
+Add new input variable:
+
+```markdown
+## Input Context
+Environment variables provided by the workflow:
+- `REPO`: The GitHub repository
+- `PR_NUMBER`: The pull request number
+- `COMMENT_AUTHOR`: The person who wrote the comment
+- `COMMENT_BODY`: The content of the comment
+- `COMMENT_TYPE`: Either "inline" (line-specific) or "general" (PR-level)
+- `COMMENT_ID`: The database ID of the comment (for inline comments only)
+```
+
+Add a helper function section:
+
+```markdown
+## Helper Functions
+
+### Post Response
+
+Use this function to post responses:
+
+**For inline comments:**
+```bash
+gh api repos/$REPO/pulls/comments/$COMMENT_ID/replies -f body="$RESPONSE"
+```
+
+**For general comments:**
+```bash
+gh pr comment $PR_NUMBER --body "$RESPONSE"
+```
+```
+
+Update the Type B (Question) and Type C (Discussion) response instructions:
+
+```markdown
+**Type B: Question**
+Action:
+- Provide a clear, helpful answer using the appropriate comment posting method
+- No plan updates needed
+
+**Type C: Discussion**
+Action:
+- Engage thoughtfully in the discussion using the appropriate comment posting method
+- No plan updates unless you agree and it improves the plan
+- If you do update, follow Type A process
+```
+
+## References
+
+- Related issues: #10
+- Related PRs: #9 (example of inline comment that wasn't replied to)
+- Documentation:
+  - GitHub REST API: https://docs.github.com/en/rest/pulls/comments#create-a-reply-for-a-review-comment
+  - GitHub Actions event payload: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_review_comment