Skip to content

fix(ci): validate every changed skill, and trim the II description - #455

Merged
sea-snake merged 2 commits into
mainfrom
fix/validate-every-changed-skill
Nov 9, 2026
Merged

sea-snake merged 2 commits into
mainfrom
fix/validate-every-changed-skill

Conversation

@sea-snake

Copy link
Copy Markdown
Contributor

checks / validate failed on the push of #454 to main (run), so deploy-ic.yml never ran and the five Internet Identity skills are not on https://skills.internetcomputer.org/.

The error

Validating skill: skills/internet-identity
Frontmatter
  ✗ description exceeds 1024 characters (1029)

Dropping two words ("It also covers" → "Also covers", "wallet integration or ICRC signer flows" → "wallet or ICRC signer flows") brings it to 1014, in line with cloud-engine-canisters (1014) and deploy-to-cloud-engine (1010). No routing term is lost: wallet-integration still appears in the same sentence as the skill to load instead.

Why CI did not catch it on the PR

skill-validator check takes exactly one path:

$ skill-validator check skills/internet-identity skills/internet-identity-sso --skip links
Error: accepts 1 arg(s), received 2
$ echo $?
3

The changed-skills step builds a space-separated list of paths, so on any PR touching two or more skills the validator exited 3 (usage error) with an empty report, and the gate only failed on exit code 1. Every such PR has been going green without being validated. On a push to main the same step passes the single path skills/, which is why the error only surfaced after the merge.

Two changes:

  • each path gets its own invocation, aggregating the exit codes with an error (1) winning over a warning (2);
  • the gate fails on any code other than 0 (clean) or 2 (warnings), so a usage error can never read as a pass again.

Verification

Per-path loop over the five skills from #454, with the description fix applied:

skills/internet-identity                  -> 2
skills/internet-identity-attributes       -> 0
skills/internet-identity-notifications    -> 0
skills/internet-identity-shared-sessions  -> 0
skills/internet-identity-sso              -> 0
aggregate=2        # warnings only, gate passes

Same loop before the description fix returns 1 for skills/internet-identity, so the new gate catches exactly the failure that got through. skill-validator check skills/ --skip links now exits 2 (was 1), and node scripts/check-project.js passes for 39 skills with the two pre-existing eval-file warnings.

No skill body changed, so no evals were re-run.

🤖 Generated with Claude Code

`skill-validator check` accepts exactly one path and exits 3 on more, so the
PR job validated nothing whenever a PR touched two or more skills: the report
came back empty and the gate, which only failed on exit code 1, went green.
Each path now gets its own invocation, and the gate fails on any code other
than 0 (clean) or 2 (warnings).

That hole is how internet-identity's 1029-character description reached main,
where skills/ is checked as a single path and the 1024-character limit failed
the run, so the site was never deployed. Two words out of the description
brings it to 1014.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sea-snake
sea-snake requested review from a team and JoshDFN as code owners November 6, 2026 11:59
@sea-snake
sea-snake requested a balanced review from Copilot November 6, 2026 11:59
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Skill Validation Report

Validating skill: /home/runner/work/icskills/icskills/skills/internet-identity

Structure

  • Pass: SKILL.md found

Frontmatter

  • Pass: name: "internet-identity" (valid)
  • Pass: description: (1014 chars)
  • Pass: license: "Apache-2.0"
  • Pass: compatibility: (31 chars)
  • Pass: metadata: (2 entries)

Tokens

  • Warning: SKILL.md body is 6255 tokens (spec recommends < 5000)

Markdown

  • Pass: no unclosed code fences found

Tokens

File Tokens
SKILL.md body 6,255
Total 6,255

Content Analysis

Metric Value
Word count 4,034
Code block ratio 0.10
Imperative ratio 0.06
Information density 0.08
Instruction specificity 0.87
Sections 26
List items 57
Code blocks 11

Contamination Analysis

Metric Value
Contamination level medium
Contamination score 0.41
Primary language category javascript
Scope breadth 3
  • Warning: Language mismatch: config (1 category differ from primary)
  • Multi-interface tool detected: aws

Result: 1 warning

Project Checks


✓ Project checks passed for 1 skills (0 warnings)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The exit-code aggregation lets an earlier warning mask later usage or execution errors, allowing the final gate to pass even when a skill was not successfully validated.

1 open issue
What changed in this PR

This PR addresses the validation failure that blocked deployment of the Internet Identity skills after #454 and updates CI to validate multi-skill changes individually.

Changes:

  • Runs the validator separately for each changed skill and aggregates results.
  • Updates the failure gate to reject exit codes other than success or warnings.
  • Shortens the Internet Identity description to fit the 1,024-character limit without changing the skill body.
File Description
skills/​internet-identity/​SKILL.md Trims the description while preserving routing terms.
.github/​workflows/​_checks.yml Adds per-path validation, result aggregation, and stricter failure conditions.

Review effort: Balanced

Comment thread .github/workflows/_checks.yml Resolved
The aggregation only raised the exit code from 0, so a warning on one skill
pinned it at 2 and a later usage error could not lift it: codes 2 then 3
aggregated to 2 and the gate passed with a skill unvalidated, which is the
failure this change set out to close. Errors, other non-zero codes and
warnings are now tracked separately and resolved in that order.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sea-snake
sea-snake merged commit 9c2bcd2 into main Nov 9, 2026
6 checks passed
@sea-snake
sea-snake deleted the fix/validate-every-changed-skill branch November 9, 2026 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants