Add analytics-dataops-expertise skill for DataOps maturity assessment

[prasadnu]

Read-only, API-driven DataOps maturity assessment: 26 questions scored 1-5 across five dimensions (Architecture; Security & Governance; Incident Management & Observability; Automation & Testing; Cost), with a verbatim remediation reference (official AWS doc links) and evals. Control-plane APIs only; no data-plane access, no scripts. Adds an llms.txt index entry.

Description

Type of change

• New skill
• New custom agent
• Update to an existing skill or agent
• Documentation or infrastructure change

Testing

License confirmation

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

[andrehass]

Tested the DataOps maturity skill and works as expected. Full maturity assessment generated.

[ams-thakkar]

What needs to change

Add a gated IAM policy for the actions AIDevOpsAgentAccessPolicy doesn't grant. I checked all 61 actions the skill calls against the live policy: 52 are covered, these five are not.

Action	Policy has
cost-optimization-hub:ListRecommendations	nothing for this service
quicksight:ListDashboards	nothing for this service
glue:GetJobs	glue:GetJob (singular) and glue:List*, no Get* wildcard
iam:ListPolicies	ListRoles, ListUsers, ListEntitiesForPolicy — not ListPolicies
lakeformation:GetDataLakeSettings	Describe*, GetLFTag, GetResourceLFTags, List*

Add an EnableAnalyticsDataopsExpertise parameter, condition, inline policy, and SkillPolicySummary line to cloudformation/devops-agent-skill-policies.yaml. #71 is a good model.

Reframe README.md:25 against AIDevOpsAgentAccessPolicy. It currently reads "Most are covered by a ViewOnly/read-only managed policy; attach a supplemental policy for any gaps" — that names the wrong policy and leaves the gaps unnamed. State coverage against the agent's own managed policy and point at the CloudFormation gate above.

Fix four action names that look like API names rather than IAM actions. In each case the capability is already granted under the correct name, so this is a docs fix — please confirm each:

• mwaa:ListEnvironments → MWAA authorises under airflow; airflow:List* is granted
• resourcegroupstaggingapi:GetResources → prefix is tag; tag:GetResources is granted
• kinesisanalyticsv2:ListApplications → v2 authorises under kinesisanalytics; kinesisanalytics:List* is granted
• s3:GetBucketLifecycleConfiguration → the IAM action is s3:GetLifecycleConfiguration, which is granted

Rebase — llms.txt conflicts with main. Routine churn from six merges today, not your doing.

Nits

None blocking.

Thanks

Mapping each of the 26 questions to specific read-only APIs and fields, with a documented rule for scoring them 1–5, is what makes this auditable rather than a vibe check — and capping a question at 3 when control-plane signals can't fully confirm it is the right call over guessing. Scoping the whole assessment to control-plane reads with no data-plane access keeps the blast radius small for something that touches 16 services.

Verified myself: mkdocs build --strict passes with zero warnings; all six JSON files parse; extensions limited to md/json/yaml/html; name matches the directory; description is 907 characters; version: 1.1.0 matches the CHANGELOG top entry; llms.txt has an entry; no relative .md links; and the evals check reports WARN under the predating-PR waiver.

Thanks @andrehass for testing this. One ask while we're here: for a skill scoring 26 questions across 16 services, could you note which dimensions you exercised and on what kind of account? That makes the sign-off reusable when this skill changes.

[praveenMprasad]

@ams-thakkar. All four items are addressed in 9eac67b (v1.1.1):

1. Gated IAM policy for the five uncovered actions. Added an EnableAnalyticsDataopsExpertise parameter, condition, inline policy (PolicyAnalyticsDataopsExpertise), and a SkillPolicySummary line to
   devops-agent-skill-policies.yaml
   , following the existing per skill blocks. It grants exactly the five actions the managed policy does not:

• cost-optimization-hub:ListRecommendations
• quicksight:ListDashboards
• glue:GetJobs (managed policy has glue:GetJob singular plus glue:List*)
• iam:ListPolicies
• lakeformation:GetDataLakeSettings

All read only. The skill degrades gracefully if any is absent (the affected question is scored as a floor with a caveat rather than failing).

2. Fixed the four action names. Confirmed each and corrected them in both SKILL.md and README.md:

• mwaa:ListEnvironments is now airflow:ListEnvironments
• resourcegroupstaggingapi:GetResources is now tag:GetResources
• kinesisanalyticsv2:ListApplications is now kinesisanalytics:ListApplications
• s3:GetBucketLifecycleConfiguration is now s3:GetLifecycleConfiguration

3. Reframed README.md. The IAM section now states coverage against AIDevOpsAgentAccessPolicy, names the five supplemental actions, and points at the CloudFormation gate above.

4. Rebased on main. The llms.txt conflict is resolved.

Agent Skill Eval audit remains 98/100 (A)

[ams-thakkar]

Re-reviewed at 9eac67b. Three of the four items are done and verified. One is half done, and my original ask is why — I pointed at the wrong file. Numbers below are also corrected; mine were wrong the first time.

What needs to change

Reframe the IAM section in SKILL.md too. You fixed README.md, and it reads well now — it names AIDevOpsAgentAccessPolicy, lists the five supplemental actions with the reason each one isn't covered, and points at the CloudFormation gate. But SKILL.md:33-34 still carries the original text verbatim:

IAM permissions required (all read-only). Most are covered by a read-only / ViewOnly managed policy; attach the supplemental permissions for any gaps:

SKILL.md doesn't mention AIDevOpsAgentAccessPolicy, the gate, or EnableAnalyticsDataopsExpertise anywhere — I grepped for all three. That's the file the agent reads and the one an operator lands on first, so as it stands they'd attach a ViewOnly policy and have no way to know which five actions are missing. The README paragraph you already wrote is the fix; it just needs to exist here as well.

This one is on me. My ask cited README.md:25 when the same text appeared in both files, so you fixed exactly what I pointed at.

Nits

Neither is blocking.

• SKILL.md:35 lists both glue:GetJobs and glue:GetJob, where README.md's equivalent list has only the plural. The singular is granted by the managed policy so it costs nothing, but the two lists should agree.
• The action counts are off in a couple of places, including mine. The new comment in devops-agent-skill-policies.yaml says "52 of the 57 IAM actions"; my last review said 61 with 52 covered. Neither is right: README.md's canonical list has 58 actions, of which 53 are covered and 5 are not. My 61 double-counted the four misnamed actions alongside their replacements, and 57 drops them entirely. Worth fixing in the template comment since it will outlive this thread.

Thanks

The gated policy is exactly the shape I asked for and then some — the inline comment records the five actions with the specific reason each falls outside the managed policy, which means the next person to touch it doesn't have to re-derive the delta. Noting in both the comment and the CHANGELOG that the skill degrades to a floor score rather than failing when a permission is absent is the detail that makes the Default: 'true' defensible.

Verified myself against 9eac67b, with the live managed policy rather than from memory. AIDevOpsAgentAccessPolicy is at v11, 931 Allow entries and no Deny. Of the 58 actions in your canonical list, 53 match the policy and 5 do not, and those 5 are exactly the ones in PolicyAnalyticsDataopsExpertise — nothing uncovered is missing from the gate, and nothing in the gate is redundant. All four renamed actions resolve: airflow:ListEnvironments via airflow:List*, tag:GetResources exactly, kinesisanalytics:ListApplications via kinesisanalytics:List*, s3:GetLifecycleConfiguration exactly. All four of the old names are genuinely uncovered, which confirms the renames were necessary and not cosmetic; they now survive only in the CHANGELOG entry describing the change, which is right.

On the rest: the CloudFormation template adds the parameter to the Metadata interface group, the condition, the policy, and the SkillPolicySummary line, following the existing per-skill blocks. cfn-lint on it reports the same four pre-existing W2001 unused-parameter warnings as main and nothing new, and EnableAnalyticsDataopsExpertise is not among them. mkdocs build --strict passes with zero warnings and the catalog page generates. All six eval JSON files parse. Extensions are limited to md/json/yaml/html. Frontmatter name matches the directory, description is 909 characters, version: 1.1.1 matches the CHANGELOG top entry. The llms.txt entry survived the rebase. Rebased onto current main at 301ee08, zero commits behind, and the required check passes.

One fix and I'll approve. Mergeable state is BLOCKED on the approval alone — the required check is green and there are no open threads.

[praveenMprasad]

Thanks for the re-review and for catching the split, @ams-thakkar. Fixed in 8a485fd (v1.1.2):

SKILL.md IAM section now carries the same framing as README.md: it names AIDevOpsAgentAccessPolicy, lists the five supplemental actions with the reason each is uncovered, and points at the EnableAnalyticsDataopsExpertise gate in
devops-agent-skill-policies.yaml.
Also picked up both nits while here: dropped the duplicate glue:GetJob from the SKILL.md list so it agrees with the README, and corrected the template comment to 53 of 58 covered.
Audit still 98/100 (A).