Add sagemaker ops review - #93
Conversation
udid-aws
added
ci-sweep
ams-thakkar
left a comment
ams-thakkar
left a comment
There was a problem hiding this comment.
What needs to change
The eval-results layout is missing, so validate-skill-evals will fail this PR. That check landed on main after you branched. A new skill is auto-enforced — no label or flag needed. Running main's validator against this head:
FAIL sagemaker-ops-review (enforced)
- missing file `skills/sagemaker-ops-review/evals/evals.json`
- missing directory `skills/sagemaker-ops-review/evals/structure/`
- missing directory `skills/sagemaker-ops-review/evals/best-practices/`
- missing directory `skills/sagemaker-ops-review/evals/functional/`
evals/eval_queries.json is here and valid, but it's only the trigger half. Adding evals.json plus the three results directories per the "Eval Results Layout" section of CONTRIBUTING.md clears it.
The strict docs build fails on five links in the skill README — skills/sagemaker-ops-review/README.md:119,151,192,193,194. deploy-docs.yml:41 runs mkdocs build --strict, and linking to README.md rather than the directory turns an INFO into a warning:
WARNING - Doc file 'skills/sagemaker-ops-review.md' contains a link '../../custom-agents/aws-operation-review/README.md', but the target '../custom-agents/aws-operation-review/README.md' is not found among documentation files.
Aborted with 5 warnings in strict mode!
I built main with only this skill overlaid: all 5 warnings are yours, 0 from anything else, so main is currently clean and this is the change that breaks the deploy. Dropping the filename fixes it — ](../../custom-agents/aws-operation-review/), ](../aiml-access-diagnostics/), ](../service-quota-check/). That's why every merged skill README has zero relative ../*.md links; the existing directory-style links only log INFO.
Merge conflicts need a rebase. git merge-tree origin/main reports content conflicts in custom-agents/aws-operation-review/README.md and custom-agents/aws-operation-review/SYSTEM_PROMPT.md — both were touched on main since you branched.
SME review confirmation is requested directly on this PR. This adds an IAM policy resource to cloudformation/devops-agent-skill-policies.yaml, so I'd like the SageMaker domain reviewer to confirm sign-off in a comment here before merge rather than it being inferred from the PR body. My review is scoped maintainer review — repo mechanics, the CI checks above, and the shape of the IAM change. It isn't a substitute for domain or security sign-off on the twenty checks themselves or on the API surface.
Nits
Non-blocking:
EnableSageMakerOpsReviewusesDefault: 'false'while every sibling gate defaults'true'. Defensible as the conservative choice and you documented the degraded path, but worth a deliberate decision rather than drift.
Thanks
The IAM work is the strongest part of this. One added action, savingsplans:DescribeSavingsPlans, checked against live AIDevOpsAgentAccessPolicy v10 rather than assumed — including confirming health:Describe* was already there so the Health check needed nothing. That's the opposite of the reflexive "add the service namespace" pattern, and the graceful degradation ("not evaluated — permission not granted", other 19 checks still run) means the gate defaulting off doesn't strand anyone. Extending aws-operation-review instead of shipping a fifth review agent is the right call, and calling out that the agent-type picker caveat probably also affects agentcore-ops-review is the kind of thing that saves the next person a debugging session. The overlap analysis against service-quota-check and aiml-access-diagnostics answered the question I was going to ask.
I verified myself: the single added action is read-only and gated behind both an AllowedValues parameter and the SkillSageMakerOpsReview condition; SkillPolicySummary is updated (line 486) and the parameter is in the Metadata group; llms.txt adds the skill entry and updates the agent's service list to include SageMaker AI; frontmatter name matches the directory, description is 588 chars, and version: "1.0.0" matches the CHANGELOG top entry; eval_queries.json, references/iam-policy.json and .skilleval.yaml all parse; file extensions are within the allowed set; and the two CI failures and the merge conflict quoted above.
Happy to approve once the evals layout lands, the README links are switched to directory form, it's rebased clean, and SME sign-off is confirmed here.
9ccb457 to
ff5c51d
Compare
|
All four items are addressed in Eval layout
For the three tool-generated types I've added The validator now passes, including with 1 skill(s) checked: 1 pass, 0 fail, 0 warn. One note on how the FAIL was producedYour run used 1 skill(s) checked: 0 pass, 0 fail, 1 warn. So a new skill isn't auto-enforced in this particular case. Flagging it only so nobody later Strict docs buildAll five links switched to directory form — Caveat: mkdocs isn't installed in my environment, so I applied the fix you pres RebaseRebased onto Nit: parameter defaultChanged to SME sign-offUnderstood that you want this on the PR rather than inferred from the body. Eddie Yao is doing the Also agreed on the scope of your review — repo mechanics and the shape of the I |
Adds a read-only operational review for Amazon SageMaker AI workloads -- endpoints, training jobs, pipelines, notebooks, feature store, model registry, and Studio domains -- covering 8 pillars and 20 checks: Security, Performance, Cost Optimization, Service Quotas, Resiliency, Operational Excellence, Sustainability, and Best Practices. Findings use a uniform High / Medium / Low scale, plus Informational for inventory checks with no pass/fail signal. Every High or Medium finding carries exactly one concrete recommendation, and the report leads with a severity-ranked Executive Summary. Findings are keyed per resource, so counts stay comparable between runs. The Best Practices pillar is grounded in the public AWS Well-Architected Machine Learning, Generative AI, and Agentic AI lenses. The skill is strictly read-only: List/Describe/Get control-plane calls and CloudWatch metric reads only, with no data-plane calls, no endpoint invocation, and no inference payload reads. IAM: AIDevOpsAgentAccessPolicy already covers every API the skill calls except savingsplans:DescribeSavingsPlans, which is an optional add-on -- without it the Savings Plan check reports "not evaluated - permission not granted" and the other 19 checks run normally. references/iam-policy.json carries that single statement. Notable check behaviour, each verified against a live account: - Autoscaling detection reads three states. Managed instance scaling, or an Application Auto Scaling target on sagemaker:variant:DesiredInstanceCount with at least one scaling policy, counts as autoscaled. A target registered without a policy only declares capacity bounds and never triggers a scaling action, so it is reported as its own finding rather than passing as healthy. - Serverless variants are scored Informational in checks covering features Serverless Inference does not support (VPC configuration, network isolation, data capture), so the report never emits a recommendation the operator cannot act on. - Service Quotas reads applied limits via servicequotas:GetServiceQuota, never the AWS defaults, and scores utilization from CloudWatch AWS/Usage ResourceCount over a trailing 24 hours at period 3600. SageMaker publishes those metrics roughly every 20 minutes with ingestion lag, so shorter windows return no datapoints. Missing usage data yields Unknown, never an inferred 0%. - The tagging check counts only user-defined tags, since SageMaker auto-injects sagemaker:domain-arn, user-profile-arn and space-arn on every Studio-created resource. Auto-generated model-monitoring-* processing jobs are excluded as they are not operator-taggable. - AWS Health findings are keyed per affected entity and filtered to the in-scope regions. ACTION_REQUIRED events that are open or upcoming carry Medium, since they represent externally-imposed deadlines. - Quotas, limits, prices, costs and percentage savings appear only when an API returned them; the skill does not estimate or recall them. Region discovery uses Cost Explorer with a sagemaker:List* sweep as fallback, so a payer-scoped Cost Explorer miss cannot produce a false "no activity" result. Checks are isolated: an AccessDenied or API error becomes an error row on that check and never aborts the review.
Wires the new sagemaker-ops-review skill into the existing operational review agent rather than shipping a second review agent, so SageMaker AI joins EKS, RDS, Aurora and Bedrock under one entry point. SYSTEM_PROMPT.md: SageMaker AI added to the Goal and to the service identification step, sagemaker-ops-review added to the skill selection list, and a SageMaker artifact naming example added. The skill's report schema is named in the existing "defer to the selected skill's report schema" guidance, alongside the bedrock-operation-review example, because sagemaker-ops-review defines its own eight pillars, a verbatim AI Disclaimer, and a severity-ranked Executive Summary that should not be forced into the generic category set. README.md: SageMaker AI added to Purpose, Key Capabilities, Prerequisites and Related, and the skill selection step now reflects that the skills are chosen per service rather than always both. The Prerequisites entry notes that AIDevOpsAgentAccessPolicy covers every API the skill calls except the optional savingsplans:DescribeSavingsPlans. CHANGELOG.md: bumped to 1.1.0. llms.txt: sagemaker-ops-review added to Available Skills, and the AWS Operation Review entry now lists SageMaker AI. cloudformation/devops-agent-skill-policies.yaml: adds the EnableSageMakerOpsReview parameter, its condition, a PolicySageMakerOpsReview resource granting savingsplans:DescribeSavingsPlans, and a SkillPolicySummary line. Every other API the skill calls is already covered by the managed policy, so this is the only addition required. Note on skill agent type: this skill's README instructs uploading with "Generic" / "All agents" selected rather than narrowing to specific agent types, because a narrowed skill does not appear in the custom agent's skill picker. The aws-operation-review README already documents that as a workaround for the EKS and RDS skills; sagemaker-ops-review states it up front instead, so the caveat is not needed for it.
Adds the eval layout. `evals/evals.json` is written by hand per CONTRIBUTING -- ten scenarios covering the report contract, the read-only invariant, dual-signal autoscaling including the target-without-policy state, the Serverless Inference feature exclusions, applied-vs-default quota limits, Savings Plan permission degradation, the no-invented-numbers rule, Health event severity and region scope, user-defined-tag-only compliance, and empty-scope precedence. `evals/exemptions.json` requests exemptions for the three tool-generated test types, which cannot be produced because the skill evaluation tool is not published in this repository. Each reason names the blocker and what was run instead, and asks for a maintainer run so the exemption can be removed. CONTRIBUTING notes an exemption should not be added without maintainer agreement, so this is a proposal for review rather than a settled decision -- happy to drop it if you would rather run the tool on our behalf first. Fixes the five strict-docs-build warnings by switching the relative links in the skill README to directory form, dropping the `README.md` filename, matching every merged skill README. No relative `*.md` links remain in the file. mkdocs is not installed locally so the strict build could not be reproduced here, only the link form corrected as prescribed. Changes the `EnableSageMakerOpsReview` default from 'false' to 'true', matching all thirteen sibling gates. The template is opt-in per deployment and this grants a single read-only action, which is narrower than several gates that already default on, so being the lone outlier was drift rather than a deliberate choice. The degraded path is unchanged: with the parameter off, the Savings Plan check reports "not evaluated - permission not granted" and the other nineteen run. Rebased onto main, resolving the conflicts in the aws-operation-review README and system prompt so Bedrock AgentCore and SageMaker AI both appear in the service lists, skill selection, report-schema guidance, and artifact naming examples.
Renames the skill to sagemaker-ai-ops-review to match the service name (Amazon SageMaker AI) and fixes the six check-logic defects raised in review, each of which would have put a factually wrong statement in a customer-facing report: - A notebook with no KmsKeyId is no longer reported "not encrypted". SageMaker AI encrypts notebook OS and ML volumes with a system-managed key when none is supplied, so the finding is the absence of a customer-managed key, not an absence of encryption. Severity stays Medium. - SKILL.md said the Service Quotas usage window was 60 minutes while references/pillar-checks.md said trailing 24 hours and documented that 60 minutes returns zero datapoints. SKILL.md is the always-loaded file, so the contradiction scored every quota Unknown and silently disabled the check. - Dropped the Feature Store and Model Registry claim from the description and the docs. No check read list-feature-groups or list-model-package-groups, so the skill activated on those prompts and returned a clean report that never mentioned them. It now says plainly that they are not assessed. - AWS Health severity reads Event.actionability. It previously tested eventScopeCode (PUBLIC|ACCOUNT_SPECIFIC|NONE) and the affected-entity statusCode (IMPAIRED|...|RESOLVED) for ACTION_REQUIRED, which neither field can hold, so the condition was unsatisfiable and urgent maintenance deadlines never reached the severity-ranked Executive Summary. - Autoscaling matches every applicable scalable dimension, not just sagemaker:variant:DesiredInstanceCount. An Inference Component endpoint scales on sagemaker:inference-component:DesiredCopyCount, returned by the same describe-scalable-targets call, so a correctly autoscaled IC endpoint earned a false Medium plus a remediation naming a dimension that does not apply to it. Includes the list-inference-components mapping, since an IC ResourceId does not contain the endpoint name. - Stale Endpoints guards on CreationTime and reads the latest non-zero invocation datapoint. Without the guard a two-day-old endpoint scored "Not invoked" and was told to be deleted; "first non-zero datapoint" returns the oldest invocation in an ascending window, so a daily-invoked endpoint reported Last Invoked: 90 days. Also addressed: explicit Savings Plan thresholds, with zero plans scored Informational rather than recommending an unmeasured multi-year commitment (this check reads plans, never spend); health, ce and savingsplans called once in us-east-1 as the global APIs they are, since per-region calls fail in a way that mimics the checks' own degradation paths; latency labelled in microseconds with a millisecond conversion; a Scope Limitations section in SKILL.md, because the README is not packaged into the uploaded skill; and .DS_Store excluded from the documented zip command. aws-operation-review now routes SageMaker AI. Its step 4 hard-coded a critical/high/medium/low scale, which would have overridden this skill's High/Medium/Low plus Informational model and broken the Executive Summary's count reconciliation; it now defers to the selected skill's scale, extending the deferral the prompt already applies to report structure. The README also gains the bedrock-operation-review prerequisite it was missing. 1.1.1, from the first live run through the router: - Inference Component endpoints are scored at both layers. Components that autoscale against a fixed host fleet hit a capacity ceiling, so the host variant earns a Medium recommending managed instance scaling, with a guard against double-counting the endpoint across layers. - Policy Count is read from describe-scaling-policies, never inferred from the presence of a target. A run reported Policy Count 1 for an endpoint with a registered target and zero policies, swallowing a Medium. - The Invocations query pins startTime to midnight UTC. CloudWatch anchors period-86400 buckets to the request time, so an unaligned start merged two calendar days into one bucket and reported Last Invoked a day early. evals.json: removed the "does not contain 'error'" assertion from all ten scenarios, which contradicted the skill's own design of recording per-check error rows, so a correct run failed it. Extended the autoscaling scenario to the Inference Component dimension and added scenarios for notebook encryption, Studio domain VpcOnly, stale endpoints and latency units — 10 to 14.
Rewriting the skill README for the rename reintroduced the `README.md`- terminated relative links that fail `mkdocs build --strict`, which review had already flagged and ff5c51d had fixed. Restored the directory form.
The exemption claimed five end-to-end runs, all predating the fixes in this revision. Run 6 exercises them and ran through the aws-operation-review router. Also names the two branches that remain logic-reviewed rather than implying full manual coverage.
ff5c51d to
5663db8
Compare
Blocking itemsEval-results layout. The three results directories are covered by Docs links. All switched to directory form — Worth flagging: I reintroduced this defect while rewriting those sections for the rename below, and caught it only by re-reading your review before pushing. The INFO-vs-warning distinction isn't visible locally without running the strict build, so it's an easy regression. If a lint rule for relative Rebase. Rebased onto Nit
Changes since your reviewRenamed Version 1.0.0 → 1.1.1, fixing six check-logic defects from domain review. Each would have put a factually wrong statement in a customer-facing report:
One change to a shared file worth your attention. The README also picks up the ValidationRun through the patched router against a live account with real SageMaker resources in us-east-1: severities stayed High/Medium/Low/Informational with no Two defects found in that run and fixed in 1.1.1: an Inference Component endpoint's host variant and its components are now scored as separate layers, because components that autoscale against a fixed host fleet hit a capacity ceiling; and |
|
Copying Eddies review feedback here. I've addressed these items in the most recent commits. hi Jack, here's the review feedback from me and also Kiro, let me know any questions: change skill name and name references across the code base to "sagemaker-ai-ops-review" to reflect the correct service name (from Kiro)The skill is well built — its anti-fabrication rules are better than most contributions in this repo. But six specific check-logic errors would each put a wrong statement in a customer-facing report, so I can't approve it yet. Five are one-line edits. Also worth noting: three severity-bearing checks have no eval scenario — encryption, domain VpcOnly (the only High rule in the skill), and stale endpoints. Two of those are blockers above. The autoscaling eval exists but asserts only the variant dimension, so it would pass while the IC bug ships. Five smaller things (not blocking) Savings Plan severity depends on "expiring soon" (no threshold defined) and "steady spend" (never measured) — recommending a multi-year financial commitment on an unmeasured premise. |
A live run emitted "Enable a customer-managed KMS key via UpdateNotebookInstance KmsKeyId" six times. UpdateNotebookInstance accepts no KmsKeyId parameter — the key is settable only at creation — so that recommendation fails the moment an operator tries it. The immutability note existed but was buried on the Severity bullet and was ignored. It is now its own rule naming the wrong API explicitly, requiring the remediation be worded as re-creating the instance with --kms-key-id and flagging that it is disruptive. Restated in SKILL.md, which is always loaded. Also adds a cross-cutting rule that every recommendation must be executable as written — this class has now produced two findings, the other being serverless endpoints told to attach a VpcConfig. Three eval assertions added so a regression fails the eval.
ams-thakkar
left a comment
ams-thakkar
left a comment
There was a problem hiding this comment.
Approving at 0e3999a. This is scoped maintainer approval — repo mechanics, the IAM change, and the specific check-logic fixes I could verify against primary sources. Detail on what I verified versus what I'm relying on others for is at the bottom, because it matters for this one.
All four of my earlier blockers are cleared. Docs links are directory-terminated and mkdocs build --strict is clean with zero warnings and no relative .md links left in the skill README. Rebased clean onto current main, zero commits behind, no conflicts. EnableSageMakerAIOpsReview now defaults 'true' in line with every sibling gate, which was my nit.
The IAM change is right, re-checked against live policy v11 rather than the v10 I used last time. savingsplans:DescribeSavingsPlans is still the only action the skill needs that the managed policy doesn't grant, and PolicySageMakerAIOpsReview grants exactly that one action, behind both an AllowedValues parameter and the SkillSageMakerAIOpsReview condition. I walked the other 21 calls — including the sagemaker:ListInferenceComponents / DescribeInferenceComponent the Inference Component fix newly needs, and health:DescribeAffectedEntities — and every one resolves through an existing wildcard. SkillPolicySummary and the Metadata parameter group are both updated.
On the eleven items in the review feedback recorded on P500283018, all eleven are addressed, and I checked the two most consequential against the service models rather than taking the fix on trust:
- AWS Health severity.
Event.actionabilityexists on the HealthEventshape with enum exactlyACTION_REQUIRED | ACTION_MAY_BE_REQUIRED | INFORMATIONAL, whileeventScopeCodeisPUBLIC | ACCOUNT_SPECIFIC | NONEandstatusCodeisopen | closed | upcoming. So the original defect was real — neither field can hold the value — and the fix reads the right field with the right enum.pillar-checks.mdnow says to read actionability "and from nowhere else," with a field table recording why. - Inference Component autoscaling.
ScalableDimensionhas exactly threesagemakervalues:sagemaker:variant:DesiredInstanceCount,sagemaker:variant:DesiredProvisionedConcurrency, andsagemaker:inference-component:DesiredCopyCount. The skill now matches all three with their resource-ID formats, so coverage is complete rather than partial — there is no fourth dimension it could still be missing.
The other nine check out on inspection: notebook encryption carries an explicit "never report a notebook instance as not encrypted" rule with the system-managed-key explanation; the quota window reads trailing 24 hours in SKILL.md and pillar-checks.md alike, with the 60-minute zero-datapoint finding recorded; the Feature Store and Model Registry claim is dropped and documented as uncovered rather than left to imply a pass; stale endpoints take the most recent non-zero datapoint with ScanBy=TimestampDescending plus a CreationTime guard that suppresses the finding below 90 days; Savings Plan severity now has real thresholds instead of an unmeasured "steady spend"; the three global APIs are pinned to a single us-east-1 call outside the per-region loop; latency is labelled microseconds in both files; the limitations moved into SKILL.md so they survive the packaged zip; and the self-contradicting does not contain 'error' assertion is gone.
Two things on the record, so nobody has to reconstruct this later.
The domain approval on P500283018 — "TFC review completed and approved per reviewer guide" — is timestamped 2026-09-18T17:33Z, which is about five hours before the first commit on this branch landed at 22:20Z. Five commits followed it, including the rename to sagemaker-ai-ops-review that the same reviewer had asked for and 0e3999a, which changes what the skill recommends about notebook KMS keys. The detailed feedback was posted on October 1 by the PR author transcribing a Slack message, and its own closing line is that it can't be approved yet. So I'm treating it as findings, which it is, and not as sign-off on this head. I'm comfortable approving because the eleven items it raises are verifiable without domain judgement and I verified them, but post-fix confirmation from the SME is still outstanding and is worth picking up when he's back — particularly on severity calibration across the twenty checks, which is the part my review does not reach.
Second: evals/exemptions.json exempts all three test types, so the required check passes on the exemption rather than on results. The exemption text asks for a maintainer run on the author's behalf. I'm merging on the exemption rather than holding the PR for it, and I'll run the three test types myself as a follow-up so the skill ends up carrying real results — at which point the exemption file should come out, as its own text says.
The manual validation behind the exemption is unusually thorough for what it is: six end-to-end runs with baselines, against an account holding a real Inference Component endpoint, notebooks with and without a CMK, a PublicInternetOnly Studio domain, and live Health events, with run 6 specifically exercising the 1.1.0 and 1.1.1 fixes through the router. Naming the two checks that remain logic-reviewed only — Savings Plan healthy coverage and the no-support-plan degradation path — rather than implying full coverage is the right instinct.
4 participants
Description
Adds a
sagemaker-ops-reviewskill for read-only operational reviews of Amazon SageMaker AIworkloads, and wires it into the existing
aws-operation-reviewcustom agent rather thanshipping a second review agent.
SageMaker AI was the remaining gap in the operation-review family alongside EKS, RDS/Aurora and
Bedrock. The skill covers 8 pillars / 20 checks — Security, Performance, Cost Optimization,
Service Quotas, Resiliency, Operational Excellence, Sustainability, and Best Practices — over
endpoints, training jobs, pipelines, notebooks, feature store, model registry and Studio domains.
Findings use a uniform High / Medium / Low scale plus Informational for inventory checks, are keyed
per resource so counts stay comparable between runs, and every High or Medium finding carries
exactly one concrete recommendation. The report leads with a severity-ranked Executive Summary. The
Best Practices pillar is grounded in the public Well-Architected ML, Generative AI and Agentic AI
lenses.
Strictly read-only:
List/Describe/Getcontrol-plane calls and CloudWatch metric reads only —no data-plane calls, no endpoint invocation, no inference payload reads.
Design notes for reviewers:
aws-operation-reviewinstead of adding an agent. SageMaker AI joins the existingservice list; the agent's "defer to the selected skill's report schema" clause already
accommodates a skill-specific report structure, so no format conflict.
CHANGELOGbumped to1.1.0.
AIDevOpsAgentAccessPolicycovers every API the skill calls exceptsavingsplans:DescribeSavingsPlans. Verified against the live managed policy (v10) — includingthat
health:Describe*is already present, so the Health check needs no addition beyond aBusiness/Enterprise Support plan. Added as an optional parameter in
cloudformation/devops-agent-skill-policies.yaml; without it the Savings Plan check reports"not evaluated — permission not granted" and the other 19 run normally.
custom agent's skill picker.
custom-agents/aws-operation-review/README.mdalready documents thisas a workaround for the EKS and RDS skills; this skill's README states it up front so the caveat
isn't needed for it. The same latent issue likely affects
agentcore-ops-review, which also shipsa companion agent.
service-quota-checkis general-purpose; this skill's Service Quotas pillar isSageMaker-scoped to seven verified quota codes with utilization-derived severity.
aiml-access-diagnosticsdiagnoses SageMaker access failures during an incident; this reviewsposture. Both are cross-referenced from the skill README.
Type of change
Testing
Five end-to-end runs through DevOps Agent against a test account with real SageMaker resources
(endpoints, notebooks, a Studio domain, pipelines, lifecycle configs, training jobs), across
us-east-1 and us-west-2, on the base
AIDevOpsAgentAccessPolicywithout the optional add-on.Baseline runs without the skill were done first for comparison — unaided, the agent produced
inconsistent structure across iterations and missed the dual-signal autoscaling case entirely.
Verified against live resources:
ManagedInstanceScalingabsent is correctly reported as autoscaled — a managed-scaling-only checkwould false-positive here. A registered target with no scaling policy is reported as its own
finding, since capacity bounds alone never trigger a scaling action.
use_awscall in the invocation trajectory inspected across runs.permission not granted", never a false "no plans found".
empty-state rows, and one-recommendation-per-High/Medium (13 recommendations for 1 High + 12
Medium; none for 7 Low).
GetServiceQuota, utilization fromAWS/UsageResourceCountover a trailing 24 hours. SageMaker publishes those metrics roughly every 20minutes with ingestion lag, so shorter windows return no datapoints; missing data yields Unknown
rather than an inferred 0%.
Serverless Inference doesn't support (VPC configuration, network isolation, data capture), so the
report never emits a recommendation the operator can't act on.
"No SageMaker AI activity detected."
Not reachable in a test account (logic-reviewed only, stated for transparency):
ml.inf*/ml.trn*quota is 0 in a fresh account — empty state onlyOpen question for maintainers: availability of
servicequotasthroughuse_awsappearsintermittent — the same account returned all seven applied limits in one run and
"service unavailable" in the next. The check now retries once and otherwise degrades to
"not evaluated" rather than inventing limits, but is
servicequotasexpected to be reliably exposed?If not, I'm happy to document it as a known limitation in the skill README.
Skill evaluation tool: not run — the tool isn't published.
evals/eval_queries.jsonis includedwith trigger and no-trigger cases. Tagging @aws/tools-for-devops-agent-admins to request an
evaluation run, and happy to iterate on the results.
License confirmation
Two things to check before you submit: