Skip to content

Add support for Action Classification from body movement#16

Merged
JamesDale merged 1 commit into
masterfrom
claude/action-classification-body-movement-wwcvvr
Jul 2, 2026
Merged

Add support for Action Classification from body movement#16
JamesDale merged 1 commit into
masterfrom
claude/action-classification-body-movement-wwcvvr

Conversation

@JamesDale

Copy link
Copy Markdown
Member

Summary

Adds Action Classification to PrototypeKit — classifying a person's action from their body movement using a Create ML / Core ML Action Classifier model.

The new ActionClassifierView is a camera-based SwiftUI view that:

  • detects human body-pose keypoints per frame with Vision's VNDetectHumanBodyPoseRequest,
  • accumulates a sliding window of frames (an action unfolds over time), and
  • feeds a [window, 3, 18] multi-array into the model, publishing the predicted label through a Binding<String>.

It mirrors the API and behaviour of the existing model-backed views (ImageClassifierView, ObjectDetectorView, HandPoseClassifierView) and works on both iOS and macOS.

@State var latestPrediction = ""

ActionClassifierView(modelURL: ActionClassifier.urlOfModelInThisBundle,
                     latestPrediction: $latestPrediction)

Details

  • ActionClassifierConfiguration exposes posesFeatureName ("poses"), labelFeatureName ("label"), predictionWindowSize (60), and predictionInterval (15) for models that differ from the Create ML template. The window size is taken from the model's own input constraint when it advertises one.
  • Graceful degradation — a bad/missing model shows the camera feed with no predictions, logs the failure, and reports it via the optional onError closure. Accepts a camera: CameraOptions?.
  • Robust windowing — frames without a detected body are zero-padded (matching how Create ML pads missing frames), so a person briefly leaving the frame doesn't corrupt the temporal window. Predictions are throttled to every predictionInterval frames once the window is full.

Docs & tests

  • Graceful-failure tests for the new view and receiver (bad URL, onError, nil-model ignores frames).
  • Documented across the README (Live Action Classification section), DocC catalog, CHANGELOG.md, the Examples/ gallery, and the plugin (skill guidance, README command table, new /prototypekit:action-classifier command).

Notes

No Swift toolchain is available in this environment, so the package was not compiled here; the code follows the established patterns of the sibling views.

🤖 Generated with Claude Code


Generated by Claude Code

Introduce ActionClassifierView, a camera-based SwiftUI view that classifies
a person's action from their body movement using a Create ML / Core ML
Action Classifier model. It detects body-pose keypoints per frame with
Vision's VNDetectHumanBodyPoseRequest, accumulates a sliding window of
frames, and feeds a [window, 3, 18] multi-array into the model, publishing
the predicted label through a binding.

- ActionClassifierConfiguration exposes the poses/label feature names,
  prediction window size, and prediction interval for models that differ
  from the Create ML template. The window size is taken from the model's
  own input constraint when advertised.
- Degrades gracefully on a bad/missing model (camera feed with no
  predictions, logged, reported via optional onError), matching the other
  model-backed views, and accepts a camera: CameraOptions?.
- Frames without a detected body are zero-padded so a person briefly
  leaving the frame doesn't corrupt the temporal window.

Adds graceful-failure tests and documents the view across the README, DocC
catalog, CHANGELOG, Examples gallery, and the plugin skill/command set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018LXbPcM8eVnaCQwtAR8y6G
@JamesDale
JamesDale marked this pull request as ready for review July 2, 2026 09:37
@JamesDale
JamesDale merged commit f1f0a7d into master Jul 2, 2026
4 checks passed
@JamesDale
JamesDale deleted the claude/action-classification-body-movement-wwcvvr branch July 2, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants