Skip to content

Add support for Object Detector models#15

Merged
JamesDale merged 2 commits into
masterfrom
claude/object-detector-support-6xailh
Jul 2, 2026
Merged

Add support for Object Detector models#15
JamesDale merged 2 commits into
masterfrom
claude/object-detector-support-6xailh

Conversation

@JamesDale

Copy link
Copy Markdown
Member

Summary

Adds support for Create ML / Core ML Object Detector models via a new SwiftUI view, ObjectDetectorView.

It runs a VNCoreMLRequest on each live camera frame and publishes the labels of the detected objects (the top label of each VNRecognizedObjectObservation) through a detectedObjects: Binding<[String]>. The design deliberately mirrors the existing model-backed views (ImageClassifierView, HandPoseClassifierView):

  • Graceful degradation — a bad/missing model doesn't crash the host app. The camera feed still shows, no detections are produced, the failure is logged via PKLog, and it's reported through the optional onError: ((PrototypeKitError) -> Void)? closure (reusing the existing .modelLoadFailed case).
  • Camera selection — accepts an optional camera: CameraOptions?.
  • Uses .scaleFill crop/scale so objects near the frame edges aren't discarded (object detectors localize across the whole frame).

Public API

public init(modelURL: URL,
            detectedObjects: Binding<[String]> = .constant([]),
            camera: CameraOptions? = nil,
            onError: ((PrototypeKitError) -> Void)? = nil)

Changes

  • New: Sources/PrototypeKit/Public/ObjectDetectorView.swift — the view + ObjectDetectorReceiver.
  • New: Tests/PrototypeKitTests/OD/ObjectDetectorTests.swift — view init, onError, and nil-model graceful-failure coverage.
  • Tests: extended GracefulFailureTests with the equivalent object-detector cases (bad-URL init, onError, nil-model receiver ignores frames).
  • Docs & tooling: README section, DocC topic entry, SKILL.md API reference + camera-view / model-view lists + corrected gotcha, CHANGELOG (Unreleased), Examples gallery entry, a new /prototypekit:object-detector scaffolding command, plugin README table row, and marketplace description.

Testing

⚠️ No Swift toolchain is available in this environment (the package depends on Apple-only frameworks — Vision/CoreML/AVFoundation/SwiftUI — so it builds and tests on macOS CI only). Changes were written to match the established patterns; CI on macOS will exercise the new tests. The receiver/view tests avoid needing a real object-detection model by covering the nil-model and bad-URL paths, consistent with the existing GracefulFailureTests.

🤖 Generated with Claude Code


Generated by Claude Code

claude added 2 commits July 2, 2026 08:31
Add ObjectDetectorView, a camera view backed by a Create ML / Core ML
Object Detector model. It runs a VNCoreMLRequest on each frame and
publishes the labels of detected objects through a
`detectedObjects: Binding<[String]>`.

Mirrors the existing model-backed views: graceful degradation on a
bad/missing model (camera feed with no detections, logged, reported via
the optional onError closure) and an optional camera: CameraOptions?.

Also updates docs and tooling: README, DocC topics, SKILL.md, CHANGELOG,
the Examples gallery, a new /prototypekit:object-detector command, plugin
README + marketplace description, and tests (view init, onError, nil-model
graceful-failure).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NURZKomSBVteJi6uoeTCqV
Object detection's value is localization, so expose it. Add a public
DetectedObject value type (label, confidence, normalized boundingBox) and a
second ObjectDetectorView initializer that publishes
detectedObjects: Binding<[DetectedObject]>.

The receiver now maps VNRecognizedObjectObservation into DetectedObject; the
existing Binding<[String]> initializer is preserved as a convenience that maps
to the top labels. Both initializers share one private designated init.

Updates tests (DetectedObject value + the new initializer paths) and docs:
README bounding-box example, SKILL.md both signatures + DetectedObject type,
DocC topic, CHANGELOG, Examples gallery box overlay, and the object-detector
scaffolding command.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NURZKomSBVteJi6uoeTCqV
@JamesDale
JamesDale marked this pull request as ready for review July 2, 2026 09:06
@JamesDale
JamesDale merged commit f6f83bf into master Jul 2, 2026
4 checks passed
@JamesDale
JamesDale deleted the claude/object-detector-support-6xailh branch July 2, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants