Pinpoint: point at an element in your running app, say what should change, and your coding agent gets it (now with native iOS)

[gowtham012]

Hi everyone 👋

I've been building Pinpoint, an open-source (MIT) tool for giving coding agents like Claude Code, Cursor and Codex precise context about what you want changed in your app.

The problem

When you ask an agent to change something on screen, you usually paste a screenshot and write "the third button on the left". The agent gets pixels and has to guess which component you meant, what it's called, and where it lives in the code.

What Pinpoint does

You click the element in your running app and type what should change in your own words. Your agent receives:

• your comment, which is the only instruction
• a CSS selector, DOM path and computed styles
• the React/Vue component chain and a source-file hint, when the dev build exposes them
• a cropped screenshot of just that element

It reaches the agent over MCP, through Claude Code hooks (notes come in with your next message), or as a pending.md file in your repo. When the agent finishes, it resolves the note with a short reply, and the pin in your browser shows what was done.

New in this update

📱 Native iOS apps, from the Simulator
Native apps have no DOM, so Pinpoint now reads the accessibility tree instead. Open http://127.0.0.1:7331/ios while the bridge runs, hover over the Simulator screenshot, click an element and say what should change. The agent gets the accessibilityIdentifier / testID, label, tree path, frame and a crop. It works with SwiftUI, UIKit, React Native and Flutter, and nothing is added to your app. It uses Xcode and Maestro, and after the first load a refresh takes about a second.

🧠 Notes can be about behaviour, not just looks
Earlier versions framed every note as a "UI change request", so a note like "this should save the draft first" tended to get the button restyled. Now the agent is told a note can be about how something looks or what it does: validation, state, data, API calls. For a behaviour change, the clicked element is where it starts, and it follows the code into the handlers, state, API calls and backend behind it.

⚙️ Setup asks what you're building
node bridge/cli.js setup now asks whether your app is web, iOS or both. For iOS it skips the browser extension, checks for Xcode, Maestro and Java (it tells you how to get anything missing and installs nothing), and opens the Simulator picker.

Quick start

git clone https://github.com/gowtham012/pinpoint
node pinpoint/bridge/cli.js setup

Every question has a default, so pressing Enter all the way through gives you a working setup. For web apps you load the extension unpacked once; Chrome doesn't allow that from a terminal.

What it's not

It isn't a browser-driving agent like Playwright MCP or computer-use. You stay in control: you point and describe, and the agent edits code. It works alongside browser automation rather than replacing it.

Honest limitations

• iOS notes have no source-file hint (the accessibility tree doesn't have one). Give important views a testID / accessibilityIdentifier so the agent can grep for it.
• Android emulator support isn't wired up yet. Maestro already reads the tree there; it mainly needs adb for screenshots.
• Firefox isn't supported yet. Chromium browsers are, and Safari can be built with Xcode.

Privacy

Everything stays on your machine. The bridge only listens on 127.0.0.1 and has no accounts, server or analytics. The only place your notes go is the coding agent you connect.

---

I'd love feedback, especially from people building React Native / Flutter / SwiftUI apps, and on whether behaviour-change notes work well with your agent. Issues and PRs are welcome! 🙌

[MadalinWolf]

Really cool project! I like the idea of being able to point at something in the UI and give the agent actual context instead of trying to explain which button or component you mean.

One thing I was wondering: could this eventually work alongside visual regression testing? For example, if a layout change shows up in a screenshot diff, you could pin the affected element and give the agent the context it needs to investigate the change. Seems like a nice way to connect spotting a visual issue with actually fixing it.

Curious to see where you take this!