The AskUI Python SDK takes a different approach to UI automation. Instead of brittle XPath and CSS selectors that break whenever a button moves, elements are found by what they look like or say, and act() turns a high-level instruction into the steps that carry it out across desktop, mobile, and HMI targets. That vision-first, intent-driven model makes it practical for RPA workflows and mobile test suites that would otherwise need constant selector maintenance.
Equally notable is the layered provider design in the BYOMP guide — Provider → Model → MessagesAPI → Client, with distinct slots for act(), get(), and locate() — and OpenAIVlmProvider, which already targets any OpenAI-compatible API. That abstraction lets a team trade cost, latency, and capability per workload without touching the agent loop, which is exactly why more endpoints belong behind it.
Proposal
I'd like to propose OrcaRouter as an additional, optional provider. It would be purely additive: no existing provider, default, or configuration path would change.
Why it may fit here
OrcaRouter exposes an OpenAI-compatible API and uses standard API-key authentication, so it maps onto the existing abstraction the same way OpenAIVlmProvider and OllamaVlmProvider do — a base URL plus a key, wired through the vlm_provider slot (and potentially image_qa_provider). Three things seem relevant to this project's users rather than generic:
- Multiple chat, reasoning, and vision models behind one endpoint. This lines up with the SDK's existing "model flexibility — bring your own" story: one provider config can address many backbones instead of several providers being maintained side by side.
- Automatic model routing and provider failover. Long
act() runs on unattended desktop and mobile machines are exactly where a single upstream hiccup currently ends the run; failover and routing absorb that.
- Usage tracking and budgets, which extend the per-provider cost tracking the SDK already has via
ModelPricing into a shared, cross-model spend view for the reporting layer.
OrcaRouter is already present in the open-source ecosystem (e.g. Dify, promptfoo, goose), so there is prior art for integrations of this shape.
Disclosure
OrcaRouter runs an optional open-source partner program: approved OSS projects can receive a 5% revenue share from OrcaRouter usage attributed to their integration. Participating is not a prerequisite for the integration, and I'm glad to follow any disclosure or governance requirements the project has.
Next steps
More detail is at https://www.orcarouter.ai/built-with. I'm an engineer on the OrcaRouter team. I'd welcome the maintainers' view on whether this fits the roadmap and which provider slot it should target, and I'm happy to prepare the implementation PR if you're open to it.
The AskUI Python SDK takes a different approach to UI automation. Instead of brittle XPath and CSS selectors that break whenever a button moves, elements are found by what they look like or say, and act() turns a high-level instruction into the steps that carry it out across desktop, mobile, and HMI targets. That vision-first, intent-driven model makes it practical for RPA workflows and mobile test suites that would otherwise need constant selector maintenance.
Equally notable is the layered provider design in the BYOMP guide — Provider → Model → MessagesAPI → Client, with distinct slots for act(), get(), and locate() — and OpenAIVlmProvider, which already targets any OpenAI-compatible API. That abstraction lets a team trade cost, latency, and capability per workload without touching the agent loop, which is exactly why more endpoints belong behind it.
Proposal
I'd like to propose OrcaRouter as an additional, optional provider. It would be purely additive: no existing provider, default, or configuration path would change.
Why it may fit here
OrcaRouter exposes an OpenAI-compatible API and uses standard API-key authentication, so it maps onto the existing abstraction the same way
OpenAIVlmProviderandOllamaVlmProviderdo — a base URL plus a key, wired through thevlm_providerslot (and potentiallyimage_qa_provider). Three things seem relevant to this project's users rather than generic:act()runs on unattended desktop and mobile machines are exactly where a single upstream hiccup currently ends the run; failover and routing absorb that.ModelPricinginto a shared, cross-model spend view for the reporting layer.OrcaRouter is already present in the open-source ecosystem (e.g. Dify, promptfoo, goose), so there is prior art for integrations of this shape.
Disclosure
OrcaRouter runs an optional open-source partner program: approved OSS projects can receive a 5% revenue share from OrcaRouter usage attributed to their integration. Participating is not a prerequisite for the integration, and I'm glad to follow any disclosure or governance requirements the project has.
Next steps
More detail is at https://www.orcarouter.ai/built-with. I'm an engineer on the OrcaRouter team. I'd welcome the maintainers' view on whether this fits the roadmap and which provider slot it should target, and I'm happy to prepare the implementation PR if you're open to it.