💡 Ambient Desktop Context Sensing: Proactive Avatar Assistance Based on Active Application Awareness

[github-actions[bot]]

Summary

The avatar proactively detects which desktop application and document the user is working in and offers contextually relevant assistance without being asked. Unlike point-and-tell screen capture (#408, #246 — user-initiated), this is opt-in ambient awareness where the avatar notices application context transitions and suggests help: "I see you just opened a quarterly report in Excel. Want me to pull the key trends?" The avatar becomes a context-aware companion rather than a passive tool waiting to be invoked.

Market Signal

ChatGPT Desktop shipped "Appshots" (Mac-only, July 2026) letting the AI reference the active window for context (TechCrunch, VoiceOS). Google Gemini reads connected apps for its daily brief (Google Blog). Microsoft Copilot Studio's computer-using agents are now GA (Microsoft). ScreenMind pitches "always-on desktop AI with screen/audio/context awareness" (MonetScope). The trend toward proactive desktop context awareness is unanimous across all major AI platforms — but none combine it with avatar personality and conversational delivery.

User Signal

Existing discussions cover user-initiated screen context: #408 (Screen Context Capture — avatar-guided visual understanding on demand) and #246 (Screenshot Context — point-and-tell mode). Discussion #234 (Proactive Workspace Monitoring) focuses on file system changes within the project directory, not cross-application desktop awareness. There is no existing proposal for proactive, ambient screen context sensing where the avatar monitors which desktop application is active and offers unprompted contextual assistance. TalkTerm's non-technical users would benefit most from proactive suggestions since they often don't know what to ask for — the avatar meets them where they are.

Technical Opportunity

Electron provides two levels of desktop context access at minimal cost:

Level	API	Data Captured	Cost	Privacy Impact
Title-only (default)	Electron BrowserWindow + OS APIs	Active window title, app name	Near-zero CPU	Low — metadata only
Visual context (opt-in)	desktopCapturer API	Screenshot of active window	Periodic, user-controlled	Higher — requires explicit consent

The avatar state machine can add a "noticing" micro-state: a subtle animation that signals the avatar has observed a context change before offering a suggestion. Contextual suggestions present via the existing ActionCard system in the left panel. The privacy architecture layers naturally onto Electron's existing permission model, and audit logging per FR27-29 provides full observability of what context was captured and when.

Three-Layer Privacy Architecture

1. Opt-in only — Disabled by default with clear "Ambient Awareness" toggle in settings
2. App exclusion list — Users blocklist sensitive applications (email, banking, HR tools); avatar never observes excluded apps
3. Title-only default — Full screenshot analysis requires explicit per-instance consent; the avatar asks "Can I take a closer look at what you're working on?" before visual capture

Assessment

Dimension	Score	Rationale
Feasibility	med	Electron APIs exist; privacy controls and context-to-suggestion pipeline need design
Impact	high	Major competitive differentiator — proactive assistance without invocation is the next UX frontier
Urgency	high	ChatGPT Appshots, Gemini connected apps, Copilot computer-use all shipping now

Adversarial Review

Strongest objection: Users working on sensitive documents (HR records, financial data, personal communications) won't want AI observing their screen. Privacy backlash could undermine trust — the opposite of TalkTerm's core value proposition.

Rebuttal: The three-layer privacy architecture is stricter than any competitor: opt-in only (ChatGPT Appshots activates on every voice interaction), app exclusion list (no competitor offers this), and title-only default with per-instance consent for visual capture. A visible indicator (avatar eye state or status badge) signals when ambient sensing is active. The audit log records every context observation per FR27-29, giving users full transparency. Users who never enable ambient sensing lose nothing — the feature is additive, not required.

Suggested Next Step

Design the three-layer privacy architecture document and prototype Electron-level active window title detection (cross-platform: macOS, Windows, Linux). Define the avatar "noticing" state machine transition and the context-to-suggestion mapping for 5-7 common application types (spreadsheets, documents, browsers, email, project management).