diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index b4d1947..32cb8bb 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -46,7 +46,7 @@ pnpm --dir Codegen~ run format:check # Prettier pnpm --dir Codegen~ run lint # ESLint pnpm --dir Codegen~ run typecheck # TypeScript pnpm --dir Codegen~ run generate # Emit C# DTOs + CSharpier-format them -pnpm --dir Codegen~ run round-trip # Serialize/deserialize each DTO via System.Text.Json +pnpm --dir Codegen~ run round-trip # Serialize/deserialize each DTO via Newtonsoft.Json pnpm --dir Codegen~ run verify:protocol-dtos # Regenerate and assert no git drift ``` @@ -61,7 +61,7 @@ emit the models (same stack the JS SDK's codegen uses). On top of that: `{ type: string, const: foo }`, strip `enum` from string fields (keep them plain `string` instead of generating C# enums), break cycles introduced by the parser's reference-based inlining of recursive schemas. -- Custom C# preset: PascalCase property names, `[JsonPropertyName("…")]` +- Custom C# preset: PascalCase property names, `[JsonProperty("…")]` attributes, `const` schemas → `init`-only string literals, required reference types get `= null!` to silence CS8618. - Post-process: inject `: IncomingSocketEvent` / `: OutgoingSocketEvent` for diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..4c965b9 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1 @@ +* @kraenhansen diff --git a/.github/ISSUE_TEMPLATE/1-bug_report.yml b/.github/ISSUE_TEMPLATE/1-bug_report.yml new file mode 100644 index 0000000..620d777 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/1-bug_report.yml @@ -0,0 +1,79 @@ +name: Bug report +description: Create a bug report for the ElevenAgents Unity SDK +labels: [bug] +body: + - type: markdown + attributes: + value: | + Thank you for reporting a bug with the **ElevenAgents Unity SDK**. Please fill out the sections below to help us understand and resolve the issue. + + **Never paste an ElevenLabs API key, signed URL, or conversation token** into this issue. + + - type: textarea + attributes: + label: Description + description: | + **Describe the bug in detail and provide clear steps to reproduce it.** + + Include information such as: + - What you were trying to achieve. + - What happened instead. + - Any relevant `ConversationOptions` or agent configuration. + placeholder: | + **Steps to reproduce:** + 1. ... + 2. ... + + **Expected behavior:** + + **Actual behavior:** + validations: + required: true + - type: input + attributes: + label: Package version + description: The `io.elevenlabs.agents` version, or the git commit from `Packages/packages-lock.json`. + placeholder: 0.1.0 (or a commit SHA) + validations: + required: true + - type: input + attributes: + label: Unity version + placeholder: 6000.3.6f1 + validations: + required: true + - type: dropdown + attributes: + label: Where does it happen? + multiple: true + options: + - Editor (Play Mode) + - Standalone (macOS) + - Standalone (Windows) + - Standalone (Linux) + - WebGL + - iOS + - Android + - XR + - Other + validations: + required: true + - type: textarea + attributes: + label: Code example + description: A minimal snippet (or a link to a small project) that reproduces the problem. + render: csharp + - type: textarea + attributes: + label: Logs + description: | + Relevant output from the Unity Console, `Editor.log` / `Player.log`, or the browser console for WebGL builds. + render: text + - type: textarea + attributes: + label: Additional context + description: Add any other context or screenshots about the problem here. + placeholder: | + - Related issues: + - Possible workaround: + - ... diff --git a/.github/ISSUE_TEMPLATE/2-feature_request.yml b/.github/ISSUE_TEMPLATE/2-feature_request.yml new file mode 100644 index 0000000..81ee730 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/2-feature_request.yml @@ -0,0 +1,37 @@ +name: Feature request +description: Propose a new feature for the ElevenAgents Unity SDK +labels: [enhancement] +body: + - type: markdown + attributes: + value: | + Thank you for taking the time to propose a new feature for the **ElevenAgents Unity SDK**. Please provide detailed information below to help us understand your proposal. + + If the feature exists in another ElevenLabs SDK (such as `@elevenlabs/client`, Swift, or Android), mention it below. We aim for parity across SDKs. + + - type: textarea + attributes: + label: Feature description + description: What is the feature, what problem does it solve, and how do you envision it working? + placeholder: | + - What is the feature? + - What problem does it solve? + - How do you envision it working? + validations: + required: true + - type: textarea + attributes: + label: Use case + description: Describe the game or app scenario where this feature would be useful, including the target platforms. + - type: textarea + attributes: + label: Alternatives considered + description: Other ways you've tried or considered solving the problem, and why they fall short. + - type: textarea + attributes: + label: Additional context + description: Any extra information, references, or screenshots that might help us understand your feature request. + placeholder: | + - Related issues or discussions: + - Equivalent API in another ElevenLabs SDK: + - Relevant links: diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..1e1e0d8 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,11 @@ +blank_issues_enabled: true +contact_links: + - name: Report a security vulnerability + url: https://github.com/elevenlabs/unity/security/advisories/new + about: Please report vulnerabilities privately, not as a public issue. See SECURITY.md. + - name: ElevenAgents documentation + url: https://elevenlabs.io/docs/eleven-agents/overview + about: Agent configuration, voices, tools, and platform questions. + - name: Ask a question (Discord) + url: https://discord.com/invite/elevenlabs + about: General questions and community help. diff --git a/.github/SECURITY.md b/.github/SECURITY.md new file mode 100644 index 0000000..a8c8930 --- /dev/null +++ b/.github/SECURITY.md @@ -0,0 +1,24 @@ +# Security policy + +## Reporting a vulnerability + +Please do not report security vulnerabilities through public GitHub issues, discussions, or pull requests. + +Report them privately through either channel: + +- **GitHub:** [Report a vulnerability](https://github.com/elevenlabs/unity/security/advisories/new) on this repository (Security tab → _Report a vulnerability_). +- **ElevenLabs Vulnerability Disclosure Program:** [elevenlabs.io/security/report](https://elevenlabs.io/security/report) or [vulnerability_disclosure_program@elevenlabs.io](mailto:vulnerability_disclosure_program@elevenlabs.io). + +Include the affected package version (or commit), the Unity version and build target, and steps to reproduce. We will acknowledge your report, investigate, and keep you updated on the fix. + +ElevenLabs does not offer cash rewards for vulnerability reports at this time. + +## Supported versions + +The SDK is pre-1.0. Security fixes land on `main` and ship in the next release; older releases are not patched. + +## Scope + +This policy covers the code in this repository: the Unity package (`Runtime/`, `Editor/`, `Plugins/`, `Samples~/`) and its build tooling. Vulnerabilities in the ElevenLabs platform or API itself should go to the ElevenLabs Vulnerability Disclosure Program above. + +Never commit or share an ElevenLabs API key in a Unity project. Players can extract anything shipped in a build, so use a public agent or have your own backend mint signed URLs or conversation tokens. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..9ca114c --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,23 @@ + + +## Changes + + + +- + +## Verification + + + +- [ ] `dotnet csharpier check .` +- [ ] `Bridge~` checks (`format:check`, `lint`, `typecheck`, `test`, `verify:primitives`, `verify:connection`) +- [ ] `Codegen~` checks (`verify:protocol-dtos`, `round-trip`) +- [ ] Edit Mode tests (`pnpm --dir TestProject run test`) +- [ ] WebGL builds + browser integration tests (`IntegrationTests~`) +- [ ] Manually tested in a Unity project. Platforms and Unity version: + +## Checklist + +- [ ] `CHANGELOG.md` updated under `[Unreleased]` for user-facing changes +- [ ] Public API changes are called out above diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index 2b27c12..2f05a8a 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -14,9 +14,14 @@ on: - "Docs~/**" - "**.md" - ".claude/**" + pull_request_target: + paths-ignore: + - "Docs~/**" + - "**.md" + - ".claude/**" concurrency: - group: integration-${{ github.ref }} + group: integration-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }} cancel-in-progress: true permissions: @@ -26,6 +31,14 @@ permissions: jobs: build-webgl: name: Build WebGL — ${{ matrix.scene.name }} (Unity 6000.3.6f1) + # Fork PRs get no secrets under pull_request, so they run via pull_request_target behind fork-ci approval. + if: >- + github.repository == 'elevenlabs/unity' && ( + github.event_name == 'push' || + (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) || + (github.event_name == 'pull_request_target' && github.event.pull_request.head.repo.full_name != github.repository) + ) + environment: ${{ github.event_name == 'pull_request_target' && 'fork-ci' || '' }} runs-on: ubuntu-latest strategy: # Run both scenes in parallel on separate runners — each gets its own @@ -50,6 +63,9 @@ jobs: steps: - name: Checkout uses: actions/checkout@v6 + with: + ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || '' }} + persist-credentials: false # Reclaim ~25 GB so the Unity image (~7 GB compressed, ~20 GB # extracted) fits on ubuntu-latest's ~14 GB root partition. @@ -68,6 +84,7 @@ jobs: df -h / - name: Cache Unity Library + if: github.event_name != 'pull_request_target' uses: actions/cache@v5 with: path: TestProject/Library @@ -80,6 +97,17 @@ jobs: Library-TestProject-WebGL- Library-TestProject- + # Fork runs write caches into main's scope, so they only restore. + - name: Restore Unity Library + if: github.event_name == 'pull_request_target' + uses: actions/cache/restore@v5 + with: + path: TestProject/Library + key: Library-TestProject-WebGL-${{ hashFiles('TestProject/Packages/packages-lock.json', 'TestProject/ProjectSettings/ProjectSettings.asset') }} + restore-keys: | + Library-TestProject-WebGL- + Library-TestProject- + # `buildMethod` runs the same entry point as the corresponding # TestProject/build-webgl*.sh script (Build → primitives, BuildConversation → conversation). - name: Build WebGL @@ -113,6 +141,9 @@ jobs: steps: - name: Checkout uses: actions/checkout@v6 + with: + ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || '' }} + persist-credentials: false - name: Download primitives WebGL build uses: actions/download-artifact@v7 @@ -141,7 +172,7 @@ jobs: uses: actions/setup-node@v6 with: node-version: lts/krypton - cache: pnpm + cache: ${{ github.event_name != 'pull_request_target' && 'pnpm' || '' }} cache-dependency-path: IntegrationTests~/pnpm-lock.yaml - name: Install IntegrationTests~ dependencies diff --git a/.github/workflows/unity-tests.yml b/.github/workflows/unity-tests.yml index fc8566a..5ca72b6 100644 --- a/.github/workflows/unity-tests.yml +++ b/.github/workflows/unity-tests.yml @@ -14,9 +14,14 @@ on: - "Docs~/**" - "**.md" - ".claude/**" + pull_request_target: + paths-ignore: + - "Docs~/**" + - "**.md" + - ".claude/**" concurrency: - group: unity-tests-${{ github.ref }} + group: unity-tests-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }} cancel-in-progress: true permissions: @@ -26,10 +31,21 @@ permissions: jobs: edit-mode: name: Edit Mode tests (Unity 6000.3.6f1) + # Fork PRs get no secrets under pull_request, so they run via pull_request_target behind fork-ci approval. + if: >- + github.repository == 'elevenlabs/unity' && ( + github.event_name == 'push' || + (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) || + (github.event_name == 'pull_request_target' && github.event.pull_request.head.repo.full_name != github.repository) + ) + environment: ${{ github.event_name == 'pull_request_target' && 'fork-ci' || '' }} runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v6 + with: + ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || '' }} + persist-credentials: false # Reclaim ~25 GB so the Unity image (~7 GB compressed, ~20 GB # extracted) fits on ubuntu-latest's ~14 GB root partition. @@ -48,6 +64,7 @@ jobs: df -h / - name: Cache Unity Library + if: github.event_name != 'pull_request_target' uses: actions/cache@v5 with: path: TestProject/Library @@ -56,6 +73,17 @@ jobs: Library-TestProject-EditMode- Library-TestProject- + # Fork runs write caches into main's scope, so they only restore. + - name: Restore Unity Library + if: github.event_name == 'pull_request_target' + uses: actions/cache/restore@v5 + with: + path: TestProject/Library + key: Library-TestProject-EditMode-${{ hashFiles('TestProject/Packages/packages-lock.json', 'TestProject/ProjectSettings/ProjectSettings.asset') }} + restore-keys: | + Library-TestProject-EditMode- + Library-TestProject- + - name: Run Edit Mode tests id: tests uses: game-ci/unity-test-runner@0ff419b913a3630032cbe0de48a0099b5a9f0ed9 # v4.3.1 diff --git a/CHANGELOG.md b/CHANGELOG.md index 8cc6b75..828e7bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,21 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Changed - -- **Breaking:** Minimum Unity version bumped from `6000.0` to `6000.3` (Unity 6.3 LTS). Required for the new native audio output engine built on [`Audio.IAudioGenerator`](https://docs.unity3d.com/6000.3/Documentation/ScriptReference/Audio.IAudioGenerator.html) / [`Audio.GeneratorInstance`](https://docs.unity3d.com/6000.3/Documentation/ScriptReference/Audio.GeneratorInstance.html), which eliminates the streaming-`AudioClip` pre-fill latency that older Unity versions imposed structurally. Unity 6.3 LTS is supported through December 2027. - ## [0.1.0] - Unreleased ### Added -- Core session management (`StartSession`, `EndSession`) -- WebSocket transport for native platforms (desktop, mobile, XR) -- `@elevenlabs/client` bridge transport for WebGL -- Audio capture and playback (default mode) across all platforms -- Core conversation events: `OnConnect`, `OnDisconnect`, `OnMessage`, `OnError`, `OnStatusChange`, `OnModeChange` -- WebGL bridge primitives: Promise-as-Task, Observer, JS-initiated Promise -- UPM distribution via git URL +- `Conversation` session API: `Conversation.StartSessionAsync(ConversationOptions)` / `EndSession()`, with `ConversationId`, `Status`, `Mode`, and `CanSendFeedback`. Async members return Unity's `Awaitable`. +- `ConversationOptions`: `AgentId` / `SignedUrl` / `ConversationToken`, `ConnectionType` (WebSocket, or WebRTC on WebGL), `Input` / `Output` device config, `OutputAudioSource`, `DynamicVariables`, `Overrides`, `UserId`, `CustomLlmExtraBody`, `EnableDebugLogging`. +- Events: `Connected`, `Disconnected`, `ErrorOccurred`, `StatusChanged`, `ModeChanged`, `CanSendFeedbackChanged`, `InitiationMetadataReceived`, `UserTranscriptReceived`, `AgentResponded`, `AgentResponseCorrected`, `AgentResponseCompleted`, `AgentChatResponsePartReceived`, `AudioReceived`, `Interrupted`, `VadScoreUpdated`, `GuardrailTriggered`, `UnhandledClientToolCall`, `AgentToolRequested`, `AgentToolResponded`, `MCPToolCallReceived`, `MCPConnectionStatusChanged`. +- Messaging: `SendUserMessage`, `SendContextualUpdate`, `SendUserActivity`, `SendMultimodalMessage`, `UploadFileAsync`, `SendFeedback`, `SendMCPToolApprovalResult`. +- Client tools: `RegisterTool` (sync and `Awaitable` handlers), `UnregisterTool`, and `ClientToolException` for reporting tool errors to the agent. +- Audio control: `SetVolume`, `SetMicMuted`, `ChangeInputDevice`, `ChangeOutputDevice`, `GetInputVolume` / `GetOutputVolume`, `GetInputByteFrequencyData` / `GetOutputByteFrequencyData`. +- Native transport for the Editor and desktop standalone players: WebSocket via `ClientWebSocket`, microphone capture via `UnityEngine.Microphone`, and low-latency playback through Unity 6.3's `IAudioGenerator`, optionally through a supplied (spatial) `AudioSource`. +- WebGL transport wrapping `@elevenlabs/client` (WebSocket and WebRTC). WebSocket audio plays through a Web Audio graph that mirrors the supplied `AudioSource`'s volume, position, and spatial settings. +- WebGL JS↔C# bridge primitives: `JsBridge`, `JsObject`, `JsFunction`, `BridgeCallback`, `BridgeException`. +- Protocol DTOs and typed event-args records (`ElevenLabs.Protocol`), generated from the Agents AsyncAPI spec. +- Samples: GettingStarted (walk-up-and-talk with spatial audio, dynamic variables, and overrides) and QuickStart (minimal transcript). +- UPM distribution via git URL. Requires Unity 6.3 LTS (`6000.3`) or later; depends on Input System 1.18.0 and Newtonsoft.Json 3.2.1. [Unreleased]: https://github.com/elevenlabs/unity/compare/v0.1.0...HEAD [0.1.0]: https://github.com/elevenlabs/unity/releases/tag/v0.1.0 diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 5d48f45..f0fd692 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -20,7 +20,19 @@ The SDK reads JS→C# callback payloads on WebGL via [`Marshal.PtrToStringUTF8`] ## Platform support -The public API (`Runtime/`) is cross-platform. WebGL is the most mature target today; native targets (desktop, mobile, XR) are implemented under `Runtime/Native/` and ship in the same package — the right implementation is selected at compile time per platform. +The public API (`Runtime/Core/`) is cross-platform. Two transports ship in the package, and the one matching the active build target is registered automatically: + +| Target | Transport | Connection types | Status | +|---|---|---|---| +| WebGL | `@elevenlabs/client` via the JS bridge | WebSocket, WebRTC | Supported, CI-tested in Chromium | +| Editor (Play Mode), desktop standalone (macOS, Windows, Linux) | Native (`ClientWebSocket`, `UnityEngine.Microphone`, `IAudioGenerator`) | WebSocket only — `ConnectionType.WebRTC` throws `NotSupportedException` | Supported | +| iOS, Android, XR | Native | WebSocket only | Untested. The SDK doesn't request microphone permission itself. | + +Editor Play Mode while the active build target is WebGL isn't supported yet ([#16](https://github.com/elevenlabs/unity/issues/16)); switch the build target to a desktop platform to iterate in the Editor. + +## Package dependencies + +Installed automatically by the Package Manager: `com.unity.inputsystem` 1.18.0 and `com.unity.nuget.newtonsoft-json` 3.2.1. ## Additional WebGL setup @@ -51,9 +63,9 @@ When you pass an `AudioSource` to `ConversationOptions.OutputAudioSource`, the f | `maxDistance` | `PannerNode.maxDistance` | Direct | | `rolloffMode` (Linear / Logarithmic) | `PannerNode.distanceModel` | `AudioRolloffMode.Custom` falls back to Logarithmic with a one-time warning | | `panStereo` | `StereoPannerNode.pan` | Effective when `spatialBlend < 1` | -| `dopplerLevel` | Sampled per-frame velocity → `PannerNode` position pre-emphasis | Approximate | +| `dopplerLevel` | ❌ Not supported | Forwarded but has no effect — Web Audio has no doppler control | | `outputAudioMixerGroup` | ❌ Not supported | One-time warning when first observed as non-null | -| Custom rolloff `AnimationCurve` | ❌ Not supported | One-time warning; falls back to `rolloffMode` | +| Custom rolloff `AnimationCurve` | ❌ Not supported | Falls back to Logarithmic (see `rolloffMode` above) | | `bypassEffects`, `bypassListenerEffects`, `bypassReverbZones` | ❌ Not supported | Silently ignored — no Web Audio analog | | `priority`, `spread`, `reverbZoneMix` | ❌ Not supported | Silently ignored | @@ -70,6 +82,6 @@ Use the SDK's cross-platform reading APIs instead — they're tuned to return co | What you want | Native-only API (silent on WebGL) | Cross-platform SDK API | |---|---|---| | Scalar volume `[0, 1]` for an envelope / meter / bob | `audioSource.GetOutputData(buf); rms(buf);` | `conversation.GetOutputVolume()` | -| Frequency-domain magnitudes | `AudioSource.GetSpectrumData(...)` | `conversation.GetByteFrequencyData(buf)` | +| Frequency-domain magnitudes | `AudioSource.GetSpectrumData(...)` | `conversation.GetOutputByteFrequencyData(buf)` | The shipped samples under `Samples~/` follow this rule — copy from them. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 65640eb..9ba0860 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,20 +11,24 @@ Runtime/ C# source compiled into the shipped package internal IConnection / IInputController / IOutputController abstractions the platform impls satisfy, and the generated protocol DTOs under Protocol/*.g.cs. No platform restrictions. - WebGL/ Editor + WebGL asmdef `ElevenLabs.Agents.WebGL`. Carries the bridge - primitives (JsObject, JsFunction, BridgeCallback handle types + their + WebGL/ Bridge primitives (JsObject, JsFunction, BridgeCallback handle types + their IJsObject / IJsFunction interface seams) and the Bridged/ folder of IConnection / IInputController / IOutputController implementations that - wrap @elevenlabs/client. - (Native/ Phase 7 — not yet present. Will mirror Runtime/WebGL/ as an - excludePlatforms=WebGL asmdef carrying native IConnection / - IInputController / IOutputController implementations.) - -Editor/ Editor-only C# (HostBuild entry points for WebGL builds, inspectors, - build post-processors, validation). + wrap @elevenlabs/client. Compiled by the Editor + WebGL asmdef + `ElevenLabs.Agents.WebGL`, which sits at `Runtime/` itself — so a new file + dropped directly into `Runtime/` lands in that assembly. + Native/ excludePlatforms=WebGL asmdef `ElevenLabs.Agents.Native`: ClientWebSocket + connection, UnityEngine.Microphone input, IAudioGenerator-based output + (UnityAudioSourceOutput + the IAudioOutputEngine implementations), and + NativeSessionLauncher. + +Editor/ Editor-only C#: HostBuild entry points for the WebGL and standalone smoke + builds, a build preprocessor that validates WebGL Player Settings, and the + link.xml injector. Tests/ - Editor/ Unity Test Runner tests (Edit Mode — run headlessly in CI). + Editor/ Edit Mode tests (run headlessly in CI). Native/ is its own asmdef. + Runtime/Native/ PlayMode characterization tests for Unity's audio APIs (run manually). Plugins/ WebGL/ .jslib bundles — Unity's required location for WebGL native plugins. @@ -70,7 +74,7 @@ TestProject/ Embedded dev Unity host project. Used to (a) run the Edit Mode integration tests. Not shipped with the package. Most of Assets/ is gitignored so a fresh clone always opens to a known-good state. -IntegrationTests~/ Vitest + Playwright tests that drive the two WebGL builds produced by +IntegrationTests~/ Vitest + Playwright tests that drive the WebGL builds produced by TestProject/ from a headless Chromium. Runs entirely outside Unity. Docs~/ Design documents and RFCs. Read these before making architectural changes; @@ -102,7 +106,7 @@ with **pnpm**. Run commands with `pnpm --dir run