Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,9 +197,9 @@ Every item below is one more layer sedimented into a default — a capability yo
- 🎨 **Style Pack Marketplace** — browse, install, and like community **style packs** from the in-app Marketplace, and publish your own (custom system prompt per pack, switchable by hotkey). Backed by a moderated marketplace backend; uploads are reviewed before they go public.
- ⚡ **Streaming insertion** — polished text is written to the cursor character by character to reduce perceived latency, with an automatic one-shot-paste fallback. Toggle in Settings → Recording.
- **Toggle and push-to-talk** recording modes, plus a **MediaPlayPause trigger** so wired-earbud inline controls can start and stop recording. `Esc` cancels at any phase, including polish and insertion.
- **Cloud ASR**: Volcengine streaming ASR (bigasr), iFlytek realtime ASR (RTASR), Alibaba Cloud Bailian (classic realtime / Qwen3 realtime / Fun-ASR-Flash file transcription), StepFun StepAudio (batch + realtime), Zhipu GLM-ASR, Xiaomi MiMo ASR, ElevenLabs Scribe, OpenAI-compatible batch transcription (OpenAI Whisper / Groq / SiliconFlow SenseVoice / OpenRouter / ZenMux), and Apple Speech (macOS).
- **Cloud ASR**: Volcengine streaming ASR (bigasr), iFlytek realtime ASR (RTASR), Alibaba Cloud Bailian (classic realtime / Qwen3 realtime / Fun-ASR-Flash file transcription), StepFun StepAudio (batch + realtime), Zhipu GLM-ASR, Xiaomi MiMo ASR, OrcaRouter audio-input Gemini, ElevenLabs Scribe, OpenAI-compatible batch transcription (OpenAI Whisper / Groq / SiliconFlow SenseVoice / OpenRouter / ZenMux), and Apple Speech (macOS).
- **Local ASR**: bundled Qwen3-ASR (0.6B / 1.7B) via vendored `Open-Less/qwen-asr` (macOS); Windows Foundry Local Whisper and sherpa-onnx (experimental) variants.
- **Polish providers**: Ark (Volcengine), DeepSeek, OpenAI, Google Gemini, Codex OAuth, SiliconFlow, Atlas Cloud, Xiaomi MiMo, CometAPI, OpenRouter, Alibaba Cloud Coding Plan, CodingPlanX, MiniMax, and StepFun — plus any OpenAI-compatible endpoint you bring.
- **Polish providers**: Ark (Volcengine), DeepSeek, OpenAI, Google Gemini, Codex OAuth, SiliconFlow, Atlas Cloud, Xiaomi MiMo, CometAPI, OpenRouter, OrcaRouter, Alibaba Cloud Coding Plan, CodingPlanX, MiniMax, and StepFun — plus any OpenAI-compatible endpoint you bring.
- **Four output modes**: raw, light polish, structured (**AI-prompt mode**), and formal. Plus a **translation hotkey** that converts speech directly into the configured target language ([#43](../../issues/43)).
- **Selection-ask QA panel** — a separate hotkey opens a floating panel that runs voice Q&A against the highlighted text in any app ([#118](../../issues/118)).
- **Main window**: Overview / History / Vocab / Style / Marketplace / Settings. Persistent tray icon, plus a mini status capsule that floats on screen and follows the display you are typing on (multi-monitor).
Expand Down
4 changes: 2 additions & 2 deletions README.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,9 +202,9 @@ OpenLess 只做一件事:**把语音变成可用的书面文字(尤其是 AI 提
- 🎨 **风格包市场**——在应用内的 Marketplace 浏览、安装、点赞社区**风格包**,并发布自己的(每个包一套自定义系统提示词,可用快捷键切换)。由经过审核的市场后端支撑;上传内容公开前会经过审核。
- ⚡ **流式插入**——润色后的文本逐字符写入光标以降低感知延迟,并带有自动的一次性粘贴回退。可在 设置 → 录音 中切换。
- **切换式与按住说话(push-to-talk)** 两种录音模式,外加 **MediaPlayPause 触发**,让有线耳机的线控也能开始 / 停止录音。`Esc` 可在任意阶段取消,包括润色与插入。
- **云端 ASR**:Volcengine 流式 ASR(bigasr)、讯飞实时语音转写(RTASR)、阿里云百炼(经典实时 / Qwen3 实时 / Fun-ASR-Flash 录音文件)、阶跃星辰 StepAudio(批式 + 实时)、智谱 GLM-ASR、小米 MiMo ASR、ElevenLabs Scribe、OpenAI 兼容批量转写(OpenAI Whisper / Groq / 硅基流动 SenseVoice / OpenRouter / ZenMux),以及 Apple Speech(macOS)。
- **云端 ASR**:Volcengine 流式 ASR(bigasr)、讯飞实时语音转写(RTASR)、阿里云百炼(经典实时 / Qwen3 实时 / Fun-ASR-Flash 录音文件)、阶跃星辰 StepAudio(批式 + 实时)、智谱 GLM-ASR、小米 MiMo ASR、OrcaRouter 音频输入 Gemini、ElevenLabs Scribe、OpenAI 兼容批量转写(OpenAI Whisper / Groq / 硅基流动 SenseVoice / OpenRouter / ZenMux),以及 Apple Speech(macOS)。
- **本地 ASR**:通过 vendored 的 `Open-Less/qwen-asr` 内置 Qwen3-ASR(0.6B / 1.7B)(macOS);Windows 上的 Foundry Local Whisper 与 sherpa-onnx(实验性)变体。
- **润色提供方**:Ark(火山方舟)、DeepSeek、OpenAI、Google Gemini、Codex OAuth、硅基流动、Atlas Cloud、小米 MiMo、CometAPI、OpenRouter、阿里云 Coding Plan、CodingPlanX、MiniMax、StepFun,以及你自带的任意 OpenAI 兼容端点。
- **润色提供方**:Ark(火山方舟)、DeepSeek、OpenAI、Google Gemini、Codex OAuth、硅基流动、Atlas Cloud、小米 MiMo、CometAPI、OpenRouter、OrcaRouter、阿里云 Coding Plan、CodingPlanX、MiniMax、StepFun,以及你自带的任意 OpenAI 兼容端点。
- **四种输出模式**:原文、轻度润色、结构化(**AI 提示词模式**)、正式。另有一个**翻译快捷键**,将语音直接转换为所配置的目标语言([#43](../../issues/43))。
- **选区问答面板**——一个独立快捷键打开浮动面板,针对任意应用中被高亮选中的文本进行语音问答([#118](../../issues/118))。
- **主窗口**:概览 / 历史 / 词典 / 风格 / 市场 / 设置。常驻托盘图标,以及一个浮于屏幕、并跟随你正在输入的显示器的迷你状态胶囊(多显示器)。
Expand Down
93 changes: 77 additions & 16 deletions openless-all/app/src-tauri/src/asr/mimo.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,19 @@ const MIMO_MAX_CHUNK_DURATION_MS: u64 = 180_000;
pub const PROVIDER_ID: &str = "xiaomi-mimo-asr";
pub const DEFAULT_ENDPOINT: &str = "https://api.xiaomimimo.com/v1";
pub const DEFAULT_MODEL: &str = "mimo-v2.5-asr";
pub const ORCAROUTER_PROVIDER_ID: &str = "orcarouter";
pub const ORCAROUTER_DEFAULT_ENDPOINT: &str = "https://api.orcarouter.ai/v1";
pub const ORCAROUTER_DEFAULT_MODEL: &str = "google/gemini-2.5-flash";
const ORCAROUTER_TRANSCRIPTION_PROMPT: &str =
"Transcribe this audio verbatim. Return only the transcript, without commentary.";

pub struct MimoBatchASR {
api_key: String,
base_url: String,
model: String,
provider_name: &'static str,
prompt: Option<&'static str>,
audio_data_url: bool,
buffer: Mutex<Vec<u8>>,
}

Expand All @@ -34,6 +42,21 @@ impl MimoBatchASR {
api_key,
base_url,
model,
provider_name: "MiMo",
prompt: None,
audio_data_url: true,
buffer: Mutex::new(Vec::new()),
}
}

pub fn new_orcarouter(api_key: String, base_url: String, model: String) -> Self {
Self {
api_key,
base_url,
model,
provider_name: "OrcaRouter",
prompt: Some(ORCAROUTER_TRANSCRIPTION_PROMPT),
audio_data_url: false,
buffer: Mutex::new(Vec::new()),
}
}
Expand All @@ -56,7 +79,7 @@ impl MimoBatchASR {

async fn transcribe_inner(&self, pcm: &[u8]) -> Result<RawTranscript> {
if self.api_key.trim().is_empty() {
anyhow::bail!("MiMo API key missing");
anyhow::bail!("{} API key missing", self.provider_name);
}

let duration_ms = pcm_duration_ms(pcm);
Expand All @@ -78,7 +101,7 @@ impl MimoBatchASR {
.map(|chunk| i16::from_le_bytes([chunk[0], chunk[1]]))
.collect();
let wav = encode_wav_16k_mono(&samples);
let body = mimo_chat_body(&self.model, &wav);
let body = audio_input_chat_body(&self.model, &wav, self.prompt, self.audio_data_url);
let url = mimo_chat_completions_url(&self.base_url)?;
let client = crate::net::http();
let resp = client
Expand All @@ -87,15 +110,18 @@ impl MimoBatchASR {
.json(&body)
.send()
.await
.context("MiMo ASR HTTP request failed")?;
.with_context(|| format!("{} ASR HTTP request failed", self.provider_name))?;

if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
anyhow::bail!("MiMo ASR API error {}: {}", status, body);
anyhow::bail!("{} ASR API error {}: {}", self.provider_name, status, body);
}

let json: Value = resp.json().await.context("parse MiMo ASR response")?;
let json: Value = resp
.json()
.await
.with_context(|| format!("parse {} ASR response", self.provider_name))?;
Ok(extract_mimo_text(&json).trim().to_string())
}

Expand Down Expand Up @@ -126,22 +152,42 @@ pub fn mimo_chat_completions_url(base_url: &str) -> Result<String> {
}

pub fn mimo_chat_body(model: &str, wav: &[u8]) -> Value {
let audio_data = format!(
"data:audio/wav;base64,{}",
base64::engine::general_purpose::STANDARD.encode(wav)
);
audio_input_chat_body(model, wav, None, true)
}

pub fn orcarouter_chat_body(model: &str, wav: &[u8]) -> Value {
audio_input_chat_body(model, wav, Some(ORCAROUTER_TRANSCRIPTION_PROMPT), false)
}

fn audio_input_chat_body(
model: &str,
wav: &[u8],
prompt: Option<&str>,
audio_data_url: bool,
) -> Value {
let encoded = base64::engine::general_purpose::STANDARD.encode(wav);
let audio_data = if audio_data_url {
format!("data:audio/wav;base64,{encoded}")
} else {
encoded
};
let mut content = Vec::with_capacity(if prompt.is_some() { 2 } else { 1 });
if let Some(text) = prompt {
content.push(serde_json::json!({ "type": "text", "text": text }));
}
content.push(serde_json::json!({
"type": "input_audio",
"input_audio": {
"data": audio_data,
"format": "wav",
},
}));
serde_json::json!({
"model": model,
"stream": false,
"messages": [{
"role": "user",
"content": [{
"type": "input_audio",
"input_audio": {
"data": audio_data,
"format": "wav",
},
}],
"content": content,
}],
})
}
Expand Down Expand Up @@ -296,6 +342,21 @@ mod tests {
.starts_with("data:audio/wav;base64,"));
}

#[test]
fn orcarouter_body_uses_prompt_and_raw_base64_audio() {
let body = orcarouter_chat_body(ORCAROUTER_DEFAULT_MODEL, b"wav");
assert_eq!(body["model"], ORCAROUTER_DEFAULT_MODEL);
assert_eq!(body["messages"][0]["content"][0]["type"], "text");
assert_eq!(
body["messages"][0]["content"][0]["text"],
ORCAROUTER_TRANSCRIPTION_PROMPT
);
let audio = &body["messages"][0]["content"][1];
assert_eq!(audio["type"], "input_audio");
assert_eq!(audio["input_audio"]["format"], "wav");
assert_eq!(audio["input_audio"]["data"], "d2F2");
}

#[test]
fn mimo_response_extracts_message_content() {
let json = serde_json::json!({
Expand Down
1 change: 1 addition & 0 deletions openless-all/app/src-tauri/src/commands/credentials.rs
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ fn llm_provider_default_endpoint(provider: &str) -> Option<&'static str> {
"mimo" => Some("https://api.xiaomimimo.com/v1"),
"cometapi" => Some("https://api.cometapi.com/v1"),
"openrouterFree" => Some("https://openrouter.ai/api/v1"),
"orcarouter" => Some("https://api.orcarouter.ai/v1"),
"alibabaCoding" => Some("https://coding-intl.dashscope.aliyuncs.com/v1"),
"codingPlanX" => Some("https://api.codingplanx.ai/v1"),
"stepfun" => Some("https://api.stepfun.com/v1"),
Expand Down
65 changes: 64 additions & 1 deletion openless-all/app/src-tauri/src/commands/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,9 @@ mod tests {
};
use crate::commands::providers::{
active_asr_is_keyless_for_validation, asr_transcriptions_url, fetch_provider_models,
is_gemini_base_url, models_url, parse_gemini_model_ids, parse_model_ids, ProviderConfig,
is_gemini_base_url, is_orcarouter_base_url, models_url, parse_gemini_model_ids,
parse_model_ids, parse_orcarouter_asr_model_ids, parse_orcarouter_model_ids,
ProviderConfig,
};
use crate::commands::settings::{
parse_latest_beta_from_atom, persist_settings, SettingsWriter,
Expand Down Expand Up @@ -686,6 +688,22 @@ mod tests {
assert!(llm_configured_for_provider("atlascloud", &ready));
}

#[test]
fn credentials_status_requires_api_key_for_orcarouter() {
let keyless = CredentialsSnapshot {
ark_endpoint: Some("https://api.orcarouter.ai/v1".into()),
ark_model_id: Some("orcarouter/fusion-flash".into()),
..snapshot()
};
assert!(!llm_configured_for_provider("orcarouter", &keyless));

let ready = CredentialsSnapshot {
ark_api_key: Some("key".into()),
..keyless
};
assert!(llm_configured_for_provider("orcarouter", &ready));
}

impl SettingsWriter for FakeSettingsWriter {
fn read_settings(&self) -> UserPreferences {
self.saved.lock().unwrap().clone().unwrap_or_default()
Expand Down Expand Up @@ -809,6 +827,51 @@ mod tests {
assert_eq!(models, vec!["a".to_string(), "b".to_string()]);
}

#[test]
fn orcarouter_catalog_filters_non_chat_models() {
let body = r#"{"data":[
{"id":"orcarouter/fusion-flash","supported_endpoint_types":["openai","anthropic"]},
{"id":"openai/gpt-5-mini","supported_endpoint_types":["openai"]},
{"id":"openai/sora-2","supported_endpoint_types":["openai-video"]},
{"id":"openai/text-embedding-3-small","supported_endpoint_types":["embeddings"]},
{"id":"legacy/chat-model"}
]}"#;
assert_eq!(
parse_orcarouter_model_ids(body).unwrap(),
vec![
"legacy/chat-model".to_string(),
"openai/gpt-5-mini".to_string(),
"orcarouter/fusion-flash".to_string(),
]
);
assert!(is_orcarouter_base_url("https://api.orcarouter.ai/v1"));
assert!(!is_orcarouter_base_url("https://orcarouter.ai/v1"));
assert!(!is_orcarouter_base_url(
"https://api.orcarouter.ai.example/v1"
));
}

#[test]
fn orcarouter_asr_catalog_keeps_audio_input_gemini_models_only() {
let body = r#"{"data":[
{"id":"google/gemini-2.5-flash","supported_endpoint_types":["openai"]},
{"id":"google/gemini-3.5-flash","supported_endpoint_types":["openai"]},
{"id":"google/gemini-2.5-flash-image-preview","supported_endpoint_types":["openai"]},
{"id":"google/gemini-2.5-flash-tts","supported_endpoint_types":["openai"]},
{"id":"openai/whisper-large-v3","supported_endpoint_types":["openai"]},
{"id":"google/gemini-legacy"},
{"id":"google/gemini-native","supported_endpoint_types":["google"]}
]}"#;
assert_eq!(
parse_orcarouter_asr_model_ids(body).unwrap(),
vec![
"google/gemini-2.5-flash".to_string(),
"google/gemini-3.5-flash".to_string(),
"google/gemini-legacy".to_string(),
]
);
}

#[test]
fn parse_gemini_model_ids_strips_models_prefix_and_dedups() {
// Google v1beta/models 真实响应的子集——name 字段带 `models/` 前缀,
Expand Down
Loading