Skip to content

Commit 7fa1aaa

Browse files
authored
Fix parakeet_runner blank priming (pytorch#16509)
### Summary `parakeet_runner` is currently missing transcription of the first part of audio. I believe this is due to a lack of "priming with blanks" that is done in https://github.com/NVIDIA-NeMo/NeMo. - SOS is defined as blank: https://github.com/NVIDIA-NeMo/NeMo/blob/bf583c980b70cecc184fa8a083a9c3ddb87f905e/nemo/collections/asr/parts/submodules/transducer_decoding/tdt_label_looping.py#L250 - Predictor priming with SOS: https://github.com/NVIDIA-NeMo/NeMo/blob/bf583c980b70cecc184fa8a083a9c3ddb87f905e/nemo/collections/asr/parts/submodules/transducer_decoding/tdt_label_looping.py#L363-L368 Tested with this audio file: [audio.wav](https://github.com/user-attachments/files/24498568/audio.wav) Which I would manually transcribe as: "Can you please add to analyze.py the ability to pass in a dash dash dataset dash path or dash d that will take the place of dot input if present? Also dot input should be renamed to dot dataset" #### ExecuTorch Before Before Executorch misses the "Can you please add to analyze" and transcribes as "py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset." ``` -> % ./cmake-out/examples/models/parakeet/parakeet_runner --model_path examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte --tokenizer_path /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model --audio_path /Users/matt/Documents/parakeet_test_audio/audio.wav I tokenizers:regex.cpp:27] Registering override fallback regex I 00:00:00.000767 executorch:main.cpp:278] Loading model from: examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte I 00:00:00.001246 executorch:main.cpp:294] Loading audio from: /Users/matt/Documents/parakeet_test_audio/audio.wav I 00:00:00.002790 executorch:wav_loader.h:98] WAV header detected, getting raw audio data. I 00:00:00.002793 executorch:wav_loader.h:105] RIFF Header: RIFF I 00:00:00.002794 executorch:wav_loader.h:106] Chunk Size: 853676 I 00:00:00.002795 executorch:wav_loader.h:113] WAVE Header: WAVE I 00:00:00.002796 executorch:wav_loader.h:120] Format Header: fmt I 00:00:00.002797 executorch:wav_loader.h:121] Format Chunk Size: 16 I 00:00:00.002798 executorch:wav_loader.h:122] Audio Format: 3 I 00:00:00.002799 executorch:wav_loader.h:123] Number of Channels: 1 I 00:00:00.002800 executorch:wav_loader.h:124] Sample Rate: 16000 I 00:00:00.002801 executorch:wav_loader.h:125] Byte Rate: 64000 I 00:00:00.002801 executorch:wav_loader.h:126] Block Align: 4 I 00:00:00.002802 executorch:wav_loader.h:127] Bits per Sample: 32 I 00:00:00.002803 executorch:wav_loader.h:132] Subchunk2Size: 853604 I 00:00:00.002824 executorch:wav_loader.h:226] Loaded 213401 audio samples from WAV file: /Users/matt/Documents/parakeet_test_audio/audio.wav I 00:00:00.002826 executorch:main.cpp:297] Loaded 213401 audio samples I 00:00:00.002838 executorch:main.cpp:308] Running preprocessor... I 00:00:00.018590 executorch:cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version I 00:00:00.018609 executorch:cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version I 00:00:00.027469 executorch:main.cpp:333] Mel spectrogram shape: [1, 128, 1334], mel_len: 1333 I 00:00:00.027480 executorch:main.cpp:336] Running encoder... I 00:00:26.206136 executorch:main.cpp:353] Encoder output shape: [1, 1024, 167], len=167 I 00:00:26.206163 executorch:main.cpp:385] Model metadata: vocab_size=8192, blank_id=8192, num_rnn_layers=2, pred_hidden=640, sample_rate=16000 I 00:00:26.206166 executorch:main.cpp:387] Running TDT greedy decode... I 00:00:29.509575 executorch:main.cpp:397] Decoded 55 tokens I 00:00:29.509588 executorch:main.cpp:400] Loading tokenizer from: /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model E tokenizers:hf_tokenizer.cpp:82] Error parsing json file: [json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - invalid literal; last read: '<U+000A><U+000E>' E tokenizers:tiktoken.cpp:59] invalid tiktoken line: I 00:00:29.514298 executorch:llm_runner_helper.cpp:77] Loaded Sentencepiece tokenizer Transcription tokens: py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset. I 00:00:29.514337 executorch:main.cpp:415] Done! ``` #### NeMo NeMo thru: ``` import nemo.collections.asr as nemo_asr asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="nvidia/parakeet-tdt-0.6b-v3") output = asr_model.transcribe(['/Users/matt/Documents/parakeet_test_audio/audio.wav']) print(output[0].text) ``` outputs "Can you please add to analyze.py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset.": ``` -> % python examples/asr/asr_chunked_inference/rnnt/test.py [NeMo W 2026-01-08 10:20:16 megatron_init:62] Megatron num_microbatches_calculator not found, using Apex version. W0108 10:20:16.612000 27527 torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs. OneLogger: Setting error_handling_strategy to DISABLE_QUIETLY_AND_REPORT_METRIC_ERROR for rank (rank=0) with OneLogger disabled. To override: explicitly set error_handling_strategy parameter. No exporters were provided. This means that no telemetry data will be collected. [NeMo I 2026-01-08 10:20:22 mixins:184] Tokenizer SentencePieceTokenizer initialized with 8192 tokens [NeMo W 2026-01-08 10:20:24 modelPT:188] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader. Train config : use_lhotse: true skip_missing_manifest_entries: true input_cfg: null tarred_audio_filepaths: null manifest_filepath: null sample_rate: 16000 shuffle: true num_workers: 2 pin_memory: true max_duration: 10.0 min_duration: 1.0 text_field: answer batch_duration: null max_tps: null use_bucketing: true bucket_duration_bins: null bucket_batch_size: null num_buckets: 30 bucket_buffer_size: 20000 shuffle_buffer_size: 10000 [NeMo W 2026-01-08 10:20:24 modelPT:195] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). Validation config : use_lhotse: true manifest_filepath: null sample_rate: 16000 batch_size: 16 shuffle: false max_duration: 40.0 min_duration: 0.1 num_workers: 2 pin_memory: true text_field: answer [NeMo I 2026-01-08 10:20:26 rnnt_models:226] Using RNNT Loss : tdt Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1} [NeMo I 2026-01-08 10:20:26 rnnt_models:226] Using RNNT Loss : tdt Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1} [NeMo W 2026-01-08 10:20:26 label_looping_base:113] No conditional node support for Cuda. Cuda graphs with while loops are disabled, decoding speed will be slower Reason: CUDA is not available [NeMo I 2026-01-08 10:20:26 rnnt_models:226] Using RNNT Loss : tdt Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1} [NeMo W 2026-01-08 10:20:26 label_looping_base:113] No conditional node support for Cuda. Cuda graphs with while loops are disabled, decoding speed will be slower Reason: CUDA is not available [NeMo I 2026-01-08 10:20:29 save_restore_connector:284] Model EncDecRNNTBPEModel was successfully restored from /Users/matt/.cache/huggingface/hub/models--nvidia--parakeet-tdt-0.6b-v3/snapshots/6d590f77001d318fb17a0b5bf7ee329a91b52598/parakeet-tdt-0.6b-v3.nemo. [NeMo W 2026-01-08 10:20:29 dataloader:760] The following configuration keys are ignored by Lhotse dataloader: use_start_end_token [NeMo W 2026-01-08 10:20:29 dataloader:498] You are using a non-tarred dataset and requested tokenization during data sampling (pretokenize=True). This will cause the tokenization to happen in the main (GPU) process,possibly impacting the training speed if your tokenizer is very large.If the impact is noticable, set pretokenize=False in dataloader config.(note: that will disable token-per-second filtering and 2D bucketing features) Transcribing: 1it [00:00, 1.11it/s] Can you please add to analyze.py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset. ``` #### ExecuTorch After After ExecuTorch "Can you please add to analyze.py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset." ``` -> % ./cmake-out/examples/models/parakeet/parakeet_runner --model_path examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte --tokenizer_path /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model --audio_path /Users/matt/Documents/parakeet_test_audio/audio.wav I tokenizers:regex.cpp:27] Registering override fallback regex I 00:00:00.002165 executorch:main.cpp:298] Loading model from: examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte I 00:00:00.002909 executorch:main.cpp:314] Loading audio from: /Users/matt/Documents/parakeet_test_audio/audio.wav I 00:00:00.005307 executorch:wav_loader.h:98] WAV header detected, getting raw audio data. I 00:00:00.005318 executorch:wav_loader.h:105] RIFF Header: RIFF I 00:00:00.005320 executorch:wav_loader.h:106] Chunk Size: 853676 I 00:00:00.005322 executorch:wav_loader.h:113] WAVE Header: WAVE I 00:00:00.005324 executorch:wav_loader.h:120] Format Header: fmt I 00:00:00.005325 executorch:wav_loader.h:121] Format Chunk Size: 16 I 00:00:00.005327 executorch:wav_loader.h:122] Audio Format: 3 I 00:00:00.005329 executorch:wav_loader.h:123] Number of Channels: 1 I 00:00:00.005330 executorch:wav_loader.h:124] Sample Rate: 16000 I 00:00:00.005332 executorch:wav_loader.h:125] Byte Rate: 64000 I 00:00:00.005333 executorch:wav_loader.h:126] Block Align: 4 I 00:00:00.005335 executorch:wav_loader.h:127] Bits per Sample: 32 I 00:00:00.005336 executorch:wav_loader.h:132] Subchunk2Size: 853604 I 00:00:00.005447 executorch:wav_loader.h:226] Loaded 213401 audio samples from WAV file: /Users/matt/Documents/parakeet_test_audio/audio.wav I 00:00:00.005458 executorch:main.cpp:317] Loaded 213401 audio samples I 00:00:00.005485 executorch:main.cpp:328] Running preprocessor... I 00:00:00.025430 executorch:cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version I 00:00:00.025449 executorch:cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version I 00:00:00.035054 executorch:main.cpp:353] Mel spectrogram shape: [1, 128, 1334], mel_len: 1333 I 00:00:00.035067 executorch:main.cpp:356] Running encoder... I 00:00:25.835932 executorch:main.cpp:373] Encoder output shape: [1, 1024, 167], len=167 I 00:00:25.835974 executorch:main.cpp:405] Model metadata: vocab_size=8192, blank_id=8192, num_rnn_layers=2, pred_hidden=640, sample_rate=16000 I 00:00:25.835977 executorch:main.cpp:407] Running TDT greedy decode... I 00:00:29.269823 executorch:main.cpp:417] Decoded 65 tokens I 00:00:29.269837 executorch:main.cpp:420] Loading tokenizer from: /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model E tokenizers:hf_tokenizer.cpp:82] Error parsing json file: [json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - invalid literal; last read: '<U+000A><U+000E>' E tokenizers:tiktoken.cpp:59] invalid tiktoken line: I 00:00:29.274029 executorch:llm_runner_helper.cpp:77] Loaded Sentencepiece tokenizer Transcription tokens: Can you please add to analyze.py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset. I 00:00:29.274057 executorch:main.cpp:435] Done! ``` ### Test plan Tested manually, see above
1 parent 535dcfb commit 7fa1aaa

1 file changed

Lines changed: 29 additions & 7 deletions

File tree

examples/models/parakeet/main.cpp

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -105,16 +105,38 @@ std::vector<int64_t> greedy_decode_executorch(
105105
static_cast<::executorch::aten::SizesType>(pred_hidden)},
106106
::executorch::aten::ScalarType::Float);
107107

108-
// Initialize decoder state with zeros
109-
std::vector<float> sos_g_data(1 * 1 * pred_hidden, 0.0f);
110-
auto sos_g = from_blob(
111-
sos_g_data.data(),
112-
{1, 1, static_cast<::executorch::aten::SizesType>(pred_hidden)},
113-
::executorch::aten::ScalarType::Float);
108+
// Prime the prediction network state with SOS (= blank_id) to match NeMo TDT
109+
// greedy label-looping decoding behavior:
110+
// - SOS is defined as blank:
111+
// https://github.com/NVIDIA-NeMo/NeMo/blob/bf583c980b70cecc184fa8a083a9c3ddb87f905e/nemo/collections/asr/parts/submodules/transducer_decoding/tdt_label_looping.py#L250
112+
// - Predictor priming with SOS:
113+
// https://github.com/NVIDIA-NeMo/NeMo/blob/bf583c980b70cecc184fa8a083a9c3ddb87f905e/nemo/collections/asr/parts/submodules/transducer_decoding/tdt_label_looping.py#L363-L368
114+
std::vector<int64_t> sos_token_data = {blank_id};
115+
auto sos_token = from_blob(
116+
sos_token_data.data(), {1, 1}, ::executorch::aten::ScalarType::Long);
117+
auto decoder_init_result = model.execute(
118+
"decoder_predict",
119+
std::vector<::executorch::runtime::EValue>{sos_token, h, c});
120+
if (!decoder_init_result.ok()) {
121+
ET_LOG(Error, "decoder_predict (SOS) failed");
122+
return hypothesis;
123+
}
124+
auto& init_outputs = decoder_init_result.get();
125+
auto g_init = init_outputs[0].toTensor();
126+
auto new_h_init = init_outputs[1].toTensor();
127+
auto new_c_init = init_outputs[2].toTensor();
128+
std::memcpy(
129+
h_data.data(),
130+
new_h_init.const_data_ptr<float>(),
131+
h_data.size() * sizeof(float));
132+
std::memcpy(
133+
c_data.data(),
134+
new_c_init.const_data_ptr<float>(),
135+
c_data.size() * sizeof(float));
114136

115137
auto g_proj_result = model.execute(
116138
"joint_project_decoder",
117-
std::vector<::executorch::runtime::EValue>{sos_g});
139+
std::vector<::executorch::runtime::EValue>{g_init});
118140
if (!g_proj_result.ok()) {
119141
ET_LOG(Error, "joint_project_decoder failed");
120142
return hypothesis;

0 commit comments

Comments
 (0)