Commit 7fa1aaa
authored
Fix parakeet_runner blank priming (pytorch#16509)
### Summary
`parakeet_runner` is currently missing transcription of the first part
of audio. I believe this is due to a lack of "priming with blanks" that
is done in https://github.com/NVIDIA-NeMo/NeMo.
- SOS is defined as blank:
https://github.com/NVIDIA-NeMo/NeMo/blob/bf583c980b70cecc184fa8a083a9c3ddb87f905e/nemo/collections/asr/parts/submodules/transducer_decoding/tdt_label_looping.py#L250
- Predictor priming with SOS:
https://github.com/NVIDIA-NeMo/NeMo/blob/bf583c980b70cecc184fa8a083a9c3ddb87f905e/nemo/collections/asr/parts/submodules/transducer_decoding/tdt_label_looping.py#L363-L368
Tested with this audio file:
[audio.wav](https://github.com/user-attachments/files/24498568/audio.wav)
Which I would manually transcribe as: "Can you please add to analyze.py
the ability to pass in a dash dash dataset dash path or dash d that will
take the place of dot input if present? Also dot input should be renamed
to dot dataset"
#### ExecuTorch Before
Before Executorch misses the "Can you please add to analyze" and
transcribes as "py the ability to pass in a dash-dataset-path or dash d
that will take the place of dot input if present. Also dot input should
be renamed dot dataset."
```
-> % ./cmake-out/examples/models/parakeet/parakeet_runner --model_path examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte --tokenizer_path /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model --audio_path /Users/matt/Documents/parakeet_test_audio/audio.wav
I tokenizers:regex.cpp:27] Registering override fallback regex
I 00:00:00.000767 executorch:main.cpp:278] Loading model from: examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte
I 00:00:00.001246 executorch:main.cpp:294] Loading audio from: /Users/matt/Documents/parakeet_test_audio/audio.wav
I 00:00:00.002790 executorch:wav_loader.h:98] WAV header detected, getting raw audio data.
I 00:00:00.002793 executorch:wav_loader.h:105] RIFF Header: RIFF
I 00:00:00.002794 executorch:wav_loader.h:106] Chunk Size: 853676
I 00:00:00.002795 executorch:wav_loader.h:113] WAVE Header: WAVE
I 00:00:00.002796 executorch:wav_loader.h:120] Format Header: fmt
I 00:00:00.002797 executorch:wav_loader.h:121] Format Chunk Size: 16
I 00:00:00.002798 executorch:wav_loader.h:122] Audio Format: 3
I 00:00:00.002799 executorch:wav_loader.h:123] Number of Channels: 1
I 00:00:00.002800 executorch:wav_loader.h:124] Sample Rate: 16000
I 00:00:00.002801 executorch:wav_loader.h:125] Byte Rate: 64000
I 00:00:00.002801 executorch:wav_loader.h:126] Block Align: 4
I 00:00:00.002802 executorch:wav_loader.h:127] Bits per Sample: 32
I 00:00:00.002803 executorch:wav_loader.h:132] Subchunk2Size: 853604
I 00:00:00.002824 executorch:wav_loader.h:226] Loaded 213401 audio samples from WAV file: /Users/matt/Documents/parakeet_test_audio/audio.wav
I 00:00:00.002826 executorch:main.cpp:297] Loaded 213401 audio samples
I 00:00:00.002838 executorch:main.cpp:308] Running preprocessor...
I 00:00:00.018590 executorch:cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version
I 00:00:00.018609 executorch:cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version
I 00:00:00.027469 executorch:main.cpp:333] Mel spectrogram shape: [1, 128, 1334], mel_len: 1333
I 00:00:00.027480 executorch:main.cpp:336] Running encoder...
I 00:00:26.206136 executorch:main.cpp:353] Encoder output shape: [1, 1024, 167], len=167
I 00:00:26.206163 executorch:main.cpp:385] Model metadata: vocab_size=8192, blank_id=8192, num_rnn_layers=2, pred_hidden=640, sample_rate=16000
I 00:00:26.206166 executorch:main.cpp:387] Running TDT greedy decode...
I 00:00:29.509575 executorch:main.cpp:397] Decoded 55 tokens
I 00:00:29.509588 executorch:main.cpp:400] Loading tokenizer from: /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model
E tokenizers:hf_tokenizer.cpp:82] Error parsing json file: [json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - invalid literal; last read: '<U+000A><U+000E>'
E tokenizers:tiktoken.cpp:59] invalid tiktoken line:
I 00:00:29.514298 executorch:llm_runner_helper.cpp:77] Loaded Sentencepiece tokenizer
Transcription tokens: py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset.
I 00:00:29.514337 executorch:main.cpp:415] Done!
```
#### NeMo
NeMo thru:
```
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="nvidia/parakeet-tdt-0.6b-v3")
output = asr_model.transcribe(['/Users/matt/Documents/parakeet_test_audio/audio.wav'])
print(output[0].text)
```
outputs "Can you please add to analyze.py the ability to pass in a
dash-dataset-path or dash d that will take the place of dot input if
present. Also dot input should be renamed dot dataset.":
```
-> % python examples/asr/asr_chunked_inference/rnnt/test.py
[NeMo W 2026-01-08 10:20:16 megatron_init:62] Megatron num_microbatches_calculator not found, using Apex version.
W0108 10:20:16.612000 27527 torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
OneLogger: Setting error_handling_strategy to DISABLE_QUIETLY_AND_REPORT_METRIC_ERROR for rank (rank=0) with OneLogger disabled. To override: explicitly set error_handling_strategy parameter.
No exporters were provided. This means that no telemetry data will be collected.
[NeMo I 2026-01-08 10:20:22 mixins:184] Tokenizer SentencePieceTokenizer initialized with 8192 tokens
[NeMo W 2026-01-08 10:20:24 modelPT:188] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
Train config :
use_lhotse: true
skip_missing_manifest_entries: true
input_cfg: null
tarred_audio_filepaths: null
manifest_filepath: null
sample_rate: 16000
shuffle: true
num_workers: 2
pin_memory: true
max_duration: 10.0
min_duration: 1.0
text_field: answer
batch_duration: null
max_tps: null
use_bucketing: true
bucket_duration_bins: null
bucket_batch_size: null
num_buckets: 30
bucket_buffer_size: 20000
shuffle_buffer_size: 10000
[NeMo W 2026-01-08 10:20:24 modelPT:195] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
Validation config :
use_lhotse: true
manifest_filepath: null
sample_rate: 16000
batch_size: 16
shuffle: false
max_duration: 40.0
min_duration: 0.1
num_workers: 2
pin_memory: true
text_field: answer
[NeMo I 2026-01-08 10:20:26 rnnt_models:226] Using RNNT Loss : tdt
Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1}
[NeMo I 2026-01-08 10:20:26 rnnt_models:226] Using RNNT Loss : tdt
Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1}
[NeMo W 2026-01-08 10:20:26 label_looping_base:113] No conditional node support for Cuda.
Cuda graphs with while loops are disabled, decoding speed will be slower
Reason: CUDA is not available
[NeMo I 2026-01-08 10:20:26 rnnt_models:226] Using RNNT Loss : tdt
Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1}
[NeMo W 2026-01-08 10:20:26 label_looping_base:113] No conditional node support for Cuda.
Cuda graphs with while loops are disabled, decoding speed will be slower
Reason: CUDA is not available
[NeMo I 2026-01-08 10:20:29 save_restore_connector:284] Model EncDecRNNTBPEModel was successfully restored from /Users/matt/.cache/huggingface/hub/models--nvidia--parakeet-tdt-0.6b-v3/snapshots/6d590f77001d318fb17a0b5bf7ee329a91b52598/parakeet-tdt-0.6b-v3.nemo.
[NeMo W 2026-01-08 10:20:29 dataloader:760] The following configuration keys are ignored by Lhotse dataloader: use_start_end_token
[NeMo W 2026-01-08 10:20:29 dataloader:498] You are using a non-tarred dataset and requested tokenization during data sampling (pretokenize=True). This will cause the tokenization to happen in the main (GPU) process,possibly impacting the training speed if your tokenizer is very large.If the impact is noticable, set pretokenize=False in dataloader config.(note: that will disable token-per-second filtering and 2D bucketing features)
Transcribing: 1it [00:00, 1.11it/s]
Can you please add to analyze.py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset.
```
#### ExecuTorch After
After ExecuTorch "Can you please add to analyze.py the ability to pass
in a dash-dataset-path or dash d that will take the place of dot input
if present. Also dot input should be renamed dot dataset."
```
-> % ./cmake-out/examples/models/parakeet/parakeet_runner --model_path examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte --tokenizer_path /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model --audio_path /Users/matt/Documents/parakeet_test_audio/audio.wav
I tokenizers:regex.cpp:27] Registering override fallback regex
I 00:00:00.002165 executorch:main.cpp:298] Loading model from: examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte
I 00:00:00.002909 executorch:main.cpp:314] Loading audio from: /Users/matt/Documents/parakeet_test_audio/audio.wav
I 00:00:00.005307 executorch:wav_loader.h:98] WAV header detected, getting raw audio data.
I 00:00:00.005318 executorch:wav_loader.h:105] RIFF Header: RIFF
I 00:00:00.005320 executorch:wav_loader.h:106] Chunk Size: 853676
I 00:00:00.005322 executorch:wav_loader.h:113] WAVE Header: WAVE
I 00:00:00.005324 executorch:wav_loader.h:120] Format Header: fmt
I 00:00:00.005325 executorch:wav_loader.h:121] Format Chunk Size: 16
I 00:00:00.005327 executorch:wav_loader.h:122] Audio Format: 3
I 00:00:00.005329 executorch:wav_loader.h:123] Number of Channels: 1
I 00:00:00.005330 executorch:wav_loader.h:124] Sample Rate: 16000
I 00:00:00.005332 executorch:wav_loader.h:125] Byte Rate: 64000
I 00:00:00.005333 executorch:wav_loader.h:126] Block Align: 4
I 00:00:00.005335 executorch:wav_loader.h:127] Bits per Sample: 32
I 00:00:00.005336 executorch:wav_loader.h:132] Subchunk2Size: 853604
I 00:00:00.005447 executorch:wav_loader.h:226] Loaded 213401 audio samples from WAV file: /Users/matt/Documents/parakeet_test_audio/audio.wav
I 00:00:00.005458 executorch:main.cpp:317] Loaded 213401 audio samples
I 00:00:00.005485 executorch:main.cpp:328] Running preprocessor...
I 00:00:00.025430 executorch:cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version
I 00:00:00.025449 executorch:cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version
I 00:00:00.035054 executorch:main.cpp:353] Mel spectrogram shape: [1, 128, 1334], mel_len: 1333
I 00:00:00.035067 executorch:main.cpp:356] Running encoder...
I 00:00:25.835932 executorch:main.cpp:373] Encoder output shape: [1, 1024, 167], len=167
I 00:00:25.835974 executorch:main.cpp:405] Model metadata: vocab_size=8192, blank_id=8192, num_rnn_layers=2, pred_hidden=640, sample_rate=16000
I 00:00:25.835977 executorch:main.cpp:407] Running TDT greedy decode...
I 00:00:29.269823 executorch:main.cpp:417] Decoded 65 tokens
I 00:00:29.269837 executorch:main.cpp:420] Loading tokenizer from: /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model
E tokenizers:hf_tokenizer.cpp:82] Error parsing json file: [json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - invalid literal; last read: '<U+000A><U+000E>'
E tokenizers:tiktoken.cpp:59] invalid tiktoken line:
I 00:00:29.274029 executorch:llm_runner_helper.cpp:77] Loaded Sentencepiece tokenizer
Transcription tokens: Can you please add to analyze.py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset.
I 00:00:29.274057 executorch:main.cpp:435] Done!
```
### Test plan
Tested manually, see above1 parent 535dcfb commit 7fa1aaa
1 file changed
Lines changed: 29 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
114 | 136 | | |
115 | 137 | | |
116 | 138 | | |
117 | | - | |
| 139 | + | |
118 | 140 | | |
119 | 141 | | |
120 | 142 | | |
| |||
0 commit comments