**Inspecting inference input data**

I'm checking the existing inference input JSON to understand its fields before running OmniASR inference with the newly installed package.


python3 - <<'PY'
import json
from pathlib import Path
path = Path('outputs/bench/maya_hf_v2_100/omniasr_7b_v2_inputs.json')
print(path.exists())
with path.open() as f:
    data = json.load(f)
print(len(data))
print(data[0].keys())
print(data[0])
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
True
1200
dict_keys(['segment_id', 'lang', 'audio_path'])
{'segment_id': 'fleurs__as_in__test__285__1764-5006634a7372640e', 'lang': 'asm_Beng', 'audio_path': 'data/bench/maya_hf_v2_100/audio/asm/fleurs__as_in__test__285__1764.wav'}


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Confirming language mapping coverage**

I'm verifying that the existing language ID mappings cover all 12 dataset languages and align with the omni language codes used in the transcription inputs, ensuring consistent language references.


python3 - <<'PY'
import json
from collections import Counter
from pathlib import Path
path = Path('outputs/bench/maya_hf_v2_100/omniasr_7b_v2_inputs.json')
langs = [row['lang'] for row in json.load(path.open())]
print(Counter(langs))
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Counter({'asm_Beng': 100, 'ben_Beng': 100, 'eng_Latn': 100, 'guj_Gujr': 100, 'hin_Deva': 100, 'kan_Knda': 100, 'mal_Mlym': 100, 'mar_Deva': 100, 'ory_Orya': 100, 'pan_Guru': 100, 'tam_Taml': 100, 'tel_Telu': 100})


Summary: auto


**Choosing model for results generation**

Decided to use the `omniASR_LLM_7B_v2` model for the pipeline since it offers improved accuracy and fits the bench audio length limits better than the "Unlimited" variant. I'll build the results.jsonl accordingly with this model choice noted.


**Planning batching and output schema**

I'll batch audio inputs (around size 2–4) to manage GPU memory for the 7B model and create results.jsonl with consistent fields including segment ID, audio path, and language hints using ISO3 codes. I'll verify evaluation conventions to decide whether to use `language_source=route` or `refs` for language metadata.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# Benchmark Results

Dataset: `data/bench/maya_hf_hin_v2` (Hindi, 200 segments)
Metric: mean WER (lower is better); CER included for context.
Inference: vLLM audio transcription, Whisper skipped.

## How We Ran These Experiments
- Pipeline: preprocess -> MMS-LID -> ASR -> ROVER -> verify. Language routing is forced from the path (`/hin/`), so Voxtral receives `language=hi`.
- Final selection in the pipeline: `final` defaults to `rover.text` unless CTC scoring or explicit router preferences are enabled. We did not enable CTC or router overrides in these runs, so `final` == ROVER output.
- Verification (accept/human) only labels segments; it does not change the transcript.

### ROVER / Ensemble Logic (inside pipeline)
- Implemented in `maya_transcribe/rover.py`:
  - Tokenize each hypothesis.
  - Choose a “centroid” hypothesis (lowest average normalized edit distance to others).
  - Align all hypotheses to the centroid.
  - Vote per token position; ties fall back to the centroid token.
  - Gap insertions are kept only if >=50% support.
- With two hypotheses, ROVER tends to pick the centroid transcript and keeps only gaps supported by both.

### Voxtral + Gemma Offline Ensemble
- We merged `results.jsonl` from Voxtral and Gemma, then ran ROVER on their `final` texts only.
- No verification gating or CTC reranking in that offline ensemble; the output is purely ROVER.

### Prompts / Decoding
- Voxtral transcription mode: vLLM’s Voxtral `TranscriptionRequest` path (audio-only, no extra text prompt). Language is set to `hi` from path routing.
- Prompt.md override: when supplied, Voxtral uses a chat prompt with text + audio; Gemma uses a custom `<start_of_turn>` prompt with `<audio_soft_token>`. `<language>` is replaced with the model-supported language name (Hindi). Output is post-processed to strip the `Output 1: Transcript` header line.
- Gemma-3n transcription mode: vLLM’s built-in Gemma3n prompt template:
  - `"Transcribe this audio into Hindi: <audio_soft_token><end_of_turn>\n<start_of_turn>model\n"`
- Voxtral proofread prompt (audio + pass1 transcript, chat mode):
  - `"Correct transcription errors using the audio. Keep the same language/script. Do not add new content. Output only the corrected transcript.\n\nTranscript:\n{transcript}"`
- Decoding: `temperature=0.0`, `top_p=1.0`. For Gemma-3n we also add stop token `<end_of_turn>` and cap output length to 12 tokens/sec (min 32, max 1024).

## Voxtral
- Voxtral-Small-24B-2507: WER 14.87%, CER 5.49% (best so far).
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096`
- Voxtral-Small-24B-2507 (2-pass proofread, audio + pass1 transcript via chat): final WER 16.62%, CER 7.32%. Raw pass stayed competitive, proofread pass degraded.
  - Final eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread/eval.json`
  - Raw eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread/eval_raw.json` (WER 15.03%, CER 5.56%)
  - Proofread eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread/eval_proofread.json` (WER 24.58%, CER 17.59%)
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-proofread`
- Voxtral-Mini-3B-2507: WER 19.32%, CER 7.85%.
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm --skip-whisper --vllm-asr-model mistralai/Voxtral-Mini-3B-2507 --vllm-gpu-memory-utilization 0.15 --vllm-max-model-len 4096`

## Gemma-3n
- Gemma-3n-E2B-it (initial run): WER 137.6%, CER 105.2%. This was dominated by runaway repetitions (hundreds/thousands of tokens) when the model did not emit a stop token.
  - Eval: `outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json`
- Gemma-3n-E2B-it (tuned stop token + dynamic max_tokens): WER 20.63%, CER 10.05%. The runaway repetition is capped, but accuracy still lags Voxtral.
  - Eval: `outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_tuned/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_tuned --skip-whisper --vllm-asr-model google/gemma-3n-E2B-it --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096`

## Ensembles
- Voxtral-Small + Gemma-3n (ROVER on both finals): WER 20.81%, CER 11.04% (worse than Voxtral alone).
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble/eval.json`

## Prompt.md Experiments (custom instruction prompt)
Prompt source: `Prompt.md` (strict verbatim transcription, `Output 1: Transcript` format). All runs below used the same Hindi bench (`data/bench/maya_hf_hin_v2`).

### Voxtral (Prompt.md)
- Voxtral-Small-24B-2507: WER 24.52%, CER 17.03% (worse than baseline).
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_prompt/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_prompt --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-transcribe-prompt-file Prompt.md`
- Voxtral-Small-24B-2507 (2-pass proofread + Prompt.md on pass1): final WER 28.50%, CER 21.95% (worse than baseline).
  - Final eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval.json`
  - Raw eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval_raw.json` (WER 25.65%, CER 18.16%)
  - Proofread eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval_proofread.json` (WER 31.84%, CER 27.36%)
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-proofread --vllm-transcribe-prompt-file Prompt.md`
- Voxtral-Mini-3B-2507: WER 98.49%, CER 99.88% (prompt caused severe degradation).
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm_prompt/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm_prompt --skip-whisper --vllm-asr-model mistralai/Voxtral-Mini-3B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-transcribe-prompt-file Prompt.md`

### Gemma-3n (Prompt.md)
- Gemma-3n-E2B-it: WER 24.26%, CER 13.90% (worse than tuned Gemma with the built-in prompt).
  - Eval: `outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_prompt/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_prompt --skip-whisper --vllm-asr-model google/gemma-3n-E2B-it --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-transcribe-prompt-file Prompt.md`

### Ensemble (Prompt.md)
- Voxtral-Small + Gemma-3n (ROVER on both finals): WER 26.82%, CER 18.12% (worse than Voxtral prompt alone).
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt/eval.json`
  - Results: `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt/results.jsonl`

## Notes
- Gemma-3n now uses a per-audio max_tokens cap (12 tokens/sec, max 1024) and a Gemma stop token (`<end_of_turn>`) to prevent runaway decoding.
- Best WER on this Hindi bench is currently Voxtral-Small (14.87%).

## Multilingual 100-sample Runs (Transcription Dumps)
Bench subset: `data/bench/maya_hf_v2_100` (12 languages x 100 samples). These runs are for JSON dumps first; eval.json was generated but not analyzed yet.

- IndicConformer 600M multilingual (HF): `outputs/bench/maya_hf_v2_100/indicconformer_600m_hf/results.jsonl`
  - Note: English segments failed with `KeyError:'joint_post_net_en'` (100 error rows).
- Whisper large-v3: `outputs/bench/maya_hf_v2_100/whisper_large_v3/results.jsonl`
- Voxtral-Small-24B-2507: `outputs/bench/maya_hf_v2_100/voxtral_small_vllm/results.jsonl`
  - Note: Voxtral only natively supports a small language set (hi/en/...) so non-supported languages likely degrade.
- Gemma-3n-E2B-it: `outputs/bench/maya_hf_v2_100/gemma3n_e2b_vllm/results.jsonl`

### Per-language WER (mean, %)
Source: `eval.json` from each run. Note: IndicConformer HF failed on English (`n/a` below), so its overall WER is computed on 1100 segments.

| Lang | IndicConformer-600M (HF) | Whisper large-v3 | Voxtral-Small | Gemma-3n-E2B |
| --- | --- | --- | --- | --- |
| asm | 19.9 | 95.9 | 107.9 | 78.0 |
| ben | 11.2 | 64.2 | 37.4 | 34.8 |
| eng | n/a | 4.5 | 3.8 | 9.2 |
| guj | 11.7 | 49.3 | 49.8 | 87.3 |
| hin | 9.2 | 30.9 | 14.9 | 18.8 |
| kan | 14.3 | 71.7 | 49.0 | 45.8 |
| mal | 29.9 | 122.5 | 62.6 | 65.1 |
| mar | 14.6 | 79.7 | 39.9 | 38.8 |
| ory | 16.3 | 115.5 | 130.3 | 120.0 |
| pan | 12.6 | 64.7 | 34.2 | 135.0 |
| tam | 20.1 | 50.9 | 52.7 | 78.8 |
| tel | 24.4 | 79.0 | 47.5 | 51.2 |
| overall | 16.7 | 69.1 | 52.5 | 63.6 |


**Confirming language_source and evaluation setup**

I'm verifying that existing eval.json uses `language_source=route` by default and that WER keys match language codes from refs. For new results lacking LID, I'll set `route.language_hint` from refs' `lang` to maintain consistency and ensure evaluation aligns with prior runs using default oracle true. Next, I'll run evaluation accordingly.


**Planning OmniASR and seamless M4T integration**

I'm preparing to run OmniASR with batch inference and error logging, adapting results formatting to include ASR outputs and errors while skipping errored entries in evaluation. For seamless M4T, I'll verify the correct model and processor imports, especially distinguishing between v1 and v2 classes, before implementing transcription with language targets.


source .venv_omni/bin/activate && python - <<'PY'
try:
    from transformers import SeamlessM4Tv2Model, AutoProcessor
    print('have SeamlessM4Tv2Model')
except Exception as e:
    print('no SeamlessM4Tv2Model', e)
try:
    from transformers import SeamlessM4TModel
    print('have SeamlessM4TModel')
except Exception as e:
    print('no SeamlessM4TModel', e)
PY


Output:
Exit code: 0
Wall time: 4.1 seconds
Output:
have SeamlessM4Tv2Model
have SeamlessM4TModel


**Confirming target language codes for model generation**

I'm verifying the supported target language codes in the processor's tokenizer to map iso3 language codes correctly for use with the model's generate function and task parameters.


source .venv_omni/bin/activate && python - <<'PY'
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained('facebook/seamless-m4t-v2-large')
# tokenizers for M4T uses lang_code_to_id
lang_map = getattr(processor.tokenizer, 'lang_code_to_id', {})
print('langs', len(lang_map))
# show relevant ones
wanted = ['eng','hin','tam','tel','kan','mal','guj','pan','mar','ben','asm','ory','ori','odi']
for w in wanted:
    if w in lang_map:
        print(w, 'OK')
PY


Output:
Exit code: 1
Wall time: 7.4 seconds
Output:
Traceback (most recent call last):
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/models/seamless_m4t/tokenization_seamless_m4t.py", line 147, in __init__
    self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False))
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/models/seamless_m4t/tokenization_seamless_m4t.py", line 429, in get_spm_processor
    model_pb2 = import_protobuf(f"The new behaviour of {self.__class__.__name__} (with `self.legacy = False`)")
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py", line 39, in import_protobuf
    from sentencepiece import sentencepiece_model_pb2
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/sentencepiece/sentencepiece_model_pb2.py", line 5, in <module>
    from google.protobuf.internal import builder as _builder
ModuleNotFoundError: No module named 'google'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/models/auto/processing_auto.py", line 396, in from_pretrained
    return processor_class.from_pretrained(
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/processing_utils.py", line 1394, in from_pretrained
    args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/processing_utils.py", line 1453, in _get_arguments_from_pretrained
    args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
    return cls._from_pretrained(
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/models/seamless_m4t/tokenization_seamless_m4t_fast.py", line 353, in _from_pretrained
    tokenizer = super()._from_pretrained(
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2151, in _from_pretrained
    slow_tokenizer = (cls.slow_tokenizer_class)._from_pretrained(
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2360, in _from_pretrained
    except import_protobuf_decode_error():
  File "/home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 88, in import_protobuf_decode_error
    raise ImportError(PROTOBUF_IMPORT_ERROR.format(error_message))
ImportError:
 requires the protobuf library but it was not found in your environment. Check out the instructions on the
installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones
that match your environment. Please note that you may need to restart your runtime after installation.


**Adding protobuf to virtual environment**


source .venv_omni/bin/activate && pip install protobuf


Output:
Exit code: 0
Wall time: 2.2 seconds
Output:
Looking in indexes: https://mcache-dsm.massedcompute.com/simple, https://pypi.org/simple
Collecting protobuf
  Downloading protobuf-6.33.4-cp39-abi3-manylinux2014_x86_64.whl.metadata (593 bytes)
Downloading protobuf-6.33.4-cp39-abi3-manylinux2014_x86_64.whl (323 kB)
Installing collected packages: protobuf
Successfully installed protobuf-6.33.4


**Rerunning to see language codes**


source .venv_omni/bin/activate && python - <<'PY'
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained('facebook/seamless-m4t-v2-large')
lang_map = getattr(processor.tokenizer, 'lang_code_to_id', {})
print('langs', len(lang_map))
wanted = ['eng','hin','tam','tel','kan','mal','guj','pan','mar','ben','asm','ory','ori','odi']
for w in wanted:
    if w in lang_map:
        print(w, 'OK')
    else:
        print(w, 'missing')
PY


Output:
Exit code: 124
Wall time: 10.1 seconds
Output:
command timed out after 10103 milliseconds
langs 0
eng missing
hin missing
tam missing
tel missing
kan missing
mal missing
guj missing
pan missing
mar missing
ben missing
asm missing
ory missing
ori missing
odi missing


**Extending command timeout**

The prior command likely timed out before fully loading tokenized languages, causing inconsistent partial output; I'll rerun it with a longer timeout to confirm.


source .venv_omni/bin/activate && python - <<'PY'
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained('facebook/seamless-m4t-v2-large')
lang_map = getattr(processor.tokenizer, 'lang_code_to_id', {})
print('langs', len(lang_map))
for k in sorted(list(lang_map.keys()))[:20]:
    print(k)
PY


Output:
Exit code: 0
Wall time: 10.2 seconds
Output:
langs 0


**Inspecting tokenizer attributes**

I'm checking the processor's tokenizer attributes to verify if the language code mapping is missing or unset, which could explain why `lang_code_to_id` is empty for M4T.


source .venv_omni/bin/activate && python - <<'PY'
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained('facebook/seamless-m4t-v2-large')
print(processor)
print('tokenizer class', processor.tokenizer.__class__)
print('has get_lang_id', hasattr(processor.tokenizer, 'get_lang_id'))
print('has lang_code_to_id', hasattr(processor.tokenizer, 'lang_code_to_id'))
print('has language_code_to_id', hasattr(processor.tokenizer, 'language_code_to_id'))
print('has id_to_lang_code', hasattr(processor.tokenizer, 'id_to_lang_code'))
# try to access special_tokens_map
print('special tokens', list(processor.tokenizer.special_tokens_map.keys()))
PY


Output:
Exit code: 0
Wall time: 11.6 seconds
Output:
SeamlessM4TProcessor:
- feature_extractor: SeamlessM4TFeatureExtractor {
  "feature_extractor_type": "SeamlessM4TFeatureExtractor",
  "feature_size": 80,
  "language_code": [
    "__afr__",
    "__amh__",
    "__arb__",
    "__ary__",
    "__arz__",
    "__asm__",
    "__azj__",
    "__bel__",
    "__ben__",
    "__bos__",
    "__bul__",
    "__cat__",
    "__ceb__",
    "__ces__",
    "__ckb__",
    "__cmn__",
    "__cmn_Hant__",
    "__cym__",
    "__dan__",
    "__deu__",
    "__ell__",
    "__eng__",
    "__est__",
    "__eus__",
    "__fin__",
    "__fra__",
    "__fuv__",
    "__gaz__",
    "__gle__",
    "__glg__",
    "__guj__",
    "__heb__",
    "__hin__",
    "__hrv__",
    "__hun__",
    "__hye__",
    "__ibo__",
    "__ind__",
    "__isl__",
    "__ita__",
    "__jav__",
    "__jpn__",
    "__kan__",
    "__kat__",
    "__kaz__",
    "__khk__",
    "__khm__",
    "__kir__",
    "__kor__",
    "__lao__",
    "__lit__",
    "__lug__",
    "__luo__",
    "__lvs__",
    "__mai__",
    "__mal__",
    "__mar__",
    "__mkd__",
    "__mlt__",
    "__mni__",
    "__mya__",
    "__nld__",
    "__nno__",
    "__nob__",
    "__npi__",
    "__nya__",
    "__ory__",
    "__pan__",
    "__pbt__",
    "__pes__",
    "__pol__",
    "__por__",
    "__ron__",
    "__rus__",
    "__sat__",
    "__slk__",
    "__slv__",
    "__sna__",
    "__snd__",
    "__som__",
    "__spa__",
    "__srp__",
    "__swe__",
    "__swh__",
    "__tam__",
    "__tel__",
    "__tgk__",
    "__tgl__",
    "__tha__",
    "__tur__",
    "__ukr__",
    "__urd__",
    "__uzn__",
    "__vie__",
    "__yor__",
    "__yue__",
    "__zlm__",
    "__zul__"
  ],
  "num_mel_bins": 80,
  "padding_side": "right",
  "padding_value": 0.0,
  "processor_class": "SeamlessM4TProcessor",
  "return_attention_mask": true,
  "sampling_rate": 16000,
  "stride": 2
}

- tokenizer: SeamlessM4TTokenizerFast(name_or_path='facebook/seamless-m4t-v2-large', vocab_size=256001, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'additional_special_tokens': ['__afr__', '__amh__', '__arb__', '__ary__', '__arz__', '__asm__', '__azj__', '__bel__', '__ben__', '__bos__', '__bul__', '__cat__', '__ceb__', '__ces__', '__ckb__', '__cmn__', '__cmn_Hant__', '__cym__', '__dan__', '__deu__', '__ell__', '__eng__', '__est__', '__eus__', '__fin__', '__fra__', '__fuv__', '__gaz__', '__gle__', '__glg__', '__guj__', '__heb__', '__hin__', '__hrv__', '__hun__', '__hye__', '__ibo__', '__ind__', '__isl__', '__ita__', '__jav__', '__jpn__', '__kan__', '__kat__', '__kaz__', '__khk__', '__khm__', '__kir__', '__kor__', '__lao__', '__lit__', '__lug__', '__luo__', '__lvs__', '__mai__', '__mal__', '__mar__', '__mkd__', '__mlt__', '__mni__', '__mya__', '__nld__', '__nno__', '__nob__', '__npi__', '__nya__', '__ory__', '__pan__', '__pbt__', '__pes__', '__pol__', '__por__', '__ron__', '__rus__', '__sat__', '__slk__', '__slv__', '__sna__', '__snd__', '__som__', '__spa__', '__srp__', '__swe__', '__swh__', '__tam__', '__tel__', '__tgk__', '__tgl__', '__tha__', '__tur__', '__ukr__', '__urd__', '__uzn__', '__vie__', '__yor__', '__yue__', '__zlm__', '__zul__']}, clean_up_tokenization_spaces=True, added_tokens_decoder={
	0: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	1: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	2: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	3: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256001: AddedToken("__afr__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256002: AddedToken("__amh__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256003: AddedToken("__arb__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256004: AddedToken("__ary__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256005: AddedToken("__arz__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256006: AddedToken("__asm__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256007: AddedToken("__azj__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256008: AddedToken("__bel__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256009: AddedToken("__ben__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256010: AddedToken("__bos__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256011: AddedToken("__bul__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256012: AddedToken("__cat__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256013: AddedToken("__ceb__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256014: AddedToken("__ces__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256015: AddedToken("__ckb__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256016: AddedToken("__cmn__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256017: AddedToken("__cmn_Hant__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256018: AddedToken("__cym__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256019: AddedToken("__dan__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256020: AddedToken("__deu__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256021: AddedToken("__ell__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256022: AddedToken("__eng__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256023: AddedToken("__est__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256024: AddedToken("__eus__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256025: AddedToken("__fin__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256026: AddedToken("__fra__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256027: AddedToken("__fuv__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256028: AddedToken("__gaz__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256029: AddedToken("__gle__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256030: AddedToken("__glg__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256031: AddedToken("__guj__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256032: AddedToken("__heb__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256033: AddedToken("__hin__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256034: AddedToken("__hrv__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256035: AddedToken("__hun__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256036: AddedToken("__hye__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256037: AddedToken("__ibo__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256038: AddedToken("__ind__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256039: AddedToken("__isl__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256040: AddedToken("__ita__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256041: AddedToken("__jav__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256042: AddedToken("__jpn__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256043: AddedToken("__kan__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256044: AddedToken("__kat__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256045: AddedToken("__kaz__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256046: AddedToken("__khk__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256047: AddedToken("__khm__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256048: AddedToken("__kir__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256049: AddedToken("__kor__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256050: AddedToken("__lao__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256051: AddedToken("__lit__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256052: AddedToken("__lug__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256053: AddedToken("__luo__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256054: AddedToken("__lvs__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256055: AddedToken("__mai__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256056: AddedToken("__mal__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256057: AddedToken("__mar__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256058: AddedToken("__mkd__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256059: AddedToken("__mlt__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256060: AddedToken("__mni__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256061: AddedToken("__mya__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256062: AddedToken("__nld__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256063: AddedToken("__nno__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256064: AddedToken("__nob__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256065: AddedToken("__npi__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256066: AddedToken("__nya__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256067: AddedToken("__ory__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256068: AddedToken("__pan__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256069: AddedToken("__pbt__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256070: AddedToken("__pes__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256071: AddedToken("__pol__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256072: AddedToken("__por__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256073: AddedToken("__ron__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256074: AddedToken("__rus__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256075: AddedToken("__sat__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256076: AddedToken("__slk__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256077: AddedToken("__slv__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256078: AddedToken("__sna__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256079: AddedToken("__snd__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256080: AddedToken("__som__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256081: AddedToken("__spa__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256082: AddedToken("__srp__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256083: AddedToken("__swe__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256084: AddedToken("__swh__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256085: AddedToken("__tam__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256086: AddedToken("__tel__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256087: AddedToken("__tgk__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256088: AddedToken("__tgl__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256089: AddedToken("__tha__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256090: AddedToken("__tur__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256091: AddedToken("__ukr__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256092: AddedToken("__urd__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256093: AddedToken("__uzn__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256094: AddedToken("__vie__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256095: AddedToken("__yor__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256096: AddedToken("__yue__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256097: AddedToken("__zlm__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	256098: AddedToken("__zul__", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
)

{
  "processor_class": "SeamlessM4TProcessor"
}

tokenizer class <class 'transformers.models.seamless_m4t.tokenization_seamless_m4t_fast.SeamlessM4TTokenizerFast'>
has get_lang_id False
has lang_code_to_id False
has language_code_to_id False
has id_to_lang_code False
special tokens ['bos_token', 'eos_token', 'unk_token', 'sep_token', 'pad_token', 'cls_token', 'additional_special_tokens']


**Clarifying language code format**

The processor's feature extractor uses tokens like "__hin__", but the target language code for Seamless M4T should be the plain code like "hin" or "eng"; under the hood, these get converted to tokens with underscores. Next, I'll verify how language IDs are derived in the processor code to confirm correct usage.


python3 - <<'PY'
import inspect
from transformers import SeamlessM4TProcessor
import transformers
print('transformers', transformers.__version__)
import transformers.models.seamless_m4t.processing_seamless_m4t as proc
print(inspect.getsource(proc.SeamlessM4TProcessor.__call__))
PY


Output:
Exit code: 1
Wall time: 3 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 2317, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 2347, in _get_module
    raise e
  File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 2345, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/seamless_m4t/processing_seamless_m4t.py", line 22, in <module>
    from ...processing_utils import ProcessingKwargs, ProcessorMixin, TextKwargs, Unpack
  File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/processing_utils.py", line 37, in <module>
    from .image_utils import ChannelDimension, ImageInput, is_vision_available
  File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/image_utils.py", line 55, in <module>
    from torchvision.transforms import InterpolationMode
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torchvision/__init__.py", line 10, in <module>
    from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils  # usort:skip
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torchvision/_meta_registrations.py", line 164, in <module>
    def meta_nms(dets, scores, iou_threshold):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/library.py", line 1069, in register
    use_lib._register_fake(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/library.py", line 219, in _register_fake
    handle = entry.fake_impl.register(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/_library/fake_impl.py", line 50, in register
    if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
RuntimeError: operator torchvision::nms does not exist

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 2320, in __getattr__
    raise ModuleNotFoundError(
ModuleNotFoundError: Could not import module 'SeamlessM4TProcessor'. Are this object's requirements defined correctly?


Summary: auto


**Switching to virtual environment for introspection**


source .venv_omni/bin/activate && python - <<'PY'
import inspect
from transformers import SeamlessM4TProcessor
from transformers.models.seamless_m4t import processing_seamless_m4t
print('processor class', SeamlessM4TProcessor)
print(inspect.getsource(processing_seamless_m4t.SeamlessM4TProcessor.__call__))
PY


Output:
Exit code: 0
Wall time: 3.7 seconds
Output:
processor class <class 'transformers.models.seamless_m4t.processing_seamless_m4t.SeamlessM4TProcessor'>
    @deprecate_kwarg("audios", version="v4.59.0", new_name="audio")
    def __call__(
        self,
        text: Optional[Union[TextInput, PreTokenizedInput, list[TextInput], list[PreTokenizedInput]]] = None,
        audios: Optional[AudioInput] = None,
        audio: Optional[AudioInput] = None,
        **kwargs: Unpack[ProcessingKwargs],
    ):
        """
        Main method to prepare for the model one or several sequences(s) and audio(s). This method forwards the `text`
        and `kwargs` arguments to SeamlessM4TTokenizerFast's [`~SeamlessM4TTokenizerFast.__call__`] if `text` is not
        `None` to encode the text. To prepare the audio(s), this method forwards the `audios` and `kwargs` arguments to
        SeamlessM4TFeatureExtractor's [`~SeamlessM4TFeatureExtractor.__call__`] if `audios` is not `None`. Please refer
        to the docstring of the above two methods for more information.

        Args:
            text (`str`, `list[str]`, `list[list[str]]`):
                The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
                (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
                `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
            audios (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`):
                The audio or batch of audios to be prepared. Each audio can be NumPy array or PyTorch tensor. In case
                of a NumPy array/PyTorch tensor, each audio should be of shape (C, T), where C is a number of channels,
                and T the sample length of the audio.
        Returns:
            [`BatchEncoding`]: A [`BatchEncoding`] with the following fields:

            - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`.
            - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when
              `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not
              `None`).
            - **input_features** -- Audio input features to be fed to a model. Returned when `audios` is not `None`.
        """
        if text is not None and audios is not None:
            raise ValueError(
                "Text and audios are mututally exclusive when passed to `SeamlessM4T`. Specify one or another."
            )
        if audio is None and audios is not None:
            logger.warning(
                "Passing `audios` as keyword argument is deprecated and will be removed in v4.63, please pass `audio` instead."
            )
            audio = audios
        return super().__call__(text=text, audio=audio, **kwargs)


**Inspecting model generate parameters**


source .venv_omni/bin/activate && python - <<'PY'
import inspect
from transformers import SeamlessM4Tv2Model
from transformers.models.seamless_m4t import modeling_seamless_m4t
print(inspect.signature(SeamlessM4Tv2Model.generate))
print('\n'.join(inspect.getsource(SeamlessM4Tv2Model.generate).splitlines()[:40]))
PY


Output:
Exit code: 0
Wall time: 3.8 seconds
Output:
(self, input_ids: Optional[torch.Tensor] = None, input_features: Optional[torch.Tensor] = None, return_intermediate_token_ids: Optional[bool] = None, tgt_lang: Optional[str] = None, speaker_id: Optional[int] = 0, generate_speech: Optional[bool] = True, **kwargs) -> Union[torch.Tensor, transformers.models.seamless_m4t_v2.modeling_seamless_m4t_v2.SeamlessM4Tv2GenerationOutput]
    @torch.no_grad()
    def generate(
        self,
        input_ids: Optional[torch.Tensor] = None,
        input_features: Optional[torch.Tensor] = None,
        return_intermediate_token_ids: Optional[bool] = None,
        tgt_lang: Optional[str] = None,
        speaker_id: Optional[int] = 0,
        generate_speech: Optional[bool] = True,
        **kwargs,
    ) -> Union[torch.Tensor, SeamlessM4Tv2GenerationOutput]:
        """
        Generates translated token ids and/or translated audio waveforms.

        <Tip>

        This method successively calls the `.generate` function of two different sub-models. You can specify keyword
        arguments at two different levels: general arguments that will be passed to both models, or prefixed arguments
        that will be passed to one of them.

        For example, calling `.generate(input_ids=input_ids, num_beams=4, speech_do_sample=True)` will successively
        perform beam-search decoding on the text model, and multinomial beam-search sampling on the speech model.

        For an overview of generation strategies and code examples, check out the [following
        guide](./generation_strategies).

        </Tip>

        Args:
            input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
                Indices of input sequence tokens in the vocabulary.

                Indices can be obtained using [`SeamlessM4TTokenizer`] or [`SeamlessM4TProcessor`]. See
                [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details.

                [What are input IDs?](../glossary#input-ids)
            input_features (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_banks)`, *optional*):
                Input audio features. This should be returned by the [`SeamlessM4TFeatureExtractor`] class or the
                [`SeamlessM4TProcessor`] class. See [`SeamlessM4TFeatureExtractor.__call__`] for details.


**Clarifying generation parameters**

I'm determining how to set `tgt_lang` and `generate_speech` for transcription versus translation in Seamless M4T, aiming to disable speech generation (`generate_speech=False`) for transcription and use `tgt_lang` to control output language tokens accordingly. Next, I'll inspect model code for `set_tgt_lang` and related usage.


source .venv_omni/bin/activate && python - <<'PY'
import inspect
from transformers.models.seamless_m4t_v2 import modeling_seamless_m4t_v2
src = inspect.getsource(modeling_seamless_m4t_v2.SeamlessM4Tv2Model.generate)
print('tgt_lang' in src)
# find token conversion
for line in src.splitlines():
    if 'tgt_lang' in line:
        print(line)
PY


Output:
Exit code: 0
Wall time: 3.7 seconds
Output:
True
        tgt_lang: Optional[str] = None,
            tgt_lang (`str`, *optional*):
        if generate_speech and tgt_lang is None:
            raise ValueError("You must specify a `tgt_lang` to generate translated speech.")
        if tgt_lang is not None:
            tgt_lang = tgt_lang.replace("__", "")
                elif tgt_lang not in lang_code_to_id:
                        f"""`tgt_lang={tgt_lang}` is not supported by this model.
                    Please specify a `tgt_lang` in {",".join(lang_code_to_id.keys())}. Note that SeamlessM4Tv2 supports
        # overwrite text_decoder_input_ids if tgt_lang is passed. The latter gets priority over decoder_input_ids.
        if tgt_lang is not None:
            # tgt_lang gets priority over decoder input ids
            text_tgt_lang_id = self.generation_config.text_decoder_lang_to_code_id.get(tgt_lang)
            text_decoder_input_ids = torch.tensor([[text_tgt_lang_id]] * batch_size, device=self.device)
        vocoder_tgt_lang_id = self.generation_config.vocoder_lang_code_to_id.get(tgt_lang)
        vocoder_tgt_lang_id = torch.tensor([[vocoder_tgt_lang_id]] * len(unit_ids), device=self.device)
            input_ids=unit_ids, speaker_id=speaker_id, lang_id=vocoder_tgt_lang_id


**Clarifying language codes and audio input format**

The target language code should be like "eng" without underscores, matching entries in the model's text_decoder_lang_to_code_id map. Audio input likely works as a 1D float32 array at 16kHz sample rate without manual reshaping, but I'll confirm with the processor output shape.


**Planning batch audio processing and scripting**

I'll process audio in batches using the processor with padding and return tensors for efficiency, passing both input features and attention masks to generate with generate_speech=False. Next, I'll create reproducible scripts for OmniASR and Seamless M4T runs and inspect the evaluation functions to correctly build results.jsonl and output keys.


grep -n "def extract_all_candidates" -n maya_transcribe/eval.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
249:def extract_all_candidates(result_row: dict[str, Any]) -> dict[str, str]:


sed -n '240,340p' maya_transcribe/eval.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        if isinstance(asr, dict):
            entry = asr.get(key) or {}
            if isinstance(entry, dict) and isinstance(entry.get("text"), str):
                return entry["text"]
        return None

    return None

def extract_all_candidates(result_row: dict[str, Any]) -> dict[str, str]:
    """
    Collect all text candidates available in a pipeline result row.
    Keys are stable and namespaced (final/rover/asr:<key>).
    """
    candidates: dict[str, str] = {}

    final = extract_hypothesis_text(result_row, "final")
    if final is not None:
        candidates["final"] = final

    rover = extract_hypothesis_text(result_row, "rover")
    if rover is not None:
        candidates["rover"] = rover

    asr = result_row.get("asr") or {}
    if isinstance(asr, dict):
        for key, entry in asr.items():
            if not isinstance(key, str):
                continue
            if not isinstance(entry, dict):
                continue
            text = entry.get("text")
            if isinstance(text, str) and text.strip():
                candidates[f"asr:{key}"] = text

    # Also include normalized hypotheses if present (useful for debugging).
    normalized = result_row.get("normalized") or {}
    if isinstance(normalized, dict):
        for key, text in normalized.items():
            if not isinstance(key, str) or not isinstance(text, str):
                continue
            if text.strip():
                candidates[f"norm:{key}"] = text

    return candidates

def duration_bucket(duration_sec: float | None) -> str | None:
    if duration_sec is None:
        return None
    if duration_sec < 3.0:
        return "0-3s"
    if duration_sec < 10.0:
        return "3-10s"
    if duration_sec < 30.0:
        return "10-30s"
    if duration_sec < 60.0:
        return "30-60s"
    return "60s+"

def _percentile(xs: list[float], q: float) -> float | None:
    if not xs:
        return None
    xs_sorted = sorted(xs)
    idx = int(round((len(xs_sorted) - 1) * q))
    idx = max(0, min(len(xs_sorted) - 1, idx))
    return float(xs_sorted[idx])


**Designing minimal ASR result format**

Planning a minimal JSON structure for ASR outputs to support oracle evaluation with `final.text` matching the transcript and `asr` entries for model and text metadata. Optional fields include `asr_errors` and minimal `config` info for traceability. Next is scripting omni ASR evaluation with references loaded and pipeline instantiated, potentially using CUDA.


**Designing robust batch transcription scripts**

Planning GPU-based batch transcription with fallback to per-sample retries capturing errors per sample; batching by language for M4T due to single-language batch constraints. Implementing audio loading with stereo-to-mono and resampling checks, and structured output rows including errors and config info. Preparing scripts for OmniASR and seamless M4T with mapping of ISO3 codes to model language tokens, plus plans for WER evaluation and results table updates.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 140
drwxrwxr-x 12 ubuntu ubuntu  4096 Jan 15 21:34 .
drwxr-x--- 42 ubuntu ubuntu  4096 Jan 15 22:18 ..
-rw-rw-r--  1 ubuntu ubuntu 41725 Jan 13 04:08 Agents.md
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 14 06:13 .claude
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 15 18:49 configs
drwxrwxr-x  4 ubuntu ubuntu  4096 Jan 13 07:15 data
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 13 03:12 docs
-rw-rw-r--  1 ubuntu ubuntu  3704 Jan 13 05:07 .env
drwxrwxr-x  3 ubuntu ubuntu  4096 Jan 15 21:27 external
-rw-rw-r--  1 ubuntu ubuntu   144 Jan 13 04:16 .gitignore
-rw-rw-r--  1 ubuntu ubuntu  9299 Jan 13 04:09 master.md
drwxrwxr-x  3 ubuntu ubuntu  4096 Jan 14 01:03 maya_transcribe
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 13 04:29 maya_transcribe.egg-info
drwxrwxr-x 12 ubuntu ubuntu  4096 Jan 13 07:31 outputs
-rw-rw-r--  1 ubuntu ubuntu  1090 Jan 15 04:44 Prompt.md
-rw-rw-r--  1 ubuntu ubuntu  1277 Jan 14 01:03 pyproject.toml
-rw-rw-r--  1 ubuntu ubuntu  1280 Jan 13 05:39 README.md
-rw-rw-r--  1 ubuntu ubuntu  9529 Jan 15 21:04 results.md
-rw-rw-r--  1 ubuntu ubuntu  2627 Jan 13 02:35 task.md
drwxrwxr-x  6 ubuntu ubuntu  4096 Jan 15 21:35 .venv_omni
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 13 06:26 .vscode


**Creating scripts directory and helper scripts**

Planning to add a new `scripts/` folder with two ASCII-only script files for different model runs, implementing a chunked helper, and handling JSONL writing manually to avoid extra dependencies.


**Drafting detailed ASR and M4T scripts**

Developed comprehensive plans for two scripts: one using OmniASRInferencePipeline with robust batch error handling and language inference, another loading audio with optional resampling for SeamlessM4Tv2Model, grouping by language and managing device computation. Next step, implement these scripts precisely.


source .venv_omni/bin/activate && python - <<'PY'
try:
    import torchaudio
    print('torchaudio', torchaudio.__version__)
except Exception as e:
    print('torchaudio missing', e)
PY


Output:
Exit code: 0
Wall time: 1.8 seconds
Output:
torchaudio 2.8.0+cu128


**Implementing efficient audio resampling**

Torchaudio will be used for resampling with a cached resampler per input sample rate to reduce overhead; audio is converted to float32 before processing. Batch processing decodes texts per segment, associating results and errors by segment ID while including language hints and config data.


**Preparing evaluation scripts with environment setup**

Planning to add new evaluation scripts under `scripts/` while ensuring they run inside a dedicated virtual environment for dependencies; will create the `scripts/` directory first if needed before adding the new files via patch.


Output:
Exit code: 0
Wall time: 0 seconds
Output:


Summary: auto


**Creating file with patch**


Input:
*** Begin Patch
*** Add File: scripts/run_omniasr_7b_v2.py
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Iterable, Iterator
+
+from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline
+
+MODEL_CARD = "omniASR_LLM_7B_v2"
+INPUTS_PATH = Path("outputs/bench/maya_hf_v2_100/omniasr_7b_v2_inputs.json")
+REFS_PATH = Path("data/bench/maya_hf_v2_100/refs.jsonl")
+OUT_DIR = Path("outputs/bench/maya_hf_v2_100/omniasr_llm_7b_v2")
+OUT_PATH = OUT_DIR / "results.jsonl"
+BATCH_SIZE = 4
+
+
+def chunked(items: list[dict[str, str]], size: int) -> Iterator[list[dict[str, str]]]:
+    for idx in range(0, len(items), size):
+        yield items[idx : idx + size]
+
+
+def load_refs_lang(path: Path) -> dict[str, str]:
+    refs: dict[str, str] = {}
+    with path.open("r", encoding="utf-8") as handle:
+        for line in handle:
+            row = json.loads(line)
+            segment_id = str(row.get("segment_id") or "").strip()
+            lang = str(row.get("lang") or "").strip()
+            if segment_id and lang:
+                refs[segment_id] = lang
+    return refs
+
+
+def infer_lang_from_path(audio_path: str | None) -> str | None:
+    if not audio_path:
+        return None
+    parts = Path(audio_path).parts
+    if "audio" in parts:
+        idx = parts.index("audio")
+        if idx + 1 < len(parts):
+            return parts[idx + 1]
+    return None
+
+
+def write_row(handle, row: dict[str, object]) -> None:
+    handle.write(json.dumps(row, ensure_ascii=False) + "\n")
+
+
+def main() -> None:
+    OUT_DIR.mkdir(parents=True, exist_ok=True)
+
+    refs_lang = load_refs_lang(REFS_PATH)
+    with INPUTS_PATH.open("r", encoding="utf-8") as handle:
+        inputs: list[dict[str, str]] = json.load(handle)
+
+    pipeline = ASRInferencePipeline(model_card=MODEL_CARD)
+    asr_key = f"omniasr:{MODEL_CARD}"
+
+    with OUT_PATH.open("w", encoding="utf-8") as handle:
+        for batch in chunked(inputs, BATCH_SIZE):
+            audio_paths = [row.get("audio_path") for row in batch]
+            langs = [row.get("lang") for row in batch]
+
+            texts: list[str | None] = [None] * len(batch)
+            errors: list[str | None] = [None] * len(batch)
+
+            try:
+                out_texts = pipeline.transcribe(audio_paths, lang=langs, batch_size=len(batch))
+                for idx, text in enumerate(out_texts):
+                    texts[idx] = text
+            except Exception:
+                for idx, row in enumerate(batch):
+                    try:
+                        text = pipeline.transcribe([row.get("audio_path")], lang=[row.get("lang")], batch_size=1)[0]
+                        texts[idx] = text
+                    except Exception as exc:
+                        errors[idx] = f"{type(exc).__name__}: {exc}"
+
+            for row, text, err in zip(batch, texts, errors):
+                segment_id = str(row.get("segment_id") or "").strip()
+                if not segment_id:
+                    continue
+                route_lang = refs_lang.get(segment_id) or infer_lang_from_path(row.get("audio_path"))
+                out_row: dict[str, object] = {
+                    "schema_version": 1,
+                    "segment_id": segment_id,
+                    "src_path": row.get("audio_path"),
+                    "route": {"language_hint": route_lang, "language_hint_source": "refs"},
+                    "config": {"model_card": MODEL_CARD, "batch_size": BATCH_SIZE},
+                    "asr_errors": {},
+                }
+                if err:
+                    out_row["asr_errors"][asr_key] = err
+                if text:
+                    out_row["asr"] = {
+                        asr_key: {
+                            "model": MODEL_CARD,
+                            "text": text,
+                            "language": row.get("lang"),
+                        }
+                    }
+                    out_row["final"] = {"text": text, "source": f"asr:{asr_key}"}
+                write_row(handle, out_row)
+
+
+if __name__ == "__main__":
+    main()
*** End Patch


Output: execution error: Io(Os { code: 2, kind: NotFound, message: "No such file or directory" })


**Checking directory existence**

I'm verifying that the `scripts` folder was created properly and exists at the expected path to rule out directory-not-found errors in apply_patch.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 8
drwxrwxr-x  2 ubuntu ubuntu 4096 Jan 15 22:22 .
drwxrwxr-x 13 ubuntu ubuntu 4096 Jan 15 22:22 ..


**Retrying patch application despite nested directories issue**


Input:
*** Begin Patch
*** Add File: scripts/run_omniasr_7b_v2.py
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Iterator
+
+from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline
+
+MODEL_CARD = "omniASR_LLM_7B_v2"
+INPUTS_PATH = Path("outputs/bench/maya_hf_v2_100/omniasr_7b_v2_inputs.json")
+REFS_PATH = Path("data/bench/maya_hf_v2_100/refs.jsonl")
+OUT_DIR = Path("outputs/bench/maya_hf_v2_100/omniasr_llm_7b_v2")
+OUT_PATH = OUT_DIR / "results.jsonl"
+BATCH_SIZE = 4
+
+
+def chunked(items: list[dict[str, str]], size: int) -> Iterator[list[dict[str, str]]]:
+    for idx in range(0, len(items), size):
+        yield items[idx : idx + size]
+
+
+def load_refs_lang(path: Path) -> dict[str, str]:
+    refs: dict[str, str] = {}
+    with path.open("r", encoding="utf-8") as handle:
+        for line in handle:
+            row = json.loads(line)
+            segment_id = str(row.get("segment_id") or "").strip()
+            lang = str(row.get("lang") or "").strip()
+            if segment_id and lang:
+                refs[segment_id] = lang
+    return refs
+
+
+def infer_lang_from_path(audio_path: str | None) -> str | None:
+    if not audio_path:
+        return None
+    parts = Path(audio_path).parts
+    if "audio" in parts:
+        idx = parts.index("audio")
+        if idx + 1 < len(parts):
+            return parts[idx + 1]
+    return None
+
+
+def write_row(handle, row: dict[str, object]) -> None:
+    handle.write(json.dumps(row, ensure_ascii=False) + "\n")
+
+
+def main() -> None:
+    OUT_DIR.mkdir(parents=True, exist_ok=True)
+
+    refs_lang = load_refs_lang(REFS_PATH)
+    with INPUTS_PATH.open("r", encoding="utf-8") as handle:
+        inputs: list[dict[str, str]] = json.load(handle)
+
+    pipeline = ASRInferencePipeline(model_card=MODEL_CARD)
+    asr_key = f"omniasr:{MODEL_CARD}"
+
+    with OUT_PATH.open("w", encoding="utf-8") as handle:
+        for batch in chunked(inputs, BATCH_SIZE):
+            audio_paths = [row.get("audio_path") for row in batch]
+            langs = [row.get("lang") for row in batch]
+
+            texts: list[str | None] = [None] * len(batch)
+            errors: list[str | None] = [None] * len(batch)
+
+            try:
+                out_texts = pipeline.transcribe(audio_paths, lang=langs, batch_size=len(batch))
+                for idx, text in enumerate(out_texts):
+                    texts[idx] = text
+            except Exception:
+                for idx, row in enumerate(batch):
+                    try:
+                        text = pipeline.transcribe([row.get("audio_path")], lang=[row.get("lang")], batch_size=1)[0]
+                        texts[idx] = text
+                    except Exception as exc:
+                        errors[idx] = f"{type(exc).__name__}: {exc}"
+
+            for row, text, err in zip(batch, texts, errors):
+                segment_id = str(row.get("segment_id") or "").strip()
+                if not segment_id:
+                    continue
+                route_lang = refs_lang.get(segment_id) or infer_lang_from_path(row.get("audio_path"))
+                out_row: dict[str, object] = {
+                    "schema_version": 1,
+                    "segment_id": segment_id,
+                    "src_path": row.get("audio_path"),
+                    "route": {"language_hint": route_lang, "language_hint_source": "refs"},
+                    "config": {"model_card": MODEL_CARD, "batch_size": BATCH_SIZE},
+                    "asr_errors": {},
+                }
+                if err:
+                    out_row["asr_errors"][asr_key] = err
+                if text:
+                    out_row["asr"] = {
+                        asr_key: {
+                            "model": MODEL_CARD,
+                            "text": text,
+                            "language": row.get("lang"),
+                        }
+                    }
+                    out_row["final"] = {"text": text, "source": f"asr:{asr_key}"}
+                write_row(handle, out_row)
+
+
+if __name__ == "__main__":
+    main()
*** End Patch


Output: execution error: Io(Os { code: 2, kind: NotFound, message: "No such file or directory" })


**Switching to shell command for file creation**

I'll create the new file using a shell `cat` command with heredoc syntax in the repo root, bypassing apply_patch issues.


cat <<'PY' > scripts/run_omniasr_7b_v2.py
#!/usr/bin/env python3
from __future__ import annotations

import json
from pathlib import Path
from typing import Iterator

from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline

MODEL_CARD = "omniASR_LLM_7B_v2"
INPUTS_PATH = Path("outputs/bench/maya_hf_v2_100/omniasr_7b_v2_inputs.json")
REFS_PATH = Path("data/bench/maya_hf_v2_100/refs.jsonl")
OUT_DIR = Path("outputs/bench/maya_hf_v2_100/omniasr_llm_7b_v2")
OUT_PATH = OUT_DIR / "results.jsonl"
BATCH_SIZE = 4

def chunked(items: list[dict[str, str]], size: int) -> Iterator[list[dict[str, str]]]:
    for idx in range(0, len(items), size):
        yield items[idx : idx + size]

def load_refs_lang(path: Path) -> dict[str, str]:
    refs: dict[str, str] = {}
    with path.open("r", encoding="utf-8") as handle:
        for line in handle:
            row = json.loads(line)
            segment_id = str(row.get("segment_id") or "").strip()
            lang = str(row.get("lang") or "").strip()
            if segment_id and lang:
                refs[segment_id] = lang
    return refs

def infer_lang_from_path(audio_path: str | None) -> str | None:
    if not audio_path:
        return None
    parts = Path(audio_path).parts
    if "audio" in parts:
        idx = parts.index("audio")
        if idx + 1 < len(parts):
            return parts[idx + 1]
    return None

def write_row(handle, row: dict[str, object]) -> None:
    handle.write(json.dumps(row, ensure_ascii=False) + "\n")

def main() -> None:
    OUT_DIR.mkdir(parents=True, exist_ok=True)

    refs_lang = load_refs_lang(REFS_PATH)
    with INPUTS_PATH.open("r", encoding="utf-8") as handle:
        inputs: list[dict[str, str]] = json.load(handle)

    pipeline = ASRInferencePipeline(model_card=MODEL_CARD)
    asr_key = f"omniasr:{MODEL_CARD}"

    with OUT_PATH.open("w", encoding="utf-8") as handle:
        for batch in chunked(inputs, BATCH_SIZE):
            audio_paths = [row.get("audio_path") for row in batch]
            langs = [row.get("lang") for row in batch]

            texts: list[str | None] = [None] * len(batch)
            errors: list[str | None] = [None] * len(batch)

            try:
                out_texts = pipeline.transcribe(audio_paths, lang=langs, batch_size=len(batch))
                for idx, text in enumerate(out_texts):
                    texts[idx] = text
            except Exception:
                for idx, row in enumerate(batch):
                    try:
                        text = pipeline.transcribe([row.get("audio_path")], lang=[row.get("lang")], batch_size=1)[0]
                        texts[idx] = text
                    except Exception as exc:
                        errors[idx] = f"{type(exc).__name__}: {exc}"

            for row, text, err in zip(batch, texts, errors):
                segment_id = str(row.get("segment_id") or "").strip()
                if not segment_id:
                    continue
                route_lang = refs_lang.get(segment_id) or infer_lang_from_path(row.get("audio_path"))
                out_row: dict[str, object] = {
                    "schema_version": 1,
                    "segment_id": segment_id,
                    "src_path": row.get("audio_path"),
                    "route": {"language_hint": route_lang, "language_hint_source": "refs"},
                    "config": {"model_card": MODEL_CARD, "batch_size": BATCH_SIZE},
                    "asr_errors": {},
                }
                if err:
                    out_row["asr_errors"][asr_key] = err
                if text:
                    out_row["asr"] = {
                        asr_key: {
                            "model": MODEL_CARD,
                            "text": text,
                            "language": row.get("lang"),
                        }
                    }
                    out_row["final"] = {"text": text, "source": f"asr:{asr_key}"}
                write_row(handle, out_row)

if __name__ == "__main__":
    main()
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Creating M4T script**


cat <<'PY' > scripts/run_seamless_m4t_v2_large.py
#!/usr/bin/env python3
from __future__ import annotations

import json
from pathlib import Path
from typing import Iterator

import soundfile as sf
import torch
import torchaudio
from transformers import AutoProcessor, SeamlessM4Tv2Model

MODEL_ID = "facebook/seamless-m4t-v2-large"
BENCH_DIR = Path("data/bench/maya_hf_v2_100")
REFS_PATH = BENCH_DIR / "refs.jsonl"
OUT_DIR = Path("outputs/bench/maya_hf_v2_100/seamless_m4t_v2_large")
OUT_PATH = OUT_DIR / "results.jsonl"
BATCH_SIZE = 4
TARGET_SR = 16000
DEVICE = "cuda"

LANG_MAP = {
    "asm": "asm",
    "ben": "ben",
    "eng": "eng",
    "guj": "guj",
    "hin": "hin",
    "kan": "kan",
    "mal": "mal",
    "mar": "mar",
    "ory": "ory",
    "pan": "pan",
    "tam": "tam",
    "tel": "tel",
}

def chunked(items: list[dict[str, object]], size: int) -> Iterator[list[dict[str, object]]]:
    for idx in range(0, len(items), size):
        yield items[idx : idx + size]

def load_refs(path: Path) -> list[dict[str, object]]:
    rows: list[dict[str, object]] = []
    with path.open("r", encoding="utf-8") as handle:
        for line in handle:
            row = json.loads(line)
            if row.get("segment_id") and row.get("lang") and row.get("audio_relpath"):
                rows.append(row)
    return rows

def load_audio(path: Path, resamplers: dict[int, torchaudio.transforms.Resample]) -> list[float]:
    audio, sr = sf.read(path)
    if audio.ndim == 2:
        audio = audio.mean(axis=1)
    audio = audio.astype("float32")
    if sr != TARGET_SR:
        resampler = resamplers.get(sr)
        if resampler is None:
            resampler = torchaudio.transforms.Resample(orig_freq=sr, new_freq=TARGET_SR)
            resamplers[sr] = resampler
        wav = torch.from_numpy(audio).float().unsqueeze(0)
        audio = resampler(wav).squeeze(0).cpu().numpy()
    return audio

def write_row(handle, row: dict[str, object]) -> None:
    handle.write(json.dumps(row, ensure_ascii=False) + "\n")

def main() -> None:
    OUT_DIR.mkdir(parents=True, exist_ok=True)

    rows = load_refs(REFS_PATH)
    by_lang: dict[str, list[dict[str, object]]] = {}
    for row in rows:
        lang = str(row.get("lang"))
        by_lang.setdefault(lang, []).append(row)

    processor = AutoProcessor.from_pretrained(MODEL_ID)
    model = SeamlessM4Tv2Model.from_pretrained(
        MODEL_ID,
        torch_dtype=torch.float16,
        low_cpu_mem_usage=True,
    ).to(DEVICE)
    model.eval()

    asr_key = f"m4t:{MODEL_ID}"
    resamplers: dict[int, torchaudio.transforms.Resample] = {}

    with OUT_PATH.open("w", encoding="utf-8") as handle:
        for lang, lang_rows in sorted(by_lang.items()):
            tgt_lang = LANG_MAP.get(lang)
            if not tgt_lang:
                for row in lang_rows:
                    segment_id = str(row.get("segment_id") or "").strip()
                    if not segment_id:
                        continue
                    out_row = {
                        "schema_version": 1,
                        "segment_id": segment_id,
                        "src_path": str(BENCH_DIR / "audio" / str(row.get("audio_relpath"))),
                        "route": {"language_hint": lang, "language_hint_source": "refs"},
                        "config": {"model_id": MODEL_ID, "batch_size": BATCH_SIZE},
                        "asr_errors": {asr_key: f"unsupported_lang:{lang}"},
                    }
                    write_row(handle, out_row)
                continue

            for batch in chunked(lang_rows, BATCH_SIZE):
                text_by_id: dict[str, str] = {}
                errors: dict[str, str] = {}

                audios: list[list[float]] = []
                valid_rows: list[dict[str, object]] = []

                for row in batch:
                    audio_relpath = str(row.get("audio_relpath"))
                    audio_path = BENCH_DIR / "audio" / audio_relpath
                    segment_id = str(row.get("segment_id") or "").strip()
                    if not segment_id:
                        continue
                    try:
                        audio = load_audio(audio_path, resamplers)
                        audios.append(audio)
                        valid_rows.append(row)
                    except Exception as exc:
                        errors[segment_id] = f"{type(exc).__name__}: {exc}"

                if valid_rows:
                    inputs = processor(audio=audios, sampling_rate=TARGET_SR, return_tensors="pt", padding=True)
                    inputs = {k: v.to(DEVICE) for k, v in inputs.items()}
                    with torch.no_grad():
                        output = model.generate(
                            **inputs,
                            tgt_lang=tgt_lang,
                            generate_speech=False,
                        )
                    tokens = output.sequences if hasattr(output, "sequences") else output
                    texts = processor.batch_decode(tokens, skip_special_tokens=True)
                    for row, text in zip(valid_rows, texts):
                        segment_id = str(row.get("segment_id") or "").strip()
                        if segment_id:
                            text_by_id[segment_id] = text

                for row in batch:
                    segment_id = str(row.get("segment_id") or "").strip()
                    if not segment_id:
                        continue
                    audio_relpath = str(row.get("audio_relpath"))
                    audio_path = BENCH_DIR / "audio" / audio_relpath
                    text = text_by_id.get(segment_id)
                    err = errors.get(segment_id)

                    out_row: dict[str, object] = {
                        "schema_version": 1,
                        "segment_id": segment_id,
                        "src_path": str(audio_path),
                        "route": {"language_hint": lang, "language_hint_source": "refs"},
                        "config": {"model_id": MODEL_ID, "batch_size": BATCH_SIZE},
                        "asr_errors": {},
                    }
                    if err:
                        out_row["asr_errors"][asr_key] = err
                    if text:
                        out_row["asr"] = {
                            asr_key: {
                                "model": MODEL_ID,
                                "text": text,
                                "language": tgt_lang,
                            }
                        }
                        out_row["final"] = {"text": text, "source": f"asr:{asr_key}"}
                    write_row(handle, out_row)

if __name__ == "__main__":
    main()
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Preparing to run OmniASR script**

The script will run within the virtual environment using a Python command without changing executable flags, and a longer timeout will be set for execution.


source .venv_omni/bin/activate && python scripts/run_omniasr_7b_v2.py