Thoroughly explore the /home/ubuntu/training/qwen3-asr-1.7b-phase2-sft repository. I need to understand:
1. The model architecture - what model is being fine-tuned, how is audio processed
2. The inference/generation code - how to load a checkpoint and generate transcriptions from audio
3. The training script structure - what framework (transformers, custom), data format expected
4. Any existing evaluation or benchmark scripts
5. The data collator and how audio features are processed
6. What tokenizer/processor is used

Read all Python files, config files, and shell scripts. Give me a comprehensive summary of how inference works and what's needed to generate transcriptions from audio files.


I'll thoroughly explore this repository to understand its architecture, inference, training, and data processing. Let me start by examining the structure and then diving into the key files.


Bash

find /home/ubuntu/training/qwen3-asr-1.7b-phase2-sft -type f -name "*.py" -o -name "*.json" -o -name "*.yaml" -o -name "*.yml" -o -name "*.sh" | head -50


Let me continue exploring and get a better overview of the directory structure:


ls -la /home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/ | head -30


total 212
drwxrwxr-x 11 ubuntu ubuntu  4096 Mar 25 21:21 .
drwxrwxr-x 10 ubuntu ubuntu  4096 Mar 25 22:53 ..
drwxrwxr-x  2 ubuntu ubuntu  4096 Mar 25 02:16 assets
drwxrwxr-x  2 ubuntu ubuntu  4096 Mar 25 02:16 docker
-rw-rw-r--  1 ubuntu ubuntu   655 Mar 25 02:16 .env
drwxrwxr-x  2 ubuntu ubuntu  4096 Mar 25 02:16 examples
drwxrwxr-x  2 ubuntu ubuntu  4096 Mar 25 21:14 finetuning
drwxrwxr-x  8 ubuntu ubuntu  4096 Mar 25 22:07 .git
drwxrwxr-x  4 ubuntu ubuntu  4096 Mar 25 02:16 .github
-rw-rw-r--  1 ubuntu ubuntu   387 Mar 25 02:16 .gitignore
-rwxrwxr-x  1 ubuntu ubuntu  1401 Mar 25 02:16 gpu_idle_watchdog.sh
-rwxrwxr-x  1 ubuntu ubuntu  1881 Mar 25 21:14 launch_production.sh
-rwxrwxr-x  1 ubuntu ubuntu  4893 Mar 25 21:14 launch_resilient.sh
-rw-rw-r--  1 ubuntu ubuntu 11343 Mar 25 02:16 LICENSE
-rw-rw-r--  1 ubuntu ubuntu   228 Mar 25 02:16 MANIFEST.in
-rwxrwxr-x  1 ubuntu ubuntu  4230 Mar 25 02:16 monitor_training.sh
-rw-rw-r--  1 ubuntu ubuntu 18555 Mar 25 02:16 OVERNIGHT_TODO.md
-rwxrwxr-x  1 ubuntu ubuntu  1902 Mar 25 02:16 push_checkpoint_r2.sh
-rw-rw-r--  1 ubuntu ubuntu  1304 Mar 25 02:16 pyproject.toml
drwxrwxr-x  6 ubuntu ubuntu  4096 Mar 25 21:23 qwen_asr
drwxrwxr-x  2 ubuntu ubuntu  4096 Mar 25 21:21 qwen_asr.egg-info
-rw-rw-r--  1 ubuntu ubuntu 60115 Mar 25 02:16 README.md
-rwxrwxr-x  1 ubuntu ubuntu  2885 Mar 25 02:16 run_bench_final.sh
-rwxrwxr-x  1 ubuntu ubuntu  4093 Mar 25 02:16 run_benchmarks.sh
-rwxrwxr-x  1 ubuntu ubuntu  3805 Mar 25 02:16 run_benchmarks_v2.sh
-rwxrwxr-x  1 ubuntu ubuntu  2835 Mar 25 02:16 run_bucketed_smoke.sh
-rwxrwxr-x  1 ubuntu ubuntu  4469 Mar 25 02:16 run_canary_1k.sh
-rwxrwxr-x  1 ubuntu ubuntu  2590 Mar 25 02:16 run_quick_smoke.sh
-rwxrwxr-x  1 ubuntu ubuntu  2116 Mar 25 02:16 smoke_test_profile.sh


Now let me explore the qwen_asr directory structure and read key files:


find /home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/qwen_asr -type f -name "*.py" | sort


Now let me read the key files. I'll start with the model architecture, configuration, and processing:


     1→# coding=utf-8
     2→# Copyright 2026 The Qwen team, Alibaba Group and the HuggingFace Inc. team. All rights reserved.
     3→#
     4→# Licensed under the Apache License, Version 2.0 (the "License");
     5→# you may not use this file except in compliance with the License.
     6→# You may obtain a copy of the License at
     7→#
     8→#     http://www.apache.org/licenses/LICENSE-2.0
     9→#
    10→# Unless required by applicable law or agreed to in writing, software
    11→# distributed under the License is distributed on an "AS IS" BASIS,
    12→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    13→# See the License for the specific language governing permissions and
    14→# limitations under the License.
    15→from transformers.configuration_utils import PretrainedConfig
    16→from transformers.utils import logging
    17→
    18→
    19→logger = logging.get_logger(__name__)
    20→
    21→
    22→class Qwen3ASRAudioEncoderConfig(PretrainedConfig):
    23→    r"""
    24→    This is the configuration class to store the configuration of a [`Qwen3ASRAudioEncoder`]. It is used to instantiate a
    25→    Qwen3-ASR audio encoder according to the specified arguments, defining the model architecture. Instantiating a
    26→    configuration with the defaults will yield a similar configuration to that of the audio encoder of the Qwen2-Audio
    27→    architecture.
    28→
    29→    e.g. [Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)
    30→
    31→    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    32→    documentation from [`PretrainedConfig`] for more information.
    33→
    34→    Args:
    35→        num_mel_bins (`int`, *optional*, defaults to 128):
    36→            Number of mel features used per input features. Should correspond to the value used in the
    37→            `Qwen3ASRProcessor` class.
    38→        encoder_layers (`int`, *optional*, defaults to 32):
    39→            Number of encoder layers.
    40→        encoder_attention_heads (`int`, *optional*, defaults to 20):
    41→            Number of attention heads for each attention layer in the Transformer encoder.
    42→        encoder_ffn_dim (`int`, *optional*, defaults to 5120):
    43→            Dimensionality of the "intermediate" (often named feed-forward) layer in encoder.
    44→        d_model (`int`, *optional*, defaults to 1280):
    45→            Dimensionality of the layers.
    46→        dropout (`float`, *optional*, defaults to 0.0):
    47→            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
    48→        attention_dropout (`float`, *optional*, defaults to 0.0):
    49→            The dropout ratio for the attention probabilities.
    50→        activation_function (`str`, *optional*, defaults to `"gelu"`):
    51→            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
    52→            `"relu"`, `"silu"` and `"gelu_new"` are supported.
    53→        activation_dropout (`float`, *optional*, defaults to 0.0):
    54→            The dropout ratio for activations inside the fully connected layer.
    55→        scale_embedding (`bool`, *optional*, defaults to `False`):
    56→            Scale embeddings by diving by sqrt(d_model).
    57→        initializer_range (`float`, *optional*, defaults to 0.02):
    58→            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
    59→        max_source_positions (`int`, *optional*, defaults to 1500):
    60→            The maximum sequence length of log-mel filter-bank features that this model might ever be used with.
    61→        n_window (`int`, *optional*, defaults to 100):
    62→            The chunk for conv and flash attn in AudioEncoder.
    63→        output_dim (`int`, *optional*, defaults to 3584):
    64→            The output dimension of AudioEncoder.
    65→
    66→    Example:
    67→
    68→    ```python
    69→    >>> from transformers import Qwen3ASRAudioEncoderConfig, Qwen3ASRAudioEncoder
    70→
    71→    >>> # Initializing a Qwen3ASRAudioEncoderConfig
    72→    >>> configuration = Qwen3ASRAudioEncoderConfig()
    73→
    74→    >>> # Initializing a Qwen3ASRAudioEncoder (with random weights)
    75→    >>> model = Qwen3ASRAudioEncoder(configuration)
    76→
    77→    >>> # Accessing the model configuration
    78→    >>> configuration = model.config
    79→    ```"""
    80→
    81→    model_type = "qwen3_asr_audio_encoder"
    82→
    83→    def __init__(
    84→        self,
    85→        num_mel_bins=128,
    86→        encoder_layers=32,
    87→        encoder_attention_heads=20,
    88→        encoder_ffn_dim=5120,
    89→        d_model=1280,
    90→        dropout=0,
    91→        attention_dropout=0,
    92→        activation_function="gelu",
    93→        activation_dropout=0,
    94→        scale_embedding=False,
    95→        initializer_range=0.02,
    96→        max_source_positions=1500,
    97→        n_window=100,
    98→        output_dim=3584,
    99→        n_window_infer=400,
   100→        conv_chunksize=500,
   101→        downsample_hidden_size=480,
   102→        **kwargs,
   103→    ):
   104→        super().__init__(**kwargs)
   105→
   106→        self.num_mel_bins = num_mel_bins
   107→        self.d_model = d_model
   108→        self.encoder_layers = encoder_layers
   109→        self.encoder_attention_heads = encoder_attention_heads
   110→        self.encoder_ffn_dim = encoder_ffn_dim
   111→        self.dropout = dropout
   112→        self.attention_dropout = attention_dropout
   113→        self.activation_function = activation_function
   114→        self.activation_dropout = activation_dropout
   115→        self.num_hidden_layers = encoder_layers
   116→        self.initializer_range = initializer_range
   117→        self.scale_embedding = scale_embedding  # scale factor will be sqrt(d_model) if True
   118→        self.max_source_positions = max_source_positions
   119→        self.n_window = n_window
   120→        self.output_dim = output_dim
   121→        self.n_window_infer = n_window_infer
   122→        self.conv_chunksize = conv_chunksize
   123→        self.downsample_hidden_size = downsample_hidden_size
   124→
   125→
   126→class Qwen3ASRTextConfig(PretrainedConfig):
   127→    r"""
   128→    This is the configuration class to store the configuration of a [`Qwen3ASRTextModel`]. It is used to instantiate a
   129→    Qwen3-ASR model according to the specified arguments, defining the model architecture. Instantiating a configuration
   130→    with the defaults will yield a similar configuration to that of
   131→    Qwen3-ASR-1.7B [Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)
   132→
   133→    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
   134→    documentation from [`PretrainedConfig`] for more information.
   135→
   136→    Args:
   137→        vocab_size (`int`, *optional*, defaults to 151936):
   138→            Vocabulary size of the Qwen3ASR model. Defines the number of different tokens that can be represented by the
   139→            `inputs_ids` passed when calling [`Qwen3ASRModel`]
   140→        hidden_size (`int`, *optional*, defaults to 4096):
   141→            Dimension of the hidden representations.
   142→        intermediate_size (`int`, *optional*, defaults to 22016):
   143→            Dimension of the MLP representations.
   144→        num_hidden_layers (`int`, *optional*, defaults to 32):
   145→            Number of hidden layers in the Transformer encoder.
   146→        num_attention_heads (`int`, *optional*, defaults to 32):
   147→            Number of attention heads for each attention layer in the Transformer encoder.
   148→        num_key_value_heads (`int`, *optional*, defaults to 32):
   149→            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
   150→            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
   151→            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
   152→            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
   153→            by meanpooling all the original heads within that group. For more details, check out [this
   154→            paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `32`.
   155→        head_dim (`int`, *optional*, defaults to 128):
   156→            The dimension of the head. If not specified, will default to `hidden_size // num_attention_heads`.
   157→        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
   158→            The non-linear activation function (function or string) in the decoder.
   159→        max_position_embeddings (`int`, *optional*, defaults to 128000):
   160→            The maximum sequence length that this model might ever be used with.
   161→        initializer_range (`float`, *optional*, defaults to 0.02):
   162→            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
   163→        rms_norm_eps (`float`, *optional*, defaults to 1e-06):
   164→            The epsilon used by the rms normalization layers.
   165→        use_cache (`bool`, *optional*, defaults to `True`):
   166→            Whether or not the model should return the last key/values attentions (not used by all models). Only
   167→            relevant if `config.is_decoder=True`.
   168→        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
   169→            Whether the model's input and output word embeddings should be tied.
   170→        rope_theta (`float`, *optional*, defaults to 5000000.0):
   171→            The base period of the RoPE embeddings.
   172→        rope_scaling (`Dict`, *optional*):
   173→            Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
   174→            and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
   175→            accordingly.
   176→            Expected contents:
   177→                `rope_type` (`str`):
   178→                    The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
   179→                    'llama3'], with 'default' being the original RoPE implementation.
   180→                `factor` (`float`, *optional*):
   181→                    Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
   182→                    most scaling types, a `factor` of x will enable the model to handle sequences of length x *
   183→                    original maximum pre-trained length.
   184→                `original_max_position_embeddings` (`int`, *optional*):
   185→                    Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during
   186→                    pretraining.
   187→                `attention_factor` (`float`, *optional*):
   188→                    Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention
   189→                    computation. If unspecified, it defaults to value recommended by the implementation, using the
   190→                    `factor` field to infer the suggested value.
   191→                `beta_fast` (`float`, *optional*):
   192→                    Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear
   193→                    ramp function. If unspecified, it defaults to 32.
   194→                `beta_slow` (`float`, *optional*):
   195→                    Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear
   196→                    ramp function. If unspecified, it defaults to 1.
   197→                `short_factor` (`list[float]`, *optional*):
   198→                    Only used with 'longrope'. The scaling factor to be applied to short contexts (<
   199→                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
   200→                    size divided by the number of attention heads divided by 2
   201→                `long_factor` (`list[float]`, *optional*):
   202→                    Only used with 'longrope'. The scaling factor to be applied to long contexts (<
   203→                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
   204→                    size divided by the number of attention heads divided by 2
   205→                `low_freq_factor` (`float`, *optional*):
   206→                    Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE
   207→                `high_freq_factor` (`float`, *optional*):
   208→                    Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE
   209→        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
   210→            Whether to use a bias in the query, key, value and output projection layers during self-attention.
   211→        attention_dropout (`float`, *optional*, defaults to 0.0):
   212→            The dropout ratio for the attention probabilities.
   213→
   214→    ```python
   215→    >>> from transformers import Qwen3ASRTextModel, Qwen3ASRTextConfig
   216→
   217→    >>> # Initializing a Qwen3ASR style configuration
   218→    >>> configuration = Qwen3ASRTextConfig()
   219→
   220→    >>> # Initializing a model from the Qwen3-VL-7B style configuration
   221→    >>> model = Qwen3ASRTextModel(configuration)
   222→
   223→    >>> # Accessing the model configuration
   224→    >>> configuration = model.config
   225→    ```"""
   226→
   227→    model_type = "qwen3_asr_text"
   228→    base_config_key = "text_config"
   229→
   230→    def __init__(
   231→        self,
   232→        vocab_size=151936,
   233→        hidden_size=4096,
   234→        intermediate_size=22016,
   235→        num_hidden_layers=32,
   236→        num_attention_heads=32,
   237→        num_key_value_heads=32,
   238→        head_dim=128,
   239→        hidden_act="silu",
   240→        max_position_embeddings=128000,
   241→        initializer_range=0.02,
   242→        rms_norm_eps=1e-6,
   243→        use_cache=True,
   244→        tie_word_embeddings=False,
   245→        rope_theta=5000000.0,
   246→        rope_scaling=None,
   247→        attention_bias=False,
   248→        attention_dropout=0.0,
   249→        **kwargs,
   250→    ):
   251→        self.vocab_size = vocab_size
   252→        self.max_position_embeddings = max_position_embeddings
   253→        self.hidden_size = hidden_size
   254→        self.intermediate_size = intermediate_size
   255→        self.num_hidden_layers = num_hidden_layers
   256→        self.num_attention_heads = num_attention_heads
   257→
   258→        # for backward compatibility
   259→        if num_key_value_heads is None:
   260→            num_key_value_heads = num_attention_heads
   261→
   262→        self.num_key_value_heads = num_key_value_heads
   263→        self.head_dim = head_dim
   264→        self.hidden_act = hidden_act
   265→        self.initializer_range = initializer_range
   266→        self.rms_norm_eps = rms_norm_eps
   267→        self.use_cache = use_cache
   268→        self.rope_theta = rope_theta
   269→        self.rope_scaling = rope_scaling
   270→        self.attention_bias = attention_bias
   271→        self.attention_dropout = attention_dropout
   272→        # Validate the correctness of rotary position embeddings parameters
   273→        # BC: if there is a 'type' field, move it to 'rope_type'.
   274→        if self.rope_scaling is not None and "type" in self.rope_scaling:
   275→            self.rope_scaling["rope_type"] = self.rope_scaling["type"]
   276→
   277→        super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)
   278→
   279→
   280→class Qwen3ASRThinkerConfig(PretrainedConfig):
   281→    r"""
   282→    This is the configuration class to store the configuration of a [`Qwen3ASRThinker`]. It is used to instantiate a
   283→    Qwen3-ASR-Thinker model according to the specified arguments, defining the model architecture. Instantiating a
   284→    configuration with the defaults will yield a similar configuration to that of the thinker component of the Qwen3-Omni
   285→    architecture.
   286→
   287→    e.g. [Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)
   288→
   289→    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
   290→    documentation from [`PretrainedConfig`] for more information.
   291→
   292→    Args:
   293→        audio_config (`dict`, *optional*):
   294→            The config dictionary of the audio backbone.
   295→        text_config (`dict`, *optional*):
   296→            The config dictionary of the text backbone.
   297→        audio_token_id (`int`, *optional*, defaults to 151646):
   298→            The audio token id to encode the audio prompt.
   299→        audio_start_token_id (`int`, *optional*, defaults to 151647):
   300→            The audio start token id to encode the audio prompt.
   301→        user_token_id (`int`, *optional*, defaults to 872):
   302→            The user token id to encode the user token.
   303→        initializer_range (`float`, *optional*, defaults to 0.02):
   304→            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
   305→
   306→    Example:
   307→
   308→    ```python
   309→    >>> from transformers import Qwen3ASRThinkerModel, Qwen3ASRThinkerConfig
   310→
   311→    >>> # Initializing a default Qwen3ASRThinkerConfig
   312→    >>> configuration = Qwen3ASRThinkerConfig()
   313→
   314→    >>> # Initializing a model (with random weights) from the default configuration
   315→    >>> model = Qwen3ASRThinkerModel(configuration)
   316→
   317→    >>> # Accessing the model configuration
   318→    >>> configuration = model.config
   319→    ```"""
   320→
   321→    model_type = "qwen3_asr_thinker"
   322→
   323→    attribute_map = {}
   324→    sub_configs = {
   325→        "audio_config": Qwen3ASRAudioEncoderConfig,
   326→        "text_config": Qwen3ASRTextConfig,
   327→    }
   328→
   329→    def __init__(
   330→        self,
   331→        audio_config=None,
   332→        text_config=None,
   333→        audio_token_id=151646,
   334→        audio_start_token_id=151647,
   335→        user_token_id=872,
   336→        initializer_range=0.02,
   337→        **kwargs,
   338→    ):
   339→        super().__init__(**kwargs)
   340→        self.user_token_id = user_token_id
   341→        self.audio_start_token_id = audio_start_token_id
   342→        self.initializer_range = initializer_range
   343→
   344→        if isinstance(audio_config, dict):
   345→            audio_config = Qwen3ASRAudioEncoderConfig(**audio_config)
   346→        elif audio_config is None:
   347→            audio_config = Qwen3ASRAudioEncoderConfig()
   348→        self.audio_config = audio_config
   349→
   350→        if isinstance(text_config, dict):
   351→            text_config = Qwen3ASRTextConfig(**text_config)
   352→        elif text_config is None:
   353→            text_config = Qwen3ASRTextConfig()
   354→        self.text_config = text_config
   355→        self.audio_token_id = audio_token_id
   356→
   357→
   358→class Qwen3ASRConfig(PretrainedConfig):
   359→    """
   360→    This is the configuration class to store the configuration of a [`Qwen3ASRForConditionalGeneration`]. It is used to instantiate a Qwen3ASR
   361→    model according to the specified sub-models configurations, defining the model architecture.
   362→
   363→    Instantiating a configuration with the defaults will yield a similar configuration to that of the
   364→    [Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) architecture.
   365→
   366→    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
   367→    documentation from [`PretrainedConfig`] for more information.
   368→
   369→    Args:
   370→        thinker_config (`dict`, *optional*): Configuration of the underlying thinker sub-model.
   371→        support_languages (`List[str]`, *optional*): The languages supported by the model.
   372→
   373→    Example:
   374→
   375→    ```python
   376→    >>> from transformers import (
   377→    ...     Qwen3ASRThinkerConfig,
   378→    ...     Qwen3ASRForConditionalGeneration,
   379→    ...     Qwen3ASRConfig,
   380→    ... )
   381→
   382→    >>> # Initializing a Qwen3ASR style configuration
   383→    >>> configuration = Qwen3ASRConfig()
   384→
   385→    >>> # Initializing a model from the configuration
   386→    >>> model = Qwen3ASRForConditionalGeneration(configuration)
   387→
   388→    >>> # Accessing the model configuration
   389→    >>> configuration = model.config
   390→    ```"""
   391→
   392→    model_type = "qwen3_asr"
   393→    sub_configs = {
   394→        "thinker_config": Qwen3ASRThinkerConfig,
   395→    }
   396→
   397→    def __init__(
   398→        self,
   399→        thinker_config=None,
   400→        support_languages=None,
   401→        **kwargs,
   402→    ):
   403→        super().__init__(**kwargs)
   404→        if thinker_config is None:
   405→            thinker_config = {}
   406→
   407→        self.thinker_config = Qwen3ASRThinkerConfig(**thinker_config)
   408→        self.support_languages = support_languages
   409→
   410→    def get_text_config(self, decoder=False) -> "PretrainedConfig":
   411→        """
   412→        Returns the config that is meant to be used with text IO. On most models, it is the original config instance
   413→        itself. On specific composite models, it is under a set of valid names.
   414→
   415→        Args:
   416→            decoder (`Optional[bool]`, *optional*, defaults to `False`):
   417→                If set to `True`, then only search for decoder config names.
   418→        """
   419→        # Overridden for deeply nested config like Qwen2.5-Omni. We don't have any omni model
   420→        # except for Qwen yet. This has to be generalized if more deeply nested configs are
   421→        # added. NOTE: currently method used only by vLLM
   422→        return self.thinker_config.get_text_config()
   423→
   424→
   425→__all__ = ["Qwen3ASRConfig", "Qwen3ASRThinkerConfig", "Qwen3ASRAudioEncoderConfig"]
   426→


     1→# coding=utf-8
     2→# Copyright 2026 The Qwen team, Alibaba Group and the HuggingFace Inc. team. All rights reserved.
     3→#
     4→# Licensed under the Apache License, Version 2.0 (the "License");
     5→# you may not use this file except in compliance with the License.
     6→# You may obtain a copy of the License at
     7→#
     8→#     http://www.apache.org/licenses/LICENSE-2.0
     9→#
    10→# Unless required by applicable law or agreed to in writing, software
    11→# distributed under the License is distributed on an "AS IS" BASIS,
    12→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    13→# See the License for the specific language governing permissions and
    14→# limitations under the License.
    15→import re
    16→
    17→import numpy as np
    18→
    19→from transformers.audio_utils import AudioInput
    20→from transformers.feature_extraction_utils import BatchFeature
    21→from transformers.processing_utils import ProcessingKwargs, ProcessorMixin
    22→from transformers.tokenization_utils_base import TextInput
    23→
    24→
    25→class Qwen3ASRProcessorKwargs(ProcessingKwargs, total=False):
    26→    _defaults = {
    27→        "text_kwargs": {
    28→            "padding": False,
    29→            "padding_side": "left",
    30→        },
    31→        "audio_kwargs": {
    32→            "sampling_rate": 16000,
    33→            "padding": True,
    34→            "return_attention_mask": True,
    35→        },
    36→    }
    37→
    38→
    39→def _get_feat_extract_output_lengths(input_lengths):
    40→    """
    41→    Computes the output length of the convolutional layers and the output length of the audio encoder
    42→    """
    43→
    44→    input_lengths_leave = input_lengths % 100
    45→    feat_lengths = (input_lengths_leave - 1) // 2 + 1
    46→    output_lengths = ((feat_lengths - 1) // 2 + 1 - 1) // 2 + 1 + (input_lengths // 100) * 13
    47→    return output_lengths
    48→
    49→
    50→class Qwen3ASRProcessor(ProcessorMixin):
    51→    r"""
    52→    Constructs a Qwen3ASR processor.
    53→    [`Qwen3ASRProcessor`] offers all the functionalities of [`WhisperFeatureExtractor`], and [`Qwen2TokenizerFast`]. See the
    54→    [`~Qwen3ASRProcessor.__call__`] and [`~Qwen3ASRProcessor.decode`] for more information.
    55→
    56→    Args:
    57→        feature_extractor ([`WhisperFeatureExtractor`], *optional*):
    58→            The audio feature extractor.
    59→        tokenizer ([`Qwen2TokenizerFast`], *optional*):
    60→            The text tokenizer.
    61→        chat_template (`Optional[str]`, *optional*):
    62→            The Jinja template to use for formatting the conversation. If not provided, the default chat template is used.
    63→    """
    64→
    65→    attributes = ["feature_extractor", "tokenizer"]
    66→    feature_extractor_class = "WhisperFeatureExtractor"
    67→    tokenizer_class = ("Qwen2Tokenizer", "Qwen2TokenizerFast")
    68→
    69→    def __init__(
    70→        self, feature_extractor=None, tokenizer=None, chat_template=None
    71→    ):
    72→        super().__init__(feature_extractor, tokenizer, chat_template=chat_template)
    73→        self.audio_token = self.tokenizer.audio_token
    74→        self.audio_bos_token = self.tokenizer.audio_bos_token
    75→        self.audio_eos_token = self.tokenizer.audio_eos_token
    76→
    77→    def __call__(
    78→        self,
    79→        text: TextInput = None,
    80→        audio: AudioInput = None,
    81→        **kwargs,
    82→    ) -> BatchFeature:
    83→        """
    84→        Main method to prepare for the model one or several sequences(s) and audio(s). This method forwards the `text`
    85→        and `kwargs` arguments to Qwen2TokenizerFast's [`~Qwen2TokenizerFast.__call__`] if `text` is not `None` to encode
    86→        the text. To prepare the audio(s), this method forwards the `audio` and `kwargs` arguments to
    87→        WhisperFeatureExtractor's [`~WhisperFeatureExtractor.__call__`] if `audio` is not `None`. Please refer to the doctsring
    88→        of the above two methods for more information.
    89→
    90→        Args:
    91→            text (`str`, `List[str]`, `List[List[str]]`):
    92→                The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
    93→                (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
    94→                `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
    95→            audio (`np.ndarray`, `List[np.ndarray]`):
    96→                The audio or batch of audio to be prepared. Each audio can be a NumPy array.
    97→        """
    98→
    99→        if text is None:
   100→            raise ValueError("You need to specify either a `text` input to process.")
   101→
   102→        output_kwargs = self._merge_kwargs(
   103→            Qwen3ASRProcessorKwargs,
   104→            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
   105→            **kwargs,
   106→        )
   107→
   108→        if audio is not None:
   109→            output_kwargs["audio_kwargs"]["padding"] = True
   110→            output_kwargs["audio_kwargs"]["truncation"] = False
   111→            audio_inputs = self.feature_extractor(audio, **output_kwargs["audio_kwargs"])
   112→            audio_inputs["feature_attention_mask"] = audio_inputs.pop(
   113→                "attention_mask"
   114→            )  # rename feature_attention_mask to prevent conflicts later on
   115→            audio_inputs["input_features"] = audio_inputs.pop(
   116→                "input_features"
   117→            )  # rename input_features to prevent conflicts later on
   118→            audio_lengths = iter(_get_feat_extract_output_lengths(audio_inputs["feature_attention_mask"].sum(-1)))
   119→        else:
   120→            audio_inputs = {}
   121→            audio_lengths = iter([])
   122→
   123→        if not isinstance(text, list):
   124→            text = [text]
   125→
   126→        text = self.replace_multimodal_special_tokens(
   127→            text,
   128→            audio_lengths,
   129→        )
   130→
   131→        texts_inputs = self.tokenizer(text, **output_kwargs["text_kwargs"])
   132→
   133→        return BatchFeature(
   134→            data={**texts_inputs, **audio_inputs},
   135→            tensor_type=kwargs.get("return_tensors"),
   136→        )
   137→
   138→    def replace_multimodal_special_tokens(
   139→        self,
   140→        text,
   141→        audio_lengths,
   142→    ):
   143→
   144→        processed_text = []
   145→        for sample in text:
   146→            positions = []
   147→            special_tokens = [re.escape(tok) for tok in [self.audio_token]]
   148→            pattern = "|".join(special_tokens)
   149→            positions = sorted([(match.start(), match.group()) for match in re.finditer(pattern, sample)])
   150→            positions.sort(key=lambda x: x[0])
   151→
   152→            for _, special_token in positions:
   153→                if special_token == self.audio_token:
   154→                    sample = sample.replace(self.audio_token, "<|audio_placeholder|>" * next(audio_lengths), 1)
   155→
   156→            sample = sample.replace("<|audio_placeholder|>", self.audio_token)
   157→            processed_text.append(sample)
   158→        return processed_text
   159→
   160→    def get_chunked_index(self, token_indices: np.ndarray, tokens_per_chunk: int) -> list[tuple[int, int]]:
   161→        """
   162→        Splits token index list into chunks based on token value ranges.
   163→
   164→        Given a list of token indices, returns a list of (start, end) index tuples representing
   165→        slices of the list where the token values fall within successive ranges of `t_ntoken_per_chunk`.
   166→
   167→        For example, if `t_ntoken_per_chunk` is 1000, the function will create chunks such that:
   168→        - the first chunk contains token values < 1000,
   169→        - the second chunk contains values >= 1000 and < 2000, and so on.
   170→
   171→        Parameters:
   172→            token_indices (`np.ndarray`): A monotonically increasing list of token index values.
   173→            t_ntoken_per_chunk (`int`): Number of tokens per chunk (used as the chunk size threshold).
   174→
   175→        Returns:
   176→            `list[tuple[int, int]]`: A list of tuples, each representing the start (inclusive)
   177→                                and end (exclusive) indices of a chunk in `token_indices`.
   178→        """
   179→
   180→        def _iter():
   181→            i, start_idx = 0, 0  # skip bos token
   182→            current_chunk = 1
   183→            while i < len(token_indices):  # skip eos token
   184→                if token_indices[i] >= current_chunk * tokens_per_chunk:
   185→                    yield (start_idx, i)
   186→                    start_idx = i
   187→                    current_chunk += 1
   188→                i += 1
   189→            yield (start_idx, len(token_indices))
   190→
   191→        return list(_iter())
   192→
   193→    def apply_chat_template(self, conversations, chat_template=None, **kwargs):
   194→        return super().apply_chat_template(conversations, chat_template, **kwargs)
   195→
   196→    @property
   197→    def model_input_names(self):
   198→        tokenizer_input_names = self.tokenizer.model_input_names
   199→        feature_extractor_input_names = self.feature_extractor.model_input_names
   200→        return list(
   201→            dict.fromkeys(
   202→                tokenizer_input_names
   203→                + feature_extractor_input_names
   204→                + ["feature_attention_mask"]
   205→            )
   206→        )
   207→
   208→
   209→__all__ = ["Qwen3ASRProcessor"]
   210→


Error: File content (17813 tokens) exceeds maximum allowed tokens (10000). Use offset and limit parameters to read specific portions of the file, or search for specific content instead of reading the whole file.


Now let me read the modeling file in chunks:


     1→# coding=utf-8
     2→# Copyright 2026 The Qwen team, Alibaba Group and the HuggingFace Inc. team. All rights reserved.
     3→#
     4→# Licensed under the Apache License, Version 2.0 (the "License");
     5→# you may not use this file except in compliance with the License.
     6→# You may obtain a copy of the License at
     7→#
     8→#     http://www.apache.org/licenses/LICENSE-2.0
     9→#
    10→# Unless required by applicable law or agreed to in writing, software
    11→# distributed under the License is distributed on an "AS IS" BASIS,
    12→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    13→# See the License for the specific language governing permissions and
    14→# limitations under the License.
    15→import math
    16→from dataclasses import dataclass
    17→from typing import Callable, Optional, Union
    18→
    19→import numpy as np
    20→import torch
    21→from torch import nn
    22→from torch.nn import functional as F
    23→
    24→from transformers.activations import ACT2FN
    25→from transformers.cache_utils import Cache, DynamicCache
    26→from transformers.generation import GenerationMixin
    27→from transformers.integrations import use_kernel_forward_from_hub
    28→from transformers.masking_utils import create_causal_mask
    29→from transformers.modeling_flash_attention_utils import FlashAttentionKwargs
    30→from transformers.modeling_layers import GradientCheckpointingLayer
    31→from transformers.modeling_outputs import (
    32→    BaseModelOutput,
    33→    BaseModelOutputWithPast,
    34→    MoeCausalLMOutputWithPast,
    35→)
    36→from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
    37→from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
    38→from transformers.processing_utils import Unpack
    39→from transformers.utils import auto_docstring, can_return_tuple
    40→from transformers.utils.deprecation import deprecate_kwarg
    41→from transformers.utils.generic import TransformersKwargs, check_model_inputs
    42→
    43→from .configuration_qwen3_asr import (
    44→    Qwen3ASRAudioEncoderConfig,
    45→    Qwen3ASRConfig,
    46→    Qwen3ASRThinkerConfig,
    47→)
    48→
    49→
    50→@use_kernel_forward_from_hub("RMSNorm")
    51→class Qwen3ASRTextRMSNorm(nn.Module):
    52→    def __init__(self, hidden_size, eps: float = 1e-6) -> None:
    53→        """
    54→        Qwen3ASRTextRMSNorm is equivalent to T5LayerNorm
    55→        """
    56→        super().__init__()
    57→        self.weight = nn.Parameter(torch.ones(hidden_size))
    58→        self.variance_epsilon = eps
    59→
    60→    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
    61→        input_dtype = hidden_states.dtype
    62→        hidden_states = hidden_states.to(torch.float32)
    63→        variance = hidden_states.pow(2).mean(-1, keepdim=True)
    64→        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
    65→        return self.weight * hidden_states.to(input_dtype)
    66→
    67→    def extra_repr(self):
    68→        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
    69→
    70→
    71→def rotate_half(x):
    72→    """Rotates half the hidden dims of the input."""
    73→    x1 = x[..., : x.shape[-1] // 2]
    74→    x2 = x[..., x.shape[-1] // 2 :]
    75→    return torch.cat((-x2, x1), dim=-1)
    76→
    77→
    78→def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
    79→    """
    80→    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
    81→    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
    82→    """
    83→    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
    84→    if n_rep == 1:
    85→        return hidden_states
    86→    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
    87→    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
    88→
    89→
    90→def eager_attention_forward(
    91→    module: nn.Module,
    92→    query: torch.Tensor,
    93→    key: torch.Tensor,
    94→    value: torch.Tensor,
    95→    attention_mask: Optional[torch.Tensor],
    96→    scaling: float,
    97→    dropout: float = 0.0,
    98→    **kwargs: Unpack[TransformersKwargs],
    99→):
   100→    key_states = repeat_kv(key, module.num_key_value_groups)
   101→    value_states = repeat_kv(value, module.num_key_value_groups)
   102→
   103→    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
   104→    if attention_mask is not None:
   105→        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
   106→        attn_weights = attn_weights + causal_mask
   107→
   108→    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
   109→    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
   110→    attn_output = torch.matmul(attn_weights, value_states)
   111→    attn_output = attn_output.transpose(1, 2).contiguous()
   112→
   113→    return attn_output, attn_weights
   114→
   115→
   116→def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
   117→    """Applies Rotary Position Embedding to the query and key tensors.
   118→
   119→    Args:
   120→        q (`torch.Tensor`): The query tensor.
   121→        k (`torch.Tensor`): The key tensor.
   122→        cos (`torch.Tensor`): The cosine part of the rotary embedding.
   123→        sin (`torch.Tensor`): The sine part of the rotary embedding.
   124→        position_ids (`torch.Tensor`, *optional*):
   125→            Deprecated and unused.
   126→        unsqueeze_dim (`int`, *optional*, defaults to 1):
   127→            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
   128→            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
   129→            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
   130→            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
   131→            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
   132→            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
   133→    Returns:
   134→        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
   135→    """
   136→    cos = cos.unsqueeze(unsqueeze_dim)
   137→    sin = sin.unsqueeze(unsqueeze_dim)
   138→    q_embed = (q * cos) + (rotate_half(q) * sin)
   139→    k_embed = (k * cos) + (rotate_half(k) * sin)
   140→    return q_embed, k_embed
   141→
   142→
   143→class Qwen3ASRTextAttention(nn.Module):
   144→    """Multi-headed attention from 'Attention Is All You Need' paper"""
   145→
   146→    def __init__(self, config: Qwen3ASRConfig, layer_idx: int):
   147→        super().__init__()
   148→        self.config = config
   149→        self.layer_idx = layer_idx
   150→        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
   151→        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
   152→        self.scaling = self.head_dim**-0.5
   153→        self.attention_dropout = config.attention_dropout
   154→        self.is_causal = True
   155→
   156→        self.q_proj = nn.Linear(
   157→            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
   158→        )
   159→        self.k_proj = nn.Linear(
   160→            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
   161→        )
   162→        self.v_proj = nn.Linear(
   163→            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
   164→        )
   165→        self.o_proj = nn.Linear(
   166→            config.num_attention_heads * self.head_dim, config.hidden_size, bias=config.attention_bias
   167→        )
   168→        self.q_norm = Qwen3ASRTextRMSNorm(
   169→            self.head_dim, eps=config.rms_norm_eps
   170→        )  # unlike olmo, only on the head dim!
   171→        self.k_norm = Qwen3ASRTextRMSNorm(
   172→            self.head_dim, eps=config.rms_norm_eps
   173→        )  # thus post q_norm does not need reshape
   174→
   175→    @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
   176→    def forward(
   177→        self,
   178→        hidden_states: torch.Tensor,
   179→        position_embeddings: tuple[torch.Tensor, torch.Tensor],
   180→        attention_mask: Optional[torch.Tensor],
   181→        past_key_values: Optional[Cache] = None,
   182→        cache_position: Optional[torch.LongTensor] = None,
   183→        **kwargs: Unpack[FlashAttentionKwargs],
   184→    ) -> tuple[torch.Tensor, Optional[torch.Tensor]]:
   185→        input_shape = hidden_states.shape[:-1]
   186→        hidden_shape = (*input_shape, -1, self.head_dim)
   187→
   188→        query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
   189→        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
   190→        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
   191→
   192→        cos, sin = position_embeddings
   193→        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
   194→
   195→        if past_key_values is not None:
   196→            # sin and cos are specific to RoPE models; cache_position needed for the static cache
   197→            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
   198→            key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx, cache_kwargs)
   199→
   200→        attention_interface: Callable = eager_attention_forward


   200→        attention_interface: Callable = eager_attention_forward
   201→        if self.config._attn_implementation != "eager":
   202→            attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
   203→
   204→        attn_output, attn_weights = attention_interface(
   205→            self,
   206→            query_states,
   207→            key_states,
   208→            value_states,
   209→            attention_mask,
   210→            dropout=0.0 if not self.training else self.attention_dropout,
   211→            scaling=self.scaling,
   212→            **kwargs,
   213→        )
   214→
   215→        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
   216→        attn_output = self.o_proj(attn_output)
   217→        return attn_output, attn_weights
   218→
   219→
   220→class Qwen3ASRTextMLP(nn.Module):
   221→    def __init__(self, config):
   222→        super().__init__()
   223→        self.config = config
   224→        self.hidden_size = config.hidden_size
   225→        self.intermediate_size = config.intermediate_size
   226→        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
   227→        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
   228→        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
   229→        self.act_fn = ACT2FN[config.hidden_act]
   230→
   231→    def forward(self, x):
   232→        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
   233→        return down_proj
   234→
   235→
   236→class Qwen3ASRThinkerTextDecoderLayer(GradientCheckpointingLayer):
   237→    def __init__(self, config: Qwen3ASRConfig, layer_idx: int):
   238→        super().__init__()
   239→        self.hidden_size = config.hidden_size
   240→
   241→        self.self_attn = Qwen3ASRTextAttention(config=config, layer_idx=layer_idx)
   242→
   243→        self.mlp = Qwen3ASRTextMLP(config)
   244→        self.input_layernorm = Qwen3ASRTextRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
   245→        self.post_attention_layernorm = Qwen3ASRTextRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
   246→
   247→    @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
   248→    def forward(
   249→        self,
   250→        hidden_states: torch.Tensor,
   251→        position_embeddings: tuple[torch.Tensor, torch.Tensor],
   252→        attention_mask: Optional[torch.Tensor] = None,
   253→        position_ids: Optional[torch.LongTensor] = None,
   254→        past_key_values: Optional[Cache] = None,
   255→        use_cache: Optional[bool] = False,
   256→        cache_position: Optional[torch.LongTensor] = None,
   257→        **kwargs: Unpack[TransformersKwargs],
   258→    ) -> torch.Tensor:
   259→        residual = hidden_states
   260→        hidden_states = self.input_layernorm(hidden_states)
   261→        # Self Attention
   262→        hidden_states, _ = self.self_attn(
   263→            hidden_states=hidden_states,
   264→            attention_mask=attention_mask,
   265→            position_ids=position_ids,
   266→            past_key_values=past_key_values,
   267→            use_cache=use_cache,
   268→            cache_position=cache_position,
   269→            position_embeddings=position_embeddings,
   270→            **kwargs,
   271→        )
   272→        hidden_states = residual + hidden_states
   273→
   274→        # Fully Connected
   275→        residual = hidden_states
   276→        hidden_states = self.post_attention_layernorm(hidden_states)
   277→        hidden_states = self.mlp(hidden_states)
   278→        hidden_states = residual + hidden_states
   279→        return hidden_states
   280→
   281→
   282→@auto_docstring
   283→class Qwen3ASRPreTrainedModel(PreTrainedModel):
   284→    config: Qwen3ASRConfig
   285→    base_model_prefix = "model"
   286→    supports_gradient_checkpointing = True
   287→    _skip_keys_device_placement = "past_key_values"
   288→    _supports_flash_attn = True
   289→    _supports_sdpa = True
   290→
   291→    _can_compile_fullgraph = True
   292→    _supports_attention_backend = True
   293→    _can_record_outputs = {
   294→        "attentions": Qwen3ASRTextAttention,
   295→    }
   296→
   297→
   298→@dataclass
   299→class Qwen3ASRThinkerCausalLMOutputWithPast(MoeCausalLMOutputWithPast):
   300→    r"""
   301→    Args:
   302→        rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*):
   303→            The rope index difference between sequence length and multimodal rope.
   304→    """
   305→
   306→    rope_deltas: Optional[torch.LongTensor] = None
   307→
   308→
   309→def _get_feat_extract_output_lengths(input_lengths):
   310→    """
   311→    Computes the output length of the convolutional layers and the output length of the audio encoder
   312→    """
   313→
   314→    input_lengths_leave = input_lengths % 100
   315→    feat_lengths = (input_lengths_leave - 1) // 2 + 1
   316→    output_lengths = ((feat_lengths - 1) // 2 + 1 - 1) // 2 + 1 + (input_lengths // 100) * 13
   317→    return output_lengths
   318→
   319→
   320→class Qwen3ASRPreTrainedModelForConditionalGeneration(Qwen3ASRPreTrainedModel):
   321→    def _prepare_4d_causal_attention_mask_with_cache_position(
   322→        self,
   323→        attention_mask: torch.Tensor,
   324→        sequence_length: int,
   325→        target_length: int,
   326→        dtype: torch.dtype,
   327→        device: torch.device,
   328→        min_dtype: float,
   329→        cache_position: torch.Tensor,
   330→        batch_size: int,
   331→    ):
   332→        """
   333→        Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape
   334→        `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing.
   335→
   336→        Args:
   337→            attention_mask (`torch.Tensor`):
   338→                A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`.
   339→            sequence_length (`int`):
   340→                The sequence length being processed.
   341→            target_length (`int`):
   342→                The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet.
   343→            dtype (`torch.dtype`):
   344→                The dtype to use for the 4D attention mask.
   345→            device (`torch.device`):
   346→                The device to place the 4D attention mask on.
   347→            min_dtype (`float`):
   348→                The minimum value representable with the dtype `dtype`.
   349→            cache_position (`torch.Tensor`):
   350→                Indices depicting the position of the input sequence tokens in the sequence.
   351→            batch_size (`torch.Tensor`):
   352→                Batch size.
   353→        """
   354→        if attention_mask is not None and attention_mask.dim() == 4:
   355→            # In this case we assume that the mask comes already in inverted form and requires no inversion or slicing.
   356→            causal_mask = attention_mask
   357→        else:
   358→            causal_mask = torch.full(
   359→                (sequence_length, target_length), fill_value=min_dtype, dtype=dtype, device=device
   360→            )
   361→            if sequence_length != 1:
   362→                causal_mask = torch.triu(causal_mask, diagonal=1)
   363→            causal_mask *= torch.arange(target_length, device=device) > cache_position.reshape(-1, 1)
   364→            causal_mask = causal_mask[None, None, :, :].expand(batch_size, 1, -1, -1)
   365→            if attention_mask is not None:
   366→                causal_mask = causal_mask.clone()  # copy to contiguous memory for in-place edit
   367→                mask_length = attention_mask.shape[-1]
   368→                padding_mask = causal_mask[:, :, :, :mask_length] + attention_mask[:, None, None, :]
   369→                padding_mask = padding_mask == 0
   370→                causal_mask[:, :, :, :mask_length] = causal_mask[:, :, :, :mask_length].masked_fill(
   371→                    padding_mask, min_dtype
   372→                )
   373→
   374→        return causal_mask
   375→
   376→
   377→    def get_chunked_index(
   378→        self, token_indices: torch.Tensor, tokens_per_chunk: int, remove_index: int
   379→    ) -> list[tuple[int, int]]:
   380→        """
   381→        Splits token index list into chunks based on token value ranges.
   382→
   383→        Given a list of token indices, returns a list of (start, end) index tuples representing
   384→        slices of the list where the token values fall within successive ranges of `t_ntoken_per_chunk`.
   385→
   386→        For example, if `t_ntoken_per_chunk` is 1000, the function will create chunks such that:
   387→        - the first chunk contains token values < 1000,
   388→        - the second chunk contains values >= 1000 and < 2000, and so on.
   389→
   390→        Parameters:
   391→            token_indices (`torch.Tensor` of shape `(seq_len, )`): A monotonically increasing list of
   392→                                token index values.
   393→            t_ntoken_per_chunk (`int`): Number of tokens per chunk (used as the chunk size threshold).
   394→            remove_index (`int`) An index id to subtract from `token_indices` before chunking
   395→
   396→        Returns:
   397→            `list[tuple[int, int]]`: A list of tuples, each representing the start (inclusive)
   398→                                and end (exclusive) indices of a chunk in `token_indices`.
   399→        """


Let me continue reading the modeling file and also check the inference code:


     1→# coding=utf-8
     2→# Copyright 2026 The Alibaba Qwen team.
     3→# SPDX-License-Identifier: Apache-2.0
     4→#
     5→# Licensed under the Apache License, Version 2.0 (the "License");
     6→# you may not use this file except in compliance with the License.
     7→# You may obtain a copy of the License at
     8→#
     9→#     http://www.apache.org/licenses/LICENSE-2.0
    10→#
    11→# Unless required by applicable law or agreed to in writing, software
    12→# distributed under the License is distributed on an "AS IS" BASIS,
    13→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    14→# See the License for the specific language governing permissions and
    15→# limitations under the License.
    16→from dataclasses import dataclass
    17→from typing import Any, Dict, List, Optional, Union
    18→
    19→import numpy as np
    20→import torch
    21→from qwen_asr.core.transformers_backend import (
    22→    Qwen3ASRConfig,
    23→    Qwen3ASRForConditionalGeneration,
    24→    Qwen3ASRProcessor,
    25→)
    26→from transformers import AutoConfig, AutoModel, AutoProcessor
    27→
    28→AutoConfig.register("qwen3_asr", Qwen3ASRConfig)
    29→AutoModel.register(Qwen3ASRConfig, Qwen3ASRForConditionalGeneration)
    30→AutoProcessor.register(Qwen3ASRConfig, Qwen3ASRProcessor)
    31→
    32→from .qwen3_forced_aligner import Qwen3ForcedAligner
    33→from .utils import (
    34→    MAX_ASR_INPUT_SECONDS,
    35→    MAX_FORCE_ALIGN_INPUT_SECONDS,
    36→    SAMPLE_RATE,
    37→    SUPPORTED_LANGUAGES,
    38→    AudioChunk,
    39→    AudioLike,
    40→    chunk_list,
    41→    merge_languages,
    42→    normalize_audios,
    43→    normalize_language_name,
    44→    parse_asr_output,
    45→    split_audio_into_chunks,
    46→    validate_language,
    47→)
    48→
    49→try:
    50→    from qwen_asr.core.vllm_backend import Qwen3ASRForConditionalGeneration
    51→    from vllm import ModelRegistry
    52→    ModelRegistry.register_model("Qwen3ASRForConditionalGeneration", Qwen3ASRForConditionalGeneration)
    53→except:
    54→    pass
    55→
    56→
    57→@dataclass
    58→class ASRTranscription:
    59→    """
    60→    One transcription result.
    61→
    62→    Attributes:
    63→        language (str):
    64→            Merged language string for the sample, e.g. "Chinese" or "Chinese,English".
    65→            Empty string if unknown or silent audio.
    66→        text (str):
    67→            Transcribed text.
    68→        time_stamps (Optional[Any]):
    69→            Forced aligner output (ForcedAlignResult).
    70→            Present only when return_time_stamps=True.
    71→    """
    72→    language: str
    73→    text: str
    74→    time_stamps: Optional[Any] = None
    75→
    76→
    77→@dataclass
    78→class ASRStreamingState:
    79→    """
    80→    Streaming ASR state for one audio stream (single utterance).
    81→
    82→    Attributes:
    83→        unfixed_chunk_num (int):
    84→            For the first N chunks, do not use previous ASR result as prefix prompt (reset prefix to "").
    85→        unfixed_token_num (int):
    86→            When chunk_id >= unfixed_chunk_num, rollback the last K tokens from the accumulated text
    87→            before using it as prefix prompt, to reduce boundary jitter.
    88→        chunk_size_sec (float):
    89→            Chunk size in seconds. Audio will be fed to the model in increments of this length.
    90→        chunk_size_samples (int):
    91→            Chunk size in samples at 16kHz (derived from chunk_size_sec).
    92→        chunk_id (int):
    93→            Current chunk index (0-based).
    94→        buffer (np.ndarray):
    95→            Buffered PCM samples that are not yet consumed into a full chunk.
    96→        audio_accum (np.ndarray):
    97→            Accumulated audio from the beginning of the stream up to current time (no padding).
    98→        prompt_raw (str):
    99→            Base prompt generated by chat template (with generation prompt), without appended prefix text.
   100→        context (str):
   101→            Context string.
   102→        force_language (Optional[str]):
   103→            If provided, force output to be text-only by appending "language X<asr_text>" in prompt_raw,
   104→            consistent with non-streaming transcribe().
   105→        language (str):
   106→            Latest parsed language (updated after each chunk decode). Empty if unknown/silent.
   107→        text (str):
   108→            Latest parsed transcription text (updated after each chunk decode).
   109→        _raw_decoded (str):
   110→            Internal accumulated decoded raw text (before parse_asr_output normalization).
   111→            Used for rollback/token trimming and as prefix for prompting.
   112→    """
   113→    unfixed_chunk_num: int
   114→    unfixed_token_num: int
   115→    chunk_size_sec: float
   116→    chunk_size_samples: int
   117→
   118→    chunk_id: int
   119→    buffer: np.ndarray
   120→    audio_accum: np.ndarray
   121→
   122→    prompt_raw: str
   123→    context: str
   124→    force_language: Optional[str]
   125→
   126→    language: str
   127→    text: str
   128→    _raw_decoded: str
   129→
   130→
   131→class Qwen3ASRModel:
   132→    """
   133→    Unified inference wrapper for Qwen3-ASR with two backends:
   134→      - Transformers backend
   135→      - vLLM backend
   136→
   137→    It optionally supports time stamp output via Qwen3-ForcedAligner.
   138→
   139→    Notes:
   140→      - Each request uses a context text and exactly one audio.
   141→      - If language is provided, the prompt will force the output to be text-only by appending
   142→        "language {Language}<asr_text>" to the assistant prompt.
   143→    """
   144→
   145→    def __init__(
   146→        self,
   147→        backend: str,
   148→        model: Any,
   149→        processor: Any,
   150→        sampling_params: Optional[Any] = None,
   151→        forced_aligner: Optional[Qwen3ForcedAligner] = None,
   152→        max_inference_batch_size: int = -1,
   153→        max_new_tokens: int = 512,
   154→    ):
   155→        self.backend = backend  # "transformers" | "vllm"
   156→        self.model = model
   157→        self.processor = processor
   158→        self.sampling_params = sampling_params
   159→        self.forced_aligner = forced_aligner
   160→        self.max_inference_batch_size = int(max_inference_batch_size)
   161→        self.max_new_tokens = max_new_tokens
   162→
   163→        if backend == "transformers":
   164→            self.device = getattr(model, "device", None)
   165→            if self.device is None:
   166→                try:
   167→                    self.device = next(model.parameters()).device
   168→                except StopIteration:
   169→                    self.device = torch.device("cpu")
   170→            self.dtype = getattr(model, "dtype", torch.float32)
   171→        else:
   172→            self.device = None
   173→            self.dtype = None
   174→
   175→    @classmethod
   176→    def from_pretrained(
   177→        cls,
   178→        pretrained_model_name_or_path: str,
   179→        forced_aligner: Optional[str] = None,
   180→        forced_aligner_kwargs: Optional[Dict[str, Any]] = None,
   181→        max_inference_batch_size: int = 32,
   182→        max_new_tokens: Optional[int] = 512,
   183→        **kwargs,
   184→    ) -> "Qwen3ASRModel":
   185→        """
   186→        Initialize using Transformers backend.
   187→
   188→        Args:
   189→            pretrained_model_name_or_path:
   190→                HuggingFace repo id or local directory.
   191→            forced_aligner:
   192→                Optional forced aligner model path/repo id.
   193→            forced_aligner_kwargs:
   194→                Optional kwargs forwarded to Qwen3ForcedAligner.from_pretrained(...).
   195→            max_inference_batch_size:
   196→                Batch size limit for inference. -1 means no chunking. Small values can avoid OOM.
   197→            max_new_tokens:
   198→                Maximum number of tokens to generate.
   199→            **kwargs:
   200→                Forwarded to AutoModel.from_pretrained(...).
   201→
   202→        Returns:
   203→            Qwen3ASRModel
   204→        """
   205→
   206→        model = AutoModel.from_pretrained(pretrained_model_name_or_path, **kwargs)
   207→
   208→        processor = AutoProcessor.from_pretrained(pretrained_model_name_or_path, fix_mistral_regex=True)
   209→
   210→        forced_aligner_model = None
   211→        if forced_aligner is not None:
   212→            forced_aligner_model = Qwen3ForcedAligner.from_pretrained(
   213→                forced_aligner, **(forced_aligner_kwargs or {})
   214→            )
   215→
   216→        return cls(
   217→            backend="transformers",
   218→            model=model,
   219→            processor=processor,
   220→            sampling_params=None,
   221→            forced_aligner=forced_aligner_model,
   222→            max_inference_batch_size=max_inference_batch_size,
   223→            max_new_tokens=max_new_tokens,
   224→        )
   225→
   226→    @classmethod
   227→    def LLM(
   228→        cls,
   229→        model: str,
   230→        forced_aligner: Optional[str] = None,
   231→        forced_aligner_kwargs: Optional[Dict[str, Any]] = None,
   232→        max_inference_batch_size: int = -1,
   233→        max_new_tokens: Optional[int] = 4096,
   234→        **kwargs,
   235→    ) -> "Qwen3ASRModel":
   236→        """
   237→        Initialize using vLLM backend.
   238→
   239→        Import is isolated to keep vLLM optional.
   240→
   241→        Args:
   242→            model:
   243→                Model path/repo for vLLM.
   244→            forced_aligner:
   245→                Optional forced aligner model path/repo id.
   246→            forced_aligner_kwargs:
   247→                Optional kwargs forwarded to Qwen3ForcedAligner.from_pretrained(...).
   248→            max_inference_batch_size:
   249→                Batch size limit for inference. -1 means no chunking. Small values can avoid OOM.
   250→            max_new_tokens:
   251→                Maximum number of tokens to generate.
   252→            **kwargs:
   253→                Forwarded to vllm.LLM(...).
   254→
   255→        Returns:
   256→            Qwen3ASRModel
   257→
   258→        Raises:
   259→            ImportError: If vLLM is not installed.
   260→        """
   261→        try:
   262→            from vllm import LLM as vLLM
   263→            from vllm import SamplingParams
   264→        except Exception as e:
   265→            raise ImportError(
   266→                "vLLM is not available. Install with: pip install qwen-asr[vllm]"
   267→            ) from e
   268→
   269→        llm = vLLM(model=model, **kwargs)
   270→
   271→        processor = Qwen3ASRProcessor.from_pretrained(model, fix_mistral_regex=True)
   272→        sampling_params = SamplingParams(**({"temperature": 0.0, "max_tokens": max_new_tokens}))
   273→
   274→        forced_aligner_model = None
   275→        if forced_aligner is not None:
   276→            forced_aligner_model = Qwen3ForcedAligner.from_pretrained(
   277→                forced_aligner, **(forced_aligner_kwargs or {})
   278→            )
   279→
   280→        return cls(
   281→            backend="vllm",
   282→            model=llm,
   283→            processor=processor,
   284→            sampling_params=sampling_params,
   285→            forced_aligner=forced_aligner_model,
   286→            max_inference_batch_size=max_inference_batch_size,
   287→            max_new_tokens=None,
   288→        )
   289→
   290→    def get_supported_languages(self) -> List[str]:
   291→        """
   292→        Returns the supported language list.
   293→
   294→        Returns:
   295→            List[str]: Canonical language names.
   296→        """
   297→        return list(SUPPORTED_LANGUAGES)
   298→
   299→    @torch.no_grad()
   300→    def transcribe(


     1→# coding=utf-8
     2→# Copyright 2026 The Alibaba Qwen team.
     3→# SPDX-License-Identifier: Apache-2.0
     4→#
     5→# Licensed under the Apache License, Version 2.0 (the "License");
     6→# you may not use this file except in compliance with the License.
     7→# You may obtain a copy of the License at
     8→#
     9→#     http://www.apache.org/licenses/LICENSE-2.0
    10→#
    11→# Unless required by applicable law or agreed to in writing, software
    12→# distributed under the License is distributed on an "AS IS" BASIS,
    13→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    14→# See the License for the specific language governing permissions and
    15→# limitations under the License.
    16→import argparse
    17→import io
    18→import json
    19→import math
    20→import os
    21→import random
    22→import re
    23→import shutil
    24→import time
    25→from collections import OrderedDict
    26→
    27→# Use file_system sharing to avoid /dev/shm exhaustion with many workers
    28→import torch.multiprocessing
    29→torch.multiprocessing.set_sharing_strategy('file_system')
    30→from dataclasses import dataclass, field
    31→from pathlib import Path
    32→from typing import Any, Dict, Iterable, List, Optional, Tuple
    33→
    34→import numpy as np
    35→import pyarrow.parquet as pq
    36→import soundfile as sf
    37→import torch
    38→import torchaudio
    39→from qwen_asr import Qwen3ASRModel
    40→from torch.utils.data import DataLoader, IterableDataset, get_worker_info
    41→from transformers import (GenerationConfig, Trainer, TrainerCallback,
    42→                          TrainingArguments)
    43→
    44→
    45→LANG_CODE_TO_NAME = {
    46→    "en": "English",
    47→    "zh": "Chinese",
    48→    "yue": "Cantonese",
    49→    "ar": "Arabic",
    50→    "de": "German",
    51→    "fr": "French",
    52→    "es": "Spanish",
    53→    "pt": "Portuguese",
    54→    "id": "Indonesian",
    55→    "it": "Italian",
    56→    "ko": "Korean",
    57→    "ru": "Russian",
    58→    "th": "Thai",
    59→    "vi": "Vietnamese",
    60→    "ja": "Japanese",
    61→    "tr": "Turkish",
    62→    "hi": "Hindi",
    63→    "ms": "Malay",
    64→    "nl": "Dutch",
    65→    "sv": "Swedish",
    66→    "da": "Danish",
    67→    "fi": "Finnish",
    68→    "pl": "Polish",
    69→    "cs": "Czech",
    70→    "fil": "Filipino",
    71→    "fa": "Persian",
    72→    "el": "Greek",
    73→    "ro": "Romanian",
    74→    "hu": "Hungarian",
    75→    "mk": "Macedonian",
    76→    # Indic languages
    77→    "as": "Assamese",
    78→    "bn": "Bengali",
    79→    "gu": "Gujarati",
    80→    "kn": "Kannada",
    81→    "ml": "Malayalam",
    82→    "mr": "Marathi",
    83→    "or": "Odia",
    84→    "pa": "Punjabi",
    85→    "ta": "Tamil",
    86→    "te": "Telugu",
    87→}
    88→
    89→_CKPT_RE = re.compile(r"^checkpoint-(\d+)$")
    90→
    91→# Module-level batch stats for ProfilingCallback / crash breadcrumbs.
    92→_BATCH_STATS = {
    93→    "batch_size": 0,
    94→    "audio_secs": 0.0,
    95→    "seq_len": 0,
    96→    "attn_tokens_sum": 0,
    97→    "audio_tokens_max": 0,
    98→    "label_tokens_sum": 0,
    99→    "skip_reason": "",
   100→    "debug_json": "",
   101→}
   102→
   103→# Markup tags that leak into native transcripts
   104→_MARKUP_RE = re.compile(r'\[(?:laugh|cough|singing|breath|noise|cry|sigh|music|applause|beep|হাসি)\]'
   105→                         r'|<(?:laugh|breath|singing|articulated|reduced|inaudible)>', re.IGNORECASE)
   106→
   107→# Unicode script ranges for Indic language detection
   108→_SCRIPT_TO_LANG = {
   109→    'DEVANAGARI': 'hi',
   110→    'BENGALI': 'bn',
   111→    'GUJARATI': 'gu',
   112→    'GURMUKHI': 'pa',
   113→    'KANNADA': 'kn',
   114→    'MALAYALAM': 'ml',
   115→    'ORIYA': 'or',
   116→    'TAMIL': 'ta',
   117→    'TELUGU': 'te',
   118→}
   119→
   120→_LANG_VALID_SCRIPTS = {
   121→    'hi': {'DEVANAGARI'},
   122→    'mr': {'DEVANAGARI'},
   123→    'bn': {'BENGALI'},
   124→    'as': {'BENGALI'},
   125→    'gu': {'GUJARATI'},
   126→    'pa': {'GURMUKHI'},
   127→    'kn': {'KANNADA'},
   128→    'ml': {'MALAYALAM'},
   129→    'or': {'ORIYA'},
   130→    'ta': {'TAMIL'},
   131→    'te': {'TELUGU'},
   132→    'en': {'LATIN'},
   133→}
   134→
   135→
   136→def clean_transcript(text: str) -> str:
   137→    """Remove audio markup tags from transcript text."""
   138→    return _MARKUP_RE.sub('', text).strip()
   139→
   140→
   141→def detect_script_language(text: str, declared_lang: str) -> str:
   142→    """If the transcript's dominant script doesn't match declared language, return the correct lang code."""
   143→    import unicodedata
   144→    scripts = {}
   145→    for ch in text:
   146→        if ch.isalpha():
   147→            name = unicodedata.name(ch, '')
   148→            if name:
   149→                script = name.split()[0]
   150→                if script in _SCRIPT_TO_LANG or script == 'LATIN':
   151→                    scripts[script] = scripts.get(script, 0) + 1
   152→    if not scripts:
   153→        return declared_lang
   154→
   155→    dominant = max(scripts, key=scripts.get)
   156→    valid = _LANG_VALID_SCRIPTS.get(declared_lang, set())
   157→    if valid and dominant not in valid:
   158→        return _SCRIPT_TO_LANG.get(dominant, declared_lang)
   159→    return declared_lang
   160→
   161→
   162→def patch_outer_forward(model):
   163→    cls = model.__class__
   164→    if getattr(cls, "_forward_patched", False):
   165→        return
   166→
   167→    if not hasattr(model, "thinker") or not hasattr(model.thinker, "forward"):
   168→        raise RuntimeError(
   169→            "Cannot patch forward: model has no `.thinker.forward`. "
   170→            "Your qwen3_asr model may be incompatible."
   171→        )
   172→
   173→    def forward(
   174→        self,
   175→        input_ids=None,
   176→        attention_mask=None,
   177→        input_features=None,
   178→        feature_attention_mask=None,
   179→        labels=None,
   180→        **kwargs,
   181→    ):
   182→        return self.thinker.forward(
   183→            input_ids=input_ids,
   184→            attention_mask=attention_mask,
   185→            input_features=input_features,
   186→            feature_attention_mask=feature_attention_mask,
   187→            labels=labels,
   188→            **kwargs,
   189→        )
   190→
   191→    cls.forward = forward
   192→    cls._forward_patched = True
   193→
   194→
   195→def find_latest_checkpoint(output_dir: str) -> Optional[str]:
   196→    if not output_dir or not os.path.isdir(output_dir):
   197→        return None
   198→    best_step = None
   199→    best_path = None
   200→    for name in os.listdir(output_dir):
   201→        m = _CKPT_RE.match(name)
   202→        if not m:
   203→            continue
   204→        step = int(m.group(1))
   205→        path = os.path.join(output_dir, name)
   206→        if os.path.isdir(path) and (best_step is None or step > best_step):
   207→            best_step = step
   208→            best_path = path
   209→    return best_path
   210→
   211→
   212→def get_dist_info() -> Tuple[int, int]:
   213→    if torch.distributed.is_available() and torch.distributed.is_initialized():
   214→        return torch.distributed.get_rank(), torch.distributed.get_world_size()
   215→    return 0, 1
   216→
   217→
   218→def _as_str(x: Any) -> str:
   219→    if x is None:
   220→        return ""
   221→    return str(x)
   222→
   223→
   224→def _get_feat_extract_output_lengths(input_lengths: torch.Tensor) -> torch.Tensor:
   225→    input_lengths_leave = input_lengths % 100
   226→    feat_lengths = (input_lengths_leave - 1) // 2 + 1
   227→    return ((feat_lengths - 1) // 2 + 1 - 1) // 2 + 1 + (input_lengths // 100) * 13
   228→
   229→
   230→# ---------------------------------------------------------------------------
   231→# IndexedTarReader: O(1) seek+read using pre-built .index.json files
   232→# Replaces TarAudioLRUCache (~400x faster per file read)
   233→# ---------------------------------------------------------------------------
   234→
   235→class IndexedTarReader:
   236→    """Direct byte-offset tar reader using pre-built index files.
   237→
   238→    ~400x faster than tarfile.extractfile() because:
   239→    - No tarfile.getmembers() scan (1.4s -> 0ms)
   240→    - Plain file handle seek+read instead of tarfile extraction
   241→    - LRU cache of open file handles
   242→    """
   243→
   244→    def __init__(self, max_open_files: int = 64, max_cached_indices: Optional[int] = None):
   245→        self.max_open_files = int(max(1, max_open_files))
   246→        # Loading one .index.json per tar is fast, but each index is ~1-2 MB in this
   247→        # dataset. Keep it LRU-bounded or workers will OOM as they touch new shards.
   248→        self.max_cached_indices = int(max(1, max_cached_indices or max_open_files))
   249→        self._file_cache: OrderedDict = OrderedDict()  # real tar_path -> file handle
   250→        self._index_cache: OrderedDict = OrderedDict()  # real tar_path -> {member_name: {offset, size}}
   251→
   252→    def _load_index(self, tar_path: str) -> dict:
   253→        real_path = os.path.realpath(tar_path)
   254→        if real_path in self._index_cache:
   255→            self._index_cache.move_to_end(real_path)
   256→            return self._index_cache[real_path]
   257→
   258→        index_path = real_path + ".index.json"
   259→        with open(index_path, "r") as f:
   260→            index = json.load(f)
   261→        self._index_cache[real_path] = index
   262→        if len(self._index_cache) > self.max_cached_indices:
   263→            self._index_cache.popitem(last=False)
   264→        return index
   265→
   266→    def _get_file(self, tar_path: str):
   267→        real_path = os.path.realpath(tar_path)
   268→        if real_path in self._file_cache:
   269→            self._file_cache.move_to_end(real_path)
   270→            return self._file_cache[real_path]
   271→
   272→        fh = open(real_path, "rb")
   273→        self._file_cache[real_path] = fh
   274→        if len(self._file_cache) > self.max_open_files:
   275→            _, old_fh = self._file_cache.popitem(last=False)
   276→            old_fh.close()
   277→        return fh
   278→
   279→    def read_member(self, tar_path: str, member_name: str) -> bytes:
   280→        index = self._load_index(tar_path)
   281→
   282→        entry = index.get(member_name)
   283→        if entry is None:
   284→            bare = member_name.lstrip("./")
   285→            entry = index.get(bare) or index.get("./" + member_name)
   286→        if entry is None:
   287→            raise FileNotFoundError(f"Member {member_name} not found in index for {tar_path}")
   288→
   289→        fh = self._get_file(tar_path)
   290→        fh.seek(entry["offset"])
   291→        return fh.read(entry["size"])
   292→
   293→    def close(self):
   294→        for fh in self._file_cache.values():
   295→            fh.close()
   296→        self._file_cache.clear()
   297→        self._index_cache.clear()
   298→
   299→
   300→# Keep legacy TarAudioLRUCache as fallback when indices don't exist


Let me continue reading the training script and look at data collator:


     1→# coding=utf-8
     2→# Copyright 2026 The Alibaba Qwen team.
     3→# SPDX-License-Identifier: Apache-2.0
     4→#
     5→# Licensed under the Apache License, Version 2.0 (the "License");
     6→# you may not use this file except in compliance with the License.
     7→# You may obtain a copy of the License at
     8→#
     9→#     http://www.apache.org/licenses/LICENSE-2.0
    10→#
    11→# Unless required by applicable law or agreed to in writing, software
    12→# distributed under the License is distributed on an "AS IS" BASIS,
    13→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    14→# See the License for the specific language governing permissions and
    15→# limitations under the License.
    16→"""
    17→Examples for Qwen3ASRModel (Transformers backend).
    18→
    19→Covers:
    20→  - single-sample inference (URL audio)
    21→  - batch inference (mixed URL / base64 / (np.ndarray, sr))
    22→  - forcing language (text-only output)
    23→  - returning time_stamps (single + batch) via Qwen3ForcedAligner
    24→"""
    25→
    26→import base64
    27→import io
    28→import urllib.request
    29→from typing import Tuple
    30→
    31→import numpy as np
    32→import soundfile as sf
    33→import torch
    34→
    35→from qwen_asr import Qwen3ASRModel
    36→
    37→
    38→ASR_MODEL_PATH = "Qwen/Qwen3-ASR-1.7B"
    39→FORCED_ALIGNER_PATH = "Qwen/Qwen3-ForcedAligner-0.6B"
    40→
    41→URL_ZH = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_zh.wav"
    42→URL_EN = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"
    43→
    44→
    45→def _download_audio_bytes(url: str, timeout: int = 30) -> bytes:
    46→    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    47→    with urllib.request.urlopen(req, timeout=timeout) as resp:
    48→        return resp.read()
    49→
    50→
    51→def _read_wav_from_bytes(audio_bytes: bytes) -> Tuple[np.ndarray, int]:
    52→    with io.BytesIO(audio_bytes) as f:
    53→        wav, sr = sf.read(f, dtype="float32", always_2d=False)
    54→    return np.asarray(wav, dtype=np.float32), int(sr)
    55→
    56→
    57→def _to_data_url_base64(audio_bytes: bytes, mime: str = "audio/wav") -> str:
    58→    b64 = base64.b64encode(audio_bytes).decode("utf-8")
    59→    return f"data:{mime};base64,{b64}"
    60→
    61→
    62→def _print_result(title: str, results) -> None:
    63→    print(f"\n===== {title} =====")
    64→    for i, r in enumerate(results):
    65→        print(f"[sample {i}] language={r.language!r}")
    66→        print(f"[sample {i}] text={r.text!r}")
    67→        if r.time_stamps is not None and len(r.time_stamps) > 0:
    68→            head = r.time_stamps[0]
    69→            tail = r.time_stamps[-1]
    70→            print(f"[sample {i}] ts_first: {head.text!r} {head.start_time}->{head.end_time} s")
    71→            print(f"[sample {i}] ts_last : {tail.text!r} {tail.start_time}->{tail.end_time} s")
    72→
    73→
    74→def test_single_url(asr: Qwen3ASRModel) -> None:
    75→    results = asr.transcribe(
    76→        audio=URL_ZH,
    77→        language=None,
    78→        return_time_stamps=False,
    79→    )
    80→    assert isinstance(results, list) and len(results) == 1
    81→    _print_result("single-url (no forced language, no timestamps)", results)
    82→
    83→
    84→def test_batch_mixed(asr: Qwen3ASRModel) -> None:
    85→    zh_bytes = _download_audio_bytes(URL_ZH)
    86→    en_bytes = _download_audio_bytes(URL_EN)
    87→
    88→    zh_b64 = _to_data_url_base64(zh_bytes, mime="audio/wav")
    89→    en_wav, en_sr = _read_wav_from_bytes(en_bytes)
    90→
    91→    results = asr.transcribe(
    92→        audio=[URL_ZH, zh_b64, (en_wav, en_sr)],
    93→        context=["", "交易 停滞", ""],
    94→        language=[None, "Chinese", "English"],
    95→        return_time_stamps=False,
    96→    )
    97→    assert len(results) == 3
    98→    _print_result("batch-mixed (forced language for some)", results)
    99→
   100→
   101→def test_single_with_timestamps(asr: Qwen3ASRModel) -> None:
   102→    results = asr.transcribe(
   103→        audio=URL_EN,
   104→        language="English",
   105→        return_time_stamps=True,
   106→    )
   107→    assert len(results) == 1
   108→    assert results[0].time_stamps is not None
   109→    _print_result("single-url (forced language + timestamps)", results)
   110→
   111→
   112→def test_batch_with_timestamps(asr: Qwen3ASRModel) -> None:
   113→    zh_bytes = _download_audio_bytes(URL_ZH)
   114→    zh_b64 = _to_data_url_base64(zh_bytes, mime="audio/wav")
   115→
   116→    results = asr.transcribe(
   117→        audio=[URL_ZH, zh_b64, URL_EN],
   118→        context=["", "交易 停滞", ""],
   119→        language=["Chinese", "Chinese", "English"],
   120→        return_time_stamps=True,
   121→    )
   122→    assert len(results) == 3
   123→    assert all(r.time_stamps is not None for r in results)
   124→    _print_result("batch (forced language + timestamps)", results)
   125→
   126→
   127→def main() -> None:
   128→    asr = Qwen3ASRModel.from_pretrained(
   129→        ASR_MODEL_PATH,
   130→        dtype=torch.bfloat16,
   131→        device_map="cuda:0",
   132→        # attn_implementation="flash_attention_2",
   133→        forced_aligner=FORCED_ALIGNER_PATH,
   134→        forced_aligner_kwargs=dict(
   135→            dtype=torch.bfloat16,
   136→            device_map="cuda:0",
   137→            # attn_implementation="flash_attention_2",
   138→        ),
   139→        max_inference_batch_size=32,
   140→        max_new_tokens=256,
   141→    )
   142→
   143→    test_single_url(asr)
   144→    test_batch_mixed(asr)
   145→    test_single_with_timestamps(asr)
   146→    test_batch_with_timestamps(asr)
   147→
   148→
   149→if __name__ == "__main__":
   150→    main()
   151→