o ei\@sddlZddlmZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZmZdd lmZdd lmZdd lmZmZmZdd lmZddlmZddlmZmZmZm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,Gddde,eZ-Gddde*Z.Gddde#Z/Gddde"Z0Gdddej1Z2Gdd d eZ3Gd!d"d"eZ4Gd#d$d$e&Z5Gd%d&d&e%Z6Gd'd(d(e(Z7ed)d*Gd+d,d,e7Z8Gd-d.d.e'Z9gd/Z:dS)0N)Callable) Tokenizer)Unigram)nn)create_bidirectional_mask)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TokenizersBackend)TransformersKwargsauto_docstringcan_return_tuple)merge_with_config_defaults)capture_outputs)LlamaAttentionLlamaRotaryEmbeddingapply_rotary_pos_embeager_attention_forward)ParakeetCTCConfigParakeetEncoderConfig)ParakeetEncoderBlock ParakeetEncoderConvolutionModuleParakeetForCTCParakeetPreTrainedModel)ParakeetProcessor) T5Tokenizerc s^eZdZ       dfdd Z   dd eeeBd ed edBd edef ddZZ S) LasrTokenizerdNc s<tjd|||||||d|tt|jddd|_dS)N) eos_token unk_token pad_token extra_idsadditional_special_tokensvocab vocab_filerF)unk_id byte_fallback)super__init__rr _vocab_scores _tokenizer) selfr$r%r&r'r(r)r*kwargs __class__r-c/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/lasr/modular_lasr.pyr/,s$   zLasrTokenizer.__init__FT token_idsskip_special_tokensclean_up_tokenization_spaces group_tokensreturnc sTt|tr|g}|rddt|D}fdd|D}tjf|||d|S)NcSsg|]}|dqS)rr-).0 token_groupr-r-r6 Tsz)LasrTokenizer._decode..csg|] }|jkr|qSr-) pad_token_id)r<tokenr2r-r6r>Ws)r7r8r9) isinstanceint itertoolsgroupbyr _decode)r2r7r8r9r:r3r-rAr6rFIs zLasrTokenizer._decode)r r!r"r#NNN)FNT) __name__ __module__ __qualname__r/rClistboolstrrF __classcell__r-r-r4r6r+s.  rc@ eZdZdS) LasrProcessorNrGrHrIr-r-r-r6rObsrOcsZeZdZdZddddddddd d d d d dd d d dddddgddgddffdd ZZS)LasrEncoderConfiga This is the configuration class to store the configuration of a [`LasrEncoder`]. It is used to instantiate a `LasrEncoder` model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 512): Dimension of the layers and the hidden states. num_hidden_layers (`int`, *optional*, defaults to 17): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer encoder. intermediate_size (`int`, *optional*, defaults to 2048): Dimension of the "intermediate" (often named feed-forward) layer in the Transformer encoder. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the encoder and pooler. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the attention layers. convolution_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in convolutions of the conformer's convolution module. conv_kernel_size (`int`, *optional*, defaults to 32): The kernel size of the convolution layers in the Conformer block. subsampling_conv_channels (`int`, *optional*, defaults to 256): The number of channels in the subsampling convolution layers. subsampling_conv_kernel_size (`int`, *optional*, defaults to 5): The kernel size of the subsampling convolution layers. subsampling_conv_stride (`int`, *optional*, defaults to 2): The stride of the subsampling convolution layers. num_mel_bins (`int`, *optional*, defaults to 128): Number of mel features. dropout (`float`, *optional*, defaults to 0.1): The dropout ratio for all fully connected layers in the embeddings, encoder, and pooler. dropout_positions (`float`, *optional*, defaults to 0.0): The dropout ratio for the positions in the input sequence. layerdrop (`float`, *optional*, defaults to 0.1): The dropout ratio for the layers in the encoder. activation_dropout (`float`, *optional*, defaults to 0.1): The dropout ratio for activations inside the fully connected layer. attention_dropout (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention layers. max_position_embeddings (`int`, *optional*, defaults to 10000): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. layer_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the layer normalization layers. feed_forward_residual_weights (`tuple[float, float]`, *optional*, defaults to `[1.5, 0.5]`): The residual weights for the feed forward layers. conv_residual_weights (`tuple[float, float]`, *optional*, defaults to `[2.0, 1.0]`): The residual weights for the convolution layers. batch_norm_momentum (`float`, *optional*, defaults to 0.01): The momentum for the batch normalization layers. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. Example: ```python >>> from transformers import LasrEncoderModel, LasrEncoderConfig >>> # Initializing a `LasrEncoder` configuration >>> configuration = LasrEncoderConfig() >>> # Initializing a model from the configuration >>> model = LasrEncoderModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` This configuration class is based on the LasrEncoder architecture from Google Health AI. You can find more details and pre-trained models at [TODO/TODO](https://huggingface.co/TODO/TODO). isiluF rg?i'g{Gz?gư>g?g?g@g?g{Gz?Nc s||_||_||_||_||_tjdid|d|d|d|d|d|d|d|d | d | d | d | d | d|d|d|d|d|d|||`|`dS)N hidden_sizenum_hidden_layersnum_attention_headsintermediate_size hidden_actattention_biasconvolution_biasconv_kernel_sizesubsampling_conv_channels num_mel_binssubsampling_conv_kernel_sizesubsampling_conv_stridedropoutdropout_positions layerdropactivation_dropoutattention_dropoutmax_position_embeddingsinitializer_ranger-) rope_parameterslayer_norm_epsfeed_forward_residual_weightsconv_residual_weightsbatch_norm_momentumr.r/subsampling_factor scale_input)r2r[r\r]r^r_r`rarbrcrerfrdrgrhrirjrkrlrmrorprqrrrnr3r4r-r6r/s^      zLasrEncoderConfig.__init__)rGrHrI__doc__r/rMr-r-r4r6rQfs6OrQcsBeZdZdZ     d deeBffdd Zed d ZZ S) LasrCTCConfiga This is the configuration class to store the configuration of a [`LasrForCTC`]. It is used to instantiate a Lasr CTC model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 512): Vocabulary size of the model. ctc_loss_reduction (`str`, *optional*, defaults to `"mean"`): Specifies the reduction to apply to the output of `torch.nn.CTCLoss`. Only relevant when training an instance of [`LasrForCTC`]. ctc_zero_infinity (`bool`, *optional*, defaults to `True`): Whether to zero infinite losses and the associated gradients of `torch.nn.CTCLoss`. Infinite losses mainly occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance of [`LasrForCTC`]. encoder_config (`Union[dict, LasrEncoderConfig]`, *optional*): The config object or dictionary of the encoder. pad_token_id (`int`, *optional*, defaults to 0): Padding token id. Also used as blank token id. Example: ```python >>> from transformers import LasrForCTC, LasrCTCConfig >>> # Initializing a Lasr configuration >>> configuration = LasrCTCConfig() >>> # Initializing a model from the configuration >>> model = LasrForCTC(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` This configuration class is based on the Lasr CTC architecture from Google Health AI. You can find more details and pre-trained models at [TODO/TODO](https://huggingface.co/TODO/TODO). rRmeanTNrencoder_configc s"tjd|||||d|dS)N) vocab_sizectc_loss_reductionctc_zero_infinityrxr?r-)r.r/)r2ryrzr{rxr?r3r4r-r6r/s  zLasrCTCConfig.__init__cCs |jjdS)Nr)rxrfrAr-r-r6inputs_to_logits_ratio%s z$LasrCTCConfig.inputs_to_logits_ratio)rRrwTNr) rGrHrIrudictrQr/propertyr|rMr-r-r4r6rvs#rvcs8eZdZdeffdd ZdejdejfddZZS)LasrEncoderSubsamplingconfigcsttt|j|j|_tj|j|j|j|j d|_ tj|j|j |j|j d|_ t|j |j|_ t|_dS)N) kernel_sizestride)r.r/rLinearrdr[dense_0Conv1drerfconv_0rcconv_1dense_1ReLUact_fnr2rr4r-r6r/+s zLasrEncoderSubsampling.__init__input_featuresr;cCsR|||}|dd}|||}|||}|dd}||S)Nr)rr transposerrr)r2r hidden_statesr-r-r6forward=s    zLasrEncoderSubsampling.forward) rGrHrIrQr/torchTensorrrMr-r-r4r6r*src@rN)LasrEncoderRotaryEmbeddingNrPr-r-r-r6rFs rc sreZdZdedeffdd Z  d dejdeejejfdBdejdBd e e d eejejff d d Z Z S)LasrEncoderAttentionr layer_idxcst||d|_dS)NF)r.r/ is_causalr2rrr4r-r6r/Js zLasrEncoderAttention.__init__Nrposition_embeddingsattention_maskr3r;cKs|jdd}g|d|jR}|||dd}|||dd}|||dd} |\} } t||| | \}}t |j j t } | |||| |f|j sVdn|j|jd|\} }| jg|dR} || } | |fS)NrrrZ)rgscaling)shapehead_dimq_projviewrk_projv_projrr get_interfacer_attn_implementationrtrainingrkrreshape contiguouso_proj)r2rrrr3 input_shape hidden_shape query_states key_states value_statescossinattention_interface attn_output attn_weightsr-r-r6rNs2  zLasrEncoderAttention.forwardNN) rGrHrIrQrCr/rrtupler r rrMr-r-r4r6rIsrcs$eZdZddeffdd ZZS)LasrEncoderConvolutionModuleNrcs,t||d|_tj|j|jd|_dS)Nsame)momentum)r.r/paddingr BatchNorm1dr[rrnorm)r2r module_configr4r-r6r/tsz%LasrEncoderConvolutionModule.__init__N)rGrHrIrQr/rMr-r-r4r6rssrc s^eZdZdedeffdd Z  d dejdejdBdejdBd ee d ejf d d Z Z S)LasrEncoderBlockrrcst|||j|_|j|_tj|j|jdd|_tj|j|jdd|_ tj|j|jdd|_ tj|j|jdd|_ tj|j|jdd|_ dS)NF)bias) r.r/rprqr LayerNormr[ronorm_feed_forward1 norm_self_att norm_convnorm_feed_forward2norm_outrr4r-r6r/{szLasrEncoderBlock.__init__Nrrrr3r;c Ks|}|||}|jd||jd|}||}|jd|||d|\}}||}|j|||d} |jd||jd| }|}|| |}|jd||jd|}| |}|S)Nrr)rrr)rr-) feed_forward1rrpr self_attnconvrrq feed_forward2rr) r2rrrr3residualnormalized_hidden_statesr_ conv_outputr-r-r6rs*   zLasrEncoderBlock.forwardr) rGrHrIrQrCr/rrr r rrMr-r-r4r6rzsrc@s(eZdZdZddZdejfddZdS)LasrPreTrainedModelFcCst|dSr)r _init_weights)r2moduler-r-r6rsz!LasrPreTrainedModel._init_weights input_lengthscCsLt|jtr |jjn|j}|j}|j}d}t|D] }|||d}q|S)Nrr)rBrrvrxrerfrange)r2rrxrr num_layersrr-r-r6_get_subsampling_output_lengths z2LasrPreTrainedModel._get_subsampling_output_lengthN)rGrHrI_supports_flex_attnrrrrr-r-r-r6rsrzh The LasrEncoder model, based on the Conformer architecture](https://arxiv.org/abs/2005.08100). ) custom_introc sjeZdZUeed<dZdeffdd Zeee e d de j de j dBde ed efd d ZZS) LasrEncoderrencodercstd|_j|_j|_j|_t|_t|_ t fddt j D|_t jjjdd|_|dS)NFcsg|]}t|qSr-)r)r<rrr-r6r>sz(LasrEncoder.__init__..)epsr)r.r/gradient_checkpointingrgrhrir subsamplerr rotary_embr ModuleListrr\layersrr[roout_norm post_initrr4rr6r/s    zLasrEncoder.__init__Nrrr3r;c Ks||}||tj|jd|jdd\}}tjj ||j |j d}tjj ||j |j d}tjj ||j |j d}|durH|j ||jdd}t |j||d}|jD]"}d}|j rftg} | |jkrfd }|su||f|||fd |}qS||}t|d S) a Example: ```python >>> from transformers import AutoProcessor, LasrEncoder >>> from datasets import load_dataset, Audio >>> model_id = TODO >>> processor = AutoProcessor.from_pretrained(model_id) >>> encoder = ParakeetEncoder.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> encoder_outputs = encoder(**inputs) >>> print(encoder_outputs.last_hidden_state.shape) ``` r)devicer)prN) target_length)r inputs_embedsrFT)rr)last_hidden_state)rrrarangerr unsqueezer functionalrgrrh_get_output_attention_maskrrrrandrirr) r2rrr3rrr encoder_layerto_dropdropout_probabilityr-r-r6rs@      zLasrEncoder.forwardr)rGrHrIrQ__annotations__base_model_prefixr/rrrrrrr r rrrMr-r-r4r6rs$ rcseZdZfddZZS) LasrForCTCc stjdi|S)a Example: ```python >>> from transformers import AutoProcessor, LasrForCTC >>> from datasets import load_dataset, Audio >>> model_id = TODO >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = LasrForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> predicted_ids = model.generate(**inputs) >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> print(transcription) ``` Nr-)r.generate) super_kwargsr4r-r6rszLasrForCTC.generate)rGrHrIrrMr-r-r4r6rsr)rrrrOrQrvr);rDcollections.abcrr tokenizersrtokenizers.modelsrr masking_utilsrmodeling_outputsrmodeling_utilsr r processing_utilsr tokenization_utils_tokenizersr utilsr rr utils.genericrutils.output_capturingrllama.modeling_llamarrrrparakeet.configuration_parakeetrrparakeet.modeling_parakeetrrrrparakeet.processing_parakeetrt5.tokenization_t5rrrOrQrvModulerrrrrrrr__all__r-r-r-r6sH            7 9*1[