o ei@sddlZddlmZddlZddlZddlmZddlmZddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZmZddlmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'e#(e)Z*GdddeZ+GdddeZ,GdddeZ-Gdddej.Z/Gdddej.Z0Gdddej.Z1Gdd d ej.Z2  !dGd"ej.d#ej3d$ej3d%ej3d&ej3dBd'e4dBd(e4d)ee!fd*d+Z5Gd,d-d-ej.Z6Gd.d/d/ej.Z7Gd0d1d1eZ8Gd2d3d3ej.Z9e"Gd4d5d5eZ:  dHd6e;ef d;d<Z?e"Gd=d>d>e:Z@dZAe"d?d@GdAdBdBe:ZBe"dCd@GdDdEdEe:ZCgdFZDdS)IN)Callable)nn)CrossEntropyLoss)initialization)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel*get_torch_context_manager_or_global_device)Unpack)TransformersKwargsauto_docstringlogging)is_flash_attention_requested) SEWConfigc&eZdZdfdd ZddZZS)SEWNoLayerNormConvLayerrcsjt|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvrfeat_extract_activation activationselfconfiglayer_id __class__b/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/sew/modeling_sew.pyr /s  z SEWNoLayerNormConvLayer.__init__cCs||}||}|SN)r(r*r, hidden_statesr1r1r2forward=s  zSEWNoLayerNormConvLayer.forwardr__name__ __module__ __qualname__r r6 __classcell__r1r1r/r2r.srcr)SEWLayerNormConvLayerrcs|t|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)NrrrT)elementwise_affine)rr r!r"r#rr$r%r&r'r( LayerNorm layer_normrr)r*r+r/r1r2r Ds  zSEWLayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)N)r( transposer@r*r4r1r1r2r6Ss     zSEWLayerNormConvLayer.forwardr7r8r1r1r/r2r=Csr=cr)SEWGroupNormConvLayerrcst|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)NrrrT) num_groups num_channelsaffine)rr r!r"r#rr$r%r&r'r(rr)r* GroupNormr@r+r/r1r2r _s   zSEWGroupNormConvLayer.__init__cC"||}||}||}|Sr3)r(r@r*r4r1r1r2r6os   zSEWGroupNormConvLayer.forwardr7r8r1r1r/r2rD^srDc$eZdZfddZddZZS)SEWPositionalConvEmbeddingcs(ttj|j|j|j|jd|j|jd|_tj j }t tj j dr)tj j j }t r}ddl}|jj|jjdd||jddd|_Wdn1sNwYt |jdrf|jj jj}|jj jj}n|jj}|jj}|j|||j||n ||jddd|_t|j|_t|j|_dS) N)rpaddinggroupsr weight_normr modifier_rankweight)namedimparametrizations)rr rr$ hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupssqueeze_factorr(utilsrOhasattrrUr deepspeedzeroGatheredParametersrR original0 original1weight_gweight_vregister_external_parameterSEWSamePadLayerrMrr)r*)r,r-rOr\rarbr/r1r2r ws6      z#SEWPositionalConvEmbedding.__init__cCrIr3)r(rMr*r4r1r1r2r6s   z"SEWPositionalConvEmbedding.forwardr8r1r1r/r2rKvs "rKcrJ)rdcs*t|ddkrd|_dSd|_dS)NrLrr)rr num_pad_remove)r,rWr/r1r2r s  zSEWSamePadLayer.__init__cCs,|jdkr|ddddd|j f}|SNr)rer4r1r1r2r6s zSEWSamePadLayer.forwardr8r1r1r/r2rds rdcrJ) SEWUpsamplingcs:tt|j|j|j|_t|j|_ |j|_dSr3) rr rLinearrVrY projectionrr)r*r,r-r/r1r2r s   zSEWUpsampling.__init__cCsd||}||}|jdkr0|\}}}||j}||j}||||j|}||||}|S)Nr)rir*rYsizereshape)r,r5bszsrc_len src_embed_dimtgt_len tgt_embed_dimr1r1r2r6s     zSEWUpsampling.forwardr8r1r1r/r2rgs rgcs0eZdZdZfddZddZddZZS)SEWFeatureEncoderz.Construct the features from raw audio waveformcstjdkr tddgfddtjdD}njdkr2fddtjD}n td jd t||_ d |_ d |_ dS) Ngrouprr.csg|] }t|ddqS)rrt)r.0ir-r1r2 sz.SEWFeatureEncoder.__init__..rlayercsg|]}t|dqS)rt)r=rurxr1r2rysz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rr feat_extract_normrDrangenum_feat_extract_layers ValueErrorr ModuleList conv_layersgradient_checkpointing_requires_grad)r,r-rr/rxr2r s       zSEWFeatureEncoder.__init__cCs|D]}d|_qd|_dSNF) parameters requires_gradrr,paramr1r1r2_freeze_parameterss  z$SEWFeatureEncoder._freeze_parameterscCs:|dddf}|jr|jrd|_|jD]}||}q|S)NT)rtrainingrr)r, input_valuesr5 conv_layerr1r1r2r6s    zSEWFeatureEncoder.forward)r9r:r;__doc__r rr6r<r1r1r/r2rrs  rrmodulequerykeyvalueattention_maskscalingdropoutkwargsc Ks|dur |dd}t||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd } | |fS)NrBrLrrT)prr) rktorchmatmulrCr functionalsoftmaxrr contiguous) rrrrrrrr attn_weights attn_outputr1r1r2eager_attention_forwards  rcseZdZdZ     ddededed ed ed ed edBffd d Z   dde j de j dBde j dBdedBde e de e j e j dBe e j dBff ddZZS) SEWAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr is_decoderr is_causalr-cst||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)r)rr rrrhead_dimr-r~rrrrrhk_projv_projq_projout_proj)r,rrrrrrr-r/r1r2r s&    zSEWAttention.__init__r5key_value_statesroutput_attentionsrreturncKs|du}|jdd\}}|r|jdn|} ||d|jf} || d|jf} ||j| dd} |r4|n|} || j| dd}|| j| dd}t|j j t }||| |||f|j sbdn|j |j|d|\}}|||d}||}||dfS)z#Input shape: Batch x Time x ChannelNrBrrLr)rrr)shaperrviewrCrrr get_interfacer-_attn_implementationrrrrrlrr)r,r5rrrris_cross_attentionrmrprn q_input_shapekv_input_shape query_statescurrent_states key_states value_statesattention_interfacerrr1r1r2r6&s8    zSEWAttention.forward)rFTFN)NNF)r9r:r;rintfloatboolrr rTensorrr tupler6r<r1r1r/r2rsL" rcrJ)SEWFeedForwardcsptt|j|_t|j|j|_ t |j t r"t |j |_n|j |_t|j|j|_t|j|_dSr3)rr rDropoutactivation_dropoutintermediate_dropoutrhrVintermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrjr/r1r2r [s  zSEWFeedForward.__init__cCs6||}||}||}||}||}|Sr3)rrrrrr4r1r1r2r6hs     zSEWFeedForward.forwardr8r1r1r/r2rZs  rcs&eZdZfddZdddZZS)SEWEncoderLayercshtt|j|j|jd|d|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_dS)NF)rrrrr-eps)rr rrVnum_attention_headsattention_dropout attentionrrrrr?layer_norm_epsr@r feed_forwardfinal_layer_normrjr/r1r2r ss  zSEWEncoderLayer.__init__NFcCsf|}|j|||d\}}}||}||}||}|||}||}|f}|r1||f7}|S)Nrr)rrr@rr)r,r5rr attn_residualr_outputsr1r1r2r6s    zSEWEncoderLayer.forwardrr8r1r1r/r2rrs rcs.eZdZfddZ    dddZZS) SEWEncodercst|_t|_tjj|_tj j j d|_ t j|_tfddtjD|_t|_d|_dS)Nrcsg|]}tqSr1)rrvrrxr1r2rysz'SEWEncoder.__init__..F)rr r-rKpos_conv_embedr AvgPool1drYpoolr?rVrr@rrrrr|num_hidden_layerslayersrgupsamplerrjr/rxr2r s     zSEWEncoder.__init__NFTc Cs|rdnd}|r dnd}|dur|ddd|jd}t|jr4d||<|dur1d|vr1|nd}nfd||<|d} | |jj} |jd|jj} tj d| | j d dd | jdd} | | ddk}d|ddddddfj |jd }|t|jj}| |jdd|jd|jd}|jd} |dd}||}||}t|d|d}|d d|f|d d|f}|dd}||}||}tpt|}|jD]7}|r||f}tg}|jo||jjk}|r|r ||||d }|d}|rd }|r||df}q|r$||f}||}|jd| krAtj |ddd| |jdf}|sPt!d d|||fDSt"|||dS)Nr1rBrrLrrdeviceg?dtype.rNNcss|] }|dur|VqdSr3r1)rvvr1r1r2 sz%SEWEncoder.forward..last_hidden_stater5 attentions)# unsqueezerepeatrrr-longsumrYrarangerrexpandtorfinfominrCrrrkr@rrr rrandr layerdroprrrpadrr )r,r5rroutput_hidden_states return_dictall_hidden_statesall_self_attentionsexpand_attention_mask input_lengthsoutput_lengthsmax_encoder_length attention_idsn_input_timestepsposition_embeddingspooled_hidden_states min_length synced_gpusrzdropout_probabilityskip_the_layer layer_outputsr1r1r2r6st      &               zSEWEncoder.forward)NFFTr8r1r1r/r2rs rc@sjeZdZUeed<dZdZdZdZdZ dZ dZ e ddZd e jeBfd d Zd ed e jfddZdS)SEWPreTrainedModelr-sewraudioTFc Cst|tr'tj|jjddtd|jjd|jj dt |jj dnt|t j r9tj|jd|jjdnvt|t jt jfrOt|j t|jn`t|t jrtrddl}t|drt|dr|jj|j|jgdd t|jWdn1swYn&|jj|jdd t|jWdn1swYnt|jt|t j t jfr|j durt|j dSdSdS) zInitialize the weightsrrLr)meanstdrNrbrarP)rrKinitnormal_r(rRmathsqrtr in_channels constant_rrrhr-initializer_ranger?rHzeros_ones_r$rr\r[r]r^rbrakaiming_normal_)r,rr\r1r1r2 _init_weightss8      z SEWPreTrainedModel._init_weightsrcCs4dd}t|jj|jjD] \}}||||}q |S)zH Computes the output length of the convolutional layers cSstj|||dddS)Nfloor) rounding_moder)rdiv) input_lengthrrr1r1r2_conv_out_length+szMSEWPreTrainedModel._get_feat_extract_output_lengths.._conv_out_length)zipr-r%r&)r,rrrrr1r1r2 _get_feat_extract_output_lengths&sz3SEWPreTrainedModel._get_feat_extract_output_lengthsfeature_vector_lengthrcCs~||dtj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dg d dg }|S)NrBr)rrrr) rrrrrrzerosrrrflipcumsumr)r,rrr batch_sizer1r1r2"_get_feature_vector_attention_mask5s "z5SEWPreTrainedModel._get_feature_vector_attention_maskN)r9r:r;r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnrno_gradr LongTensorrrr$r1r1r1r2rs  rr mask_prob mask_length min_masksrcs|\}dkr tdkrtdddtjdfdd}|dur:|d n fd d t|D}tj |ft d }g} |} | d krZ|S|D];} || } tjj t | d| dd} t | d kr}d}n| d }t| tj| | tjd |g} | | q\t| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkrd| | dk<t|| dd |S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr}|d|kr*t|dd}|S)z;Given input length, compute how many spans should be maskedrr)rmax)rnum_masked_spanepsilonr0r/r1sequence_lengthr1r2compute_num_masked_spanhs  z6_compute_mask_indices..compute_num_masked_spanNrBcsg|]}qSr1r1r)r7r1r2ry{sz)_compute_mask_indices..rrF)replace)r~nprandomritemdetachrtolistr|r rchoicerlen concatenateonesint32appendarray broadcast_torlr3put_along_axis)rr/r0rr1r#r8r spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrr4spec_aug_mask_idxdummy_mask_idxoffsetsr1r5r2_compute_mask_indicesBs\    rNcseZdZdeffdd Z  ddejdejdBdejdBfdd Ze     dd ej dBdej dBdejdBd e dBd e dBd e dBde e BfddZZS)SEWModelr-cst|||_t||_tj|jd|jd|_ |jd|j k|_ |j r1t |jd|j |_ t|j|_|jdksB|jdkrNtt|j |_t||_|dS)NrBrr)rr r-rrfeature_extractorrr?r!rr@rVproject_featuresrhfeature_projectionrfeat_proj_dropoutfeature_dropoutmask_time_probmask_feature_prob Parameterrruniform_masked_spec_embedrencoder post_initrjr/r1r2r s    zSEWModel.__init__Nr5mask_time_indicesrcCst|jdds |S|\}}}|dur|j|j||<n-|jjdkrK|jrKt||f|jj|jj ||jj d}t j ||j t jd}|j|j||<|jjdkr|jrt||f|jj|jj|jjd}t j ||j t jd}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)r/r0rr1)rr)r/r0r1rB)getattrr-rkrYrrrUrrNmask_time_lengthmask_time_min_masksrtensorrrrVmask_feature_lengthmask_feature_min_masksr)r,r5r\rr#r7rVmask_feature_indicesr1r1r2_mask_hidden_statess4 zSEWModel._mask_hidden_statesrrrrrc Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}||}|dd}||}|jr6||}| |} |durH| | j d|}|j | |d} |j | ||||d} | d} |sh| f| ddSt| | j| jdS)a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. NrrL)r\rrrrrr)r-rruse_return_dictrPrCr@rQrRrTr$rrerZr r5r) r,rrr\rrrrextract_featuresr5encoder_outputsr1r1r2r6s8     zSEWModel.forwardrNNNNN)r9r:r;rr r FloatTensorr.rerrrrr r6r<r1r1r/r2rOs@ . rOzk SEW Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ) custom_introcseZdZddedBffdd ZddZddZd d Ze     dd e j dBd e j dBd e dBde dBde dBde j dBde e BfddZZS) SEWForCTCN target_langcs~t|t||_t|j|_||_|j dur#t d|j dt |dr.|j r.|jn|j}t||j |_|dS)a- target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`SEWForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SEWForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)rr rOr rr final_dropoutrrn vocab_sizer~r0r[rooutput_hidden_sizerVrhlm_headr[)r,r-rnrrr/r1r2r >s     zSEWForCTC.__init__cKsttdkr dS|j}|dur"t|jdddur"td|d|dur6t|jdddur6tddS|durC|j |dddSdS) a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. metaNadapter_attn_dimzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load) rrrrnr^r-r~loggerinfo load_adapter)r,rrnr1r1r2 tie_weights[szSEWForCTC.tie_weightscC|jjdSz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. Nr rPrr,r1r1r2freeze_feature_encodersz SEWForCTC.freeze_feature_encodercC|jD]}d|_qdSz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNr rrrr1r1r2freeze_base_modelzzSEWForCTC.freeze_base_modelrrrrrlabelsrc Ks||dur|n|jj}|dur||jjkrtd|jj|j|||||d}|d} || } || } d} |dur|durC|ntj |tj d}| | d tj } |dk} | d}|| }tjj| dtjddd}tjjjd d tjj||| ||jj|jj|jjd } Wdn1swY|s| f|td}| dur| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rfrrrB)rTrrF)enabled)blank reduction zero_infinitylosslogitsr5r)r-rgr3rqr~r rrsr ones_likerrrr masked_selectrr log_softmaxfloat32rCbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONr r5r)r,rrrrrrrrr5rrr labels_masktarget_lengthsflattened_targets log_probsoutputr1r1r2r6sN      zSEWForCTC.forwardr3rj)r9r:r;rr rzrrrrrrrr r6r<r1r1r/r2rm8s4 rmz SEW Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. cseZdZfddZddZddZe     ddejdBd ejdBd e dBd e dBd e dBd ejdBde e BfddZ Z S)SEWForSequenceClassificationcst|t|dr|jrtdt||_|jd}|jr*t t |||_ t |j|j|_t |j|j|_|dS)NrozZSequence classification does not support the use of SEW adapters (config.add_adapter=True)r)rr r[ror~rOr ruse_weighted_layer_sumrrWrrB layer_weightsrhrVclassifier_proj_size projector num_labels classifierr[)r,r- num_layersr/r1r2r s    z%SEWForSequenceClassification.__init__cCr{r|r}r~r1r1r2rrz3SEWForSequenceClassification.freeze_feature_encodercCrrrrr1r1r2rrz.SEWForSequenceClassification.freeze_base_modelNrrrrrrrcKsz|dur|n|jj}|jjrdn|}|j|||||d}|jjrB|t} tj| dd} tjj |j dd} | | dddj dd} n|d} | | } |durV| jdd} n+|| jd|} | ddd| jd} d | | <| j dd| j dd dd} || }d}|durt}|| d|jj| d}|s|f|td}|dur|f|S|St|||j|jd S) a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SEWProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrfrrrBrrLrr)r-rgrr rrstackrrrrrrrr r$rrrrrrrr5r)r,rrrrrrrrr5 norm_weights pooled_output padding_maskexpand_padding_maskrrloss_fctrr1r1r2r6sH    z$SEWForSequenceClassification.forwardrj)r9r:r;r rrrrrrrrr6r<r1r1r/r2rs2  r)rmrrOr)Nrrf)Ercollections.abcrnumpyr:rrtorch.nnrrr  activationsrintegrations.deepspeedrintegrations.fsdpr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr r rmodeling_utilsrrrprocessing_utilsrrZrrr utils.genericrconfiguration_sewr get_loggerr9rwrr=rDModulerKrdrgrrrrrrrrrrrrr.ndarrayrNrOrrmr__all__r1r1r1r2s             + , V$fI  w{ h