o ei@s,ddlZddlmZddlmZddlZddlZddlm Z ddlm Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+e),e-Z.ee(ddGdddeZ/Gddde j0Z1Gddde j0Z2GdddeZ3GdddeZ4Gdd d eZ5Gd!d"d"e j0Z6Gd#d$d$e j0Z7  %dVd&e j0d'ej8d(ej8d)ej8d*ej8dBd+e9dBd,e9d-e%e'fd.d/Z:Gd0d1d1e j0Z;Gd2d3d3e j0ZGd8d9d9e j0Z?Gd:d;d;eZ@Gdd?d?e j0ZBe(Gd@dAdAe"ZC  dWdBeDeEeEfdCe9dDeEd*ejFdBdEeEdFejGf dGdHZHeZIe(GdIdJdJeCZJe(dKdGdLdMdMeCZKdNZLe(dOdGdPdQdQeCZMe(dRdGdSdTdTeCZNgdUZOdS)XN)Callable) dataclass)CrossEntropyLoss)initialization)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutput ModelOutputSequenceClassifierOutputWav2Vec2BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel*get_torch_context_manager_or_global_device)Unpack)TransformersKwargsauto_docstringlogging)UniSpeechConfigzh Output type of [`UniSpeechForPreTrainingOutput`], with potential hidden states and attentions. ) custom_introc@seZdZUdZdZejdBed<dZejdBed<dZ ejdBed<dZ ejdBed<dZ e ejdBed<dZ e ejdBed<dS) UniSpeechForPreTrainingOutputa loss (*optional*, returned when model is in train mode, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentions)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrr r!tupler"r+r+n/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/unispeech/modeling_unispeech.pyr5s rc$eZdZfddZddZZS)UniSpeechSamePadLayercs*t|ddkrd|_dSd|_dS)Nrr)super__init__num_pad_remove)selfnum_conv_pos_embeddings __class__r+r,r1Ss  zUniSpeechSamePadLayer.__init__cCs,|jdkr|ddddd|j f}|SNr)r2r3r!r+r+r,forwardWs zUniSpeechSamePadLayer.forwardr#r$r%r1r9 __classcell__r+r+r5r,r.Rs r.cr-) UniSpeechPositionalConvEmbeddingcs$ttj|j|j|j|jd|jd|_tjj }t tjj dr'tjj j }t r{ddl }|jj|jjdd||jddd|_Wdn1sLwYt |jdrd|jj jj}|jj jj}n|jj}|jj}|j|||j||n ||jddd|_t|j|_t|j|_dS) Nr/) kernel_sizepaddinggroups weight_normr) modifier_rankweight)namedimparametrizations)r0r1nnConv1d hidden_sizer4num_conv_pos_embedding_groupsconvutilsr@hasattrrEr deepspeedzeroGatheredParametersrB original0 original1weight_gweight_vregister_external_parameterr.r>rfeat_extract_activation activation)r3configr@rMrRrSr5r+r,r1^s4     z)UniSpeechPositionalConvEmbedding.__init__cCs:|dd}||}||}||}|dd}|S)Nrr/) transposerJr>rVr8r+r+r,r9s     z(UniSpeechPositionalConvEmbedding.forwardr:r+r+r5r,r<]s !r<c&eZdZdfdd ZddZZS)UniSpeechNoLayerNormConvLayerrcsjt|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrrr=stridebias)r0r1conv_dim in_conv_dim out_conv_dimrFrG conv_kernel conv_stride conv_biasrJrrUrVr3rWlayer_idr5r+r,r1s  z&UniSpeechNoLayerNormConvLayer.__init__cCs||}||}|SN)rJrVr8r+r+r,r9s  z%UniSpeechNoLayerNormConvLayer.forwardrr:r+r+r5r,rZsrZcrY)UniSpeechLayerNormConvLayerrcs|t|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)Nrrr[T)elementwise_affine)r0r1r^r_r`rFrGrarbrcrJ LayerNorm layer_normrrUrVrdr5r+r,r1s  z$UniSpeechLayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)N)rJrXrkrVr8r+r+r,r9s     z#UniSpeechLayerNormConvLayer.forwardrgr:r+r+r5r,rhsrhcrY)UniSpeechGroupNormConvLayerrcst|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)Nrrr[T) num_groups num_channelsaffine)r0r1r^r_r`rFrGrarbrcrJrrUrV GroupNormrkrdr5r+r,r1s   z$UniSpeechGroupNormConvLayer.__init__cCs"||}||}||}|Srf)rJrkrVr8r+r+r,r9s   z#UniSpeechGroupNormConvLayer.forwardrgr:r+r+r5r,rnsrncs0eZdZdZfddZddZddZZS)UniSpeechFeatureEncoderz.Construct the features from raw audio waveformcstjdkr tddgfddtjdD}njdkr2fddtjD}n td jd t||_ d |_ d |_ dS) Ngrouprrecsg|] }t|ddqS)rru)rZ.0irWr+r, sz4UniSpeechFeatureEncoder.__init__..rlayercsg|]}t|dqS)ru)rhrvryr+r,rzs z`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) r0r1feat_extract_normrnrangenum_feat_extract_layers ValueErrorrF ModuleList conv_layersgradient_checkpointing_requires_grad)r3rWrr5ryr,r1s        z UniSpeechFeatureEncoder.__init__cCs|D]}d|_qd|_dSNF) parameters requires_gradrr3paramr+r+r,_freeze_parameterss  z*UniSpeechFeatureEncoder._freeze_parameterscCs:|dddf}|jr|jrd|_|jD]}||}q|S)NT)rtrainingrr)r3 input_valuesr! conv_layerr+r+r,r9s    zUniSpeechFeatureEncoder.forward)r#r$r%r&r1rr9r;r+r+r5r,rss  rscr-)UniSpeechFeatureProjectioncsJttj|jd|jd|_t|jd|j|_ t |j |_ dS)Nrmeps) r0r1rFrjr^layer_norm_epsrkLinearrH projectionDropoutfeat_proj_dropoutdropoutr3rWr5r+r,r1s z#UniSpeechFeatureProjection.__init__cCs&||}||}||}||fSrf)rkrr)r3r!norm_hidden_statesr+r+r,r9s   z"UniSpeechFeatureProjection.forwardr:r+r+r5r,rs rmodulequerykeyvalueattention_maskscalingrkwargsc Ks|dur |dd}t||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd } | |fS)Nrmr/rrD)prr) sizer'matmulrXrF functionalsoftmaxrr contiguous) rrrrrrrr attn_weights attn_outputr+r+r,eager_attention_forward s  rcseZdZdZ     ddededed ed ed ed edBffd d Z   dde j de j dBde j dBdedBde e de e j e j dBe e j dBff ddZZS)UniSpeechAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr is_decoderr] is_causalrWcst||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)r])r0r1rrrhead_dimrWrrrrrFrk_projv_projq_projout_proj)r3rrrrr]rrWr5r+r,r1)s&    zUniSpeechAttention.__init__r!key_value_statesroutput_attentionsrreturncKs|du}|jdd\}}|r|jdn|} ||d|jf} || d|jf} ||j| dd} |r4|n|} || j| dd}|| j| dd}t|j j t }||| |||f|j sbdn|j |j|d|\}}|||d}||}||dfS)z#Input shape: Batch x Time x ChannelNrmrr/r)rrr)shaperrviewrXrrr get_interfacerW_attn_implementationrrrrreshaperr)r3r!rrrris_cross_attentionbsztgt_lensrc_len q_input_shapekv_input_shape query_statescurrent_states key_states value_statesattention_interfacerrr+r+r,r9Hs8    zUniSpeechAttention.forward)rFTFN)NNF)r#r$r%r&intfloatboolrr1r'Tensorrr r*r9r;r+r+r5r,r&sL" rcr-)UniSpeechFeedForwardcsptt|j|_t|j|j|_ t |j t r"t |j |_n|j |_t|j|j|_t|j|_dSrf)r0r1rFractivation_dropoutintermediate_dropoutrrHintermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrr5r+r,r1}s  zUniSpeechFeedForward.__init__cCs6||}||}||}||}||}|Srf)rrrrrr8r+r+r,r9s     zUniSpeechFeedForward.forwardr:r+r+r5r,r|s  rcs&eZdZfddZdddZZS)UniSpeechEncoderLayercshtt|j|j|jd|d|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_dS)NFrrrrrWr)r0r1rrHnum_attention_headsattention_dropout attentionrFrrrrjrrkr feed_forwardfinal_layer_normrr5r+r,r1s  zUniSpeechEncoderLayer.__init__NFcCsf|}|j|||d\}}}||}||}||}|||}||}|f}|r1||f7}|SNrr)rrrkrrr3r!rr attn_residualr_outputsr+r+r,r9s    zUniSpeechEncoderLayer.forwardrr:r+r+r5r,rs rc sLeZdZfddZ    d dejdejdBded ed ef d d ZZ S)UniSpeechEncodercft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nrcg|]}tqSr+)rrwrryr+r,rzz-UniSpeechEncoder.__init__..Fr0r1rWr<pos_conv_embedrFrjrHrrkrrrrr}num_hidden_layerslayersrrr5ryr,r1s    zUniSpeechEncoder.__init__NFTr!rroutput_hidden_states return_dictcCs.|rdnd}|r dnd}|dur"|ddd|jd}d||<t|j||d}||} || }||}||}tpCt |} |j D]3} |rP||f}t g} |j o]| |jjk} | rb| rm| |||d}|d}| rqd}|rz||df}qG|r||f}|std d |||fDSt|||d S) Nr+rmrr/rrW inputs_embedsrrNNcs|] }|dur|VqdSrfr+rwvr+r+r, z+UniSpeechEncoder.forward..last_hidden_stater!r") unsqueezerepeatrr rWrrkrrr rr'randr layerdropr*r r3r!rrrrall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusr{dropout_probabilityskip_the_layer layer_outputsr+r+r,r9sN          zUniSpeechEncoder.forwardNFFT) r#r$r%r1r'tensorrrr9r;r+r+r5r,rs"  rcs,eZdZfddZdejfddZZS)UniSpeechAttnAdapterLayercsZt|j|_|j|_t|j|_t |j|j|_ t |_ t |j|j|_ dS)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)r0r1adapter_attn_dim input_dimrH hidden_dimrFrjnormrlinear_1ReLUact_fnlinear_2rr5r+r,r1s  z"UniSpeechAttnAdapterLayer.__init__r!cCs,||}||}||}||}|Srf)r rrrr8r+r+r,r9s    z!UniSpeechAttnAdapterLayer.forward)r#r$r%r1r'r(r9r;r+r+r5r,r s r cs@eZdZfddZ  d dejdejdBdefdd ZZS) $UniSpeechEncoderLayerStableLayerNormcstt|j|j|jd|d|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_t|dddurAt||_dSd|_dS)NFrrr )r0r1rrHrrrrFrrrrjrrkrrrgetattrr  adapter_layerrr5r+r,r1s   z-UniSpeechEncoderLayerStableLayerNorm.__init__NFr!rrcCsz|}||}|j|||d\}}}||}||}||||}|jdur1|||}|f}|r;||f7}|Sr)rkrrrrrrr+r+r,r9-s    z,UniSpeechEncoderLayerStableLayerNorm.forwardr) r#r$r%r1r'rrr9r;r+r+r5r,rs rcs.eZdZfddZ    dddZZS) UniSpeechEncoderStableLayerNormcr)Nrcrr+)rrryr+r,rzOrz.Frrr5ryr,r1Hs   z(UniSpeechEncoderStableLayerNorm.__init__NFTcCs.|rdnd}|r dnd}|dur"|ddd|jd}d||<t|j||d}||} || }||}tp>t|} |j D]3} |rK||f}t g} |j oX| |jj k} | r]| rh| |||d}|d}| rld}|ru||df}qB||}|r||f}|std d |||fDSt|||d S) Nr+rmrr/rrrrcsrrfr+rr+r+r,rrz:UniSpeechEncoderStableLayerNorm.forward..r)rrrr rWrrrr rr'rrrrkr*r rr+r+r,r9SsN          z'UniSpeechEncoderStableLayerNorm.forwardrr:r+r+r5r,rGs rcs4eZdZdZfddZeddZddZZS)UniSpeechGumbelVectorQuantizerz Vector quantization using gumbel softmax. See `[CATEGORICAL REPARAMETERIZATION WITH GUMBEL-SOFTMAX](https://huggingface.co/papers/1611.01144) for more information. cst|j|_|j|_|j|jdkr"td|jd|jdt t d|j|j|j|j|_ t |jd|j|j|_d|_dS)Nrz`config.codevector_dim z5 must be divisible by `config.num_codevector_groups` z for concatenationrrmr/)r0r1num_codevector_groupsronum_codevectors_per_groupnum_varscodevector_dimrrF Parameterr'r( codevectorsrr^ weight_proj temperaturerr5r+r,r1s   z'UniSpeechGumbelVectorQuantizer.__init__cCs2|jdd}ttjt||dd }|S)Nrrrm)meanr'expsumxlogy)probsmarginal_probs perplexityr+r+r,_compute_perplexitys "z2UniSpeechGumbelVectorQuantizer._compute_perplexityc Cs|j\}}}||}||||jd}|jr?tjj||j dd |}t j ||||jddd}| |}n$|jdd}|j|jd|ddd}||||jd}| |}|||d}|d|j} | |||j|jd} | d||d} | |fS)NrmT)tauhardrrg?rl)rrrrorrFrgumbel_softmaxrrtype_asr'rr&argmax new_zerosscatter_rrrr!) r3r! batch_sizesequence_lengthrHcodevector_probscodevector_soft_distr%codevector_idxcodevectors_per_grouprr+r+r,r9s0       z&UniSpeechGumbelVectorQuantizer.forward) r#r$r%r&r1 staticmethodr&r9r;r+r+r5r,rs   rc@sjeZdZUeed<dZdZdZdZdZ dZ dZ e ddZde jeBfd d Zd ed e jfd dZdS)UniSpeechPreTrainedModelrW unispeechraudioTc Cst|trtj|jjdddt|jjt|j dSt|t rFtj|j jddt d|j jd|j jdt|j jddSt|trlt d|jj}tj|jj| |dtj|jj| |ddSt|tjrtj|jd|jjd|jdurt|jdSdSt|tjtjfrt|jt|jdSt|tjrt|j|jdurt |j|j|jd}tj|j| |ddSdSdS)zInitialize the weightsrr)rstdrr/)abN)rrinitnormal_rrBzeros_r]uniform_rr<rJmathsqrtr= in_channels constant_rr in_featuresrFrrWinitializer_rangerjrrones_rGkaiming_normal_r?)r3rkr+r+r, _init_weightss<          z&UniSpeechPreTrainedModel._init_weights input_lengthscCs4dd}t|jj|jjD] \}}||||}q |S)zH Computes the output length of the convolutional layers cSstj|||dddS)Nfloor) rounding_moder)r'div) input_lengthr=r\r+r+r,_conv_out_length szSUniSpeechPreTrainedModel._get_feat_extract_output_lengths.._conv_out_length)ziprWrarb)r3rIrNr=r\r+r+r, _get_feat_extract_output_lengthssz9UniSpeechPreTrainedModel._get_feat_extract_output_lengthsfeature_vector_lengthrcCs|jdddddf}||tj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dgd dg }|S)Nrmrr)dtypedevicer)rS) cumsumrPtor'longrzerosrRrSarangeflipr)r3rQrnon_padded_lengthsoutput_lengthsr.r+r+r,"_get_feature_vector_attention_masks "z;UniSpeechPreTrainedModel._get_feature_vector_attention_maskN)r#r$r%rr)base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr'no_gradrH LongTensorrrPr\r+r+r+r,r5s  !r5r mask_prob mask_length min_masksrcs|\}dkr tdkrtdddtjdfdd}|dur:|d n fd d t|D}tj |ft d }g} |} | d krZ|S|D];} || } tjj t | d| dd} t | d kr}d}n| d }t| tj| | tjd |g} | | q\t| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkrd| | dk<t|| dd |S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr}|d|kr*t|dd}|S)z;Given input length, compute how many spans should be maskedrr)rmax)rMnum_masked_spanepsilonrgrfrhr/r+r,compute_num_masked_spanKs  z6_compute_mask_indices..compute_num_masked_spanNrmcsg|]}qSr+r+r)r/r+r,rz^sz)_compute_mask_indices..rRrF)replace)rnprandomritemdetachr!tolistr}rWrchoicerXlen concatenateonesint32appendarray broadcast_torrjput_along_axis)rrfrgrrhr.rnrI spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrMrkspec_aug_mask_idxdummy_mask_idxoffsetsr+rlr,_compute_mask_indices%s\    rcseZdZdeffdd Z  ddejdejdBdejdBfdd Ze     dd ej dBdej dBdejdBd e dBd e dBd e dBde e BfddZZS)UniSpeechModelrWcszt|||_t||_t||_|jdks|jdkr)t t |j |_|jr2t||_nt||_|dSNr)r0r1rWrsfeature_extractorrfeature_projectionmask_time_probmask_feature_probrFrr'rrHr>masked_spec_embeddo_stable_layer_normrencoderr post_initrr5r+r,r1s      zUniSpeechModel.__init__Nr!mask_time_indicesrcCst|jdds |S|\}}}|dur|j|j||<n-|jjdkrK|jrKt||f|jj|jj ||jj d}t j ||j t jd}|j|j||<|jjdkr|jrt||f|jj|jj|jjd}t j ||j t jd}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)rfrgrrh)rSrR)rfrgrhrm)rrWrrrUrRrrrmask_time_lengthmask_time_min_masksr'rrSrrmask_feature_lengthmask_feature_min_masksexpand)r3r!rrr.r/rHmask_feature_indicesr+r+r,_mask_hidden_statess4 z"UniSpeechModel._mask_hidden_statesrrrrrc Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}||}|dd}|dur6||jd|}||\} }|j | ||d} |j | ||||d} | d} |s_| |f| ddSt | || j | j dS)a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Nrr/)rrrrrrr)rextract_featuresr!r")rWrruse_return_dictrrXr\rrrrUniSpeechBaseModelOutputr!r") r3rrrrrrrrr!encoder_outputsr+r+r,r9s8  zUniSpeechModel.forwardrNNNNN)r#r$r%rr1r'r(rerrrrr*rr9r;r+r+r5r,rs@ . rzZ UniSpeech Model with a vector-quantization module and ctc loss for pre-training. cseZdZdeffdd ZdefddZddZe dd e j d e j d e j defd dZ e    dde j dBde j dBdedBdedBdedBdeeBf ddZZS)UniSpeechForPreTrainingrWcs~t|t||_t|j|_t||_ t |j |j |_ t |j |j|_t |j|j|_t|j|_|dSrf)r0r1rr6rFrfeat_quantizer_dropoutdropout_featuresr quantizerrrproj_codevector_dim project_qrH project_hidnum_ctc_classesctc_proj final_dropoutrrrr5r+r,r1s    z UniSpeechForPreTraining.__init__rcCs ||j_dS)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rr)r3rr+r+r,set_gumbel_temperature,s z.UniSpeechForPreTraining.set_gumbel_temperaturecC|jjdSz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. Nr6rrr3r+r+r,freeze_feature_encoder2z.UniSpeechForPreTraining.freeze_feature_encoderrtarget_featuresnegative_featurespredicted_featurescCs@tj||gdd}tj||dd}||}||}|S)z Compute logits for contrastive loss based using cosine similarity as the distance measure between `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied. rrrm)r'catcosine_similarityrr*)rrrrlogitsr+r+r,compute_contrastive_logits9s   z2UniSpeechForPreTraining.compute_contrastive_logitsNrrrrrrcKsJ|dur|n|jj}|j|||||d}|d}||d} || \} } || |jjj} | | } t | d| d |jj} | dd} t | |j} | dd} | d} || d| | d}||}||}d}|s|dur||| | f|ddS|| | f|ddSt||| | |j|jdS) a Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, UniSpeechForPreTraining >>> feature_extractor = AutoFeatureExtractor.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> model = UniSpeechForPreTraining.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> # TODO: Add full pretraining example ```Nrrrrmrr/)rrrr r!r")rWrr6rrrrUrBrRrr'emptyrfill_ replace_probrX bernoullirrSr masked_fillrrrr!r")r3rrrrrrrtransformer_featuresrquantized_featuresr prob_replace_matrixsampled_replace_matrixrrr+r+r,r9MsL        zUniSpeechForPreTraining.forward)r)NNNN)r#r$r%rr1rrrr4r'r(rrrrr*rr9r;r+r+r5r,rsB rr/zq UniSpeech Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). cseZdZddedBffdd ZddZddZd d Ze     dd e j dBd e j dBd e dBde dBde dBde j dBde e BfddZZS)UniSpeechForCTCN target_langcs~t|t||_t|j|_||_|j dur#t d|j dt |dr.|j r.|jn|j}t||j |_|dS)a3 target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`UniSpeechForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `UniSpeechForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)r0r1rr6rFrrrr vocab_sizerr6rLroutput_hidden_sizerHrlm_headr)r3rWrrr5r+r,r1s     zUniSpeechForCTC.__init__cKsttdkr dS|j}|dur"t|jdddur"td|d|dur6t|jdddur6tddS|durC|j |dddSdS) a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. metaNr zCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load) rr'rSrrrWrloggerinfo load_adapter)r3rrr+r+r, tie_weightsszUniSpeechForCTC.tie_weightscCrrrrr+r+r,rrz&UniSpeechForCTC.freeze_feature_encodercC|jD]}d|_qdSz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNr6rrrr+r+r,freeze_base_modelz!UniSpeechForCTC.freeze_base_modelrrrrrlabelsrc Ks||dur|n|jj}|dur||jjkrtd|jj|j|||||d}|d} || } || } d} |dur|durC|ntj |tj d}| | d tj } |dk} | d}|| }tjj| dtjddd}tjjjd d tjj||| ||jj|jj|jjd } Wdn1swY|s| f|td}| dur| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rrrorm)rDrRrF)enabled)blank reduction zero_infinityrrr!r")rWrrjrrr6rrr' ones_likerVrPr!rU masked_selectrFr log_softmaxfloat32rXbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr!r")r3rrrrrrrrr!rrrI labels_masktarget_lengthsflattened_targets log_probsoutputr+r+r,r9sN      zUniSpeechForCTC.forwardrfr)r#r$r%rr1rrrrr'rrr*rr9r;r+r+r5r,rs4 rz UniSpeech Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. cseZdZfddZddZddZe     ddejdBd ejdBd e dBd e dBd e dBd ejdBde e BfddZ Z S)"UniSpeechForSequenceClassificationcst|t|dr|jrtdt||_|jd}|jr*t t |||_ t |j|j|_t |j|j|_|dS)Nrz`Sequence classification does not support the use of UniSpeech adapters (config.add_adapter=True)r)r0r1rLrrrr6ruse_weighted_layer_sumrFrr'ry layer_weightsrrHclassifier_proj_size projector num_labels classifierr)r3rW num_layersr5r+r,r13s    z+UniSpeechForSequenceClassification.__init__cCrrrrr+r+r,rDrz9UniSpeechForSequenceClassification.freeze_feature_encodercCrrrrr+r+r,rKrz4UniSpeechForSequenceClassification.freeze_base_modelNrrrrrrrcKsz|dur|n|jj}|jjrdn|}|j|||||d}|jjrB|t} tj| dd} tjj |j dd} | | dddj dd} n|d} | | } |durV| jdd} n+|| jd|} | ddd| jd} d | | <| j dd| j dd dd} || }d}|durt}|| d|jj| d}|s|f|td}|dur|f|S|St|||j|jd S) a  input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`UniSpeechProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrrrrmrr/rr)rWrrr6rr'stackrFrrrrr!rrr\rrrrrrrr!r")r3rrrrrrrrr! norm_weights pooled_output padding_maskexpand_padding_maskrrloss_fctrr+r+r,r9SsH    z*UniSpeechForSequenceClassification.forwardr)r#r$r%r1rrrr'rrr*rr9r;r+r+r5r,r,s2  r)rrrrr5rr7)Pr?collections.abcr dataclassesrnumpyrqr'torch.nnrFrrr; activationsrintegrations.deepspeedrintegrations.fsdpr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr rrrrmodeling_utilsrrrprocessing_utilsrrKrrrconfiguration_unispeechr get_loggerr#rrModuler.r<rZrhrnrsrrrrrrrrr rrrr5r*rrendarrayrrrrrrr__all__r+r+r+r,s               -) V$H.LFO  wwz h