o i@sdZddlZddlmZmZddlZddlmZddlmZddl m Z m Z m Z ddl mZdd lmZdd lmZdd lmZdd lmZmZdd lmZddlmZmZmZddlmZddl m!Z!e"e#Z$dZ%dZ&dZ'gdZ(dZ)dZ*Gdddej+Z,Gdddej+Z-Gdddej+Z.Gdddej+Z/Gdd d ej+Z0Gd!d"d"ej+Z1Gd#d$d$ej+Z2Gd%d&d&ej+Z3Gd'd(d(eZ4Gd)d*d*eZ5d+Z6d,Z7Gd-d.d.e5Z8e d/e6Gd0d1d1e5Z9e d2e6Gd3d4d4e5Z:gd5Z;dS)6zPyTorch M-CTC-T model.N)OptionalUnion)nn)ACT2FN)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forward)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputCausalLMOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)logging) MCTCTConfigrzspeechbrain/m-ctc-t-large)rizY"Mr. Quilter is the apostle of the middle classes, and we're glad to welcome his gospel."gv@cs(eZdZdZfddZddZZS)MCTCTConv1dSubsamplerz Convolutional subsampler: a stack of 1D convolution (along temporal dimension) followed by non-linear activation via gated linear units (https://huggingface.co/papers/1911.08460) cst|_|j_t|j_|j _ |j |j _ j dkr1|jdur,td|j_nd_|jd_|j_|j_tfddtjD_dS)NrzbNeed to specify `conv_channels` configuration in `MCTCTConfig` to use multiple convolution layers.c3s\|])\}}tj|dkrjnj||jdkrj|nj|j|ddVqdS)rrvalid) kernel_sizestridepaddingN)rConv1d in_channels mid_channels num_layers out_channelsr).0ikselfg/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/deprecated/mctct/modeling_mctct.py Us z1MCTCTConv1dSubsampler.__init__..)super__init__config conv_glu_dimglu_dimrDropout conv_dropoutdropoutnum_conv_layersr!input_feat_per_channelinput_channelsr conv_channels ValueErrorr hidden_sizer" conv_kernelr conv_strider ModuleList enumerate conv_layersr'r- __class__r&r)r,9s&     zMCTCTConv1dSubsampler.__init__cCstdd|jD}tjj|dd||fdd}|dd}|jD]}||}tjj ||j d}| |}q#|dd}|S)Ncss|]}|dVqdS)rNr()r#sizer(r(r)r*csz0MCTCTConv1dSubsampler.forward..rconstantrrdim) sumrtorchr functionalpad transpose contiguousr=glur/r2)r'input_featuresr hidden_statesconvr(r(r)forward`s  zMCTCTConv1dSubsampler.forward__name__ __module__ __qualname____doc__r,rO __classcell__r(r(r?r)r3s 'rcs,eZdZdZfddZ dddZZS) MCTCTEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ t |_ t|j|_|jdt|jddd|jdtj|jtj|jjddddS)N) padding_idx position_ids)rF) persistenttoken_type_idsdtypedevice)r+r,r Embedding vocab_sizer8 pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddingsMCTCTLayerNorm LayerNormr0hidden_dropout_probr2register_bufferrFarangeexpandzerosrXrAlongr^r>r?r(r)r,ss  zMCTCTEmbeddings.__init__Nrc Cs|dur|n|dd}|d}|dur%|jdd|||f}|durOt|drD|jddd|f}||d|} | }n tj|tj|jjd}|durX| |}| |} || } | | } | | } | S)NrYrr[rr\) rArXhasattrr[rlrFrmrnr^rbrfrhr2) r'rLr[rX inputs_embedspast_key_values_length input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrf embeddingsr(r(r)rOs"      zMCTCTEmbeddings.forward)NNNNrrPr(r(r?r)rVps  rVcsDeZdZfddZddZddZddZ d d d ZZS)MCTCTSelfAttentioncst|j|jdkrt|dstd|jd|jd|j|_|j|_|j|j|_t j |j|jdd|_ t j |j|jdd|_ t j |j|jdd|_ t |j|_|j|_t d|jd |j|_|j|_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()Fbiasrr)r+r,r8num_attention_headsror7attention_head_dimattention_head_size all_head_sizerLinearquerykeyvaluer0attention_probs_dropout_probr2rcr_distance_embedding is_decoderr>r?r(r)r,s"   zMCTCTSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrYrrr)rAr|r~viewpermute)r'x new_x_shaper(r(r)transpose_for_scoress z'MCTCTSelfAttention.transpose_for_scorescCsFt|jdkr|jttt|j}|jt|jttt|S)Nr)lenshaperreversedrangereshape)r'rrr(r(r)reshape_fortrans z"MCTCTSelfAttention.reshape_fortrancCs|dddd}|j\}}}}tj|tj||||f|jdfdd}||||||d|g}|ddd||d|f}|||||d||g}|d}|dd|||fdd}|ddddS)Nrrrrr^rC)rrrFcatrmr^rrI)r'scoresbatch hidden_stateseq_lenheads halfpointr(r(r)"relative_position_embedding_rotates&  z5MCTCTSelfAttention.relative_position_embedding_rotateNFcCs||}|t|j}|||}|||}||}t|| dd} |j j } t d| | dd} | | } | | } |durL| |} tjj| dd} || } |dura| |} t| |} | ddddjdd } |r{| | f}|S| f}|S) NrYzlh, bche -> bclerrrCrr) start_dim)rmathsqrtr~rrrrFmatmulrIrweighteinsumrrrGsoftmaxr2rflatten)r'rMattention_mask head_maskoutput_attentionsmixed_query_layer key_layer value_layer query_layerattention_scorespositional_embeddingrelative_position_scoresattention_probs context_layeroutputsr(r(r)rOs,      zMCTCTSelfAttention.forwardNNF) rQrRrSr,rrrrOrUr(r(r?r)rws rwc$eZdZfddZddZZS)rgcs2tttd|_ttd|_dSNr) r+r,r ParameterrFonessingleton_weightrmsingleton_biasr&r?r(r)r,s zMCTCTLayerNorm.__init__cCs||j|jSN)rrr'rMr(r(r)rOszMCTCTLayerNorm.forwardrQrRrSr,rOrUr(r(r?r)rgs rgcr)MCTCTSelfOutputcsLt||_tj|j|jdd|_tj|j|jd|_t |j |_ dSNFrz)eps) r+r,r-rrr8denserhlayer_norm_epsr0rir2r>r?r(r)r,!s zMCTCTSelfOutput.__init__cC&||}||}|||}|Srrr2rhr'rM input_tensorr(r(r)rO(  zMCTCTSelfOutput.forwardrr(r(r?r)r s rcs4eZdZfddZddZ   d ddZZS) MCTCTAttentioncs*tt||_t||_t|_dSr)r+r,rwr'routputset pruned_headsr>r?r(r)r,0s    zMCTCTAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrC)rrr'r|r~rrrrrrrrunion)r'rindexr(r(r) prune_heads6s zMCTCTAttention.prune_headsNFcCs6|||||}||d|}|f|dd}|S)Nrr)r'r)r'rMrrr self_outputsattention_outputrr(r(r)rOHszMCTCTAttention.forwardr)rQrRrSr,rrOrUr(r(r?r)r/s rcr)MCTCTIntermediatecsHttj|j|jdd|_t|jt rt |j|_ dS|j|_ dS)NFrz) r+r,rrr8intermediate_sizer isinstance hidden_actstrrintermediate_act_fnr>r?r(r)r,\s   zMCTCTIntermediate.__init__cCs||}||}|Sr)rrrr(r(r)rOds  zMCTCTIntermediate.forwardrr(r(r?r)r[s rcr) MCTCTOutputcsFttj|j|jdd|_tj|j|jd|_t |j |_ dSr) r+r,rrrr8rrhrr0rir2r>r?r(r)r,ks zMCTCTOutput.__init__cCrrrrr(r(r)rOqrzMCTCTOutput.forwardrr(r(r?r)rjs rcs:eZdZdeffdd Z   d ddZdd ZZS) MCTCTLayerr-csBtd|_|j|_t||_t||_|j|_t ||_ dSr) r+r, seq_len_dimchunk_size_feed_forwardr intermediater attentionrrrr>r?r(r)r,ys   zMCTCTLayer.__init__NFc CsH|j||||d}|d}|dd}t|j|j|j|}|f|}|S)N)rrr)rrfeed_forward_chunkrr) r'rMrrrself_attention_outputsrr layer_outputr(r(r)rOs  zMCTCTLayer.forwardcCs||}|||}|Sr)rr)r'rintermediate_outputrr(r(r)rs  zMCTCTLayer.feed_forward_chunkr)rQrRrSrr,rOrrUr(r(r?r)rxs rc@sFeZdZUdZeed<dZdZdZddZ de j fd d Z d d Z d S)MCTCTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r-mctctrLTcCs |jj}t|tjr|jjjd|d|jdur|jj nDt|tj r=|jjjd|d|j dur<|jj|j  n&t|tj rQ|jj |jj dnt|trc|jj d|jj t|tjtjfr|jjjd|d|jdur|jj dSdSdS)zInitialize the weightsg)meanstdNg?)r-initializer_rangerrrrdatanormal_r{zero_r_rWrhfill_rgrrr)r'modulerr(r(r) _init_weightss.          z"MCTCTPreTrainedModel._init_weights input_lengthscCshd}tt|jj|jj|jjD]!\}}}|d}|d|||dd}tj||ddd}q|S)zH Computes the output length of the convolutional layers rrtrunc) rounding_mode)ziprr-r3r9r:rFdiv)r'rdilation_ kernel_szrrr(r(r) _get_feat_extract_output_lengthssz5MCTCTPreTrainedModel._get_feat_extract_output_lengthscCst|jdkr|dddddf}||d}|d}tj||f|j|jd}d|tj ||jd|df<| dg d dg }|S)NrrYrr\rr) rrrrErArFrmr]r^rkflipcumsumrn)r'feature_vector_lengthrsubsampled_lengthsbszr(r(r)"_get_feature_vector_attention_masks z7MCTCTPreTrainedModel._get_feature_vector_attention_maskN)rQrRrSrTr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointingrrF LongTensorrrr(r(r(r)rs  raH This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MCTCTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_features (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`Wav2Vec2CTCTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. cs^eZdZdeffdd Z   ddejdejdejd ed ed ed ee e ffd dZ Z S) MCTCTEncoderr-csPtj|_t|_t|_tfddt j D|_ d|_ dS)Ncsg|]}tqSr()r)r#rr-r(r) sz)MCTCTEncoder.__init__..F) r+r,rirg layer_normrrNrr;rnum_hidden_layerslayersgradient_checkpointingr>r?rr)r,s    zMCTCTEncoder.__init__FTrLrrroutput_hidden_states return_dictreturncCs|dur|n|jj}|dur|n|jj}|dur|n|jj}||}||}|dur5||jd|}tj j ||j |j d}|durJt ||j}|rNdnd} |rTdnd} |durw|dt|jkrwtdt|jd|ddtp}t|} t|jD]5\} } |r| |f} tg}|j o||jjk}|r| r| |||d}|d}|rd }|r| |df} q|r| |f} |std d || | fDSt|| | d S) Nr)ptrainingr(rz&The head_mask should be specified for z layers, but it is for .)rMrr)NNcss|] }|dur|VqdSrr()r#vr(r(r)r*`sz'MCTCTEncoder.forward..last_hidden_staterM attentions)r-rruse_return_dictrrNrrrrGr2rir r r]rArrr7r r r<rFrand layerdroptupler)r'rLrrrrrrprMencoder_statesall_attentions synced_gpusidx encoder_layerdropout_probabilityskip_the_layer layer_outputsr(r(r)rOsZ          zMCTCTEncoder.forward)FFT) rQrRrSrr,rFTensorboolrrrrOrUr(r(r?r)rs( rzaThe bare M-CTC-T Model transformer outputting raw hidden-states without any specific head on top.cseZdZfddZeedeee e de d     dde j dee j d ee j d eed eed eed eee ffddZZS) MCTCTModelcs(t|||_t||_|dSr)r+r,r-rencoder post_initr>r?r(r)r,ks   zMCTCTModel.__init__zbatch_size, sequence_lengthaudio) checkpoint output_type config_classmodalityexpected_outputNrLrrrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|j||||||d}|d}|s@|f|ddSt||j|jdS)Nz#You have to specify input_features.rrrrrrrr ) r-rrrr7rrrMr) r'rLrrrrrencoder_outputssequence_outputr(r(r)rOts,zMCTCTModel.forward)NNNNN)rQrRrSr,r MCTCT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErFrrrrrrOrUr(r(r?r)rfs<    rzcMCTCT Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).cseZdZfddZeeeeee e e d      dde j dee j dee j deed eed eed ee jd eeeffd dZZS) MCTCTForCTCcsTt|t||_|jdurtd|jd|j}t ||j|_ | dS)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `MCTCTForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) r+r,rrr`r7r@r8rrctc_headr)r'r-output_hidden_sizer?r(r)r,s     zMCTCTForCTC.__init__)r!r"r#r% expected_lossNrLrrrrrlabelsrc Cs~|dur||jjkrtd|jj|dur|n|jj}|j||||||d}|d} || } d} |dur|dur?|n tj|j ddtj d}| | d tj } |dk} | d}|| }tjj| dtjddd}tjjjd d tjj||| ||jj|jj|jjd } Wdn1swY|s| f|td}| dur| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: r&rrY)r])rDr]rF)enabled)blank reduction zero_infinity)losslogitsrMr)maxr-r`r7rrr/rFrrrnrrEto masked_selectrrG log_softmaxfloat32rIbackendscudnnflagsctc_lossractc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONrrMr)r'rLrrrrrr2rrMr8r7r labels_masktarget_lengthsflattened_targets log_probsrr(r(r)rOsR     zMCTCTForCTC.forward)NNNNNN)rQrRrSr,r r)rr+rr,_CTC_EXPECTED_OUTPUT_CTC_EXPECTED_LOSSrFrrrrrrrOrUr(r(r?r)r.sB   r.)r.rr)s^          =:l ,'E  W8 d