o eiz$@sdZddlZddlZddlZddlmZddlmZmZm Z ddl m Z ddl mZddlmZmZmZdd lmZdd lmZdd lmZdd lmZmZdd lmZddlmZm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(m)Z)ddl*m+Z+m,Z,e)-e.Z/dZ0dej1de2de2fddZ3 ddej1de2dej1dBfddZ4  dde5e2e2fde6de2dej7dBd e2d!ej8f d"d#Z9Gd$d%d%eZ:Gd&d'd'eZ;Gd(d)d)eZGd,d-d-ej=Z?Gd.d/d/ej=Z@Gd0d1d1ejj=ZAGd2d3d3ej=ZBGd4d5d5ej=ZCGd6d7d7ej=ZDGd8d9d9ej=ZEGd:d;d;ej=ZFGdd?d?ej=ZHGd@dAdAej=e%ZIGdBdCdCej=e%ZJGdDdEdEej=e%ZKGdFdGdGej=ZLGdHdIdIej=ZMGdJdKdKeZNGdLdMdMeZOe(GdNdOdOe&ZPGdPdQdQePZQGdRdSdSePZRGdTdUdUePZSGdVdWdWePZTGdXdYdYePZUGdZd[d[ePZVGd\d]d]ePZWGd^d_d_ePZXGd`dadaej=ZYGdbdcdcej=ZZe(dddeGdfdgdgePZ[e(dhdeGdidjdjePeZ\   k l m  n nddoePdej]dpej]dBdej7dBdqe6dre6dse6dtej=dBdue^dve^d!ej]e5ej]ej]fBfdwdxZ_e(dydeGdzd{d{ePZ`e(d|deGd}d~d~ePZaGdddej=Zbe(ddeGddde&ZcgdZddS)zPyTorch SpeechT5 model.N)nn)BCEWithLogitsLossCrossEntropyLossL1Loss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutputSeq2SeqSpectrogramOutput)EmbeddingAccessMixinPreTrainedModel)auto_docstringlogging)SpeechT5ConfigSpeechT5HifiGanConfig input_ids pad_token_iddecoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|dur*td||dk||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrr shifted_input_idsr(l/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/speecht5/modeling_speecht5.pyshift_tokens_right4s (r* input_valuesreduction_factorattention_maskcCs|dkr"|dd|dd|f}|dur"|dd|dd|f}||j}|ddddf|ddddf<||dkd||fS)zw Shift input spectrograms one timestep to the right. Also applies the reduction factor to the sequence length. rNr!Y)r"r#r$r&)r+r,r-shifted_input_valuesr(r(r)shift_spectrograms_rightDs (r1r# mask_prob mask_length min_masksreturncs|\}dkr tdkrtdddtjdfdd}|dur:|d n fd d t|D}tj |ft d }g} |} | d krZ|S|D];} || } tjj t | d| dd} t | d kr}d}n| d }t| tj| | tjd |g} | | q\t| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkrd| | dk<t|| dd |S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr}|d|kr*t|dd}|S)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonr3r2r4sequence_lengthr(r)compute_num_masked_spans  z6_compute_mask_indices..compute_num_masked_spanNr!csg|]}qSr(r(.0_)r=r(r) sz)_compute_mask_indices..dtyperF)replace)r%nprandomranditemdetachsumtolistrangezerosboolchoicearangelen concatenateonesint32appendarray broadcast_toreshaper8put_along_axis)r#r2r3r-r4 batch_sizer> input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanr9r:spec_aug_mask_idxdummy_mask_idxoffsetsr(r;r)_compute_mask_indicesZs\    rcc&eZdZdfdd ZddZZS)SpeechT5NoLayerNormConvLayerrcsjt|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvrfeat_extract_activation activationselfconfiglayer_id __class__r(r)rks  z%SpeechT5NoLayerNormConvLayer.__init__cCs||}||}|SN)rsrurw hidden_statesr(r(r)forward  z$SpeechT5NoLayerNormConvLayer.forwardr__name__ __module__ __qualname__rkr __classcell__r(r(rzr)resrecrd)SpeechT5LayerNormConvLayerrcs|t|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)NrrrfT)elementwise_affine)rjrkrlrmrnrrorprqrrrs LayerNorm layer_normrrtrurvrzr(r)rks  z#SpeechT5LayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)Nr!)rs transposerrur}r(r(r)rs     z"SpeechT5LayerNormConvLayer.forwardrrr(r(rzr)rsrcrd)SpeechT5GroupNormConvLayerrcst|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)NrrrfT) num_groups num_channelsaffine)rjrkrlrmrnrrorprqrrrsrrtru GroupNormrrvrzr(r)rks   z#SpeechT5GroupNormConvLayer.__init__cCs"||}||}||}|Sr|)rsrrur}r(r(r)rs   z"SpeechT5GroupNormConvLayer.forwardrrr(r(rzr)rsrc seZdZdZddedededBffdd ZddedededBfd d ZeddedededBfd d Ze dde j defddZ dde j dededBfddZ ZS)%SpeechT5SinusoidalPositionalEmbeddingzDThis module produces sinusoidal positional embeddings of any length.N num_positions embedding_dim padding_idxcs:td|_||_||_||_|||j||dSN)rjrkoffsetrrr make_weights)rwrrrrzr(r)rk s z.SpeechT5SinusoidalPositionalEmbedding.__init__num_embeddingscCsB||||}t|dr|j|jj|jjd}|jd|dddS)NweightsrDdeviceF persistent) get_embeddinghasattrtorrDrregister_buffer)rwrrr emb_weightsr(r(r)r(s z2SpeechT5SinusoidalPositionalEmbedding.make_weightscCs|d}td|d}ttj|tjd| }tj|tjdd|d}tjt |t |gdd |d}|ddkrUtj|t |dgdd}|durad||ddf<| tS) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'rrCrdimr!N)mathlogtorchexprQint64float unsqueezecatsincosviewrNrget_default_dtype)rrrhalf_dimembr(r(r)r0s $& z3SpeechT5SinusoidalPositionalEmbedding.get_embeddingrrpast_key_values_lengthcCs||\}}|||j||j}|jd|}||jdkr-|||j|j|j|j d| d ||d S)Nrrr!) size"create_position_ids_from_input_idsrrrrrrr index_selectrrJ)rwrrbszseq_len position_idsmax_posr(r(r)rBs "z-SpeechT5SinusoidalPositionalEmbedding.forwardcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)ner7rcumsumtype_aslong)rwrrrmaskincremental_indicesr(r(r)rQs  zHSpeechT5SinusoidalPositionalEmbedding.create_position_ids_from_input_idsr|r)rrr__doc__r7rkr staticmethodrrno_gradTensorrrrr(r(rzr)rs  rc$eZdZfddZddZZS)SpeechT5PositionalConvEmbeddingcs$ttj|j|j|j|jd|jd|_tjj }t tjj dr'tjj j }t r{ddl }|jj|jjdd||jddd|_Wdn1sLwYt |jdrd|jj jj}|jj jj}n|jj}|jj}|j|||j||n ||jddd|_t|j|_t|j|_dS) Nr)rgpaddinggroups weight_normr) modifier_rankweight)namerparametrizations)rjrkrro hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupsrsutilsrrrr deepspeedzeroGatheredParametersr original0 original1weight_gweight_vregister_external_parameterSpeechT5SamePadLayerrrrtru)rwrxrrrrrzr(r)rkds4     z(SpeechT5PositionalConvEmbedding.__init__cCs:|dd}||}||}||}|dd}|SNrr)rrsrrur}r(r(r)rs     z'SpeechT5PositionalConvEmbedding.forwardrr(r(rzr)rcs !rcs*eZdZdZdfdd ZddZZS) SpeechT5ScaledPositionalEncodingu[ Scaled positional encoding, see §3.2 in https://huggingface.co/papers/1809.08895 cst||}td|d}ttjd|dtjdtd| }t |||dddddf<t |||dddddf<|d}t |j d|ddtj|d |_||_||_ttd |_dS) NrrrrC@peFrp?)rrNrQrrrrrrrrrjrkrrDropoutdropoutrmax_len Parametertensoralpha)rwrrrrpositiondiv_termrzr(r)rks .$$  z)SpeechT5ScaledPositionalEncoding.__init__cCs4||j|jddd|df}||}|S)Nr)rrrr)rwrr(r(r)rs& z(SpeechT5ScaledPositionalEncoding.forward)r)rrrrrkrrr(r(rzr)rsrcrd)"SpeechT5RelativePositionalEncodingcs.t||_||_tjd|||_dSr)rjrkr max_lengthrr Embeddingpe_k)rwrrrzr(r)rks z+SpeechT5RelativePositionalEncoding.__init__cCs|jd}td|j|jtjd}|dddf|dddf}t||j k|j |}t||jk|jd|}||j}||S)NrrrrD) r#rrQrrrwhererr)rwr~rpos_seqr(r(r)rs    z*SpeechT5RelativePositionalEncoding.forward)rrr(r(rzr)rsrcr)rcs*t|ddkrd|_dSd|_dS)Nrrr)rjrknum_pad_remove)rwrrzr(r)rks  zSpeechT5SamePadLayer.__init__cCs,|jdkr|ddddd|j f}|SNr)rr}r(r(r)rs zSpeechT5SamePadLayer.forwardrr(r(rzr)rs rcs0eZdZdZfddZddZddZZS)SpeechT5FeatureEncoderz.Construct the features from raw audio waveformcstjdkr tddgfddtjdD}njdkr2fddtjD}n td jd t||_ d |_ d |_ dS) Ngrouprrycsg|] }t|ddqS)rr)rer@irxr(r)rBsz3SpeechT5FeatureEncoder.__init__..rlayercg|]}t|dqS)r)rrrr(r)rBs z`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rjrkfeat_extract_normrrMnum_feat_extract_layersr%r ModuleList conv_layersgradient_checkpointing_requires_grad)rwrxrrzrr)rks        zSpeechT5FeatureEncoder.__init__cCs|D]}d|_qd|_dS)NF) parameters requires_gradr)rwparamr(r(r)_freeze_parameterss  z)SpeechT5FeatureEncoder._freeze_parameterscCs:|dddf}|jr|jrd|_|jD]}||}q|SNT)rtrainingrr)rwr+r~ conv_layerr(r(r)rs    zSpeechT5FeatureEncoder.forward)rrrrrkrrrr(r(rzr)rs  rcr)SpeechT5FeatureProjectioncsJttj|jd|jd|_t|jd|j|_ t |j |_ dS)Nr!eps) rjrkrrrllayer_norm_epsrLinearr projectionrfeat_proj_dropoutrrwrxrzr(r)rks z"SpeechT5FeatureProjection.__init__cCs&||}||}||}||fSr|)rrr)rwr~norm_hidden_statesr(r(r)rs   z!SpeechT5FeatureProjection.forwardrr(r(rzr)r s r cseZdZfddZddZ  ddejdejdBdejdBfd d Z d e dejfd d Z deje BfddZ  ddejdejdBdejdBfddZ ZS)SpeechT5SpeechEncoderPrenetcs|t||_t||_t||_|jdks|jdkr(t t |j |_t||_t|j|jd|j |j|_dS)Nr/r)rjrkrxrfeature_encoderr feature_projectionmask_time_probmask_feature_probrrrrruniform_masked_spec_embedrpos_conv_embedrmax_speech_positionsrpos_sinusoidal_embedrrzr(r)rks     z$SpeechT5SpeechEncoderPrenet.__init__cCs|jdSr|)rrrwr(r(r)freeze_feature_encodersz2SpeechT5SpeechEncoderPrenet.freeze_feature_encoderNr+r-mask_time_indicesc Cs||}|dd}|dur||jd|}||\}}|j|||d}||}||}|dur<|d}nt j |jddt j|j d}| |}||}||fS)Nrr)r!r-r) rr"_get_feature_vector_attention_maskr#r_mask_hidden_statesrrrrrNrr) rwr+r-r!extract_featuresr~positional_conv_embedding padding_mask positional_sinusoidal_embeddingsr(r(r)rs&    z#SpeechT5SpeechEncoderPrenet.forwardfeature_vector_lengthcCs|jdddddf}||tj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dgd dg }|S)Nr!rrrrr) r _get_feat_extract_output_lengthsrrrr#rNrDrrQfliprO)rwr(r-non_padded_lengthsoutput_lengthsr[r(r(r)r"9s "z>SpeechT5SpeechEncoderPrenet._get_feature_vector_attention_maskr\cCs4dd}t|jj|jjD] \}}||||}q |S)zH Computes the output length of the convolutional layers cSstj|||dddS)Nfloor) rounding_moder)rdiv)r9rgrhr(r(r)_conv_out_lengthNszVSpeechT5SpeechEncoderPrenet._get_feat_extract_output_lengths.._conv_out_length)ziprxrprq)rwr\r1rgrhr(r(r)r*Isz.)rjrkrxrrrMspeech_decoder_prenet_layerslayersrr@r final_layerrpositional_dropoutrencode_positionsspeaker_embedding_dimspeaker_embeds_layerrrzrr)rks   z$SpeechT5SpeechDecoderPrenet.__init__cCsJtj|d|d}|d|ddd}t|dk|ddd|S)Nrrr)r bernoullirrepeatrr)rw inputs_embedsrr all_masksr(r(r)_consistent_dropoutsz/SpeechT5SpeechDecoderPrenet._consistent_dropoutNr+speaker_embeddingscCs|}|jD]}tj||}|||jj}q||}||}|durKtj |}| d d| dd}t j||gdd}tj||}|S)Nrr!r)rBr functionalrelurLrxspeech_decoder_prenet_dropoutrCrE normalizerr9rrrrG)rwr+rMrJrr(r(r)rs    z#SpeechT5SpeechDecoderPrenet.forwardr|) rrrrkrLrrrrr(r(rzr)r>s r>crd)SpeechT5BatchNormConvLayerrcst|dkr |j}n|j}||jdkr|j}n|j}tj|||jd|jdddd|_t ||_ ||jdkrCt |_ nd|_ t |j|_dS)NrrrF)rgrhrri)rjrkr?speech_decoder_postnet_unitsspeech_decoder_postnet_layersrrospeech_decoder_postnet_kernelrs BatchNorm1d batch_normTanhrurspeech_decoder_postnet_dropoutr)rwrxryrmrnrzr(r)rks(    z#SpeechT5BatchNormConvLayer.__init__cCs6||}||}|jdur||}||}|Sr|)rsrWrurr}r(r(r)rs     z"SpeechT5BatchNormConvLayer.forwardrrr(r(rzr)rRsrRcs<eZdZfddZdejfddZdejfddZZS)SpeechT5SpeechDecoderPostnetcs^t|_tjjj|_tjj|_ t fddt j D|_ dS)Ncsg|]}t|qSr()rRrrr(r)rBsz9SpeechT5SpeechDecoderPostnet.__init__..)rjrkrxrrrr?r,feat_outprob_outrrMrTrBrrzrr)rks  z%SpeechT5SpeechDecoderPostnet.__init__r~cCsJ|||dd|jj}||}|||dd}|||fS)Nrr!)r[rrrxr?postnetr\)rwr~outputs_before_postnetoutputs_after_postnetlogitsr(r(r)rs  z$SpeechT5SpeechDecoderPostnet.forwardcCs0|dd}|jD]}||}q ||ddSr)rrB)rwr~ layer_outputrr(r(r)r]s   z$SpeechT5SpeechDecoderPostnet.postnet) rrrrkrrrr]rr(r(rzr)rZs  rZcs,eZdZfddZdejfddZZS)SpeechT5TextEncoderPrenetcs>t||_t|j|j|j|_t |j |j|j |_ dSr|) rjrkrxrr vocab_sizerr embed_tokensrrDmax_text_positionsrErrzr(r)rks  z"SpeechT5TextEncoderPrenet.__init__rcCs||}||}|Sr|)rdrE)rwrrJr(r(r)rrz!SpeechT5TextEncoderPrenet.forward)rrrrkrrrrr(r(rzr)rbs  rbcsDeZdZfddZ  d dejdejdBdedBfddZZ S) SpeechT5TextDecoderPrenetcsnt||_t|j|_|jrt |j nd|_ t |j |j |j|_t|j|jd|j |j|_dS)Nrr)rjrkrxrrrDrscale_embeddingrsqrtr embed_scalerrcrrdrreembed_positionsrrzr(r)rks  z"SpeechT5TextDecoderPrenet.__init__Nrr-past_key_valuescCst|dur|}|d|d}ntd|durdn|}|||}|||j}||7}||}||fS)Nr!z'You have to specify `decoder_input_ids`r)rrr%get_seq_lengthrjrdrir)rwrr-rk input_shaper positionsrJr(r(r)rs  z!SpeechT5TextDecoderPrenet.forwardr;) rrrrkrrr<r rrr(r(rzr)rfs rfcs<eZdZfddZdejfddZddZdd ZZ S) SpeechT5TextDecoderPostnetcs*t||_tj|j|jdd|_dS)NFri)rjrkrxrrrrclm_headrrzr(r)rk4s z#SpeechT5TextDecoderPostnet.__init__r~cCs ||Sr|rqr}r(r(r)r9 z"SpeechT5TextDecoderPostnet.forwardcCs|jSr|rrrr(r(r)get_output_embeddings<sz0SpeechT5TextDecoderPostnet.get_output_embeddingscCs ||_dSr|rrrwnew_embeddingsr(r(r)set_output_embeddingsArsz0SpeechT5TextDecoderPostnet.set_output_embeddings) rrrrkrrrrtrwrr(r(rzr)ro3s  rocseZdZdZ    ddedededBd edBd edBd edBf fd d Z      ddej dej dBde dBdej dBdej dBdedej dBde ej ej dBe dBffddZ Z S)SpeechT5Attentionz Multi-headed attention from 'Attention Is All You Need' paper with relative position bias (see https://aclanthology.org/N18-2074.pdf) r/FTN embed_dim num_headsr is_decoderri layer_idxcst||_||_||_|||_|j||jkr'td|jd|d|jd|_||_||_ t j |||d|_ t j |||d|_ t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).grp)rjrkryrzrhead_dimr%scalingr{r|rrk_projv_projq_projout_proj)rwryrzrr{rir|rzr(r)rkKs$    zSpeechT5Attention.__init__r~key_value_statesrkr- position_biasoutput_attentionscache_positionr5cCs|du}|\} } } |||j} d} |dur1t|tr/|j|j} |r+|j}n|j }n|}|r5|n|}|rN|durN| rN|j |jj }|j |jj }nJ| |}||}|| d|j|jdd}|| d|j|jdd}|dur|s||nd}||||jd|i\}}|rt|trd|j|j<| |jd|jf}| | | |j|jdd} | j|} |j|}|j|}|d}t| |dd}|| |j| |fkrtd| |j| |fd ||dur#| | |jd|jd d}t||d d}|d d| |j|d |d}||7}|durX|| d| |fkrCtd | d| |fd ||| |j| ||}|| |j| |}tjj|dd }|rw|| |j| |}|| |j| |}nd}tjj||j|jd}t||}|| |j| |jfkrtd| |j| |jfd ||| |j| |j}|dd}|| | |j}| |}||fS)z#Input shape: Batch x Time x ChannelNFr!rrrTz$Attention weights should be of size z , but is rrz!Attention mask should be of size r)rr z `attn_output` should be of size )!rrr~ isinstancer is_updatedgetr|cross_attention_cacheself_attention_cacherBkeysvaluesrrrrzr}rupdaterYrbmmr% contiguousmatmulrrNsoftmaxrr ryr)rwr~rrkr-rrris_cross_attentionrtgt_lenrA query_statesrcurr_past_key_valuescurrent_states key_states value_states proj_shapesrc_len attn_weights reshape_q rel_pos_biasattn_weights_reshaped attn_probs attn_outputr(r(r)rhs           "     zSpeechT5Attention.forward)r/FTN)NNNNFN)rrrrr7rrOrkrrr tuplerrr(r(rzr)rxEsT   rxcr)SpeechT5FeedForwardcsltt|j|_t|j||_t |j t r!t |j |_ n|j |_ t||j|_t|j|_dSr|)rjrkrractivation_dropoutintermediate_dropoutrrintermediate_denser hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropout)rwrxintermediate_sizerzr(r)rks  zSpeechT5FeedForward.__init__cCs6||}||}||}||}||}|Sr|)rrrrrr}r(r(r)rs     zSpeechT5FeedForward.forwardrr(r(rzr)rs  rc sReZdZdeffdd Z   d dejdejdBdejdBd efd d ZZ S) SpeechT5EncoderLayerrxcsjtt|j|j|jdd|_t|j |_ tj |j|j d|_ t||j|_tj |j|j d|_dS)NF)ryrzrr{r )rjrkrxrencoder_attention_headsattention_dropout attentionrrrrrrrrencoder_ffn_dim feed_forwardfinal_layer_normrrzr(r)rks zSpeechT5EncoderLayer.__init__NFr~r-rrcCsf|}|j||||d\}}||}||}||}|||}||}|f}|r1||f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. position_bias (`torch.FloatTensor`): relative position embeddings of size `(seq_len, seq_len, hidden_size // encoder_attention_heads)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r~r-rr)rrrrr)rwr~r-rrresidualroutputsr(r(r)rs     zSpeechT5EncoderLayer.forward)NNF) rrrrrkrrrOrrr(r(rzr)rsrcseZdZddeffdd Z       ddejdejdBd ejdBd ejdBd edBd edBd edBdejdBfddZ Z S)SpeechT5DecoderLayerNrxcstt|j|j|jd|d|_t|j |_ tj |j|j d|_ t|j|j|jd|d|_tj |j|j d|_t||j|_tj |j|j d|_dS)NT)ryrzrr{r|r )rr{r|)rjrkrxrdecoder_attention_headsr self_attnrrrrrrself_attn_layer_norm encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrr)rwrxr|rzr(r)rk2s( zSpeechT5DecoderLayer.__init__FTr~r-encoder_hidden_statesencoder_attention_maskrkr use_cacherc Cs|} |j|||||d\}} ||}| |}||}d} |dur?|} |j||||||d\}} ||}| |}||}|||}||}|f} |rV| | | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, hidden_size)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r~rkr-rrN)r~rr-rkrr)rrrrrrr) rwr~r-rrrkrrrrself_attn_weightscross_attn_weightsrr(r(r)rJs>      zSpeechT5DecoderLayer.forwardr|)NNNNFTN) rrrrrkrrr rOrrr(r(rzr)r1s4 rc@s>eZdZUeed<dZdZdZdZe de j fddZ d S) SpeechT5PreTrainedModelrxspeecht5r+audioTmodulec Cs4|jj}t|tr,tj|jjddt d|jj d|jj dt |jj dn^t|trt|j|j|j}}t||}td|d}ttjd|dtjdtd| }t|||dddddf<t|||dddddf<|d}t|j|nt|trt d|j j!}tj"|j j| |dtj"|j j | |dnt|t#j$rtj|jd |d|j durt%|j nt|t#j&t#j't#j(fr t%|j t|jt)|d ddurt%|j*t|j+t%|j,nt|t#j-r5t.|j|j dur4t |j/|j |j d}tj"|j | |dnUt|t#j0r]tj|jd |d|j1dur\t)|jd d s\t%|j|j1n-t|t2rx|3|j4|j5|j6|j1} t|j7| nt|t8rt%|j9t|j:t;|d rt"|j<dSdS)zInitialize the weightsrrr)meanstdrCrN)abr/ running_mean_is_hf_initializedFr)=rxinitializer_rangerrinitnormal_rsrrrhrg in_channels constant_rirones_rrrrrNrQrrrrrrrcopy_rr r in_featuresrrrzeros_rrrVr4r running_varnum_batches_trackedrokaiming_normal_rrrrrrrrrSpeechT5HifiGanrscalerr) rwrrrrrrrkrr(r(r) _init_weightssr     .$$                 z%SpeechT5PreTrainedModel._init_weightsN)rrrr__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointingrrrModulerr(r(r(r)rs rcjeZdZdZdeffdd Z    ddejdejdBde dBd e dBd e dBd e e Bf d d Z Z S)SpeechT5Encoderzu Transformer encoder consisting of *config.encoder_layers* layers. Each layer is a [`SpeechT5EncoderLayer`]. rxcs~ttjjjd|_tj|_ j |_ t fddt jD|_tjjj|_d|_|dS)Nr csg|]}tqSr()rr?rr(r)rBz,SpeechT5Encoder.__init__..F)rjrkrrrrrrrrencoder_layerdrop layerdroprrMencoder_layersrBrrencoder_max_relative_positionrjr post_initrrzrr)rks   zSpeechT5Encoder.__init__Nr~r-routput_hidden_states return_dictr5cKsB|dur|n|jj}|dur|n|jj}|dur|n|jj}t|j||d}||}||}||}tp;t |}|r@dnd} |rFdnd} t |j D]7\} } |rX| |f} d} |j rgt g}||jk} | rk|rw| ||||d}|d}| r{d}|r| |df} qM|r| |f} |std d || | fDSt|| | d S) a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the encoder prenet. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. N)rxrJr-r(F)r-rrrr;rcs|] }|dur|VqdSr|r(r@vr(r(r) 9sz*SpeechT5Encoder.forward..last_hidden_stater~ attentions)rxrruse_return_dictrrrrjr r enumeraterBr rrHrrr)rwr~r-rrrkwargsr synced_gpusall_hidden_statesall_self_attentionsidx encoder_layerskip_the_layerdropout_probability layer_outputsr(r(r)rsX         zSpeechT5Encoder.forwardNNNNrrrrrrkrr=rrOrrrrr(r(rzr)rs(rcr)SpeechT5EncoderWithSpeechPrenetz Wrapper around SpeechT5Encoder that applies SpeechT5SpeechEncoderPrenet to convert the audio waveform data to hidden features. rxc,t|t||_t||_|dSr|)rjrkrprenetrwrapped_encoderrrrzr(r)rkH    z(SpeechT5EncoderWithSpeechPrenet.__init__Nr+r-rrrr5c Ks(|||\}}|j|||||d}|SN)r~r-rrrrr rwr+r-rrrrr~rr(r(r)rPs z'SpeechT5EncoderWithSpeechPrenet.forwardrrr(r(rzr)rBs( rcszeZdZdZdeffdd ZddZddZ dd ej d ej d Bd e d Bd e d Bde d Bde e Bf ddZZS)SpeechT5EncoderWithTextPrenetz| Wrapper around SpeechT5Encoder that applies SpeechT5TextEncoderPrenet to convert the input_ids to hidden features. rxcrr|)rjrkrbrrrrrrzr(r)rkkrz&SpeechT5EncoderWithTextPrenet.__init__cC |jSr|rget_input_embeddingsrr(r(r)r srsz2SpeechT5EncoderWithTextPrenet.get_input_embeddingscC|j|dSr|rset_input_embeddingsrwvaluer(r(r)rvz2SpeechT5EncoderWithTextPrenet.set_input_embeddingsNr+r-rrrr5c Ks"||}|j|||||d}|Srrrr(r(r)rys z%SpeechT5EncoderWithTextPrenet.forwardr)rrrrrrkr rrr=rrOrrrrr(r(rzr)r fs,r cr)SpeechT5EncoderWithoutPrenet This wrapper class is a helper class to correctly load pretrained checkpoints when used in combination with [`SpeechT5Model`]. rxc"t|t||_|dSr|)rjrkrrrrrzr(r)rk   z%SpeechT5EncoderWithoutPrenet.__init__Nr+r-rrrr5cKs|j|||||dSr)r)rwr+r-rrrrr(r(r)rs z$SpeechT5EncoderWithoutPrenet.forwardrrr(r(rzr)rs( rceZdZdZdeffdd Z          ddejdBdejdBdejdBd ejdBd e dBd e dBd e dBd e dBde dBdej dBde e BfddZZS)SpeechT5Decoderzt Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`SpeechT5DecoderLayer`] rxcsFtj|_tfddtjD|_d|_ | dS)Ncr))r|)rrrr(r)rBz,SpeechT5Decoder.__init__..F) rjrkdecoder_layerdroprrrrMdecoder_layersrBrrrrzrr)rks   zSpeechT5Decoder.__init__Nr~r-rrrkrrrrrr5c  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |jr7|jr7|r7tdd}|rJ|durJt t |jdt |jd}|durR| nd} | durft j | | |jd|jd} t|j||| |d}|dur|durt|j|||d }tpt|} |rd nd}|rd nd}|r|durd nd}t|jD]C\}}|r||f}d}|jrt g}||jk}|r| sq||||||||| d }|d}|r||df}|dur||d f}q|r||f}| std d|||||fDSt|||||dS)aQ Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the decoder prenet. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrrr))rxrJr-rrk)rxrJr-rr()rrkrrrrcsrr|r(rr(r(r)rHsz*SpeechT5Decoder.forward..)rrkr~rcross_attentions)rxrrrrrr logger warning_oncer r rlrrQr#rrrr rrrBrHrrr)rwr~r-rrrkrrrrrrrrrrall_cross_attentionsr decoder_layerrrrr(r(r)rs:           zSpeechT5Decoder.forward NNNNNNNNNNrrrrrrkrr=r<r rOrrrrrr(r(rzr)rsH     rcseZdZdZdeffdd Z           ddejdBdejdBdejdBd ejdBd ej dBd e dBd e dBd e dBde dBde dBdej dBde e BfddZZS)SpeechT5DecoderWithSpeechPrenetz Wrapper around SpeechT5Decoder that applies SpeechT5SpeechDecoderPrenet to convert log-mel filterbanks to hidden features. rxcrr|)rjrkr>rrwrapped_decoderrrrzr(r)rk]rz(SpeechT5DecoderWithSpeechPrenet.__init__Nr+r-rrrMrkrrrrrr5c  Ks.|||} |j| ||||||| | | d }|SN) r~r-rrrkrrrrrrr$)rwr+r-rrrMrkrrrrrrdecoder_hidden_statesrr(r(r)res  z'SpeechT5DecoderWithSpeechPrenet.forward) NNNNNNNNNNN)rrrrrrkrr=r<rr rOrrrrr(r(rzr)r#WsN     r#cseZdZdZdeffdd ZddZddZ dd ej d Bd ej d Bd ej d Bd ej d Bde d Bde d Bde d Bde d Bde d Bdej d BdeeBfddZZS)SpeechT5DecoderWithTextPrenetz{ Wrapper around SpeechT5Decoder that applies SpeechT5TextDecoderPrenet to convert input tokens to hidden features. rxcrr|)rjrkrfrrr$rrrzr(r)rkrz&SpeechT5DecoderWithTextPrenet.__init__cCr r|r rr(r(r)r rsz2SpeechT5DecoderWithTextPrenet.get_input_embeddingscCr r|rrr(r(r)rrz2SpeechT5DecoderWithTextPrenet.set_input_embeddingsNr+r-rrrkrrrrrr5c  Ks4||||\} }|j| |||||||| | d } | Sr%r&)rwr+r-rrrkrrrrrrr'rr(r(r)rs z%SpeechT5DecoderWithTextPrenet.forwardr!)rrrrrrkr rrr=r<r rOrrrrrr(r(rzr)r(sL    r(cr)SpeechT5DecoderWithoutPrenetrrxcrr|)rjrkrr$rrrzr(r)rkrz%SpeechT5DecoderWithoutPrenet.__init__Nr+r-rrrkrrrrrr5c Ks"|j||||||||| | d } | Sr%)r$) rwr+r-rrrkrrrrrrrr(r(r)rs z$SpeechT5DecoderWithoutPrenet.forwardr!r"r(r(rzr)r)sH     r)cs\eZdZdZdeffdd Zdejdejdejdej fd d Z d d Z e d dZ ZS)$SpeechT5GuidedMultiheadAttentionLossz Guided attention loss from the paper [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://huggingface.co/papers/1710.08969), adapted for multi-head attention. rxcst|j|_|j|_dSr|)rjrkguided_attention_loss_sigmasigmaguided_attention_loss_scalerrrzr(r)rks  z-SpeechT5GuidedMultiheadAttentionLoss.__init__r input_masks output_masksr5cCsX||||j}|d|d@}||jd}||}t||}|j|S)aY Compute the attention loss. Args: attentions (`torch.FloatTensor` of shape `(batch_size, layers * heads, output_sequence_length, input_sequence_length)`): Batch of multi-head attention weights input_masks (`torch.BoolTensor` of shape `(batch_size, input_sequence_length)`): Input attention mask as booleans. output_masks (`torch.BoolTensor` of shape `(batch_size, output_sequence_length)`): Target attention mask as booleans. Returns: `torch.Tensor` with the loss value r!rr)_make_guided_attention_masksrrrrr masked_selectr)rwrr.r/guided_attn_masksmaskslosseslossr(r(r)rs  z,SpeechT5GuidedMultiheadAttentionLoss.forwardc Cs|d}|d}tjt||jd|jdf|d}tt||D]\}\}} ||| |j|||d| d|f<q#| dS)Nr!rr)) rKrrNrRr#rr2_make_guided_attention_maskr,r) rwr.r/rr\r-r2rilenolenr(r(r)r0s  $& zASpeechT5GuidedMultiheadAttentionLoss._make_guided_attention_maskscCsdtjtj||dtj||ddd\}}||}||}dt||d d|dS)Nr)xy)indexingrr)rmeshgridrQrr)r9 output_lengthr,rgrid_ygrid_xr(r(r)r6s    $z@SpeechT5GuidedMultiheadAttentionLoss._make_guided_attention_mask)rrrrrrkrr= BoolTensorrrr0rr6rr(r(rzr)r*s  r*csbeZdZdZdeffdd Z ddejdejdejd ejd ejd ejdBd ej fd dZ Z S)SpeechT5SpectrogramLossz; Loss computation used by SpeechT5ForTextToSpeech. rxcsTt|j|_|j|_|j|_t|_tt dd|_ |jr(t ||_ dSdS)Ng@) pos_weight) rjrkuse_guided_attention_lossguided_attention_loss_num_headsr,r l1_criterionrrr bce_criterionr*attn_criterionrrzr(r)rk#s z SpeechT5SpectrogramLoss.__init__Nr-r^r_r`labelsrr5cs<|dk}||}||}||}||||}|dddddf} tj| dt| dd| jgdd} | ddddf| } || }|| } || } j rtjfdd|Ddd} |dk}|dddddf}j dkr|ddj ddj f} | ||}| |7} | S)Nr.rrrrcs"g|] }|dddjfqSr|)rC)r@xrr(r)rBPs"z3SpeechT5SpectrogramLoss.forward..) r1rDrrrTrrrrErBr,rF)rwr-r^r_r`rGrr&l1_lossr3 stop_labelsbce_lossr5attnr.r/ attn_lossr(rr)r/s(   .   zSpeechT5SpectrogramLoss.forwardr|) rrrrrrkrr<r=rrrr(r(rzr)r@s&r@zv The bare SpeechT5 Encoder-Decoder Model outputting raw hidden-states without any specific pre- or post-nets.  custom_introcseZdZ  ddedejdBdejdBffdd ZddZd d Zd d Z e            dd e j dBde j dBde j dBde j dBdeee jdBdedBdedBde jdBdedBdedBdedBde j dBdee jeBfddZZS) SpeechT5ModelNrxencoderdecodercsJt|||_|durt|n||_|durt|n||_|dS)z encoder (`PreTrainedModel`, *optional*): The encoder model to use. decoder (`PreTrainedModel`, *optional*): The decoder model to use. N)rjrkrxrrQr)rRr)rwrxrQrRrzr(r)rkas  zSpeechT5Model.__init__cCs0t|jtr |jSt|jtr|jStr|)rrQr r rRr(NotImplementedErrorrr(r(r)r us    z"SpeechT5Model.get_input_embeddingscCs8t|jtr |j|t|jtr|j|dSdSr|)rrQr rrRr(rr(r(r)r|s   z"SpeechT5Model.set_input_embeddingscCs t|jtr|jjdSdSz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rrQrrr rr(r(r)r s z$SpeechT5Model.freeze_feature_encoderr+r-decoder_input_valuesdecoder_attention_maskencoder_outputsrkrrMrrrrr5c  Ksb| dur| n|jj} | dur| n|jj} |dur|n|jj}| dur$| n|jj} |dur7|j||| | | d}n$| r[t|ts[t|dt|dkrL|dndt|dkrW|dndd}|durst|jt rs|jj |dj d|}n|}t|j trd|i}ni}|j d |||d|||| | | | d |}| s||St|j|j|j|j|j|j|j|jd S) a input_values (`torch.Tensor` of shape `(batch_size, sequence_length)`): Depending on which encoder is being used, the `input_values` are either: float values of the input raw speech waveform, or indices of input sequence tokens in the vocabulary, or hidden states. decoder_input_values (`torch.Tensor` of shape `(batch_size, target_sequence_length)`, *optional*): Depending on which decoder is being used, the `decoder_input_values` are either: float values of log-mel filterbank features extracted from the raw speech waveform, or indices of decoder input sequence tokens in the vocabulary, or hidden states. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. N)r+r-rrrrrrrrM) r+r-rrrkrrrrr)rrkr'decoder_attentionsrencoder_last_hidden_staterencoder_attentionsr()rxrrrrrQrrrRrrr"r#rRr#rrrkr~rr)rwr+r-rUrVrWrkrrMrrrrrr decoder_argsdecoder_outputsr(r(r)rsj#   zSpeechT5Model.forwardr; NNNNNNNNNNNN)rrrrrrrkr rr rrrr<rr=r rOrrrr(r(rzr)rP[sj       rPzB SpeechT5 Model with a speech encoder and a text decoder. cseZdZddiZdeffdd ZddZdd Zd d Ze dd e j d Bde j d Bde j d Bde j d Bde e e j d Bded Bded Bded Bded Bded Bde j d Bde jd Bde eBfddZZS)SpeechT5ForSpeechToTextz#text_decoder_postnet.lm_head.weightz+speecht5.decoder.prenet.embed_tokens.weightrxc\t||jdurtd|jdt|}t|}t||||_t ||_ | dS)NYou are trying to instantiate a with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SpeechT5ForSpeechToText.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) rjrkrcr%r{rr(rPrrotext_decoder_postnetr)rwrxspeech_encoder text_decoderrzr(r)rk     z SpeechT5ForSpeechToText.__init__cC|jdSrT get_encoderrr rr(r(r)r  z.SpeechT5ForSpeechToText.freeze_feature_encodercCr r|)rartrr(r(r)rtrsz-SpeechT5ForSpeechToText.get_output_embeddingscCr r|)rarwrur(r(r)rwrz-SpeechT5ForSpeechToText.set_output_embeddingsNr+r-decoder_input_idsrVrWrkrrrrrGrr5c  Ks| dur| n|jj} | dur|durt| |jj|jj}|j||||||||| d| d }||d}d}| durJt}||d|jj | d}| s`|f|dd}|dur^|f|S|St |||j |j |j |j|j|j|jd S)a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) SpeechT5 uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Label indices can be obtained using [`SpeechT5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToText >>> from datasets import load_dataset >>> dataset = load_dataset( ... "hf-internal-testing/librispeech_asr_demo", "clean", split="validation" ... ) # doctest: +IGNORE_RESULT >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_asr") >>> model = SpeechT5ForSpeechToText.from_pretrained("microsoft/speecht5_asr") >>> # audio file is decoded on the fly >>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> predicted_ids = model.generate(**inputs, max_length=100) >>> # transcribe speech >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> transcription[0] 'mister quilter is the apostle of the middle classes and we are glad to welcome his gospel' ``` ```python >>> inputs["labels"] = processor(text_target=dataset[0]["text"], return_tensors="pt").input_ids >>> # compute loss >>> loss = model(**inputs).loss >>> round(loss.item(), 2) 19.68 ``` NT) r+r-rUrVrWrkrrrrrrr!r) r5r`rkr'rXrrYrrZ)rxrr*rr rrarrrcrrkr'rXrrYrrZ)rwr+r-rirVrWrkrrrrrGrrrr`r5loss_fctoutputr(r(r)rsLTzSpeechT5ForSpeechToText.forwardr])rrr_tied_weights_keysrrkr rtrwrrr=r<rr rOrrrrr(r(rzr)r^s\     r^?r/4@FmodelrM threshold minlenratio maxlenratiovocoderoutput_cross_attentionsreturn_output_lengthsc " s|durtd|durd||jjk} n|} |d} |jj|| dd} | j} t|jjt r?|jjj | dj d| } t| d||jj }t| d||jj }| | d|jj}g}g}d}d}i |d7}|jj ||}|jjj|ddddfd| | |d|dd}|r|tj|jdd|jd}|j}|j|}|| |jj |jj}|||dddddf| d|jj}tj||fdd}t|j|}||krql||krtj|dd|k}t|d}nt t!|}fd d |D}t!|dkr3t"|}|#dd$dd }|j%|}|D] }|||<q)t!| kr;nqmfd d t t!D}| s| dkrU|dn tj&j'j(j)|dd }|duri||}n|}|rtj|d d}| dkr|j| t|d| g|ddR}||f}|Sgt | D] } || dq|durtj&j'j(j)|dd }|f}ngtj&j'j(j)|dd }||fdd D}!|!f}|rtj|d d}|j| t|d| g|ddR}g||R}|S)Na`speaker_embeddings` must be specified. For example, you can use a speaker embeddings by following the code snippet provided in this link: https://huggingface.co/datasets/Matthijs/cmu-arctic-xvectors rrT)r+r-rr!)r~r-rrrkrrrrcsg|]}|vr|qSr(r(rresult_spectrogramr(r)rBrz$_generate_speech..rcsg|]}|qSr(r(rrvr(r)rB r) batch_firstcs&g|]}tdt|qSr)r7rr8r)spectrogram_lengths waveformsr(r)rB s&)*r%rxrr7rrrQrrrrr"r#r,r"r?rRr$rVrrrsqueezerkspeech_decoder_postnetr[rsigmoidr\rKrrLrMrRstackrflattenr]rrrnn pad_sequence)"ror+rMr-rprqrrrsrtrurr encoder_outrYmaxlenminlenoutput_sequence spectrogramrrkrr' decoder_outlast_decoder_outputspectrumnew_spectrogramprobmeet_thresholds meet_indexes spectrograms meet_indexrrwaveform_lengthsr()rwr{r|r)_generate_speechs       $   5&      rzB SpeechT5 Model with a text encoder and a speech decoder. c!seZdZdZdZdeffdd ZedefddZ e d'de j d Bd e j d Bd e j d Bd e j d Bd eee j d Bded Bded Bded Bded Bded Bde j d Bde j d Bde jd Bde jd BdeeBfddZe      d(de j d e j d Bde j d Bdededed ejd Bd!ed"ede j ee j e j fBfd#d$Ze      d(de j de j d Bd e j d Bdededed ejd Bd!ed"ede j ee j e j fBfd%d&ZZS))SpeechT5ForTextToSpeech)textrrxcr_)Nr`a with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SpeechT5ForTextToSpeech.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) rjrkrcr%r{r r#rPrrZr~r)rwrx text_encoderspeech_decoderrzr(r)rk2 rdz SpeechT5ForTextToSpeech.__init__r5cCsdSr r()clsr(r(r) can_generateF sz$SpeechT5ForTextToSpeech.can_generateNr-rUrVrWrkrrrrrMrGrJrcKs| dur| n|jj} | dur"|durt| |jj|\}}|jjr"d}|j|||||||| || d|d }||d\}}}d}| durRt|j}|||||| |j}| sh|f|dd}|durf|f|S|St |||j |j |j |j|j |j|jd S)ab input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`): Float values of input mel spectrogram. SpeechT5 uses an all-zero spectrum as the starting token for `decoder_input_values` generation. If `past_key_values` is used, optionally only the last `decoder_input_values` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. labels (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`, *optional*): Float values of target mel spectrogram. Timesteps set to `-100.0` are ignored (masked) for the loss computation. Spectrograms can be obtained using [`SpeechT5Processor`]. See [`SpeechT5Processor.__call__`] for details. stop_labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Binary tensor indicating the position of the stop token in the sequence. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan, set_seed >>> import torch >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_tts") >>> model = SpeechT5ForTextToSpeech.from_pretrained("microsoft/speecht5_tts") >>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan") >>> inputs = processor(text="Hello, my dog is cute", return_tensors="pt") >>> speaker_embeddings = torch.zeros((1, 512)) # or load xvectors from a file >>> set_seed(555) # make deterministic >>> # generate speech >>> speech = model.generate(inputs["input_ids"], speaker_embeddings=speaker_embeddings, vocoder=vocoder) >>> speech.shape torch.Size([15872]) ``` NT r+r-rUrVrWrkrrMrrrrrr r5rrkr'rXrrYrrZ)rxrr1r,rBrr~r@rrrkr'rXrYrrZ)rwrr-rUrVrWrkrrrrrMrGrJrrrr^r_r`r5 criterionrkr(r(r)rM s`F   zSpeechT5ForTextToSpeech.forwardrmr/rnFrprqrrrsrtruc Ks^|dur"|d} |d| kr"|ddkr|| d}ntdt|||||||||| S)aE Converts a sequence of input tokens into a sequence of mel spectrograms, which are subsequently turned into a speech waveform using a vocoder. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Attention mask from the tokenizer, required for batched inference to signal to the model where to ignore padded tokens from the input_ids. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. threshold (`float`, *optional*, defaults to 0.5): The generated sequence ends when the predicted stop token probability exceeds this value. minlenratio (`float`, *optional*, defaults to 0.0): Used to calculate the minimum required length for the output sequence. maxlenratio (`float`, *optional*, defaults to 20.0): Used to calculate the maximum allowed length for the output sequence. vocoder (`nn.Module`, *optional*): The vocoder that converts the mel spectrogram into a speech waveform. If `None`, the output is the mel spectrogram. output_cross_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of the decoder's cross-attention layers. return_output_lengths (`bool`, *optional*, defaults to `False`): Whether or not to return the concrete spectrogram/waveform lengths. Returns: `tuple(torch.FloatTensor)` comprising various elements depending on the inputs: - when `return_output_lengths` is False - **spectrogram** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrogram. - **waveform** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(num_frames,)` -- The predicted speech waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. - when `return_output_lengths` is True - **spectrograms** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrograms that are padded to the maximum length. - **spectrogram_lengths** (*optional*, returned when no `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each spectrogram. - **waveforms** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, num_frames)` -- The predicted speech waveforms that are padded to the maximum length. - **waveform_lengths** (*optional*, returned when a `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(batch_size, config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. NrrzUThe first dimension of speaker_embeddings must be either 1 or the same as batch_size.rrIr%r) rwrr-rMrprqrrrsrtrurr[r(r(r)generate s(E z SpeechT5ForTextToSpeech.generatec Cs^|dur"|d} |d| kr"|ddkr|| d}ntdt|||||||||| S)a Converts a sequence of input tokens into a sequence of mel spectrograms, which are subsequently turned into a speech waveform using a vocoder. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) threshold (`float`, *optional*, defaults to 0.5): The generated sequence ends when the predicted stop token probability exceeds this value. minlenratio (`float`, *optional*, defaults to 0.0): Used to calculate the minimum required length for the output sequence. maxlenratio (`float`, *optional*, defaults to 20.0): Used to calculate the maximum allowed length for the output sequence. vocoder (`nn.Module`, *optional*, defaults to `None`): The vocoder that converts the mel spectrogram into a speech waveform. If `None`, the output is the mel spectrogram. output_cross_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of the decoder's cross-attention layers. return_output_lengths (`bool`, *optional*, defaults to `False`): Whether or not to return the concrete spectrogram/waveform lengths. Returns: `tuple(torch.FloatTensor)` comprising various elements depending on the inputs: - when `return_output_lengths` is False - **spectrogram** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrogram. - **waveform** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(num_frames,)` -- The predicted speech waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. - when `return_output_lengths` is True - **spectrograms** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrograms that are padded to the maximum length. - **spectrogram_lengths** (*optional*, returned when no `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each spectrogram. - **waveforms** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, num_frames)` -- The predicted speech waveforms that are padded to the maximum length. - **waveform_lengths** (*optional*, returned when a `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(batch_size, config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. NrrzUThe first dimension of speaker_embeddings must be either 1 or the same as batch size.r) rwrrMr-rprqrrrsrtrur[r(r(r)generate_speech& s(I z'SpeechT5ForTextToSpeech.generate_speechNNNNNNNNNNNNNNNNrmr/rnNFF)rrrrrrrk classmethodrOrrrr<r=rr rrrrrrrrrrr(r(rzr)r) s      |   [   rzD SpeechT5 Model with a speech encoder and a speech decoder. c!sJeZdZdeffdd ZddZe              d$dejdBdej dBd ejdBd ej dBd e e ejdBd e dBd e dBde dBde dBde dBdejdBdejdBdej dBdej dBde eBfddZe        d%dejdejdBdej dBdedededejdBd e d!e dejfd"d#ZZS)&SpeechT5ForSpeechToSpeechrxcs@t|t|}t|}t||||_t||_|dSr|) rjrkrr#rPrrZr~r)rwrxrbrrzr(r)rk s   z"SpeechT5ForSpeechToSpeech.__init__cCrerTrfrr(r(r)r  rhz0SpeechT5ForSpeechToSpeech.freeze_feature_encoderNr+r-rUrVrWrkrrrrrMrGrJrr5cKs| dur| n|jj} | dur|durt| |jj|\}}|j|||||||| || d|d }||d\}}}d}| sO|f|dd}|durM|f|S|St|||j|j|j |j |j |j |j d S)a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. decoder_input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`): Float values of input mel spectrogram. SpeechT5 uses an all-zero spectrum as the starting token for `decoder_input_values` generation. If `past_key_values` is used, optionally only the last `decoder_input_values` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. labels (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`, *optional*): Float values of target mel spectrogram. Spectrograms can be obtained using [`SpeechT5Processor`]. See [`SpeechT5Processor.__call__`] for details. stop_labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Binary tensor indicating the position of the stop token in the sequence. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToSpeech, SpeechT5HifiGan, set_seed >>> from datasets import load_dataset >>> import torch >>> dataset = load_dataset( ... "hf-internal-testing/librispeech_asr_demo", "clean", split="validation" ... ) # doctest: +IGNORE_RESULT >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_vc") >>> model = SpeechT5ForSpeechToSpeech.from_pretrained("microsoft/speecht5_vc") >>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan") >>> # audio file is decoded on the fly >>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> speaker_embeddings = torch.zeros((1, 512)) # or load xvectors from a file >>> set_seed(555) # make deterministic >>> # generate speech >>> speech = model.generate_speech(inputs["input_values"], speaker_embeddings, vocoder=vocoder) >>> speech.shape torch.Size([77824]) ``` NTrrrr)rxrr1r,rr~rrkr'rXrrYrrZ)rwr+r-rUrVrWrkrrrrrMrGrJrrrrArr`r5rkr(r(r)r sHM z!SpeechT5ForSpeechToSpeech.forwardrmr/rnFrprqrrrsrtruc Cs2|dur tjd|jd}t|||||||||| S)a' Converts a raw speech waveform into a sequence of mel spectrograms, which are subsequently turned back into a speech waveform using a vocoder. Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) threshold (`float`, *optional*, defaults to 0.5): The generated sequence ends when the predicted stop token probability exceeds this value. minlenratio (`float`, *optional*, defaults to 0.0): Used to calculate the minimum required length for the output sequence. maxlenratio (`float`, *optional*, defaults to 20.0): Used to calculate the maximum allowed length for the output sequence. vocoder (`nn.Module`, *optional*, defaults to `None`): The vocoder that converts the mel spectrogram into a speech waveform. If `None`, the output is the mel spectrogram. output_cross_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of the decoder's cross-attention layers. return_output_lengths (`bool`, *optional*, defaults to `False`): Whether or not to return the concrete spectrogram/waveform lengths. Returns: `tuple(torch.FloatTensor)` comprising various elements depending on the inputs: - when `return_output_lengths` is False - **spectrogram** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrogram. - **waveform** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(num_frames,)` -- The predicted speech waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. - when `return_output_lengths` is True - **spectrograms** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrograms that are padded to the maximum length. - **spectrogram_lengths** (*optional*, returned when no `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each spectrogram. - **waveforms** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, num_frames)` -- The predicted speech waveforms that are padded to the maximum length. - **waveform_lengths** (*optional*, returned when a `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(batch_size, config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. N)rir))rrNrr) rwr+rMr-rprqrrrsrtrur(r(r)r sJz)SpeechT5ForSpeechToSpeech.generate_speechrr)rrrrrkr rrr=r<rr rOrrrrrrrrrr(r(rzr)r s       w   rcs@eZdZdfdd ZdddZd d Zd d Zd dZZS)HifiGanResidualBlockrrr皙?csbt|_tfddttD_tfddttD_dS)Nc s2g|]}tjd||dqSr)rhdilationrrro get_paddingrchannelsrrgrwr(r)rBy s z1HifiGanResidualBlock.__init__..c s*g|]}tjddddqSrrr?)rrgrwr(r)rB s  ) rjrkleaky_relu_sloperrrMrRconvs1convs2)rwrrgrrrzrr)rkt s     zHifiGanResidualBlock.__init__rcCs|||dSrr()rwrgrr(r(r)r rz HifiGanResidualBlock.get_paddingcCsLtjj}ttjjdrtjjj}|jD]}||q|jD]}||qdSNr)rrrrrrrrwrrr(r(r)apply_weight_norm s     z&HifiGanResidualBlock.apply_weight_normcCs4|jD]}tj|q|jD]}tj|qdSr|)rrrremove_weight_normrrwrr(r(r)r s  z'HifiGanResidualBlock.remove_weight_normcCsXt|j|jD]"\}}|}tj||j}||}tj||j}||}||}q|Sr|)r2rrrrN leaky_relur)rwr~conv1conv2rr(r(r)r s zHifiGanResidualBlock.forward)rrrrz) rrrrkrrrrrr(r(rzr)rs s   rz HiFi-GAN vocoder. csleZdZUeed<dZdeffdd ZfddZddZd d Z e d d de j d e j fddZ ZS)rrxrc sNt|t|j|_t|j|_tj|j |j dddd|_ t |_ tt|j|jD]$\}\}}|j tj|j d||j d|d||||ddq-t |_tt|j D]#}|j d|d}t|j|jD]\}}|jt||||jqpq^tj|ddddd|_|dt|j |dt|j |dS)Nrr)rgrhrrrr)rjrkrRresblock_kernel_sizes num_kernelsupsample_rates num_upsamplesrro model_in_dimupsample_initial_channelconv_prer upsamplerrr2upsample_kernel_sizesrVConvTranspose1d resblocksrMresblock_dilation_sizesrr conv_postrrrNrTr)rwrxr upsample_ratergrrrzr(r)rk s>        zSpeechT5HifiGan.__init__cs6t|t|trt|jt|jdSdSr|) rjrrrrrrrr)rwrrzr(r)r s   zSpeechT5HifiGan._init_weightscCs`tjj}ttjjdrtjjj}||j|jD]}||q|jD]}|q"||j dSr) rrrrrrrrrrrr(r(r)r s      z!SpeechT5HifiGan.apply_weight_normcCsLtj|j|jD]}tj|q |jD]}|qtj|jdSr|)rrrrrrrrr(r(r)r s    z"SpeechT5HifiGan.remove_weight_norma Converts a log-mel spectrogram into a speech waveform. Passing a batch of log-mel spectrograms returns a batch of speech waveforms. Passing a single, un-batched log-mel spectrogram returns a single, un-batched speech waveform. rNr5c Ks|jjr ||j|j}|dk}|s|d}|dd}||}t|j D]8}t j ||jj }|j||}|j||j|}td|jD]}||j||j||7}qK||j}q)t j |}||}t|}|s|dddd}|S|d}|S)a spectrogram (`torch.FloatTensor`): Tensor containing the log-mel spectrograms. Can be batched and of shape `(batch_size, sequence_length, config.model_in_dim)`, or un-batched and of shape `(sequence_length, config.model_in_dim)`. Returns: `torch.FloatTensor`: Tensor containing the speech waveform. If the input spectrogram is batched, will be of shape `(batch_size, num_frames,)`. If un-batched, will be of shape `(num_frames,)`. rrrrr!)rxnormalize_beforerrrrrrrMrrrNrrrrrrrtanhr}r) rwrr is_batchedr~r res_statejwaveformr(r(r)r s,         zSpeechT5HifiGan.forward)rrrrrrrkrrrrrr=rrr(r(rzr)r s  &  r)r^rrrPrr)rNrr)errnumpyrFrrtorch.nnrrrrr activationsr cache_utilsr r r generationr integrations.deepspeedr integrations.fsdpr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrrrrrconfiguration_speecht5rr get_loggerrr_HIDDEN_STATES_START_POSITIONrr7r*r1rrr<ndarrayrcrerrrrrrrrrr rr>rRrZrbrfrorxrrrrrrr rrr#r(r)r*r@rPr^r=rOrrrrr__all__r(r(r(r)s            xF- )4(%9]Bq$)*/3*;=*   [h> v