o wi=@sdZddlZddlmZmZddlZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZm Z ddl!m"Z"ddl#m$Z$m%Z%ddl&m'Z'm(Z(e%)e*Z+dZ,dej-de.de.fddZ/ ddej-de.deej-fddZ0  dde1e.e.fde2de.deej3de.d ej4f d!d"Z5Gd#d$d$eZ6Gd%d&d&eZ7Gd'd(d(eZ8Gd)d*d*e j9Z:Gd+d,d,e j9Z;Gd-d.d.e j9ZGd3d4d4e j9Z?Gd5d6d6e j9Z@Gd7d8d8e j9ZAGd9d:d:e j9ZBGd;d<dd>e j9ZDGd?d@d@e j9ZEGdAdBdBe j9ZFGdCdDdDe j9ZGGdEdFdFe j9ZHGdGdHdHe j9ZIGdIdJdJeZJGdKdLdLeZKe$GdMdNdNe"ZLGdOdPdPeLZMGdQdRdReLZNGdSdTdTeLZOGdUdVdVeLZPGdWdXdXeLZQGdYdZdZeLZRGd[d\d\eLZSGd]d^d^eLZTGd_d`d`e j9ZUGdadbdbe j9ZVe$dcddGdedfdfeLZWe$dgddGdhdidieLeZX   j k l  m mddneLdejYdoeejYdeej3dpe2dqe2dre2dsee j9dteZdueZd eejYe1ejYejYfffdvdwZ[e$dxddGdydzdzeLZ\e$d{ddGd|d}d}eLZ]Gd~dde j9Z^e$dddGddde"Z_gdZ`dS)zPyTorch SpeechT5 model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossL1Loss)ACT2FN)GenerationMixin)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutputSeq2SeqSpectrogramOutput)PreTrainedModel)auto_docstringlogging)SpeechT5ConfigSpeechT5HifiGanConfig input_ids pad_token_iddecoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|dur*td||dk||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrrshifted_input_idsr%k/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/speecht5/modeling_speecht5.pyshift_tokens_right5s (r' input_valuesreduction_factorattention_maskcCs|dkr"|dd|dd|f}|dur"|dd|dd|f}||j}|ddddf|ddddf<||dkd||fS)zw Shift input spectrograms one timestep to the right. Also applies the reduction factor to the sequence length. rNrY)rr r!r#)r(r)r*shifted_input_valuesr%r%r&shift_spectrograms_rightEs (r.r mask_prob mask_length min_masksreturncs|\}dkr tdkrtdddtjdfdd}|dur:|d n fd d t|D}tj |ft d }g} |} | d krZ|S|D];} || } tjj t | d| dd} t | d kr}d}n| d }t| tj| | tjd |g} | | q\t| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkrd| | dk<t|| dd |S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr}|d|kr*t|dd}|S)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonr0r/r1sequence_lengthr%r&compute_num_masked_spans  z6_compute_mask_indices..compute_num_masked_spanNrcsg|]}qSr%r%.0_)r:r%r& sz)_compute_mask_indices..dtyperF)replace)r"nprandomranditemdetachsumtolistrangezerosboolchoicearangelen concatenateonesint32appendarray broadcast_toreshaper5put_along_axis)r r/r0r*r1 batch_sizer; input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanr6r7spec_aug_mask_idxdummy_mask_idxoffsetsr%r8r&_compute_mask_indices[s\    r`c&eZdZdfdd ZddZZS)SpeechT5NoLayerNormConvLayerrcsjt|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__r%r&rhs  z%SpeechT5NoLayerNormConvLayer.__init__cCs||}||}|SN)rprrrt hidden_statesr%r%r&forward  z$SpeechT5NoLayerNormConvLayer.forwardr__name__ __module__ __qualname__rhr| __classcell__r%r%rwr&rbsrbcra)SpeechT5LayerNormConvLayerrcs|t|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)NrrrcT)elementwise_affine)rgrhrirjrkrrlrmrnrorp LayerNorm layer_normr rqrrrsrwr%r&rhs  z#SpeechT5LayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)Nr)rp transposerrrrzr%r%r&r|s     z"SpeechT5LayerNormConvLayer.forwardr~rr%r%rwr&rsrcra)SpeechT5GroupNormConvLayerrcst|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)NrrrcT) num_groups num_channelsaffine)rgrhrirjrkrrlrmrnrorpr rqrr GroupNormrrsrwr%r&rhs   z#SpeechT5GroupNormConvLayer.__init__cCs"||}||}||}|Sry)rprrrrzr%r%r&r|s   z"SpeechT5GroupNormConvLayer.forwardr~rr%r%rwr&rsrc seZdZdZddededeeffdd Zddededeefd d Zeddededeefd d Z e dde j defddZ dde j dedeefddZZS)%SpeechT5SinusoidalPositionalEmbeddingzDThis module produces sinusoidal positional embeddings of any length.N num_positions embedding_dim padding_idxcs4td|_||_||_|||j||dSN)rgrhoffsetrr make_weights)rtrrrrwr%r&rh!s z.SpeechT5SinusoidalPositionalEmbedding.__init__num_embeddingscCsB||||}t|dr|j|jj|jjd}|jd|dddS)NweightsrAdeviceF persistent) get_embeddinghasattrtorrArregister_buffer)rtrrr emb_weightsr%r%r&r(s z2SpeechT5SinusoidalPositionalEmbedding.make_weightscCs|d}td|d}ttj|tjd| }tj|tjdd|d}tjt |t |gdd |d}|ddkrUtj|t |dgdd}|durad||ddf<| tS) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'rr@rdimrN)mathlogtorchexprNint64float unsqueezecatsincosviewrKrget_default_dtype)rrrhalf_dimembr%r%r&r0s $& z3SpeechT5SinusoidalPositionalEmbedding.get_embeddingrrpast_key_values_lengthcCs||\}}|||j||j}|jd|}||jdkr-|||j|j|j|j d| d ||d S)Nrrr) size"create_position_ids_from_input_idsrrrrrrr index_selectrrG)rtrrbszseq_len position_idsmax_posr%r%r&r|Bs "z-SpeechT5SinusoidalPositionalEmbedding.forwardcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)ner4rcumsumtype_aslong)rtrrrmaskincremental_indicesr%r%r&rQs  zHSpeechT5SinusoidalPositionalEmbedding.create_position_ids_from_input_idsryr~)rrr__doc__r4rrhr staticmethodrrno_gradTensorr|rrr%r%rwr&rs  rc$eZdZfddZddZZS)SpeechT5PositionalConvEmbeddingcs$ttj|j|j|j|jd|jd|_tjj }t tjj dr'tjj j }t r{ddl }|jj|jjdd||jddd|_Wdn1sLwYt |jdrd|jj jj}|jj jj}n|jj}|jj}|j|||j||n ||jddd|_t|j|_t|j|_dS) Nr)rdpaddinggroups weight_normr) modifier_rankweight)namerparametrizations)rgrhrrl hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupsrputilsrrrr deepspeedzeroGatheredParametersr original0 original1weight_gweight_vregister_external_parameterSpeechT5SamePadLayerrr rqrr)rtrurrrrrwr%r&rhds4     z(SpeechT5PositionalConvEmbedding.__init__cCs:|dd}||}||}||}|dd}|SNrr)rrprrrrzr%r%r&r|s     z'SpeechT5PositionalConvEmbedding.forwardrr%r%rwr&rcs !rcs*eZdZdZdfdd ZddZZS) SpeechT5ScaledPositionalEncodingu[ Scaled positional encoding, see §3.2 in https://huggingface.co/papers/1809.08895 cst||}td|d}ttjd|dtjdtd| }t |||dddddf<t |||dddddf<|d}t |j d|ddtj|d |_||_tjtd |_dS) Nrrrr@g@peFrp?)rrKrNrrrrrrrrrgrhrrDropoutdropoutr Parametertensoralpha)rtrrmax_lenrpositiondiv_termrwr%r&rhs .$$  z)SpeechT5ScaledPositionalEncoding.__init__cCs4||j|jddd|df}||}|S)Nr)rrrr)rtrr%r%r&r|s& z(SpeechT5ScaledPositionalEncoding.forward)r)rrrrrhr|rr%r%rwr&rs rcra)"SpeechT5RelativePositionalEncodingcs.t||_||_tjd|||_dSr)rgrhr max_lengthrr Embeddingpe_k)rtrrrwr%r&rhs z+SpeechT5RelativePositionalEncoding.__init__cCs|jd}td|j|jtjd}|dddf|dddf}|j |||j k<|jd|||jk<||j}||S)NrrrrA)r rrNrrrrr)rtr{rpos_seqr%r%r&r|s    z*SpeechT5RelativePositionalEncoding.forward)rrr%r%rwr&rsrcr)rcs*t|ddkrd|_dSd|_dS)Nrrr)rgrhnum_pad_remove)rtrrwr%r&rhs  zSpeechT5SamePadLayer.__init__cCs,|jdkr|ddddd|j f}|SNr)rrzr%r%r&r|s zSpeechT5SamePadLayer.forwardrr%r%rwr&rs rcs0eZdZdZfddZddZddZZS)SpeechT5FeatureEncoderz.Construct the features from raw audio waveformcstjdkr tddgfddtjdD}njdkr2fddtjD}n td jd t||_ d |_ d |_ dS) Ngrouprrvcsg|] }t|ddqS)rr)rbr=irur%r&r?sz3SpeechT5FeatureEncoder.__init__..rlayercsg|]}t|dqS)r)rrrr%r&r?s z`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rgrhfeat_extract_normrrJnum_feat_extract_layersr"r ModuleList conv_layersgradient_checkpointing_requires_grad)rtrurrwrr&rhs        zSpeechT5FeatureEncoder.__init__cCs|D]}d|_qd|_dS)NF) parameters requires_gradr)rtparamr%r%r&_freeze_parameterss  z)SpeechT5FeatureEncoder._freeze_parameterscCs:|dddf}|jr|jrd|_|jD]}||}q|SNT)rtrainingrr)rtr(r{ conv_layerr%r%r&r|s    zSpeechT5FeatureEncoder.forward)rrrrrhrr|rr%r%rwr&rs  rcr)SpeechT5FeatureProjectioncsJttj|jd|jd|_t|jd|j|_ t |j |_ dS)Nreps) rgrhrrrilayer_norm_epsrLinearr projectionrfeat_proj_dropoutrrtrurwr%r&rhs z"SpeechT5FeatureProjection.__init__cCs&||}||}||}||fSry)rr r)rtr{norm_hidden_statesr%r%r&r|s   z!SpeechT5FeatureProjection.forwardrr%r%rwr&rs rcseZdZfddZddZ  ddejdeejdeej fd d Z d e dejfd d Z de eje ffddZ  ddej deej deejfddZZS)SpeechT5SpeechEncoderPrenetcs|t||_t||_t||_|jdks|jdkr(t t |j |_t||_t|j|jd|j |j|_dS)Nr,r)rgrhrurfeature_encoderrfeature_projectionmask_time_probmask_feature_probrrrrruniform_masked_spec_embedrpos_conv_embedrmax_speech_positionsrpos_sinusoidal_embedr rwr%r&rhs     z$SpeechT5SpeechEncoderPrenet.__init__cCs|jdSry)rrrtr%r%r&freeze_feature_encodersz2SpeechT5SpeechEncoderPrenet.freeze_feature_encoderNr(r*mask_time_indicesc Cs||}|dd}|dur||jd|}||\}}|j|||d}||}||}|dur<|d}nt j |jddt j|j d}| |}||}||fS)Nrr)rr*r) rr"_get_feature_vector_attention_maskr r_mask_hidden_statesrrrrrKrr) rtr(r*rextract_featuresr{positional_conv_embedding padding_mask positional_sinusoidal_embeddingsr%r%r&r|s&    z#SpeechT5SpeechEncoderPrenet.forwardfeature_vector_lengthcCs|jdddddf}||tj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dgd dg }|S)Nrrrrrr) r _get_feat_extract_output_lengthsrrrr rKrArrNfliprL)rtr"r*non_padded_lengthsoutput_lengthsrXr%r%r&r8s "z>SpeechT5SpeechEncoderPrenet._get_feature_vector_attention_maskrYcCs4dd}t|jj|jjD] \}}||||}q |S)zH Computes the output length of the convolutional layers cSstj|||dddS)Nfloor) rounding_moder)rdiv)r6rdrer%r%r&_conv_out_lengthMszVSpeechT5SpeechEncoderPrenet._get_feat_extract_output_lengths.._conv_out_length)ziprurmrn)rtrYr+rdrer%r%r&r$Hsz.)rgrhrurrrJspeech_decoder_prenet_layerslayersr r:r final_layerrpositional_dropoutrencode_positionsspeaker_embedding_dimspeaker_embeds_layerr rwrr&rhs   z$SpeechT5SpeechDecoderPrenet.__init__cCsJtj|d|d}|d|ddd}t|dk|ddd|S)Nrrr)r bernoullirrepeatrwhere)rt inputs_embedsrr all_masksr%r%r&_consistent_dropoutsz/SpeechT5SpeechDecoderPrenet._consistent_dropoutNr(speaker_embeddingscCs|}|jD]}tj||}|||jj}q||}||}|durKtj |}| d d| dd}t j||gdd}tj||}|S)Nrrr)r<r functionalrelurGruspeech_decoder_prenet_dropoutr=r? normalizerr3rrrrA)rtr(rHrErr%r%r&r|s    z#SpeechT5SpeechDecoderPrenet.forwardry) rrrrhrGrrrr|rr%r%rwr&r8s r8cra)SpeechT5BatchNormConvLayerrcst|dkr |j}n|j}||jdkr|j}n|j}tj|||jd|jdddd|_t ||_ ||jdkrCt |_ nd|_ t |j|_dS)NrrrF)rdrerrf)rgrhr9speech_decoder_postnet_unitsspeech_decoder_postnet_layersrrlspeech_decoder_postnet_kernelrp BatchNorm1d batch_normTanhrrrspeech_decoder_postnet_dropoutr)rtrurvrjrkrwr%r&rhs(    z#SpeechT5BatchNormConvLayer.__init__cCs6||}||}|jdur||}||}|Sry)rprRrrrrzr%r%r&r|s     z"SpeechT5BatchNormConvLayer.forwardr~rr%r%rwr&rMsrMcs<eZdZfddZdejfddZdejfddZZS)SpeechT5SpeechDecoderPostnetcs^t|_tjjj|_tjj|_ t fddt j D|_ dS)Ncsg|]}t|qSr%)rMrrr%r&r?sz9SpeechT5SpeechDecoderPostnet.__init__..)rgrhrurr rr9r)feat_outprob_outrrJrOr<r rwrr&rhs  z%SpeechT5SpeechDecoderPostnet.__init__r{cCsJ|||dd|jj}||}|||dd}|||fS)Nrr)rVrrrur9postnetrW)rtr{outputs_before_postnetoutputs_after_postnetlogitsr%r%r&r|s  z$SpeechT5SpeechDecoderPostnet.forwardcCs0|dd}|jD]}||}q ||ddSr)rr<)rtr{ layer_outputrr%r%r&rXs   z$SpeechT5SpeechDecoderPostnet.postnet) rrrrhrrr|rXrr%r%rwr&rUs  rUcs<eZdZfddZddZddZdejfdd ZZ S) SpeechT5TextEncoderPrenetcs>t||_t|j|j|j|_t |j |j|j |_ dSry) rgrhrurr vocab_sizerr embed_tokensrr>max_text_positionsr?r rwr%r&rhs  z"SpeechT5TextEncoderPrenet.__init__cC|jSryr_rr%r%r&get_input_embeddingsz.SpeechT5TextEncoderPrenet.get_input_embeddingscC ||_dSryrbrtvaluer%r%r&set_input_embeddings  z.SpeechT5TextEncoderPrenet.set_input_embeddingsrcCs||}||}|Sry)r_r?)rtrrEr%r%r&r| r}z!SpeechT5TextEncoderPrenet.forward) rrrrhrcrhrrr|rr%r%rwr&r]s   r]c sZeZdZfddZddZddZ  d dejd eej d ee ej fd d Z Z S)SpeechT5TextDecoderPrenetcsnt||_t|j|_|jrt |j nd|_ t |j |j |j|_t|j|jd|j |j|_dS)Nrr)rgrhrurrr>rscale_embeddingrsqrtr embed_scalerr^rr_rr`embed_positionsr rwr%r&rhs  z"SpeechT5TextDecoderPrenet.__init__cCraryrbrr%r%r&rc"rdz.SpeechT5TextDecoderPrenet.get_input_embeddingscCreryrbrfr%r%r&rh%riz.SpeechT5TextDecoderPrenet.set_input_embeddingsNrr*past_key_valuescCs~|dur|}|d|d}ntd|dur"|ddjdnd}|||}|||j}||7}||}||fS)Nrz'You have to specify `decoder_input_ids`rr)rrr"r rnr_rmr)rtrr*ro input_shaper positionsrEr%r%r&r|(s  z!SpeechT5TextDecoderPrenet.forwardr5)rrrrhrcrhrrrr6listr7r|rr%r%rwr&rjs  rjcs<eZdZfddZdejfddZddZdd ZZ S) SpeechT5TextDecoderPostnetcs*t||_tj|j|jdd|_dS)NFrf)rgrhrurr rr^lm_headr rwr%r&rh?s z#SpeechT5TextDecoderPostnet.__init__r{cCs ||Sryrurzr%r%r&r|Driz"SpeechT5TextDecoderPostnet.forwardcCraryrvrr%r%r&get_output_embeddingsGrdz0SpeechT5TextDecoderPostnet.get_output_embeddingscCreryrvrtnew_embeddingsr%r%r&set_output_embeddingsJriz0SpeechT5TextDecoderPostnet.set_output_embeddings) rrrrhrrr|rwrzrr%r%rwr&rs>s  rscseZdZdZ   ddedededed ef fd d Zd ej d edefddZ      ddej de ej de e ej de ej de ej de ej dede ej e ej e e ej ffddZ ZS)SpeechT5Attentionz Multi-headed attention from 'Attention Is All You Need' paper with relative position bias (see https://aclanthology.org/N18-2074.pdf) r,FT embed_dim num_headsr is_decoderrfcst||_||_||_|||_|j||jkr'td|jd|d|jd|_||_t j |||d|_ t j |||d|_ t j |||d|_ t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).grt)rgrhr|r}rhead_dimr"scalingr~rr k_projv_projq_projout_proj)rtr|r}rr~rfrwr%r&rhTs"    zSpeechT5Attention.__init__rrrcCs ||||j|jddSr)rr}rr contiguous)rtrrrr%r%r&_shapeos zSpeechT5Attention._shapeNr{key_value_statespast_key_valuer*layer_head_mask position_biasoutput_attentionsr2cCs|du}|\} } } |||j} |r"|dur"|d} |d}nZ|r9|||d| } |||d| }nC|durh|||d| } |||d| }tj|d| gdd} tj|d|gdd}n|||d| } |||d| }|jr| |f}| |j d|j f}|| | | j |} | j |} |j |}| d}t | | dd}|| |j | |fkrtd| |j | |fd||dur|  | |j d|j  dd}t|| d d}| dd | |j |d|d}||7}|dur:|| d| |fkr%td | d| |fd|| | |j | ||}| | |j | |}tjj|dd}|dury||j fkr^td |j fd|| dddd| | |j | |}| | |j | |}|r| | |j | |}| | |j | |}nd}tjj||j|jd }t ||}|| |j | |j fkrtd | |j | |j fd|| | |j | |j }| dd}|| | |j}||}|||fS)z#Input shape: Batch x Time x ChannelNrrrrrz$Attention weights should be of size z , but is rz!Attention mask should be of size z/Head mask for a single layer should be of size )rrz `attn_output` should be of size )rrrrrrrrr~r}rrbmmrr"rmatmulrrIsoftmaxrrrVr|r)rtr{rrr*rrris_cross_attentionrtgt_lenr> query_states key_states value_states proj_shapesrc_len attn_weights reshape_q rel_pos_biasattn_weights_reshaped attn_probs attn_outputr%r%r&r|rs      "    "    zSpeechT5Attention.forward)r,FT)NNNNNF)rrrrr4rrLrhrrrrtupler|rr%r%rwr&r{NsP   r{cr)SpeechT5FeedForwardcsltt|j|_t|j||_t |j t r!t |j |_ n|j |_ t||j|_t|j|_dSry)rgrhrractivation_dropoutintermediate_dropoutr rintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropout)rtruintermediate_sizerwr%r&rhs  zSpeechT5FeedForward.__init__cCs6||}||}||}||}||}|Sry)rrrrrrzr%r%r&r|s     zSpeechT5FeedForward.forwardrr%r%rwr&rs  rc s^eZdZdeffdd Z    d dejdeejdeejd eejd ef d d Z Z S)SpeechT5EncoderLayerrucsjtt|j|j|jdd|_t|j |_ tj |j|j d|_ t||j|_tj |j|j d|_dS)NFr|r}rr~r)rgrhr{rencoder_attention_headsattention_dropout attentionrrrrrr rrencoder_ffn_dim feed_forwardfinal_layer_normr rwr%r&rhs zSpeechT5EncoderLayer.__init__NFr{r*rrrc Csj|}|j|||||d\}}}||}||}||}|||}||}|f} |r3| |f7} | S)as Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(config.encoder_attention_heads,)`. position_bias (`torch.FloatTensor`): relative position embeddings of size `(seq_len, seq_len, hidden_size // encoder_attention_heads)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r{r*rrr)rrrrr) rtr{r*rrrresidualrr>outputsr%r%r&r|s"    zSpeechT5EncoderLayer.forward)NNNF) rrrrrhrrrrLr|rr%r%rwr&rs"rcseZdZdeffdd Z        ddejdeejd eejd eejd eejd eejd eeejdee dee fddZ Z S)SpeechT5DecoderLayerrucstt|j|j|jdd|_t|j |_ tj |j|j d|_ t|j|j|jdd|_tj |j|j d|_t||j|_tj |j|j d|_dS)NTrr)rr~)rgrhr{rdecoder_attention_headsr self_attnrrrrrr self_attn_layer_norm encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrr rwr%r&rhDs$ zSpeechT5DecoderLayer.__init__NFTr{r*encoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskrr use_cachec Cs|} |dur |ddnd} |j|| |||d\}} } ||}| |}||}d}d}|dur_|} |dur=|ddnd}|j||||||d\}}}||}| |}||}| |} |||}||}|f}|rv|| |f7}| r}|| f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, hidden_size)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. Nr)r{rr*rrr)r{rr*rrr)rrrrrrr)rtr{r*rrrrrrrrself_attn_past_key_valueself_attn_weightspresent_key_valuecross_attn_present_key_valuecross_attn_weightscross_attn_past_key_valuerr%r%r&r|ZsJ       zSpeechT5DecoderLayer.forward)NNNNNNFT) rrrrrhrrrrrLr|rr%r%rwr&rCs:   rc@s$eZdZeZdZdZdZddZdS)SpeechT5PreTrainedModelspeecht5r(Tc Cst|tr*tjj|jjddtd|jj d|jj dtj |jj ddSt|t rRtd|jj}tjj|jj| |dtjj|jj | |ddSt|tjrr|jjjd|jjd|j durp|j jdSdSt|tjtjfr|j j|jjddSt|tjrtj|j|j durt|j|j |j d}tjj|j | |ddSdSt|tjr|jjjd|jjd|jdur|jj|jdSdSdS) zInitialize the weightsrrrmeanstd)abr,Nr)rrrinitnormal_rprrrlrd in_channels constant_rfrr  in_featuresrr dataruinitializer_rangezero_rrfill_rlkaiming_normal_rrr)rtmodulekr%r%r& _init_weightss>          z%SpeechT5PreTrainedModel._init_weightsN) rrrr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrr%r%r%r&rs  rczeZdZdZdeffdd Z     ddejdeej deej d ee d ee d ee d e e e ffd dZZS)SpeechT5Encoderzu Transformer encoder consisting of *config.encoder_layers* layers. Each layer is a [`SpeechT5EncoderLayer`]. rucs~ttjjjd|_tj|_ j |_ t fddt jD|_tjjj|_d|_|dS)Nrcg|]}tqSr%)rr<rr%r&r?z,SpeechT5Encoder.__init__..F)rgrhrrrr rrrrencoder_layerdrop layerdroprrJencoder_layersr<rrencoder_max_relative_positionrnr post_initr rwrr&rhs   zSpeechT5Encoder.__init__Nr{r* head_maskroutput_hidden_states return_dictr2cCs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur(t||j}||}||}||}t p=t |}|rBdnd} |rHdnd} |durk| dt |j krktdt |j d| ddt|j D]@\} } |r{| |f} d} |jrtg}||jk} | r|r| ||||dur|| nd|d}|d}| rd }|r| |d f} qp|r| |f} |std d || | fDSt|| | d S)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the encoder prenet. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr%rz&The head_mask should be specified for  layers, but it is for .F)r*rrrr5rcs|] }|dur|VqdSryr%r=vr%r%r& Ksz*SpeechT5Encoder.forward..last_hidden_stater{ attentions)rurruse_return_dictr rArrrnr r rrOr<r" enumeraterrrErrr)rtr{r*rrrrr synced_gpusall_hidden_statesall_self_attentionsidx encoder_layerskip_the_layerdropout_probability layer_outputsr%r%r&r|sb$           zSpeechT5Encoder.forwardNNNNNrrrrrrhrr7rrrLrrrr|rr%r%rwr&rs. rcr)SpeechT5EncoderWithSpeechPrenetz Wrapper around SpeechT5Encoder that applies SpeechT5SpeechEncoderPrenet to convert the audio waveform data to hidden features. ruc,t|t||_t||_|dSry)rgrhrprenetrwrapped_encoderrr rwr%r&rhZ    z(SpeechT5EncoderWithSpeechPrenet.__init__Nr(r*rrrrr2c Cs*|||\}}|j||||||d}|SN)r{r*rrrrrr rtr(r*rrrrr{rr%r%r&r|bs  z'SpeechT5EncoderWithSpeechPrenet.forwardrrr%r%rwr&rTs.  rcseZdZdZdeffdd ZddZddZ dd ej d e ej d e ej d e e de e de e de eeffddZZS)SpeechT5EncoderWithTextPrenetz| Wrapper around SpeechT5Encoder that applies SpeechT5TextEncoderPrenet to convert the input_ids to hidden features. rucrry)rgrhr]rrrrr rwr%r&rh~rz&SpeechT5EncoderWithTextPrenet.__init__cC |jSryrrcrr%r%r&rcriz2SpeechT5EncoderWithTextPrenet.get_input_embeddingscC|j|dSryrrhrfr%r%r&rhz2SpeechT5EncoderWithTextPrenet.set_input_embeddingsNr(r*rrrrr2c Cs$||}|j||||||d}|Srrrr%r%r&r|s  z%SpeechT5EncoderWithTextPrenet.forwardr)rrrrrrhrcrhrr7rrrLrrrr|rr%r%rwr&rys2 rcr)SpeechT5EncoderWithoutPrenet This wrapper class is a helper class to correctly load pretrained checkpoints when used in combination with [`SpeechT5Model`]. ruc"t|t||_|dSry)rgrhrrrr rwr%r&rh   z%SpeechT5EncoderWithoutPrenet.__init__Nr(r*rrrrr2cCs|j||||||dSr)r)rtr(r*rrrrr%r%r&r|s z$SpeechT5EncoderWithoutPrenet.forwardrrr%r%rwr&r s.  r ceZdZdZdeffdd Z           ddeejdeej deejd eej d eej d eej d ee ejd ee dee dee dee de eeffddZZS)SpeechT5Decoderzt Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`SpeechT5DecoderLayer`] rucsFtj|_tfddtjD|_d|_ | dS)Ncrr%)rr<rr%r&r?rz,SpeechT5Decoder.__init__..F) rgrhdecoder_layerdroprrrrJdecoder_layersr<rrr rwrr&rhs   zSpeechT5Decoder.__init__Nr{r*rrrcross_attn_head_maskrorrrrr2c  Cs| dur| n|jj} | dur| n|jj} |dur|n|jj}| dur$| n|jj} |dd} |dur=|ddjdnd} t|| || }|durX|durXt||j | dd}t p^t |}|j rn|j rn|rntdd}| rrdnd}| rxdnd}| r|durdnd}|rdnd}t||gd d gD](\}}|dur|dt|jkrtd |d t|jd |ddqt|jD]p\}}| r||f}d}|j rtg}||jk}|r|sq|dur||nd}||||||dur||nd|dur||nd|| |d }|d}|r||| rdndf7}| r1||df}|dur1||df}q| r:||f}|r?|nd}| sRtdd|||||fDSt|||||dS)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the decoder prenet. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nrrr)rzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr%rrzThe `z` should be specified for rr)rrrrrrrrcsrryr%rr%r%r&rssz*SpeechT5Decoder.forward..)rror{rcross_attentions)rurrrrrr rr rAr r rrlogger warning_oncer,rOr<r"rrrErrr)rtr{r*rrrrrorrrrrprrrrall_cross_attentionsnext_decoder_cache attn_mask mask_namer decoder_layerrrrr next_cacher%r%r&r|sI             zSpeechT5Decoder.forward NNNNNNNNNNNrrrrrrhrrr7r6rrrrLrrrr|rr%r%rwr&rsN       rcseZdZdZdeffdd Z            ddeejdeej deejd eej d eej d eej d eej d ee ejdee dee dee dee de eeffddZZS)SpeechT5DecoderWithSpeechPrenetz Wrapper around SpeechT5Decoder that applies SpeechT5SpeechDecoderPrenet to convert log-mel filterbanks to hidden features. rucrry)rgrhr8rrwrapped_decoderrr rwr%r&rhrz(SpeechT5DecoderWithSpeechPrenet.__init__Nr(r*rrrHrrrorrrrr2c  Cs0|||} |j| ||||||| | | | d }|SN) r{r*rrrrrorrrrrr )rtr(r*rrrHrrrorrrrdecoder_hidden_statesrr%r%r&r|s z'SpeechT5DecoderWithSpeechPrenet.forward) NNNNNNNNNNNNrr%r%rwr&rsT       rcseZdZdZdeffdd ZddZddZ dd ee j d ee j d ee j d ee j dee j dee j dee e j deedeedeedeedeeeffddZZS)SpeechT5DecoderWithTextPrenetz{ Wrapper around SpeechT5Decoder that applies SpeechT5TextDecoderPrenet to convert input tokens to hidden features. rucrry)rgrhrjrrr rr rwr%r&rhrz&SpeechT5DecoderWithTextPrenet.__init__cCrryrrr%r%r&rcriz2SpeechT5DecoderWithTextPrenet.get_input_embeddingscCrryr rfr%r%r&rhr z2SpeechT5DecoderWithTextPrenet.set_input_embeddingsNr(r*rrrrrorrrrr2c  Cs6||||\} }|j| |||||||| | | d } | Sr!r")rtr(r*rrrrrorrrrr#rr%r%r&r|sz%SpeechT5DecoderWithTextPrenet.forwardr)rrrrrrhrcrhrrr7r6rrrrLrrrr|rr%r%rwr&r$sR      r$cr)SpeechT5DecoderWithoutPrenetr rucr ry)rgrhrr rr rwr%r&rhrz%SpeechT5DecoderWithoutPrenet.__init__Nr(r*rrrrrorrrrr2c Cs$|j||||||||| | | d } | Sr!)r ) rtr(r*rrrrrorrrrrr%r%r&r|s z$SpeechT5DecoderWithoutPrenet.forwardrrr%r%rwr&r%sN       r%cs\eZdZdZdeffdd Zdejdejdejdej fd d Z d d Z e d dZ ZS)$SpeechT5GuidedMultiheadAttentionLossz Guided attention loss from the paper [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://huggingface.co/papers/1710.08969), adapted for multi-head attention. rucst|j|_|j|_dSry)rgrhguided_attention_loss_sigmasigmaguided_attention_loss_scalescaler rwr%r&rhs  z-SpeechT5GuidedMultiheadAttentionLoss.__init__r input_masks output_masksr2cCsX||||j}|d|d@}||jd}||}t||}|j|S)aY Compute the attention loss. Args: attentions (`torch.FloatTensor` of shape `(batch_size, layers * heads, output_sequence_length, input_sequence_length)`): Batch of multi-head attention weights input_masks (`torch.BoolTensor` of shape `(batch_size, input_sequence_length)`): Input attention mask as booleans. output_masks (`torch.BoolTensor` of shape `(batch_size, output_sequence_length)`): Target attention mask as booleans. Returns: `torch.Tensor` with the loss value rrr)_make_guided_attention_masksrrrrr masked_selectr*)rtrr+r,guided_attn_masksmaskslosseslossr%r%r&r|s  z,SpeechT5GuidedMultiheadAttentionLoss.forwardc Cs|d}|d}tjt||jd|jdf|d}tt||D]\}\}} ||| |j|||d| d|f<q#| dS)Nrrr#) rHrrKrOr rr,_make_guided_attention_maskr(r) rtr+r,rrYr'r/rilenolenr%r%r&r-5s  $& zASpeechT5GuidedMultiheadAttentionLoss._make_guided_attention_maskscCsdtjtj||dtj||ddd\}}||}||}dt||d d|dS)Nr#xy)indexingrr)rmeshgridrNrr)r6 output_lengthr(rgrid_ygrid_xr%r%r&r3@s    $z@SpeechT5GuidedMultiheadAttentionLoss._make_guided_attention_mask)rrrrrrhrr7 BoolTensorrr|r-rr3rr%r%rwr&r&s  r&csbeZdZdZdeffdd Z ddejdejdejd ejd ejd e ejd ej fd dZ Z S)SpeechT5SpectrogramLossz; Loss computation used by SpeechT5ForTextToSpeech. rucsTt|j|_|j|_|j|_t|_tt dd|_ |jr(t ||_ dSdS)Ng@) pos_weight) rgrhuse_guided_attention_lossguided_attention_loss_num_headsr)r l1_criterionrrr bce_criterionr&attn_criterionr rwr%r&rhQs z SpeechT5SpectrogramLoss.__init__Nr*rYrZr[labelsrr2cs<|dk}||}||}||}||||}|dddddf} tj| dt| dd| jgdd} | ddddf| } || }|| } || } j rtjfdd|Ddd} |dk}|dddddf}j dkr|ddj ddj f} | ||}| |7} | S)Nr+rrrrcs"g|] }|dddjfqSry)r@)r=xrr%r&r?~s"z3SpeechT5SpectrogramLoss.forward..) r.rArrrQrrrrBr?r)rC)rtr*rYrZr[rDrr l1_lossr0 stop_labelsbce_lossr2attnr+r, attn_lossr%rr&r|]s(   .   zSpeechT5SpectrogramLoss.forwardry) rrrrrrhrr6r7rrr|rr%r%rwr&r=Ls&r=zv The bare SpeechT5 Encoder-Decoder Model outputting raw hidden-states without any specific pre- or post-nets.  custom_introc"s6eZdZ  d"dedeejdeejffdd ZddZd d Z d d Z d dZ ddZ e              d#deejdeejdeejdeejdeejdeejdeejdeeeejdeeeejdeedeejdeedeedeedeeejeffd d!ZZS)$ SpeechT5ModelNruencoderdecodercsJt|||_|durt|n||_|durt|n||_|dS)z encoder (`PreTrainedModel`, *optional*): The encoder model to use. decoder (`PreTrainedModel`, *optional*): The decoder model to use. N)rgrhrur rNr%rOr)rtrurNrOrwr%r&rhs  zSpeechT5Model.__init__cCs0t|jtr |jSt|jtr|jStry)rrNrrcrOr$NotImplementedErrorrr%r%r&rcs    z"SpeechT5Model.get_input_embeddingscCs8t|jtr |j|t|jtr|j|dSdSry)rrNrrhrOr$rfr%r%r&rhs   z"SpeechT5Model.set_input_embeddingscCrary)rNrr%r%r& get_encoderrdzSpeechT5Model.get_encodercCrary)rOrr%r%r& get_decoderrdzSpeechT5Model.get_decodercCs t|jtr|jjdSdSz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rrNrrrrr%r%r&rs z$SpeechT5Model.freeze_feature_encoderr(r*decoder_input_valuesdecoder_attention_maskrdecoder_head_maskrencoder_outputsrorrHrrrr2cCsf| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|dur8|j|||| | |d}n$|r\t|ts\t|dt|dkrM|dndt|dkrX|dndd}|durtt|jt rt|jj |dj d|}n|}t|j trd| i}ni}|j d |||d|||| | | | |d |}|s||St|j|j|j|j|j|j|j|jd S) a input_values (`torch.Tensor` of shape `(batch_size, sequence_length)`): Depending on which encoder is being used, the `input_values` are either: float values of the input raw speech waveform, or indices of input sequence tokens in the vocabulary, or hidden states. decoder_input_values (`torch.Tensor` of shape `(batch_size, target_sequence_length)`, *optional*): Depending on which decoder is being used, the `decoder_input_values` are either: float values of log-mel filterbank features extracted from the raw speech waveform, or indices of decoder input sequence tokens in the vocabulary, or hidden states. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. N)r(r*rrrrrrrrrH) r(r*rrrrrorrrr)rror#decoder_attentionsrencoder_last_hidden_staterencoder_attentionsr%)rurrrrrNrrrOrrrr rOrrrror{rr)rtr(r*rTrUrrVrrWrorrHrrrr decoder_argsdecoder_outputsr%r%r&r|sn)    zSpeechT5Model.forwardr5NNNNNNNNNNNNNN)rrrrrrModulerhrcrhrQrRrrrrr6r7rrLrrr|rr%r%rwr&rMsz      rMzB SpeechT5 Model with a speech encoder and a text decoder. c"s(eZdZdgZdeffdd ZddZddZd d Zd d Z d dZ e              d#de e jde e jde e jde e jde e jde e jde e jde eee jde eee jde ede ede ede ede e jdeeeffdd Zed!d"ZZS)$SpeechT5ForSpeechToTextz#text_decoder_postnet.lm_head.weightruc\t||jdurtd|jdt|}t|}t||||_t ||_ | dS)NYou are trying to instantiate a with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SpeechT5ForSpeechToText.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) rgrhr^r"rxrr$rMrrstext_decoder_postnetr)rtruspeech_encoder text_decoderrwr%r&rh3     z SpeechT5ForSpeechToText.__init__cCrryrrQrr%r%r&rQGriz#SpeechT5ForSpeechToText.get_encodercCrryrrRrr%r%r&rRJriz#SpeechT5ForSpeechToText.get_decodercC|jdSrSrQrrrr%r%r&rMz.SpeechT5ForSpeechToText.freeze_feature_encodercCrry)rbrwrr%r%r&rwTriz-SpeechT5ForSpeechToText.get_output_embeddingscCrry)rbrzrxr%r%r&rzWr z-SpeechT5ForSpeechToText.set_output_embeddingsNr(r*decoder_input_idsrUrrVrrWrorrrrrDr2cCs| dur| n|jj} |dur|durt||jj|jj}|j||||||||| | | | dd }||d}d}|durLt}||d|jj |d}| sb|f|dd}|dur`|f|S|St |||j |j |j |j|j|j|jd S)a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (*pip install soundfile*). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) SpeechT5 uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Label indices can be obtained using [`SpeechT5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToText >>> from datasets import load_dataset >>> dataset = load_dataset( ... "hf-internal-testing/librispeech_asr_demo", "clean", split="validation" ... ) # doctest: +IGNORE_RESULT >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_asr") >>> model = SpeechT5ForSpeechToText.from_pretrained("microsoft/speecht5_asr") >>> # audio file is decoded on the fly >>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> predicted_ids = model.generate(**inputs, max_length=100) >>> # transcribe speech >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> transcription[0] 'mister quilter is the apostle of the middle classes and we are glad to welcome his gospel' ``` ```python >>> inputs["labels"] = processor(text_target=dataset[0]["text"], return_tensors="pt").input_ids >>> # compute loss >>> loss = model(**inputs).loss >>> round(loss.item(), 2) 19.68 ``` NT) r(r*rTrUrrVrrWrorrrrrrr) r2r[ror#rXrrYrrZ)rurr'rrrrbrrr^rror#rXrrYrrZ)rtr(r*rkrUrrVrrWrorrrrrDrr[r2loss_fctoutputr%r%r&r|ZsPYzSpeechT5ForSpeechToText.forwardcs.d}|D]}|tfdd|Df7}q|S)Nr%c3s$|] }|d|jVqdS)rN)rrr)r= past_statebeam_idxr%r&rs"z9SpeechT5ForSpeechToText._reorder_cache..)r)rorpreordered_past layer_pastr%ror&_reorder_caches z&SpeechT5ForSpeechToText._reorder_cacher])rrr_tied_weights_keysrrhrQrRrrwrzrrrr7r6rrrLrrr|rrsrr%r%rwr&r_+sr       r_?r,4@FmodelrH threshold minlenratio maxlenratiovocoderoutput_cross_attentionsreturn_output_lengthsc " s|durtd|durd||jjk} n|} |d} |jj|| dd} | j} t|jjt r?|jjj | dj d| } t| d||jj }t| d||jj }| | d|jj}g}g}d}d}i |d7}|jj ||}|jjj|ddddfd| | |d|dd}|r|tj|jdd|jd}|j}|j|}|| |jj |jj}|||dddddf| d|jj}tj||fdd}t|j|}||krql||krtj|dd|k}t|d}nt t!|}fd d |D}t!|dkr3t"|}|#dd$dd }|j%|}|D] }|||<q)t!| kr;nqmfd d t t!D}| s| dkrU|dn tj&j'j(j)|dd }|duri||}n|}|rtj|d d}| dkr|j| t|d| g|ddR}||f}|Sgt | D] } || dq|durtj&j'j(j)|dd }|f}ngtj&j'j(j)|dd }||fdd D}!|!f}|rtj|d d}|j| t|d| g|ddR}g||R}|S)Na`speaker_embeddings` must be specified. For example, you can use a speaker embeddings by following the code snippet provided in this link: https://huggingface.co/datasets/Matthijs/cmu-arctic-xvectors rrT)r(r*rr)r{r*rrrorrrrcsg|]}|vr|qSr%r%rresult_spectrogramr%r&r?M sz$_generate_speech..rcsg|]}|qSr%r%rr~r%r&r?V r) batch_firstcs&g|]}tdt|qSr)r4rr5r)spectrogram_lengths waveformsr%r&r?p s&)*r"rurr4rrrNrrrrrr r)rr9rOr rSrrrsqueezerospeech_decoder_postnetrVrsigmoidrWrHrDrIrJrOstackrflattenrXrrrnn pad_sequence)"rwr(rHr*rxryrzr{r|r}rr encoder_outrYmaxlenminlenoutput_sequence spectrogramrrorr# decoder_outlast_decoder_outputspectrumnew_spectrogramprobmeet_thresholds meet_indexes spectrograms meet_indexrrwaveform_lengthsr%)rrrr&_generate_speechs       $   5&      rzB SpeechT5 Model with a text encoder and a speech decoder. c&seZdZdZdeffdd ZedefddZdd Z d d Z e d,de e jd e e jde e jde e jde e jde e jde e jde eee jde eee jde ede ede ede ede e jde e jde e jdeeeff"ddZe    ! !d-de jd e e jde e jd"ed#ed$ed%e ejd&ed'edee jee je jfffd(d)Ze    ! !d-de jde e jd e e jd"ed#ed$ed%e ejd&ed'edee jee je jfffd*d+ZZS).SpeechT5ForTextToSpeechrrucr`)Nraa with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SpeechT5ForTextToSpeech.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) rgrhr^r"rxrrrMrrUrr)rtru text_encoderspeech_decoderrwr%r&rh rez SpeechT5ForTextToSpeech.__init__r2cCsdSrr%)clsr%r%r& can_generate sz$SpeechT5ForTextToSpeech.can_generatecCrryrfrr%r%r&rQ riz#SpeechT5ForTextToSpeech.get_encodercCrryrgrr%r%r&rR riz#SpeechT5ForTextToSpeech.get_decoderNr*rTrUrrVrrWrorrrrrHrDrGcCs| dur| n|jj} |dur"|durt||jj|\}}|jjr"d} |j||||||||| | || | dd}||d\}}}d}|durTt|j}|||||||j}| sj|f|dd}|durh|f|S|St |||j |j |j |j|j |j|jd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`): Float values of input mel spectrogram. SpeechT5 uses an all-zero spectrum as the starting token for `decoder_input_values` generation. If `past_key_values` is used, optionally only the last `decoder_input_values` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. labels (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`, *optional*): Float values of target mel spectrogram. Timesteps set to `-100.0` are ignored (masked) for the loss computation. Spectrograms can be obtained using [`SpeechT5Processor`]. See [`SpeechT5Processor.__call__`] for details. stop_labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Binary tensor indicating the position of the stop token in the sequence. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan, set_seed >>> import torch >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_tts") >>> model = SpeechT5ForTextToSpeech.from_pretrained("microsoft/speecht5_tts") >>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan") >>> inputs = processor(text="Hello, my dog is cute", return_tensors="pt") >>> speaker_embeddings = torch.zeros((1, 512)) # or load xvectors from a file >>> set_seed(555) # make deterministic >>> # generate speech >>> speech = model.generate(inputs["input_ids"], speaker_embeddings=speaker_embeddings, vocoder=vocoder) >>> speech.shape torch.Size([15872]) ``` NTr(r*rTrUrrVrrWrorrHrrrrr r2rror#rXrrYrrZ)rurr.r)r?rrr=rrror#rXrYrrZ)rtrr*rTrUrrVrrWrorrrrrHrDrGrrYrZr[r2 criterionrmr%r%r&r| sdL   zSpeechT5ForTextToSpeech.forwardrur,rvFrxryrzr{r|r}c Ks^|dur"|d} |d| kr"|ddkr|| d}ntdt|||||||||| S)aE Converts a sequence of input tokens into a sequence of mel spectrograms, which are subsequently turned into a speech waveform using a vocoder. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Attention mask from the tokenizer, required for batched inference to signal to the model where to ignore padded tokens from the input_ids. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. threshold (`float`, *optional*, defaults to 0.5): The generated sequence ends when the predicted stop token probability exceeds this value. minlenratio (`float`, *optional*, defaults to 0.0): Used to calculate the minimum required length for the output sequence. maxlenratio (`float`, *optional*, defaults to 20.0): Used to calculate the maximum allowed length for the output sequence. vocoder (`nn.Module`, *optional*): The vocoder that converts the mel spectrogram into a speech waveform. If `None`, the output is the mel spectrogram. output_cross_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of the decoder's cross-attention layers. return_output_lengths (`bool`, *optional*, defaults to `False`): Whether or not to return the concrete spectrogram/waveform lengths. Returns: `tuple(torch.FloatTensor)` comprising various elements depending on the inputs: - when `return_output_lengths` is False - **spectrogram** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrogram. - **waveform** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(num_frames,)` -- The predicted speech waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. - when `return_output_lengths` is True - **spectrograms** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrograms that are padded to the maximum length. - **spectrogram_lengths** (*optional*, returned when no `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each spectrogram. - **waveforms** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, num_frames)` -- The predicted speech waveforms that are padded to the maximum length. - **waveform_lengths** (*optional*, returned when a `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(batch_size, config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. NrrzUThe first dimension of speaker_embeddings must be either 1 or the same as batch_size.rrCr"r) rtrr*rHrxryrzr{r|r}kwargsrXr%r%r&generate) s(E z SpeechT5ForTextToSpeech.generatec Cs^|dur"|d} |d| kr"|ddkr|| d}ntdt|||||||||| S)a Converts a sequence of input tokens into a sequence of mel spectrograms, which are subsequently turned into a speech waveform using a vocoder. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) threshold (`float`, *optional*, defaults to 0.5): The generated sequence ends when the predicted stop token probability exceeds this value. minlenratio (`float`, *optional*, defaults to 0.0): Used to calculate the minimum required length for the output sequence. maxlenratio (`float`, *optional*, defaults to 20.0): Used to calculate the maximum allowed length for the output sequence. vocoder (`nn.Module`, *optional*, defaults to `None`): The vocoder that converts the mel spectrogram into a speech waveform. If `None`, the output is the mel spectrogram. output_cross_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of the decoder's cross-attention layers. return_output_lengths (`bool`, *optional*, defaults to `False`): Whether or not to return the concrete spectrogram/waveform lengths. Returns: `tuple(torch.FloatTensor)` comprising various elements depending on the inputs: - when `return_output_lengths` is False - **spectrogram** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrogram. - **waveform** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(num_frames,)` -- The predicted speech waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. - when `return_output_lengths` is True - **spectrograms** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrograms that are padded to the maximum length. - **spectrogram_lengths** (*optional*, returned when no `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each spectrogram. - **waveforms** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, num_frames)` -- The predicted speech waveforms that are padded to the maximum length. - **waveform_lengths** (*optional*, returned when a `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(batch_size, config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. NrrzUThe first dimension of speaker_embeddings must be either 1 or the same as batch size.r) rtrrHr*rxryrzr{r|r}rXr%r%r&generate_speech s(I z'SpeechT5ForTextToSpeech.generate_speechNNNNNNNNNNNNNNNNNNrur,rvNFF)rrrrrrh classmethodrLrrQrRrrrr6r7rrrrr|rrrr^rrrr%r%rwr&r{ s          [   rzD SpeechT5 Model with a speech encoder and a speech decoder. c&seZdZdeffdd ZddZddZdd Ze d*d e e j d e e j d e e j de e j de e j de e j de e j de eee j de eee j de ede ede ede ede e j de e j de e j deeeff"ddZe    ! !d+d e j de e j d e e j d"ed#ed$ed%e ejd&ed'ede j fd(d)ZZS),SpeechT5ForSpeechToSpeechrucs@t|t|}t|}t||||_t||_|dSry) rgrhrrrMrrUrr)rtrurcrrwr%r&rh s   z"SpeechT5ForSpeechToSpeech.__init__cCrryrfrr%r%r&rQ riz%SpeechT5ForSpeechToSpeech.get_encodercCrryrgrr%r%r&rR riz%SpeechT5ForSpeechToSpeech.get_decodercCrhrSrirr%r%r&r rjz0SpeechT5ForSpeechToSpeech.freeze_feature_encoderNr(r*rTrUrrVrrWrorrrrrHrDrGr2cCs| dur| n|jj} |dur|durt||jj|\}}|j||||||||| | || | dd}||d\}}}d}| sQ|f|dd}|durO|f|S|St|||j|j|j |j |j |j |j d S)a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (*pip install soundfile*). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. decoder_input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`): Float values of input mel spectrogram. SpeechT5 uses an all-zero spectrum as the starting token for `decoder_input_values` generation. If `past_key_values` is used, optionally only the last `decoder_input_values` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. labels (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`, *optional*): Float values of target mel spectrogram. Spectrograms can be obtained using [`SpeechT5Processor`]. See [`SpeechT5Processor.__call__`] for details. stop_labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Binary tensor indicating the position of the stop token in the sequence. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToSpeech, SpeechT5HifiGan, set_seed >>> from datasets import load_dataset >>> import torch >>> dataset = load_dataset( ... "hf-internal-testing/librispeech_asr_demo", "clean", split="validation" ... ) # doctest: +IGNORE_RESULT >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_vc") >>> model = SpeechT5ForSpeechToSpeech.from_pretrained("microsoft/speecht5_vc") >>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan") >>> # audio file is decoded on the fly >>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> speaker_embeddings = torch.zeros((1, 512)) # or load xvectors from a file >>> set_seed(555) # make deterministic >>> # generate speech >>> speech = model.generate_speech(inputs["input_values"], speaker_embeddings, vocoder=vocoder) >>> speech.shape torch.Size([77824]) ``` NTrrrr)rurr.r)rrrror#rXrrYrrZ)rtr(r*rTrUrrVrrWrorrrrrHrDrGrr>rr[r2rmr%r%r&r| sLR z!SpeechT5ForSpeechToSpeech.forwardrur,rvFrxryrzr{r|r}c Cs2|dur tjd|jd}t|||||||||| S)a Converts a raw speech waveform into a sequence of mel spectrograms, which are subsequently turned back into a speech waveform using a vocoder. Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (*pip install soundfile*). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) threshold (`float`, *optional*, defaults to 0.5): The generated sequence ends when the predicted stop token probability exceeds this value. minlenratio (`float`, *optional*, defaults to 0.0): Used to calculate the minimum required length for the output sequence. maxlenratio (`float`, *optional*, defaults to 20.0): Used to calculate the maximum allowed length for the output sequence. vocoder (`nn.Module`, *optional*, defaults to `None`): The vocoder that converts the mel spectrogram into a speech waveform. If `None`, the output is the mel spectrogram. output_cross_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of the decoder's cross-attention layers. return_output_lengths (`bool`, *optional*, defaults to `False`): Whether or not to return the concrete spectrogram/waveform lengths. Returns: `tuple(torch.FloatTensor)` comprising various elements depending on the inputs: - when `return_output_lengths` is False - **spectrogram** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrogram. - **waveform** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(num_frames,)` -- The predicted speech waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. - when `return_output_lengths` is True - **spectrograms** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrograms that are padded to the maximum length. - **spectrogram_lengths** (*optional*, returned when no `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each spectrogram. - **waveforms** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, num_frames)` -- The predicted speech waveforms that are padded to the maximum length. - **waveform_lengths** (*optional*, returned when a `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(batch_size, config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. N)rir#)rrKrr) rtr(rHr*rxryrzr{r|r}r%r%r&r sIz)SpeechT5ForSpeechToSpeech.generate_speechrr)rrrrrhrQrRrrrrr7r6rrrLrrr|rrrr^rrr%r%rwr&r s        ~   rcs@eZdZdfdd ZdddZd d Zd d Zd dZZS)HifiGanResidualBlockrrr皙?csbt|_tfddttD_tfddttD_dS)Nc s2g|]}tjd||dqSr)redilationrrrl get_paddingrchannelsrrdrtr%r&r? s z1HifiGanResidualBlock.__init__..c s*g|]}tjddddqSrrr<)rrdrtr%r&r? s  ) rgrhleaky_relu_sloperrrJrOconvs1convs2)rtrrdrrrwrr&rh s     zHifiGanResidualBlock.__init__rcCs|||dSrr%)rtrdrr%r%r&r r z HifiGanResidualBlock.get_paddingcCsLtjj}ttjjdrtjjj}|jD]}||q|jD]}||qdSNr)rrrrrrrrtrrr%r%r&apply_weight_norm s     z&HifiGanResidualBlock.apply_weight_normcCs4|jD]}tj|q|jD]}tj|qdSry)rrrremove_weight_normrrtrr%r%r&r s  z'HifiGanResidualBlock.remove_weight_normcCsXt|j|jD]"\}}|}tj||j}||}tj||j}||}||}q|Sry)r,rrrrI leaky_relur)rtr{conv1conv2rr%r%r&r| s zHifiGanResidualBlock.forward)rrrr) rrrrhrrrr|rr%r%rwr&r s   rz HiFi-GAN vocoder. csbeZdZeZdZdeffdd ZddZddZd d Z e d d de j d e j fddZ ZS)SpeechT5HifiGanrruc sNt|t|j|_t|j|_tj|j |j dddd|_ t |_ tt|j|jD]$\}\}}|j tj|j d||j d|d||||ddq-t |_tt|j D]#}|j d|d}t|j|jD]\}}|jt||||jqpq^tj|ddddd|_|dt|j |dt|j |dS)Nrr)rdrerrrr*)rgrhrOresblock_kernel_sizes num_kernelsupsample_rates num_upsamplesrrl model_in_dimupsample_initial_channelconv_prer upsamplerrr,upsample_kernel_sizesrSConvTranspose1d resblocksrJresblock_dilation_sizesrr conv_postrrrKrQr)rtrur upsample_raterdrrrwr%r&rh% s>        zSpeechT5HifiGan.__init__cCsJt|tjtjfr!|jjjd|jjd|j dur#|j j dSdSdS)zInitialize the weights.r,rN) rrr rlrrrrurrfr)rtrr%r%r&rK s  zSpeechT5HifiGan._init_weightscCs`tjj}ttjjdrtjjj}||j|jD]}||q|jD]}|q"||j dSr) rrrrrrrrrrrr%r%r&rR s      z!SpeechT5HifiGan.apply_weight_normcCsLtj|j|jD]}tj|q |jD]}|qtj|jdSry)rrrrrrrrr%r%r&r^ s    z"SpeechT5HifiGan.remove_weight_norma Converts a log-mel spectrogram into a speech waveform. Passing a batch of log-mel spectrograms returns a batch of speech waveforms. Passing a single, un-batched log-mel spectrogram returns a single, un-batched speech waveform. rKr2cCs|jjr ||j|j}|dk}|s|d}|dd}||}t|j D]8}t j ||jj }|j||}|j||j|}td|jD]}||j||j||7}qK||j}q)t j |}||}t|}|s|dddd}|S|d}|S)a spectrogram (`torch.FloatTensor`): Tensor containing the log-mel spectrograms. Can be batched and of shape `(batch_size, sequence_length, config.model_in_dim)`, or un-batched and of shape `(sequence_length, config.model_in_dim)`. Returns: `torch.FloatTensor`: Tensor containing the speech waveform. If the input spectrogram is batched, will be of shape `(batch_size, num_frames,)`. If un-batched, will be of shape `(num_frames,)`. rrrrr)runormalize_beforerr*rrrrrJrrrIrrrrrrrtanhrr)rtr is_batchedr{r res_statejwaveformr%r%r&r|f s,         zSpeechT5HifiGan.forward)rrrrrrrhrrrrrr7r|rr%r%rwr&r s&  r)r_rrrMrr)rNrr)arrtypingrrnumpyrCrtorch.utils.checkpointrtorch.nnrrr activationsr generationr integrations.deepspeedr integrations.fsdpr modeling_attn_mask_utilsr rmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrrrrconfiguration_speecht5rr get_loggerrr_HIDDEN_STATES_START_POSITIONrr4r'r.rrr6ndarrayr`rbrrr^rrrrrrrrr8rMrUr]rjrsr{rrrrrrrr rrr$r%r&r=rMr_r7rLrrrrr__all__r%r%r%r&s           xE- )4(+!=l%%* @04+;=@   ht> w