o :i@sddlZddlZddlZddlZddlmZmZmZddlZddlm Z m Z ej r.dZ n ejj r7dZ ndZ ddlmZmZmZmZmZmZmZmZmZmZmZmZmZd+d d ZGd d d e jZ d,de!de!defddZ"Gddde jZ#Gddde jZ$Gddde jZ%Gddde jZ&Gdddej jZ'Gdddej jZ(Gdd d ej jZ)Gd!d"d"e jZ*Gd#d$d$e jZ+Gd%d&d&e jZ,Gd'd(d(e jZ-Gd)d*d*e jZ.dS)-N)OptionalTupleUnion)Tensornncudampscpu) ActivationDropoutAndLinearBalancerBiasNormDropout2 FloatLikeIdentity ScaledLinearScheduledFloatSwooshRWhitenlimit_param_valuepenalize_abs_values_gtsoftmax'c Cs|d}tt| tjd|tj|jd|}|dkr&|dd}|d |d}tj t |t |gdd}|drUtj |t |d ddfgdd}|S) a!Create sinusoidal timestep embeddings. :param timesteps: shape of (N) or (N, T) :param dim: the dimension of the output. :param max_period: controls the minimum frequency of the embeddings. :return: an Tensor of positional embeddings. shape of (N, dim) or (T, N, dim) r)startenddtypedevice).NNdim.)torchexpmathlogarangefloat32rr transposefloatcatcossin zeros_like) timestepsr max_periodhalffreqsargs embeddingr38/home/ubuntu/LuxTTS/zipvoice/models/modules/zipformer.pytimestep_embedding6s   $r5c)seZdZdZ          d+dededeeeefdeeeefdeeeefdededededededededede dede d ed!e d"d f(fd#d$ Z d,d%e d&e e d'e e d(e e d"ee e ff d)d*Z ZS)- TTSZipformera Args: Note: all "int or Tuple[int]" arguments below will be treated as lists of the same length as downsampling_factor if they are single ints or one-element tuples. The length of downsampling_factor defines the number of stacks. downsampling_factor (Tuple[int]): downsampling factor for each encoder stack. Note: this is in addition to the downsampling factor of 2 that is applied in the frontend (self.encoder_embed). encoder_dim (Tuple[int]): embedding dimension of each of the encoder stacks, one per encoder stack. num_encoder_layers (int or Tuple[int])): number of encoder layers for each stack query_head_dim (int or Tuple[int]): dimension of query and key per attention head: per stack, if a tuple.. pos_head_dim (int or Tuple[int]): dimension of positional-encoding projection per attention head value_head_dim (int or Tuple[int]): dimension of value in each attention head num_heads: (int or Tuple[int]): number of heads in the self-attention mechanism. Must be at least 4. feedforward_dim (int or Tuple[int]): hidden dimension in feedforward modules cnn_module_kernel (int or Tuple[int])): Kernel size of convolution module pos_dim (int): the dimension of each positional-encoding vector prior to projection, e.g. 128. dropout (float): dropout rate warmup_batches (float): number of batches to warm up over; this controls dropout of encoder layers. use_time_embed: (bool): if True, take time embedding as an additional input. time_embed_dim: (int): the dimension of the time embedding. use_guidance_scale_embed (bool): if True, take guidance scale embedding as an additional input. guidance_scale_embed_dim: (int): the dimension of the guidance scale embedding. rr8 N@@TFin_dimout_dimdownsampling_factornum_encoder_layerscnn_module_kernel encoder_dimquery_head_dim pos_head_dimvalue_head_dim num_headsfeedforward_dimpos_dimdropoutwarmup_batchesuse_time_embedtime_embed_dimuse_guidance_scale_embedguidance_scale_embed_dimuse_convreturnc stt|| durtdd} ttrffdd}dd}||_||}|||_}||_||_ ||_ | |_ | |_ ||_ ||_||_|j rV|dksUJnd}||_t|||_t|||_g}t}t|D]J}t|| | ||| | |||| d }t|||||| ||d |d ||d |d d |d d }|d krt|||d}||qst||_|j rtt||d tt|d ||_nd|_|jrt ||ddd|_!dSd|_!dS)N)333333?)@皙?csRt|tr|f}t|dkr|t}|St|tkr%t|dts'J|S)zoConverts a single int or a 1-tuple of an int to a tuple with the same length as downsampling_factorrr) isinstanceintlen)xrCr3r4 _to_tuples   "z(TTSZipformer.__init__.._to_tuplecSs|ddkr |ddksJtdt|ddD]}||||ddks)Jqtt|ddt|D]}||d||dksGJq7dS)z.assert downsampling_factor follows u-net stylerrrrN)ranger[)factorsir3r3r4_assert_downsampling_factors z:TTSZipformer.__init__.._assert_downsampling_factorr) embed_dimrLrJrGrHrIrKrSrErMrrgQ??)rcrPrL warmup_begin warmup_endfinal_layerdrop_rate)r downsampleFrXbias initial_scale)"superr6__init__rrYrZrCrErFrDrGrIrJrOrQrPrRrLinearin_projout_projr[r_Zipformer2EncoderLayerZipformer2EncoderDownsampledZipformer2Encoderappend ModuleListencoders Sequentialr time_embedrguidance_scale_embed)selfrArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSr^rbrv num_encodersra encoder_layerencoder __class__r]r4rmts          zTTSZipformer.__init__r\t padding_maskguidance_scalec Cs|ddd}||}|durO|dks!|dks!J|jt||j}|durI|dks<|dks Args: x: The input tensor. Its shape is (batch_size, seq_len, feature_dim). t: A t tensor of shape (batch_size,) or (batch_size, seq_len) padding_mask: The mask for padding, of shape (batch_size, seq_len); True means masked position. May be None. guidance_scale: The guidance scale in classifier-free guidance of distillation model. Returns: Return the output embeddings. its shape is (batch_size, output_seq_len, encoder_dim) rrrN)time_embsrc_key_padding_mask attn_mask) permuteror shaper5rPryrRrx enumeratervrp) rzr\rrrrguidance_scale_embrramoduler3r3r4forwards4 "    zTTSZipformer.forward)r7r8r9r:r;r8r<r=r>r?Nr@Tr?Fr?TNNN)__name__ __module__ __qualname____doc__rZrrrr(boolrmrrr __classcell__r3r3r~r4r6Os(       r6@r\ratiorTcCstd|fd||f|dS)NrUrWdefault)r)r\rr3r3r4_whitening_schedule/src#seZdZdZdddeddddd eddddd ed d dd ed d ded d dedddd f dededededededededededededededed ed!d"f"fd#d$ Zd%e d&e d!e e fd'd(Z d%e d&e d!e fd)d*Z " " "d2d+e d,e d-e e d.e e d/e e d!e f d0d1ZZS)3rqac Args: embed_dim: the number of expected features in the input (required). nhead: the number of heads in the multiheadattention models (required). feedforward_dim: the dimension of the feedforward network model (required). dropout: the dropout value (default=0.1). cnn_module_kernel (int): Kernel size of convolution module (default=31). Examples:: >>> encoder_layer = Zipformer2EncoderLayer(embed_dim=512, nhead=8) >>> src = torch.rand(10, 32, 512) >>> pos_emb = torch.rand(32, 19, 512) >>> out = encoder_layer(src, pos_emb) rXr9TrU皙?)r@皙?)i>rUrrrU?)r@皙?rUrX)r@{Gz?)gj@rUrUrdr@g{Gz?rcrLrJrGrHrIrKrMrErSattention_skip_rateconv_skip_rateconst_attention_rate ff2_skip_rate ff3_skip_ratebypass_skip_raterTNc stt|||_t||dd|_t|dd|_t| |_ t| |_ t||_ t||_ t| |_ t|||||dd|_t||||_t||||_t||dd||_t||||_t||dd||_t|d|dd |_| |_|jrt|| |_t|| |_t||_t|d d d d dd|_ t|d ddt!dddd|_"t|d ddt!ddddddd|_#t|d ddt!ddddddd|_$t%dt&ddddd d!|_'t|d d d d"dd|_(dS)#Nr) skip_ratestraight_through_raterrU)rLrJrGrHrMr8)hidden_channelsrg?g?rg@ channel_dim min_positive max_positivemin_absmax_absrVgffffff?)rUgMbp?rr)rrrrprob)rUrU)r@rXrrrrrrrr)r@rr@rrrr num_groupswhitening_limitr grad_scalerX))rlrqrmrc BypassModulebypass bypass_midcopydeepcopyrrrrr$RelPositionMultiheadAttentionWeightsself_attn_weights SelfAttention self_attn1 self_attn2FeedforwardModule feed_forward1 feed_forward2 feed_forward3NonlinAttentionnonlin_attentionrSConvolutionModule conv_module1 conv_module2r normr balancer1r balancer_na balancer_ff2 balancer_ff3rrwhiten balancer2)rzrcrLrJrGrHrIrKrMrErSrrrrrrr~r3r4rmCs                  zZipformer2EncoderLayer.__init__r\ dropout_ratecCsR|dks|jrtjstjrdS|jd}tj|d|jd|k|j }|S)NrUrr) trainingr!jit is_scripting is_tracingrrandrtor)rzr\r batch_sizemaskr3r3r4get_sequence_dropout_masks z0Zipformer2EncoderLayer.get_sequence_dropout_maskcCs |||}|dur |S||S)zf Apply sequence-level dropout to x. x shape: (seq_len, batch_size, embed_dim) N)r)rzr\r dropout_maskr3r3r4sequence_dropouts z'Zipformer2EncoderLayer.sequence_dropoutsrcpos_embrrrcCs|}tjs tjrd}n |jrt|jnd}|j||||d}|dur*||}|||}| ||} |dd} tjsGtjrHn%|jrmt t|j krm| dd} | dk | j } | d| jddd } |||| } || dur}| n| | }|||} || dur| n| | }|jrtjstjrd} n |jrt|jnd} |dur||}|||j||d | }tjstjrd}n |jrt|jnd}||||||}|||}|||} || dur| n| | }|jr4tjstjrd} n |jrt|jnd} |dur'||}|||j||d | }tjs@tjrCd}n |jrLt|jnd}||||||}||}| |}|!||}|"|}|#|}|S) a Pass the input through the encoder layer. Args: src: the sequence to the encoder (required): shape (seq_len, batch_size, embedding_dim). pos_emb: (1, 2*seq_len-1, pos_emb_dim) or (batch_size, 2*seq_len-1, pos_emb_dim) time_emb: the embedding representing the current timestep shape (batch_size, embedding_dim) or (seq_len, batch_size, embedding_dim). attn_mask: the attention mask, of shape (batch_size, seq_len, seq_len) or (seq_len, seq_len), interpreted as (batch_size, tgt_seq_len, src_seq_len) or (tgt_seq_len, src_seq_len). True means masked position. May be None. src_key_padding_mask: the mask for padding, of shape (batch_size, seq_len); True means masked position. May be None. Returns: A tensor which has the same shape as src rU)rrkey_padding_maskNrr?rT)r keepdim)r)$r!rrrrr(rrrrrandomrrrsumrrrrSrrrrrrrrrrrrrrrrr)rzrrrrrsrc_origr attn_weightsself_attn_dropout_maskselected_attn_weightsna self_attnrrrr3r3r4rs           zZipformer2EncoderLayer.forwardr)rrrrrrZrrrmrr(rrrrrr3r3r~r4rq3s     rqcseZdZdZ  ddejdedededed ed ed ed ed dffdd Z   dde de e de e de e d e f ddZ Z S)rra&Zipformer2Encoder is a stack of N encoder layers Args: encoder_layer: an instance of the Zipformer2EncoderLayer() class (required). num_layers: the number of sub-encoder-layers in the encoder (required). pos_dim: the dimension for the relative positional encoding Examples:: >>> encoder_layer = Zipformer2EncoderLayer(embed_dim=512, nhead=8) >>> zipformer_encoder = Zipformer2Encoder(encoder_layer, num_layers=6) >>> src = torch.rand(10, 32, 512) >>> out = zipformer_encoder(src) rdrr| num_layersrcrPrLrerfinitial_layerdrop_ratergrTNc stt|ddd|_|dkrttt|||_nd|_t fddt |D|_ ||_ d|kr>|ksAJJd|||} |} t |D]} | | } t | |f| | fdd |j | j_| } qOdS) Ng333333?r)r length_factorrcsg|]}tqSr3)rr).0rar|r3r4 sz.Zipformer2Encoder.__init__..rrUr)rlrmCompactRelPositionalEncoding encoder_posrrwrrnrrur_layersrrrr)rzr|rrcrPrLrerfrrgdelta cur_beginracur_endr~rr4rms4   zZipformer2Encoder.__init__rrrrc Csf||}|jdur|dusJ||}n|dusJ|}t|jD] \}}||||||d}q#|S)aUPass the input through the encoder layers in turn. Args: src: the sequence to the encoder (required): shape (seq_len, batch_size, embedding_dim). time_emb: the embedding representing the current timestep: shape (batch_size, embedding_dim) or (seq_len, batch_size, embedding_dim) . attn_mask: the attention mask, of shape (batch_size, seq_len, seq_len) or (seq_len, seq_len), interpreted as (batch_size, tgt_seq_len, src_seq_len) or (tgt_seq_len, src_seq_len). True means masked position. May be None. src_key_padding_mask: the mask for padding, of shape (batch_size, seq_len); True means masked position. May be None. Returns: a Tensor with the same shape as src. Nrrr)rrrr) rzrrrrroutputramodr3r3r4rs     zZipformer2Encoder.forward)rdrr) rrrrrModulerZr(rmrrrrr3r3r~r4rrsN   -rrc sjeZdZdZddedddddfded ed ed ed ef fd d ZdefddZde de fddZ Z S)raA An nn.Module that implements a learnable bypass scale, and also randomized per-sequence layer-skipping. The bypass is limited during early stages of training to be close to "straight-through", i.e. to not do the bypass operation much initially, in order to force all the modules to learn something. rU)rUg?)rWrrrrrcrr scale_min scale_maxcsTttt|fd|_t||_ t||_ t||_ t||_ dS)Nrd) rlrmr Parameterr!full bypass_scalerrrrrr)rzrcrrrrr~r3r4rms    zBypassModule.__init__rcCstjs tjs |js|jSt|jt|jt|j d}t|j }|dkr7tj |df|j d|k}||}t|j }|dkrVtj |df|j d|k}t|||j}|S)N)minmaxrUrr)r!rrrrrrr(rrrrrrmaximumrr)rzransrrrr3r3r4_get_bypass_scale s&  zBypassModule._get_bypass_scalerrcCs ||jd}||||S)z Args: src_orig and src are both of shape (seq_len, batch_size, num_channels) Returns: something with the same shape as src and src_orig r)r r)rzrrrr3r3r4r%szBypassModule.forward) rrrrrrZrrmr rrrr3r3r~r4rs&  rc sbeZdZdZdejdedeffdd Z   dded e ed e ed e ed ef d dZ Z S)rsa  DownsampledZipformer2Encoder is a zipformer encoder evaluated at a reduced frame rate, after convolutional downsampling, and then upsampled again at the output, and combined with the origin input, so that the output has the same shape as the input. r}r rhcsHtt|||_t||_|j|_||_t||_ t |dd|_ dS)Nrr) rlrsrmdownsample_factorSimpleDownsamplerhrr}SimpleUpsampleupsampler out_combiner)rzr}r rhr~r3r4rm5s  z%DownsampledZipformer2Encoder.__init__NrrrrrTcCs|}||}|j}|dur|dkr|dd|}|dur+|dd|dd|f}|dur8|ddd|f}|j||||d}||}|d|jd}|||S)aDownsample, go through encoder, upsample. Args: src: the sequence to the encoder (required): shape (seq_len, batch_size, embedding_dim). time_emb: the embedding representing the current timestep: shape (batch_size, embedding_dim) or (seq_len, batch_size, embedding_dim) . feature_mask: something that broadcasts with src, that we'll multiply `src` by at every layer: if a Tensor, likely of shape (seq_len, batch_size, embedding_dim) attn_mask: the attention mask, of shape (batch_size, seq_len, seq_len) or (seq_len, seq_len), interpreted as (batch_size, tgt_seq_len, src_seq_len) or (tgt_seq_len, src_seq_len). True means masked position. May be None. src_key_padding_mask: the mask for padding, of shape (batch_size, seq_len); True means masked position. May be None. Returns: a Tensor with the same shape as src. Nr.rr)rhr r r}rrr)rzrrrrrdsr3r3r4r>s$   z$DownsampledZipformer2Encoder.forwardr) rrrrrrrZrmrrrrr3r3r~r4rs.s" rsc8eZdZdZdeffdd ZdedefddZZS) r z< Does downsampling with attention, by weighted sum. rhcs0tt|tt||_d|_||_ dSN) rlr rmrrr!zerosrjnamerh)rzrhr~r3r4rmus zSimpleDownsample.__init__rrTc Cs|j\}}}|j}||d|}|||}||jddd||jd|jd}tj||fdd}|jd||ksAJ|||||}|jjdd} | dd} || j dd} | S)z x: (seq_len, batch_size, in_channels) Returns a tensor of shape ( (seq_len+downsample-1)//downsample, batch_size, channels) rrNrrr) rrhexpandr!r)reshaperjr unsqueezer) rzrseq_lenr in_channelsr d_seq_lenpad src_extraweightsr r3r3r4r~s  ,zSimpleDownsample.forward rrrrrZrmrrrr3r3r~r4r ps r cr) r zG A very simple form of upsampling that just repeats the input. rcstt|||_dSr)rlr rmr)rzrr~r3r4rms zSimpleUpsample.__init__rrTcCs>|j}|j\}}}|d||||}|||||}|S)z x: (seq_len, batch_size, num_channels) Returns a tensor of shape ( (seq_len*upsample), batch_size, num_channels) r)rrrrr)rzrrrr num_channelsr3r3r4rs  zSimpleUpsample.forwardrr3r3r~r4r sr c sleZdZdZ  ddededededd f fd d Zdd ededd fddZ dd ededefddZ Z S)ra Relative positional encoding module. This version is "compact" meaning it is able to encode the important information about the relative position in a relatively small number of dimensions. The goal is to make it so that small differences between large relative offsets (e.g. 1000 vs. 1001) make very little difference to the embedding. Such differences were potentially important when encoding absolute position, but not important when encoding relative position because there is now no need to compare two large offsets with each other. Our embedding works by projecting the interval [-infinity,infinity] to a finite interval using the atan() function, before doing the Fourier transform of that fixed interval. The atan() function would compress the "long tails" too small, making it hard to distinguish between different magnitudes of large offsets, so we use a logarithmic function to compress large offsets to a smaller range before applying atan(). Scalings are chosen in such a way that the embedding can clearly distinguish individual offsets as long as they are quite close to the origin, e.g. abs(offset) <= about sqrt(embedding_dim) Args: embed_dim: Embedding dimension. dropout_rate: Dropout rate. max_len: Maximum input length: just a heuristic for initialization. length_factor: a heuristic scale (should be >= 1.0) which, if larger, gives less weight to small differences of offset near the origin. rrcrmax_lenrrTNcshtt|||_|ddksJ|t||_d|_|dks$J|||_|t d |dS)z0Construct a CompactRelPositionalEncoding object.rrNrrU) rlrrmrcr rMper extend_per!tensorr)rzrcrr!rr~r3r4rms z%CompactRelPositionalEncoding.__init__rr\left_context_lenc Cs^|d|}|jdur%|jd|ddkr%|jj|j|jd|_dStj|d ||jdtjd}dtj|j d|jd}|j d}|| | | t |}|j|j dt j}||}||} ||} tj|jd|j |jd} | | dddddf<| | dddddf<d | ddd f<| j|jd |_dS) zReset the positional encodings.rNrr)rrrrdrrr)r)sizer"rrrr!r%r&rrcsignabsr$r#rpiatanr*r+rr) rzr\r%Tr0compression_length x_compressed length_scalex_atancosinessinesr"r3r3r4r#s. &    z&CompactRelPositionalEncoding.extend_pecCsn||||d|}|j|jdd|d|jdd|dddf}|d}||S)aCreate positional encoding. Args: x (Tensor): Input tensor (time, batch, `*`). left_context_len: (int): Length of cached left context. Returns: positional embedding, of shape (batch, left_context_len + 2*time-1, `*`). rrrN)r#r&r"rrM)rzr\r% x_size_leftrr3r3r4rs"   z$CompactRelPositionalEncoding.forward)r r)r) rrrrrZrr(rmrr#rrr3r3r~r4rs" 3rcseZdZdZdeddfdedededed ed ed ed d ffdd Z dde de de e de e d e f ddZ de fddZ Z S)ra$Module that computes multi-head attention weights with relative position encoding. Various other modules consume the resulting attention weights: see, for example, the SimpleAttention module which allows you to compute conventional attention. This is a quite heavily modified from: "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context", we have to write up the differences. Args: embed_dim: number of channels at the input to this module, e.g. 256 pos_dim: dimension of the positional encoding vectors, e.g. 128. num_heads: number of heads to compute weights for, e.g. 8 query_head_dim: dimension of the query (and key), per head. e.g. 24. pos_head_dim: dimension of the projected positional encoding per head, e.g. 4. dropout: dropout probability for attn_output_weights. Default: 0.0. pos_emb_skip_rate: probability for skipping the pos_emb part of the scores on any given call to forward(), in training time. rUr)r@rUrcrLrJrGrHrMpos_emb_skip_raterTNc st||_||_||_||_||_t||_ d|_ |}||||} t || d|dd|_ t |tdddd|_t||dd d d d dd |_t |||ddd|_t|_t|_dS)NTgпrirrrrrg?g333333?rUgY@rFr)rlrmrcrJrGrHrMrrr3rrrorr whiten_keysr balance_keys linear_posrcopy_pos_query copy_query) rzrcrLrJrGrHrMr3 key_head_dim in_proj_dimr~r3r4rm@sH    z-RelPositionMultiheadAttentionWeights.__init__r\rrrcCs||}|j}|j}|j}|j\}} } ||} |dd| f} |d| d| f} |dd| df}|jd||ksGJ|jd||f|| } ||| } ||}| || ||} | || ||}| || ||} | dddd} | dddd}| dddd} t | | }d}t j st j rd }n|jrtt|jkrd }|r;||}d|d}| d||| dddd}t ||}t j r|j\}} }}t j|dddd }t |}|| |d}||}| d|}t j|d|d }| || ||}n(|j|| ||f|d|d|d|d|df|d|dd }||}t j sGt j rHn|jr\td kr\t|dd|jd}|j|| ||fkshJ|dur||jt jksvJ||d}|dur|j| |fksJ|j||dd}t |dd}t j st j rntdkr|js|!|t"j#j$||j$|jd}|S)aE Args: x: input of shape (seq_len, batch_size, embed_dim) pos_emb: Positional embedding tensor, of shape (1, 2*seq_len - 1, pos_dim) key_padding_mask: a bool tensor of shape (batch_size, seq_len). Positions that are True in this mask will be ignored as sources in the attention weighting. attn_mask: mask of shape (seq_len, seq_len) or (batch_size, seq_len, seq_len), interpreted as ([batch_size,] tgt_seq_len, src_seq_len) saying which positions are allowed to attend to which other positions. Returns: a tensor of attention weights, of shape (hum_heads, batch_size, seq_len, seq_len) interpreted as (hum_heads, batch_size, tgt_seq_len, src_seq_len). .rrNrrrFT)rrstep)r index)storage_offsetrXg9@g-C6?)limitpenaltyrirgMbP?)pr)%rorGrHrJrr8r4r5r7rrr!matmulrrrrrr(r3r6r%repeatrgather as_stridedstriderrrr masked_fillr_print_attn_entropyr functionalrM)rzr\rrrrGrHrJrr_ query_dimqkr@ attn_scoresuse_pos_scoresseq_len2 pos_scorestime1nrowscolsindexesrr3r3r4rs                     z,RelPositionMultiheadAttentionWeights.forwardrc Cs|j\}}}}tLtjjtdd+|tj}|d|j ddj dd }t d|j d|Wdn1sBwYWddSWddS1sZwYdS) NF)enabledg#B ;rr)rrzname=z, attn_weights_entropy = )rr!no_gradampautocast DEVICE_TYPErr&r$rmeanloggingdebugr)rzrrJrrattn_weights_entropyr3r3r4rG#s"  "z8RelPositionMultiheadAttentionWeights._print_attn_entropy)NN)rrrrrrZr(rrmrrrrGrr3r3r~r4r*sH H  rcsHeZdZdZdedededdffdd Zd ed edefd d ZZS) ra[ The simplest possible attention module. This one works with already-computed attention weights, e.g. as computed by RelPositionMultiheadAttentionWeights. Args: embed_dim: the input and output embedding dimension num_heads: the number of attention heads value_head_dim: the value dimension per head rcrJrIrTNcsTttj|||dd|_t|||ddd|_tdtdddd d d |_ dS) NTrjrrir@rrrrr) rlrmrrnrorrprrr)rzrcrJrIr~r3r4rm?s   zSelfAttention.__init__r\rcCs|j\}}}|jd}|j||||fksJ||}||||ddddd}|jd}t||}|dddd||||}||}| |}|S)ah Args: x: input tensor, of shape (seq_len, batch_size, embed_dim) attn_weights: a tensor of shape (num_heads, batch_size, seq_len, seq_len), with seq_len being interpreted as (tgt_seq_len, src_seq_len). Expect attn_weights.sum(dim=-1) == 1. Returns: a tensor with the same shape as x. rrrrr) rrorrr!rA contiguousviewrpr)rzr\rrrrcrJrIr3r3r4rVs       zSelfAttention.forwardrr3r3r~r4r4s$ rcs<eZdZdZdededeffdd Zdefdd ZZ S) rz)Feedforward module in TTSZipformer model.rcrKrMc sdtt|t|||_t|dddddd|_t||d|dd d d |_ t d t d ddd|_ dS)NrrVr?@rSwooshLrTrX) activation dropout_pdropout_shared_dimrjrkrr`rrr) rlrrmrrnror hidden_balancerr rprr out_whiten)rzrcrKrMr~r3r4rms2   zFeedforwardModule.__init__r\cCs,||}||}||}||}|Sr)rorirprj)rzr\r3r3r4rs    zFeedforwardModule.forward) rrrrrZrrmrrrr3r3r~r4r~srcsDeZdZdZdededdffdd Zded edefd d ZZS) ra`This is like the ConvolutionModule, but refactored so that we use multiplication by attention weights (borrowed from the attention module) in place of actual convolution. We also took out the second nonlinearity, the one after the attention mechanism. Args: channels (int): The number of channels of conv layers. channelsrrTNcst||_tj||ddd|_t|dtddtddd d d |_t |_ t |_ t |_ t |_t||dd d |_tdtd ddd|_tdtd ddddd|_dS)NrTr_rr)rWr)rUrc)rWgffffff?rdrdrrrirrrrrr)rlrmrrrnror rbalancerTanhtanhr identity1 identity2 identity3rrprrwhiten1whiten2)rzrkrr~r3r4rms<    zNonlinAttention.__init__r\rc Cs||}|j\}}}|j}|jddd\}}}||}||}|d|||}||}||}| |}|j\}}} |jd} |j| |||fksPJ|||| d dddd}t ||}| dddd||d}| |}||}||}||}||}|S)z. Args: x: a Tensor of shape (seq_len, batch_size, num_channels) attn_weights: a Tensor of shape (num_heads, batch_size, seq_len, seq_len) Returns: a Tensor with the same shape as x rrrrrr)rorrchunkrlrnrrrrrorr!rArprqrprs) rzr\rrrrIrsyrcrJr3r3r4rs,            zNonlinAttention.forwardrr3r3r~r4rs  /rcsLeZdZdZdededdffdd Z d ded eedefd d ZZ S) rzConvolutionModule in Zipformer2 model. Args: channels (int): The number of channels of conv layers. kernel_size (int): Kernerl size of conv layers. bias (bool): Whether to use bias in conv layers (default=True). rk kernel_sizerTNc stt||dddksJ|}t|d||_t|dtddddtd d dd d |_t |_ t |_ t |_ |ddksDJtj|||||dd |_t|dtdddtdddd |_tdtdddd|_t||dddd|_dS)z%Construct a ConvolutionModule object.rrrr)rUr)@@rrg?)rUrd)rx$@rr)r out_channelsgroupsrwpaddingr)rxrr)rWrdryr`rrrrrUr)rfrgrkN)rlrrmrrnror rrr activation1Sigmoidsigmoid activation2Conv1ddepthwise_convrrrrr rp)rzrkrwbottleneck_dimr~r3r4rms^    zConvolutionModule.__init__r\rcCs||}|jddd\}}||}||}||}||}||}|ddd}|dur=||d |d}| |}| |}|ddd}| |}| |}|S)aKCompute convolution module. Args: x: Input tensor (#time, batch, channels). src_key_padding_mask: the mask for the src keys per batch (optional): (batch, #time), contains True in masked positions. Returns: Tensor: Output tensor (#time, batch, channels). rrrrNrU)rortrrr}rrrFr expand_asrrrrp)rzr\rrur3r3r4rms         zConvolutionModule.forwardr) rrrrrZrmrrrrr3r3r~r4rs" Tr)r)r)/rr\r#rtypingrrrr!rrr is_availablerZbackendsrzipvoice.models.modules.scalingr r r r rrrrrrrrrr5rr6r(rrqrrrrsr r rrrrrrr3r3r3r4sB  < a[f<B+y J+i