o }oi K@s(ddlZddlmZddlmZmZddlZddlZddlm m Z ddlm Z ddl m Z ddlmZmZddlmZmZmZmZmZddlmZd gZGd d d e ZGd d d e ZGddde ZGddde ZGddde Zddedede fddZ!Gdd d eZ"GdddeZ#dS)N)partial)DictOptional)nn)Module) NeuralModule typecheck)BoolType FloatType LengthsType NeuralTypeSpectrogramType)loggingTransformerUNetcs<eZdZdZdeffdd ZdejdejfddZZ S) LearnedSinusoidalPosEmbz?The sinusoidal Embedding to encode time conditional informationdimcsDt|ddkrtd|d|d}tt||_dS)NrzInput dimension  is not divisible by 2!)super__init__ ValueErrorr Parametertorchrandnweights)selfrhalf_dim __class__k/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/audio/parts/submodules/transformerunet.pyr9s  z LearnedSinusoidalPosEmb.__init__treturncCsFt|d}|t|jddtj}tj||fdd}|S)z Args: t: input time tensor, shape (B) Return: fouriered: the encoded time conditional embedding, shape (B, D) zb -> b 1zd -> 1 drr) einops rearrangermathpircatsincos)rr!freqs fourieredrrr forward@s zLearnedSinusoidalPosEmb.forward) __name__ __module__ __qualname____doc__intrrTensorr. __classcell__rrrr r6srcs>eZdZdZd dededeeffdd Zd dd ZZS) ConvPositionEmbedzDThe Convolutional Embedding to encode time information of each frameNr kernel_sizegroupsc s\t|ddkrtd|d|dur|}ttj|||||ddt|_dS)Nrrz Kernel size z is divisible by 2!)r8padding)rrrr SequentialConv1dGELU dw_conv1d)rrr7r8rrr rQs   zConvPositionEmbed.__init__cCsV|dur|d}||d}t|d}||}t|d}|dur)||d}|S)z Args: x: input tensor, shape (B, T, D) Return: out: output tensor with the same shape (B, T, D) N).Nzb n c -> b c nzb c n -> b n c) masked_fillr%r&r=)rxmaskoutrrr r.]s      zConvPositionEmbed.forwardN) r/r0r1r2r3rrr.r5rrrr r6Ns  r6cs0eZdZdZfddZdejfddZZS)RMSNormzThe Root Mean Square Layer Normalization References: - Zhang et al., Root Mean Square Layer Normalization, 2019 cs*t|d|_tt||_dSNg?)rrscalerrronesgamma)rrrrr r{s  zRMSNorm.__init__r@cCstj|dd|j|jS)Nr#r$)F normalizerFrH)rr@rrr r.szRMSNorm.forward) r/r0r1r2rrr4r.r5rrrr rDts rDcsFeZdZdZd dedeeffdd Zdejdejfd d Z Z S) AdaptiveRMSNormz Adaptive Root Mean Square Layer Normalization given a conditional embedding. This enables the model to consider the conditional input during normalization. Nrcond_dimcst|dur |}|d|_t|||_t|||_tj|jj tj |jj tj|jj tj|jj dSrE) rrrFrLinearto_gammato_betainitzeros_weightones_bias)rrrLrrr rs  zAdaptiveRMSNorm.__init__r@condcCsNtj|dd|j}||||}}t|d}t|d}|||S)Nr#r$z B D -> B 1 D)rIrJrFrNrOr%r&)rr@rUnormedrHbetarrr r.s    zAdaptiveRMSNorm.forwardrC) r/r0r1r2r3rrrr4r.r5rrrr rKsrKc@s eZdZdZdejfddZdS)GEGLUz#The GeGLU activation implementationr@cCs |jddd\}}t||S)Nrr#r$)chunkrIgelu)rr@gaterrr r.sz GEGLU.forwardN)r/r0r1r2rr4r.rrrr rXsrXr>rmultdropoutc Cs@t||dd}tt||dtt|t||S)zn Return a Feed-Forward layer for the Transformer Layer. GeGLU activation is used in this FF layer r)r3rr:rMrXDropout)rr]r^ dim_innerrrr get_feedforward_layers,rbcseZdZdZ         d$d ed ed ed ed ededededeededeeffdd Zded efddZ e de e e ffddZe de e e ffddZed%deejfddZd ed!efd"d#ZZS)&ra> Implementation of the transformer Encoder Model with U-Net structure used in VoiceBox and AudioBox References: Le et al., Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale, 2023 Vyas et al., Audiobox: Unified Audio Generation with Natural Language Prompts, 2023 r\r>pFNTrdepthheadsff_mult attn_dropout ff_dropout max_positionsadaptive_rmsnormadaptive_rmsnorm_cond_dim_inuse_unet_skip_connectionskip_connect_scalec st|ddkrtd|dtg|_|j||d|r*| dur*td||_| |_|jr:t t | d} nt } | durDd |_ n| |_ t |D]8} | d }| oX||dk}|jt|rht|d|nd| |d tj|||d d | |d t|||dgqKt ||_td|jjtd|td|td||td|td|td|td|td|td|td| td| td|j dS)a4 Args: dim: Embedding dimension depth: Number of Transformer Encoder Layers heads: Number of heads in MHA ff_mult: The multiplier for the feedforward dimension (ff_dim = ff_mult * dim) attn_dropout: dropout rate for the MHA layer ff_dropout: droupout rate for the feedforward layer max_positions: The maximum time length of the input during training and inference adaptive_rmsnorm: Whether to use AdaptiveRMS layer. Set to True if the model has a conditional embedding in forward() adaptive_rms_cond_dim_in: Dimension of the conditional embedding use_unet_skip_connection: Whether to use U-Net or not skip_connect_scale: The scale of the U-Net connection. rrzNumber of layers r)rjrfNzIadaptive_rmsnorm_cond_dim_in must be provided if adaptive_rmsnorm is True)rLg;f?r$T) embed_dim num_headsr^ batch_first)rr]r^Initialized %s withz embedding dim: %sz Number of Layer: %sz feedforward dim: %sz number of heads: %sz Dropout rate of MHA: %sz Dropout rate of FF: %sz maximun time length: %sz use AdaptiveRMS: %sz Conditional dim: %sz Use UNet connection: %sz skip connect scale: %s)rrrr ModuleListlayers init_alibirkrlrrKrDrnrangeappendrMMultiheadAttentionrb final_normrdebugrr/)rrrerfrgrhrirjrkrlrmrn rmsnorm_classindlayerhas_skiprrr rs^                 zTransformerUNet.__init__cCsdd}t|stdtd|tt||d|_ dt t | dt | d}t|d }|jd |d d d S)zInitialize the Alibi bias parameters References: - Press et al., Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, 2021 cSs dd|}|td|dS)Nriro)rarange)nratiorrr get_slopes"s z.TransformerUNet.init_alibi..get_slopeszYIt is recommend to set number of attention heads to be the power of 2 for the Alibi bias!zCurrent value of heads: z B -> B 1 1r#rrozT1 T2 -> 1 T1 T2 pos_matrixF) persistentN)r'log2 is_integerrwarningrrr%r&slopesrabsr unsqueezefloatregister_buffer)rrjrfrrrrr rvs , zTransformerUNet.init_alibir"cCs,tdttdtddtdtdddS)+Returns definitions of module output ports.BTD)rrToptional)rrr@key_padding_maskadaptive_rmsnorm_cond)r r r rrrr input_types4 zTransformerUNet.input_typescCsdtdtiS)routputr)r r rrrr output_types=s zTransformerUNet.output_typesrc Cs|j^}}}g}|j||d}t} |durt|d} |jD]^\} } } } }| dur/||n||j}tj||fdd}| |}| |fi| }|dur\| }| |t d}nd}| ||||d|d\}}||}| |fi| }|||}q| |S) a Forward pass of the model. Args: input: input tensor, shape (B, C, D, T) key_padding_mask: mask tensor indicating the padding parts, shape (B, T) adaptive_rmsnorm_cond: conditional input for the model, shape (B, D) ) batch_sizeseq_lenN)rUr#r$z-infF)querykeyvaluer need_weights attn_mask) shapeget_alibi_biasdictrurxpoprnrr)rr?rz)rr@rrrr_ skip_connects alibi_biasrmsnorm_kwargs skip_combiner attn_prenormattn ff_prenormff skip_connect attn_inputfloat_key_padding_mask attn_outputff_inputrrr r.Ds:     zTransformerUNet.forwardrrcCs8|jddd|d|f}||j}||dd}|S)zK Return the alibi_bias given batch size and seqence length Nro)rrrepeat)rrrrrrrr rus zTransformerUNet.get_alibi_bias) rcr\r>r>rdFNTNNN)r/r0r1r2r3rboolrrrvpropertyrstrr rrrrr4r.rr5rrrr rs\     W 0cseZdZdZ          d(d ededededededededededeededeedeeffdd Ze de e e ffddZ e de e e ffd d!Zed"ejd#efd$d%Zed)d&d'ZZS)*SpectrogramTransformerUNeta:This model handles complex-valued inputs by stacking real and imaginary components. Stacked tensor is processed using TransformerUNet and the output is projected to generate real and imaginary components of the output channels. Convolutional Positional Embedding is applied for the input sequence ror\r>rdNT in_channels out_channelsfreq_dimrrerfrgrirhrjtime_hidden_dimconv_pos_embed_kernel_sizeconv_pos_embed_groupsrkc st||_||_||d}| dur|d} t|||_|r2tt|t|| t |_ t || | d|_ t |||||| | || dd |_||d}t|||_td|jjtd|jtd|jtd |dS) Nrr\)rr7r8T) rrerfrgrirhrjrkrlrmrsz in_channels: %sz out_channels: %sz Input frequency dimension: %s)rrrrrrMproj_inr:rSiLU sinu_pos_embr6 conv_embedrtransformerunetproj_outrr{rr/)rrrrrrerfrgrirhrjrrrrkdim_indim_outrrr rs<    z#SpectrogramTransformerUNet.__init__r"cCs,tdttdtddtdtdddS)rrCrrrTr)input input_length condition)r r r r rrrr rrz&SpectrogramTransformerUNet.input_typescCstdttdtdddS)rrrTr)r output_length)r r r rrrr rs z'SpectrogramTransformerUNet.output_typesr max_lengthcCs0t|t|||j}||dk}|S)af Return the self_attention masking according to the input length. 0 indicates the frame is in the valid range, while 1 indicates the frame is a padding frame. Args: input_length: shape (B) max_length (int): The maximum length of the input sequence return: key_padding_mask: shape (B, T) ro)rrexpandlentodevicer)rrrrrr _get_key_padding_masks z0SpectrogramTransformerUNet._get_key_padding_maskc Cs|j\}}}}||jkrtd|d|jtj|j|jgdd}t|d}| |} |j ||d} |j | | d| } |durDd} n| |} |j | | | d } || } tj| d |jd|d } t| } | |fS) a4Forward pass of the model. Args: input: input tensor, shape (B, C, D, T) input_length: length of the valid time steps for each example in the batch, shape (B,) condition: scalar condition (time) for the model, will be embedded using `self.time_embedding` zUnexpected input channel size z , expected rr$zB C RI D T -> B T (C RI D))r)rANrzB T (C RI D) -> B C D T RI)rRIr)rr RuntimeErrorrstackrealimagr%r&rrrrrrrview_as_complex contiguous) rrrrrC_inrrinput_real_imagr@rtime_embrrrr r.s      z"SpectrogramTransformerUNet.forward)rororrrrr\r>r>rdNrNTr)r/r0r1r2r3rrrrrrrr rr staticmethodrr4rrr.r5rrrr rsj      8r)r\r>)$r' functoolsrtypingrrr%rtorch.nn.functionalr functionalrItorch.nnrnemo.core.classesrrnemo.core.neural_typesr r r r r nemo.utilsr__all__rr6rDrKrXr3rrbrrrrrr s*$    &! K