o }oip@sddlZddlmZddlmZmZmZmZmZm Z ddl Z ddl m m ZddlmZGddde j jZGddde j jZGd d d e j jZGd d d eZGd ddeZGddde j jZGddde j jZdS)N)abstractmethod)CallableDictListOptionalTupleUnion)loggingcsXeZdZ      ddedededed eed ed ed effd d ZddZZS)ConvolutionLayerNTF in_channels out_channels kernel_sizestridepaddingdilationbias is_causalc std|_|r)|d|df|_|dur&t|d|d|jdd}n|durA|ddkr7tdt||dd}||_||_||_ t j j |||||||d |_ dS) a A convolutional layer that supports causal convolutions with padding. Replaces the standard MLP layer used in the original transformer. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. kernel_size (int): Size of the convolving kernel. stride (int): Stride of the convolution. padding (Optional[int]): Padding added to both sides of the input. If None, it's calculated automatically. dilation (int): Spacing between kernel elements. bias (bool): If True, adds a learnable bias to the output. is_causal (bool): If True, uses causal convolution. Nr rz@ was initialized with is_causal set to True, and padding set to z8. The provided padding value will be ignored and set to .z1`kernel_size` must be odd when `padding` is None.)rrrrr)super__init__causal_paddingr warning ValueErrorintrrrtorchnnConv1dconv) selfr r rrrrrr __class__a/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/tts/modules/transformer_2501.pyrs6    zConvolutionLayer.__init__cCs"|jr t||j}||}|SN)rFpadrr)r signal conv_signalr#r#r$forwardUs zConvolutionLayer.forward)r r Nr TF) __name__ __module__ __qualname__rrboolrr* __classcell__r#r#r!r$r s2 :r csVeZdZdddejjddfdededed ed ed ed e ffd d Z ddZ Z S)PositionwiseConvFFr FTtanh approximated_modeld_ffn p_dropoutrrr non_linearitycsPt||_||_t|||||d|_t|||||d|_tj ||_ dS)aV Positionwise Convolutional Feed-Forward layer to replace the MLP layer in transformers. Module will take the input with d_model hidden state, project it to d_ffn hidden dimension, perform nonlinear transformation, and project the state back into d_model hidden dimension. Finally, it applied dropout. Args: d_model (int): Input and output dimension of the model. d_ffn (int): Hidden dimension of the feed-forward network (usually 4 * d_model). p_dropout (float): Dropout probability. kernel_size (int): Size of the convolving kernel. bias (bool): If True, adds a learnable bias to the convolution layers. is_causal (bool): If True, uses causal convolution. non_linearity (Callable): Activation function to use (default: GELU). )rrrN) rrr4r7r projo_netrrDropoutdropout)r r4r5r6rrrr7r!r#r$r_s zPositionwiseConvFF.__init__cCs4|||dd}|||dd}|S)z x (B, T, C) r r)r7r8 transposer;r9)r xr#r#r$r*szPositionwiseConvFF.forward) r+r,r-rrGELUrfloatr.rrr*r/r#r#r!r$r0^s* "r0csPeZdZ ddedededeffdd Ze   dd ej d e ej d e ej d e ej fd dZ e de ee eeej fffddZddefddZ    dd ej d e ej d e ej d e ej de ej deej eej ff ddZ    dd ej d e ej d e ej d e ej de ej deej eej ff ddZZS) AttentionTn_headsr4r6rcst||dksJd|||_||_||_|jd|_||_tjj ||j|dd|_ tj ||_ d|_ ||_dS)aN Base Attention parent class. Users should not be instantiating this class, but rather use SelfAttention or CrossAttention classes as appropriate. Does DotProductionAttention and additionally dropout inside the module. The class does not currently support RoPE nor ALiBi. Args: n_heads (int): Number of attention heads. d_model (int): Dimension of the model. p_dropout (float): Dropout probability. is_causal (bool): Whether to use causal attention. Only supported when used in SelfAttention. rzd_model % n_head != 0gFrN)rrd_headrAr4scalerrrLinearr9r:r; use_cache _init_cachecache)r rAr4r6rr!r#r$rs   zAttention.__init__Nquery query_maskmemory memory_maskcCsdSr%r#)r rIrJrKrLr#r#r$compute_qkv_and_maskszAttention.compute_qkv_and_maskreturncCsdddddddS)NF)is_initializedself_kself_vcross_kvcross_kcross_vr#r#r#r#r$rGszAttention._init_cacheFrFcCs||_||_dSr%)rFrGrHr rFr#r#r$ reset_cacheszAttention.reset_cache attn_priorcCs|jr+|jdr&|ddddddf}|dur#|ddddfnd}nd|jd<|j||||d\}}}} |dd}|dd}|dd}|j\} } } t||dd|j} | durk| | dkt d |j r| |j d d| d| fdkt d |durd }|ddd| f}t ||}|dddf d|jdd}tj| dd |}tj|dd }ntj| dd }| dur|| dkd }||}t||}|dd| | d}||| gfS)NrOT)rIrJrKrLr rrz-inf.g:0yE>dim)rFrHrMr<shapermatmulrD masked_fill_r?r causal_masklogrepeatrAr& log_softmaxsoftmax masked_fillr; contiguousview)r rIrJrKrLrWqkvmaskBT_ attn_scoreepsattn_score_log attn_probyr#r#r$ attn_naives> "     (   zAttention.attn_naivecCs.||||||\}}|||}||fS)a Forward pass of the Attention module. Args: query (torch.Tensor): Input tensor of shape (B, T1, C). query_mask (Optional[torch.Tensor]): Mask for query tensor of shape (B, T1). memory (Optional[torch.Tensor]): Memory tensor for cross-attention of shape (B, T2, C). memory_mask (Optional[torch.Tensor]): Mask for memory tensor of shape (B, T2). attn_prior (Optional[torch.Tensor]): Prior attention weights of shape (B, T1, T2). Returns: Tuple[torch.Tensor, List[torch.Tensor]]: - y: Attention module tensor output of shape (B, T1, C). - attn_prob: List containing attention probabilities and scores. returned only in attn_naive. [0]: Attention probabilities used for logging during validation. [1]: Attention scores used for CTC loss (only in naive attention). )rtr;r9)r rIrJrKrLrWrsrrr#r#r$r*szAttention.forward)TNNNFNNNN)r+r,r-rr?r.rrrTensorrrM staticmethodrstrrrGrVrrrtr*r/r#r#r!r$r@sz  &  8r@c sneZdZ  ddededededef fdd Z dd ejd e ejd e ejde ejfddZ Z S) SelfAttentionTrAr4r6rmax_length_causal_maskc stj||||d|r/|dus|dkrtd|d|dtt||dd||tjj |d||j d d |_ dS) a Implements SelfAttention. See parent class for forward implementation. Args: n_heads (int): Number of attention heads. d_model (int): Dimension of the model. p_dropout (float): Dropout probability. is_causal (bool): Whether to use causal attention. Only supported when used in SelfAttention. max_length_causal_mask (int): Maximum sequence length for Attention module. rAr4r6rNrzUSelf Attention was called with is_causal True, but received an inappropriate valueof z for max_length_causal_maskr`r rYFrB) rrrregister_bufferrtrilonesrgrrErCqkv_net)r rAr4r6rr}r!r#r$rs("zSelfAttention.__init__NrIrJrKrLc Cs|j\}}}||||d|j|j}|jddd\} } } | d| d| d} } } |jr[|jddurQt j |jd| gdd} t j |jd| gdd} | |jd<| |jd<|durk|ddddddfnd} | | | | fS)NrYrrZrPr rQ) r]rreshaperArCchunksqueezerFrHrcat) r rIrJrKrLrlrmrnqkvrhrirjrkr#r#r$rM?s "  $ z"SelfAttention.compute_qkv_and_mask)Tr|ru) r+r,r-rr?r.rrrxrrMr/r#r#r!r$r{s4)r{c sdeZdZdedededeffdd Z   ddejd eejd eejd eejfd d Z Z S)CrossAttentionrAr4d_memoryr6cs`tj|||dd|durtdtjj|||jdd|_tjj|d||jdd|_dS)al Implements CrossAttention. See parent class for forward implementation. Must be non-causal. Args: n_heads (int): Number of attention heads. d_model (int): Dimension of the model. d_memory (int): Dimension of the conditioning / cross-attention input. p_dropout (float): Dropout probability. Fr~Nz-d_memory must be provided for cross-attentionrBr) rrrrrrErCq_netkv_net)r rAr4rr6r!r#r$rUs"zCrossAttention.__init__NrIrJrKrLcCs|j\}}}|j\}} }|||||j|j} |jr)|jddur)|jd} n|||| d|j|j} |jrL|jddurL|jd} |jd} n&| jddd\} } | d| d} } |jrr| |jd<| |jd<| |jd<|dur|ddddfnd}| | | |fS)NrRrrSrTrZ) r]rrrArCrFrHrrr)r rIrJrKrLBqTqrnBkvTkvrhkvrirjrkr#r#r$rMps"         z#CrossAttention.compute_qkv_and_maskru) r+r,r-rr?rrrxrrMr/r#r#r!r$rTs,rcseZdZdddddejjddfdededed ed ed ed e ed e edededede ffdd Z e de fddZd!ddZ   d"dejdejde ejde ejde ejde f dd ZZS)#TransformerLayerNTr|r1r2r4r5 sa_n_headsrr6 has_xattn xa_d_memory xa_n_headsrapply_norm_to_condr}conv_non_linearityc st||_tjj|dd|_t|||| | d|_|jr?| |_ tjj|dd|_ t ||||d|_ |j r?tjj|dd|_ tjj|dd|_t||||| | d|_d|_||_dS)af One layer of the Transformer. Args: d_model : Model dimension d_ffn : Feed forward dimension (usually 4*d_model) sa_n_heads : Number of attention heads used in self-attention kernel_size : Convolution kernel size for FFN p_dropout : Dropout probability has_xattn : Whether to use cross attention xa_d_memory : Hidden dimension for cross attention xa_n_heads : Number of attention heads used in cross attention is_causal : Whether to use causal attention apply_norm_to_cond : Whether to apply normalization to conditioning tensor max_length_causal_mask : Maximum length of causal mask conv_non_linearity : Convolution non-linearity FrB)rAr4r6r}r)rAr4rr6)rrr7N)rrrrr LayerNorm norm_selfr{self_attentionrnorm_xattn_queryrcross_attentionnorm_xattn_memory norm_pos_ffr0pos_ffrFrGrH) r r4r5rrr6rrrrrr}rr!r#r$rs6  zTransformerLayer.__init__rNcCs ddddS)N)self_attn_outputcross_attn_outputrKr#r#r#r#r$rGszTransformerLayer._init_cacheFcCs6||_||_|j||jr|j|dSdSr%)rFrGrHrrVrrrUr#r#r$rVs   zTransformerLayer.reset_cacher=x_maskcond cond_maskrWc CsL||d}|j|||d\}}|jr.|jddur)tj|jd|gdd}||jd<||}d}|jr|dur||} |jrP|jddurP|jd} n|j rX| |n|} |jrb| |jd<|j | || ||d\} }|jr|jd durtj|jd | gdd} | |jd <|| }|| | |}||d}|||d d S) a Args: x (B, T1, C): Input tensor x_mask (B, T1): Multiplicative mask where True means we keep the input, False we zero it out. Mask for self attention input. cond (B, T2, C): Conditioning tensor cond_mask (B, T2): Multiplicative mask where True means we keep the input, False we zero it out. Mask for cross attention input if it exists. Returns dict with keys output (B, T1, C): Output tensor attn_probabilities : Attention probabilities rX)rIrJrNr rZrK)rIrJrKrLrWr)self_attn_probabilitiescross_attn_probabilitiesoutputattn_probabilities) unsqueezerrrFrHrrrrrrrrr) r r=rrrrWx_ s_attn_prob x_attn_probx_normedrKx_resr#r#r$r*s8       zTransformerLayer.forwardrvru)r+r,r-rrr>rr?r.rrrryrrGrVrxr*r/r#r#r!r$rsf       @  rc!seZdZddddddddddejjddf ded ed ed ed ed ededede ede edededededede f fdd Z d*ddZ e ddZe dede eejeejfd e eejeejfd!e eejeejfd"e ee ed#ee eje eje ejff d$d%Z    d+d&ejd'ejde eejeejfd e eejeejfd!e eejeejfd"e ee ed#eeeejefffd(d)ZZS), Transformerr\FNTr|r1r2n_layersr4r5rrr6 p_dropout_outrrrrrapply_norm_outr}use_learnable_pos_embrcs<|r| dus | durtdttj||_||_|jdkr+tj|j|_nd|_| |_ |j r>tjj |dd|_ nd|_ tj |_ t|D]}|j t||||||| | | | ||d qK||_d|_|jrstj|||_||j|D]\}}d|vr|drtjjj|dd td |d q}dS) a Initializes a stack of transformer layers. Can be used for both encoder and decoder. Set is_causal is True for autoregressive models. Equivalent to TransformerBlock from Megatron-LM Args: n_layers : Number of transformer layers d_model : Model dimension d_ffn : Feed forward dimension (usually 4*d_model) sa_n_heads : Number of attention heads used in self-attention kernel_size : Convolution kernel size for FFN p_dropout : Dropout probability p_dropout_out : Dropout probability for output has_xattn : Whether to use cross attention xa_d_memory : Hidden dimension for cross attention; required if has_xattn is True xa_n_heads : Number of attention heads used in cross attention; required if has_xattn is True is_causal : Whether to make attention and the convolution feedforward networks causal. apply_norm_to_cond : Whether to apply normalization to conditioning tensor; conditioning tensor being the input to the memory part of cross-attention. apply_norm_out : Whether to apply normalization to output max_length_causal_mask : Maximum length of causal mask use_learnable_pos_emb : Whether to add a learnable positionable embedding inside the class conv_non_linearity : Convolution non-linearity NzWIt requires that `xa_d_memory` and `xa_n_heads` are specified when `has_xattn` is True!r\FrB) r4r5rrr6rrrrrr}rr9weight{Gz?rmeanstd)rrrrrr:r;r dropout_outrrnorm_out ModuleListlayersrangeappendrrposition_embeddings Embeddingapply_init_weights_gpt2named_parametersendswithinitnormal_mathsqrt)r rr4r5rrr6rrrrrrrr}rrrnnameparamr!r#r$rsP)     "zTransformer.__init__cCs|jD]}||qdSr%)rrV)r rFlayerr#r#r$rVss  zTransformer.reset_cachecCsft|tjjtjjtjjfrtjjj|jdddt|tjjr/|j dur1tjj |j dSdSdS)Nr\rr) isinstancerrrErrrrrrzeros_)moduler#r#r$rws zTransformer._init_weights_gpt2idxrrrWmulti_encoder_mappingrNcCs^|dur*||dur dS||||dur|||nd|dur'|||fSdfS|||fS)Nrur#)rrrrWrr#r#r$_get_layer_inputss   zTransformer._get_layer_inputsr=rcCst|trt|jt|krtdt|jdt|d|jr6tj|d|j d d}|| |}g}| |}t |jD]$\} } || ||||\} } } | ||| | | d}|d}||d qB|jd urq||}|jd ur{||}||d S) a Args: x (B, T1, C): x_mask (B, T1): Multiplicative mask where True means we keep the input, False we zero it out. Mostly used in non-causal self-attention to zero out padding values. In causal self-attention, the causal mask will be used in place of this. cond (B, T2, C) or list of such tensors (from different encoders) cond_mask (B, T2): Multiplicative mask where True means we keep the input, False we zero it out or list of such tensors (from different encoders) output (B, T1, C) multi_encoder_mapping : None or Same size as n_layers, value indicates which cond input to use for this layer Returns dict with keys: output (B, T1, C): Output tensor attn_probabilities : Attention probabilities of each layer znInsufficient Transformer layers for multiple conditionals. Each layer must cross-attend one conditional.Found z layers for z conditionals.r )devicer)rWrrNr)rlistlenrrrrarangesizerrrr; enumeraterrrr)r r=rrrrWr positionsrrr_cond _cond_mask _attn_priorout_dictr#r#r$r*s2        zTransformer.forwardrvrw)r+r,r-rrr>rr?r.rrrrVryrrrxrrrrrzr*r/r#r#r!r$rs       Z  r)rabcrtypingrrrrrrrtorch.nn.functionalr functionalr& nemo.utilsr Moduler r0r@r{rrrr#r#r#r$s   D,<: