o }oi ,@sddlZddlmZddlmZmZz ddlmZdZWney+e ddZYnwdd d Z d d Z Gdddej Z Gdddej ZGdddej ZGdddej ZGdddej ZGdddeZdddZdS)N) rearrange)Tensornn GroupNormTz2Fused optimized group norm has not been installed.F cCst||dd|dS)avCreates a group normalization layer with specified activation. Args: in_channels (int): Number of channels in the input. num_groups (int, optional): Number of groups for GroupNorm. Defaults to 32. act (str, optional): Activation function name. Defaults to "". Returns: GroupNorm: A normalization layer with optional activation. gư>T) num_groups num_channelsepsaffineactr) in_channelsr r rY/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/diffusion/vae/blocks.py Normalizes rcCs|t|S)zNonlinearity function used in temporal embedding projection. Currently implemented as a SiLU (Swish) function. Args: x (Tensor): Input tensor. Returns: Tensor: Output after applying SiLU activation. )torchsigmoid)xrrr nonlinearity+s rcs*eZdZdZd fdd Zdd ZZS) ResnetBlockaA ResNet-style block that can optionally apply a temporal embedding and shortcut projections. This block consists of two convolutional layers, normalization, and optional temporal embedding. It can adjust channel dimensions between input and output via shortcuts. NFrcst||_|dur|n|}||_||_t|dd|_tjj ||dddd|_ |dkr5tj |||_ t|dd|_ tj||_tjj ||dddd|_|j|jkrt|jrftjj ||dddd|_dStjj ||dddd|_dSdS)a Args: in_channels (int): Number of input channels. out_channels (int, optional): Number of output channels. Defaults to in_channels. conv_shortcut (bool, optional): Whether to use a convolutional shortcut. Defaults to False. dropout (float, optional): Dropout probability. Defaults to 0.0. temb_channels (int, optional): Number of channels in temporal embedding. Defaults to 0. Nsilur  kernel_sizestridepaddingr)super__init__r out_channelsuse_conv_shortcutrnorm1rrConv2dconv1Linear temb_projnorm2Dropoutdropoutconv2 conv_shortcut nin_shortcut)selfrr"r-r+ temb_channels __class__rrr!@s"  zResnetBlock.__init__cCs|}||}||}|dur#||t|ddddddf}||}||}||}|j|jkrI|j rD| |}||S| |}||S)a&Forward pass of the ResnetBlock. Args: x (Tensor): Input feature map of shape (B, C, H, W). temb (Tensor): Temporal embedding tensor of shape (B, temb_channels). Returns: Tensor: Output feature map of shape (B, out_channels, H, W). N) r$r&r(rr)r+r,rr"r#r-r.)r/rtembhrrrforward\s  &      zResnetBlock.forward)NFrr__name__ __module__ __qualname____doc__r!r5 __classcell__rrr1rr9src(eZdZdZfddZddZZS)UpsamplezUpsampling block that increases spatial resolution by a factor of 2. Can optionally include a convolution after upsampling. cs6t||_|jrtjj||dddd|_dSdS)z Args: in_channels (int): Number of input channels. with_conv (bool): If True, apply a convolution after upsampling. rrrNr r! with_convrrr%convr/rr?r1rrr!s zUpsample.__init__cCsX|j}|tjkr|tj}tjjj|ddd}|tjkr"||}|jr*| |}|S)zForward pass of the Upsample block. Args: x (Tensor): Input feature map (B, C, H, W). Returns: Tensor: Upsampled feature map (B, C, 2H, 2W). g@nearest) scale_factormode) dtyperbfloat16tofloat32r functional interpolater?r@)r/rrErrrr5s     zUpsample.forwardr6rrr1rr=zs  r=cr<) DownsamplezDownsampling block that reduces spatial resolution by a factor of 2. Can optionally include a convolution before downsampling. cs6t||_|jrtjj||dddd|_dSdS)z Args: in_channels (int): Number of input channels. with_conv (bool): If True, apply a convolution before downsampling. rrrNr>rAr1rrr!s zDownsample.__init__cCsF|jrd}tjjj||ddd}||}|Stjjj|ddd}|S)zForward pass of the Downsample block. Args: x (Tensor): Input feature map (B, C, H, W). Returns: Tensor: Downsampled feature map (B, C, H/2, W/2). )rrrrconstantr)rDvaluerL)rr)r?rrrIpadr@ avg_pool2d)r/rrOrrrr5s  zDownsample.forwardr6rrr1rrKs  rKcsJeZdZdZdeffdd ZdedefddZd edefd d ZZ S) AttnBlockzSelf-attention block that applies scaled dot-product attention to feature maps. Normalizes input, computes queries, keys, and values, then applies attention and a projection. rcsjt||_t|dd|_tj||dd|_tj||dd|_tj||dd|_ tj||dd|_ dS)W Args: in_channels (int): Number of input/output channels. rrr)rN) r r!rrnormrr%qkvproj_outr/rr1rrr!s zAttnBlock.__init__h_returnc Cs||}||}||}||}|j\}}}}t|d}t|d}t|d}tj |||}t|d||||dS)zCompute the attention over the input feature maps. Args: h_ (Tensor): Normalized input feature map (B, C, H, W). Returns: Tensor: Output after applying scaled dot-product attention (B, C, H, W). zb c h w -> b 1 (h w) czb 1 (h w) c -> b c h w)r4wcb) rSrTrUrVshaper contiguousrrIscaled_dot_product_attention) r/rYrTrUrVr]r\r4r[rrr attentions   zAttnBlock.attentionrcCs||||S)zForward pass of the AttnBlock. Args: x (Tensor): Input feature map (B, C, H, W). Returns: Tensor: Output feature map after self-attention (B, C, H, W). )rWra)r/rrrrr5s zAttnBlock.forward) r7r8r9r:intr!rrar5r;rrr1rrQs rQcs*eZdZdZdfdd ZddZZS) LinearAttentionzLinear Attention block for efficient attention computations. Uses linear attention mechanisms to reduce complexity and memory usage. rcsDt||_||}tj||dddd|_t||d|_dS)z Args: dim (int): Input channel dimension. heads (int, optional): Number of attention heads. Defaults to 4. dim_head (int, optional): Dimension per attention head. Defaults to 32. rrF)biasN)r r!headsrr%to_qkvto_out)r/dimrfdim_head hidden_dimr1rrr!s zLinearAttention.__init__c Csv|j\}}}}||}t|d|jdd\}}} |jdd}td|| } td| |} t| d|j||d } || S) zForward pass of the LinearAttention block. Args: x (Tensor): Input feature map (B, C, H, W). Returns: Tensor: Output feature map after linear attention (B, C, H, W). z*b (qkv heads c) h w -> qkv b heads c (h w)r)rfqkv)rizbhdn,bhen->bhdezbhde,bhdn->bhenz"b heads c (h w) -> b (heads c) h w)rfr4r[)r^rgrrfsoftmaxreinsumrh) r/rr]r\r4r[rlrTrUrVcontextoutrrrr5s   zLinearAttention.forward)rdrr6rrr1rrcs rccs eZdZdZfddZZS) LinAttnBlockzcWrapper class to provide a linear attention block in a form compatible with other attention blocks.cstj|d|ddS)rRr)rirfrjN)r r!rXr1rrr!$szLinAttnBlock.__init__)r7r8r9r:r!r;rrr1rrr!srrvanillacCsX|dvs Jd|dtd|d|d|dkrt|S|dkr(t|St|S) aoFactory function to create an attention block. Args: in_channels (int): Number of input/output channels. attn_type (str, optional): Type of attention block to create. Options: "vanilla", "linear", "none". Defaults to "vanilla". Returns: nn.Module: An instance of the requested attention block. )rslinearnonez attn_type z unknownzmaking attention of type 'z' with z in_channelsrsru)printrQrIdentityrr)r attn_typerrr make_attn,s  ry)rr)rs)reinopsrrrapex.contrib.group_normrOPT_GROUP_NORM ExceptionrvrrModulerr=rKrQrcrrryrrrrs&    A&$7&