o }oiD@sddlZddlmZmZmZddlZddlZddlm m Z ddl m Z mZddlmZddlmZddlmZddlmZddlmZdd lm Z Gd d d e Zd d ZGdddeZGdddeZdS)N)DictLiteralOptional)TimestepEmbeddingget_3d_sincos_pos_embed) rearrange) Rearrange)parallel_state)get_pos_emb_on_this_cp_rank)MegatronModule)nncsFeZdZdZd dedeffdd Zdejdejffd d ZZ S) ParallelTimestepEmbeddinga ParallelTimestepEmbedding is a subclass of TimestepEmbedding that initializes the embedding layers with an optional random seed for syncronization. Args: in_channels (int): Number of input channels. time_embed_dim (int): Dimension of the time embedding. seed (int, optional): Random seed for initializing the embedding layers. If None, no specific seed is set. Attributes: linear_1 (nn.Module): First linear layer for the embedding. linear_2 (nn.Module): Second linear layer for the embedding. Methods: __init__(in_channels, time_embed_dim, seed=None): Initializes the embedding layers. N in_channelstime_embed_dimcstj||d|dur0tjt||j|jWdn1s+wYt dkrXt |jj ddt |jj ddt |jj ddt |jj dddSdS)N)rrpipeline_parallelT)super__init__torchrandomfork_rng manual_seedlinear_1reset_parameterslinear_2r &get_pipeline_model_parallel_world_sizesetattrweightbias)selfrrseed __class__h/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/diffusion/models/dit/dit_embeddings.pyr2s     z"ParallelTimestepEmbedding.__init__xreturncst|jtjddS)z Computes the positional embeddings for the input tensor. Args: x (torch.Tensor): Input tensor of shape (B, T, H, W, C). Returns: torch.Tensor: Positional embeddings of shape (B, T, H, W, C). T non_blocking)rforwardtorbfloat16)rr%r!r#r$r)@s z!ParallelTimestepEmbedding.forwardN __name__ __module__ __qualname____doc__intrrTensorr) __classcell__r#r#r!r$r s"r cCst}t}tj|gdddjdd}|jg|jd||d|j|ddR}|||}|jg|jd|d|j|ddR}|S) as Adjusts the positional embeddings tensor to the current context parallel rank. Args: pos_emb (torch.Tensor): The positional embeddings tensor. seq_dim (int): The sequence dimension index in the positional embeddings tensor. Returns: torch.Tensor: The adjusted positional embeddings tensor for the current context parallel rank. cpuT)device pin_memoryr'Nr) r get_context_parallel_world_sizeget_context_parallel_rankrtensorcudaviewshape index_select)pos_embseq_dimcp_sizecp_rankcp_idxr#r#r$r Ms 4 0r csDeZdZdZ  d dededeffdd Zdejfd d ZZ S) SinCosPosEmb3DaH SinCosPosEmb3D is a 3D sine-cosine positional embedding module. Args: model_channels (int): Number of channels in the model. h (int): Length of the height dimension. w (int): Length of the width dimension. t (int): Length of the temporal dimension. spatial_interpolation_scale (float, optional): Scale factor for spatial interpolation. Default is 1.0. temporal_interpolation_scale (float, optional): Scale factor for temporal interpolation. Default is 1.0. Methods: forward(pos_ids: torch.Tensor) -> torch.Tensor: Computes the positional embeddings for the input tensor. Args: pos_ids (torch.Tensor): Input tensor of shape (B S 3). Returns: torch.Tensor: Positional embeddings of shape (B S D). ?hwtcsvtj|d||_||_||_t|j||g|||}t|d}tj |j d|j|_ tj j t|dd|j _dS)Nconfigzt hw c -> (t hw) crF) requires_grad)rrrHrIrJr hidden_sizerrr Embeddingr? pos_embedding Parameterr<r)rrLrHrIrJspatial_interpolation_scaletemporal_interpolation_scaleparamr!r#r$rxs  zSinCosPosEmb3D.__init__pos_idscCs4|d|j|j|d|j|d}||SN).r).r).r9)rHrIrP)rrUpos_idr#r#r$r)s* zSinCosPosEmb3D.forward)rGrGr-r#r#r!r$rFasrFcsJeZdZdededeffdd ZddZdd Zd ejfd d Z Z S) FactorizedLearnable3DEmbeddingrJrHrIc stj|dtj||j|_tj||j|_tj||j|_d| vr]|d}tj "t ||j r?|n |WddSWddS1sVwYdS|j rf|dSdS)NrKr )rrrr rOrNemb_temb_hemb_wkeysrrrperform_initializationcustomize_init_paramr)rrLrJrHrIkwargsr r!r#r$rs"     " z'FactorizedLearnable3DEmbedding.__init__cCs4|j|jj|j|jj|j|jjdSr,)rL init_methodrYrrZr[rr#r#r$r^sz3FactorizedLearnable3DEmbedding.customize_init_paramcCs"|j|j|jdSr,)rYrrZr[rar#r#r$rs  z/FactorizedLearnable3DEmbedding.reset_parametersrUcCs*||d||d||dSrV)rYrZr[)rrUr#r#r$r)s*z&FactorizedLearnable3DEmbedding.forward) r.r/r0r2rr^rrr3r)r4r#r#r!r$rXsrX)mathtypingrrrnumpynprtorch.nn.functionalr functionalFdiffusers.models.embeddingsrreinopsreinops.layers.torchr megatron.corer ;megatron.core.models.common.embeddings.rotary_pos_embeddingr megatron.core.transformer.moduler r rFrXr#r#r#r$s       .2