o }oi @sddlZddlZddlmZmZdejdededejfddZGd d d ejZGd d d ejZ  ddejdede de de def ddZ GdddejZ GdddejZ dS)N)TensornnposdimthetareturncCsR|ddks Jdtjd|dtj|jd|}d||}td||}|S)z Different from the original ROPE used for flux. Megatron attention takes the out product and calculate sin/cos inside, so we only need to get the freqs here in the shape of [seq, ..., dim] rzThe dimension must be even.)dtypedeviceg?z ...n,d->...nd)torcharangefloat64r einsumfloat)rrrscaleomegaoutra/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/diffusion/models/flux/layers.pyropes  rcsHeZdZdZdededeeffdd Zdejdejfd d Z Z S) EmbedNDz; Generate Rope matrix with preset axes dimensions. rraxes_dimcs t||_||_||_dSN)super__init__rrr)selfrrr __class__rrr)s  zEmbedND.__init__idsrcsnjd}tjfddt|Ddd}|ddddd}tj||gddjg|jdddRS) Ncs(g|]}td|fj|jqS).)rrr).0irrrr 4s(z#EmbedND.forward..rrr)shaper catrange unsqueezepermutestackreshape)rrn_axesembrr"rforward0s ,zEmbedND.forward) __name__ __module__ __qualname____doc__intlistrr rr0 __classcell__rrrrr$srcs<eZdZdZdedeffdd Zdedefdd ZZS) MLPEmbedderzF MLP embedder with two projection layers and Silu in between. in_dim hidden_dimcs<ttj||dd|_t|_tj||dd|_dS)NT)bias)rrrLinearin_layerSiLUsilu out_layer)rr9r:rrrr@s  zMLPEmbedder.__init__xrcCs||||Sr)r@r?r=)rrArrrr0GszMLPEmbedder.forward) r1r2r3r4r5rrr0r7rrrrr8;sr8Tr%' timesteps embedding_dimflip_sin_to_cosdownscale_freq_shiftr max_periodc Cst|jdks Jd|d}t| tjd|tj|jd}|||}t|}|dddf |dddf}||}tj t |t |gdd}|rktj |dd|df|ddd|fgdd}|ddkrytj j|d }|S) a& This matches the implementation in Denoising Diffusion Probabilistic Models: Create sinusoidal timestep embeddings. Args timesteps (torch.Tensor): a 1-D Tensor of N indices, one per batch element. These may be fractional. embedding_dim (int): the dimension of the output. flip_sin_to_cos (bool): Whether the embedding order should be `cos, sin` (if True) or `sin, cos` (if False) downscale_freq_shift (float): Controls the delta between frequencies between dimensions scale (float): Scaling factor applied to the embeddings. max_period (int): Controls the maximum frequency of the embeddings Returns torch.Tensor: an [N x dim] Tensor of positional embeddings. r%zTimesteps should be a 1d-arrayrr)startendr r Nrr$)rr%rr)lenr'mathlogr r float32r exprr(sincosr functionalpad) rCrDrErFrrGhalf_dimexponentr/rrrget_timestep_embeddingLs   $2 rUc sReZdZ    ddedededed ef fd d Zd ejd ejfddZ Z S) TimestepsTrr%rBrDrErFrrGcs,t||_||_||_||_||_dSr)rrrDrErFrrG)rrDrErFrrGrrrrs  zTimesteps.__init__rCrcCs"t||j|j|j|j|jd}|S)N)rErFrrG)rUrDrErFrrG)rrCt_embrrrr0szTimesteps.forwardTrr%rB) r1r2r3r5boolrrr rr0r7rrrrrVs"rVc sZeZdZdZ    ddededed ed ed ef fd d Zdej dej fddZ Z S)TimeStepEmbeddera A neural network module that embeds timesteps for use in models such as diffusion models. It projects the input timesteps to a higher-dimensional space and then embeds them using an MLP (Multilayer Perceptron). The projection and embedding provide a learned representation of the timestep that can be used in further computations. Args: embedding_dim (int): The dimensionality of the timestep embedding space. hidden_dim (int): The dimensionality of the hidden layer in the MLPEmbedder. flip_sin_to_cos (bool, optional): Whether to flip the sine and cosine components during the projection (default is True). downscale_freq_shift (float, optional): A scaling factor for the frequency shift during the projection (default is 0). scale (float, optional): A scaling factor applied to the timestep projections (default is 1). max_period (int, optional): The maximum period for the sine and cosine functions used in projection (default is 10000). Methods: forward: Takes a tensor of timesteps and returns their embedded representation. Trr%rBrDr:rErFrrGcs0tt|||||d|_t||d|_dS)N)rDrErFrrG)r9r:)rrrV time_projr8 time_embedder)rrDr:rErFrrGrrrrs zTimeStepEmbedder.__init__rCrcCs||}||}|Sr)r[r\)rrCtimesteps_proj timesteps_embrrrr0s  zTimeStepEmbedder.forwardrX) r1r2r3r4r5rYrrr rr0r7rrrrrZs(rZrX)rKr rrr5rModulerr8rYrrUrVrZrrrrs2  7