o wiP@slddlZddlmZddlmZmZmZddlZddlm Z ddlm Z m Z ddl m Z ddlmZz ddlmZeZWneyVdd lmZdZej jZed Ynwdd lmZdd lmZdd lmZmZddlm Z ddl!m"Z"ddlm#Z#eGddde#Z$GdddZ%Gddde j&Z'dej(de)dej(fddZ*Gddde Z+GdddeZ,dS) N) dataclass)CallableOptionalUnion)MegatronModule)einsumnn)ColumnParallelConv2dPatch)MultimodalProjectorConfig)TENorm)loggingzFailed to import Transformer Engine dependencies. `from megatron.core.extensions.transformer_engine import *`If using NeMo Run, this is expected. Otherwise, please verify the Transformer Engine installation.) VisionModule) ModelType) ModuleSpec build_module)TransformerBlock)TransformerConfig) CLIPViTConfigc@sveZdZUdZdZeed<dZeed<dZ eed<dZ eed<d Z eed <d Z eed <d Z eed <dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<ejjjZeed<dZeed<dZ eed <dZ!eed!<dZ"eed"<dZ#eed#<d$Z$eed%<d&Z%eed'<dZ&eed(<d)Z'eed*<dZ(eed+<d,Z)e*e+ed-<d2d0d1Z,d,S)3Llama4VisionConfigzlayernorm_epsilonapply_rope_fusiong?pixel_shuffle_ratiorotary_interleavedNtransformer_layer_specreturnLlama4ViTModelc CsR|j}t|tsddlm}|dd}t|||j|j|j|j |j |j |j |j d S)z9Configures and returns an instance of the Llama4ViTModel.r)get_layer_spec_teT)is_vit) ln_pre_impl ln_post_impladd_class_tokenclass_token_lenrrr model_subtype)r8 isinstancer nemo.collections.vlm.layer_specsr;r:r=r>r?r@rrrr)selfr8r;rEe/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/vlm/llama4/model/vision.pyconfigure_modelTs    z"Llama4VisionConfig.configure_model)r9r:)-__name__ __module__ __qualname____doc__rstr__annotations__rintrrrrr r"r#boolr$r%r&floatr'r(r*r+torchr functionalgelur,rr-r.r/r0r1r3r4r5r6r7r8rrrGrErErErFr3s>                           rc@s<eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd S) PackingIndexz=Defines constant indices for accessing packed token metadata.rN)rHrIrJrKZYXTIMEHEIGHTWIDTHIDX BATCH_IDX NUM_METADATA ID_CLS_TOKEN ID_PAD_TOKENrErErErFrTksrTcs<eZdZdZdeffdd ZdejdejfddZZ S) PixelShufflea8Performs pixel shuffle operation on encoded patches. Rearranges elements in a tensor of shape [B, N, C] representing encoded image patches by moving spatial dimensions from the channel dimension. Args: ps_ratio (float): The ratio for pixel shuffling (e.g., sqrt of downscaling factor). ps_ratiocst||_dS)zyInitializes the PixelShuffle module. Args: ps_ratio (float): The ratio for pixel shuffling. N)super__init__rk)rDrk __class__rErFrms  zPixelShuffle.__init__xr9cCs|jdus Jd|dksJdtt|jd}}||jd||d}t||jd}||jdd|jd}|S) aEApplies the pixel shuffle operation. Args: x (torch.Tensor): Input tensor of shape [B, N, C], where N is the number of patches. Returns: torch.Tensor: Tensor after pixel shuffle, shape [B, N', C'], where N' = N * ps_ratio^2 and C' = C / ps_ratio^2. Nz&ps_ratio is required for pixel shufflerWz0pixel shuffle requires encoded patches [B, N, C]rUrr^)rk)rkdimrNmathsqrtshapereshapepixel_shuffle_op)rDrphhwwpixel_shuffle_patchesrErErFforwards zPixelShuffle.forward) rHrIrJrKrPrmrQTensorrz __classcell__rErErnrFrjs  rjinput_xrkr9c Cs|\}}}}|||t||t||}|dddd}||t||t||t|||}|dddd}|S)aHelper function to perform the core pixel shuffle logic. Args: input_x (torch.Tensor): Input tensor of shape [n, w, h, c]. ps_ratio (float): Pixel shuffle ratio. Returns: torch.Tensor: Output tensor after pixel shuffle. rrVrUrW)sizeviewrNpermute contiguous)r}rknwhcrErErFrvs   rvc sReZdZdZ  ddededededef fd d Zd e j d e j fd dZ Z S)PixelShuffleMLPaApplies pixel shuffle followed by an MLP projection. Takes encoded patches, performs pixel shuffling, and then projects them using a configurable MLP. Args: config (TransformerConfig): Megatron core transformer configuration. ps_ratio (float): Ratio for the pixel shuffle operation. input_dim (int): Input dimension before pixel shuffle. The dimension after pixel shuffle (input to MLP) will be input_dim / (ps_ratio**2). output_dim (int): Output dimension of the MLP projection. Defaults to 4096. add_fc (bool): Whether to add an additional fully connected layer (Not Implemented). Defaults to False. r)Fconfigrk input_dimr*add_fccsPt|t||_tdt||d||ddd|_|j|_|r&t dS)z'Initializes the PixelShuffleMLP module. mcore_mlprVF)projector_type input_sizer%r(biasr/N) rlrmrj pixel_shuffler rN mlp_configrGmlpNotImplementedError)rDrrkrr*rrnrErFrms  zPixelShuffleMLP.__init__encoded_patchesr9cCs"||}|jjj}|||S)zForward pass through pixel shuffle and MLP. Args: encoded_patches (torch.Tensor): Input encoded patches of shape [B, N, C_in]. Returns: torch.Tensor: Output tensor after projection, shape [B, N', C_out]. )rrencoderr,)rDrr,rErErFrzs zPixelShuffleMLP.forward)r)F) rHrIrJrKrrPrNrOrmrQr{rzr|rErErnrFrs rcseZdZdZeeddddddfdeded eeefd eeefd e d e d e de de de ddffdd Z de jfddZd-de de de jfddZe jjdddde jd e jde jfd!d"Zd#e jddfd$d%Zd.d&e jd'ee jde jfd(d)Zd*e jde jfd+d,ZZS)/r:aDLLama vision model. Args: transformer_config (TransformerConfig): Transformer config. transformer_layer_spec (ModuleSpec): Specifies module to use for transformer layers. ln_pre_impl (ModuleSpec or type): Specifies the layer norm type to use for ln_pre. add_class_token (bool, optional): Include a class token. Defaults to True. class_token_len (int): Class token length. Defaults to 1 but 8 may be faster. patch_dim (int): Image patch size. img_h (int): Input image height. img_w (int): Input image width. TrUrrrtransformer_configr8r=r>r?r@rrrrAr9Nc stj|d||_|j|_||_||_| |_|j|jdks!J|j|jdks+J|j|j|_|j|j|_ |j|j |_ ||_ ||_|j |j rN|jnd|_ t |||j|jd|_t |||j|jd|_t|d|j|j|jdd|_t|j dd|_tj|j |j|_||_ |j r|jd } tj| td|j|j|_tj|_ t!||d ddd |_"t#||j$|j|j%d |_&|j%|_%|'|_(d S)aInitializes the Llama4 Vision Transformer model. Args: transformer_config (TransformerConfig): Transformer configuration object. transformer_layer_spec (ModuleSpec): Specification for the transformer layer module. ln_pre_impl (Union[ModuleSpec, type]): Implementation for pre-layer normalization. Defaults to NORM_IMPL. ln_post_impl (Union[ModuleSpec, type]): Implementation for post-layer normalization. Defaults to NORM_IMPL. add_class_token (bool): Whether to prepend a class token to the patch sequence. Defaults to True. class_token_len (int): Length of the class token sequence. Defaults to 1. patch_dim (int): Dimension of the square patches the image is divided into. Defaults to 14. img_h (int): Height of the input images. Defaults to 336. img_w (int): Width of the input images. Defaults to 336. model_subtype (str): Subtype identifier for the model. Defaults to "llama4". )rr)rr%epsrWF)r in_channels out_channels kernel_sizestriderrUr^gT)rspec pre_process post_processvp_stage)rrkrr*N))rlrmr@r%visual_hidden_sizerrrnum_patches_per_dim_hnum_patches_per_dim_w num_patchesr? seq_lengthrr4ln_preln_postr conv1rQarangeexpandcuda position_idsr Embeddingposition_embeddings Parameterrandn class_tokenrencoder_or_decoder model_typerdecoderrr6r*adapter get_rope_embrotary_pos_emb) rDrr8r=r>r?r@rrrrAscalernrErFrmst  zLlama4ViTModel.__init__c CsD|j}}|j||j|}}tj|j|j||tjd}|||d}tj||ddgdd}tj |d<tj |j d|j dtj dtjd}|||ddddtj f<|||ddddtjf<|ddddtjf||ddddtjf|||ddddtjf<|ddtj d}||jj|jjd}|||ddddtjfd}|||ddddtj fd} tj|| gddd dddf} | |ddddtjdfdkd} | d} tj| dd| ddfdd| j dd} | ddddddf} | S) aComputes the Rotary Position Embedding (RoPE) based on image patch coordinates. Generates 2D RoPE embeddings using the packed image index metadata. The embeddings are computed separately for X and Y coordinates and concatenated. RoPE is disabled for padding and CLS tokens. Returns: torch.Tensor: The computed RoPE tensor of shape [seq_length, 1, 1, dim], ready to be applied in the attention mechanism. )dtyperUNrrq)r^r^r^rV.)rrrrQrint32rucatrTrhemptyrtrgr`rarcfill_rdreget_rope_freqsrr%rcompute_rope_freqsrPr masked_fillsqueezestackrr) rDpatch_hpatch_widx_hidx_wimg_idxpacked_img_idx rope_freqfreqs_xfreqs_yfreqsrrErErFrms6   $$(& 0zLlama4ViTModel.get_rope_emb'rqthetacCs.d|td|dd|d|}|S)abCalculates the base frequencies for RoPE. Args: dim (int): The dimension of the embeddings for which RoPE is calculated (usually head_dim // 2). theta (int): The base period for the sinusoidal embeddings. Defaults to 10000. Returns: torch.Tensor: A 1D tensor containing the RoPE frequencies. g?rrVN)rQrrP)rDrqrrrErErFrs* zLlama4ViTModel.get_rope_freqsrF)enabledrtcCs&td||j|}|jddd}|S)aComputes RoPE frequencies for given positions `t`. Applies the base frequencies to the position indices and interleaves them. Args: freqs (torch.Tensor): Base RoPE frequencies (output of get_rope_freqs). t (torch.Tensor): Tensor containing position indices (e.g., X or Y coordinates). Returns: torch.Tensor: RoPE frequencies corresponding to the input positions `t`. z..., f -> ... frVr^r)rtyperrepeat_interleave)rDrrrErErFrs z!Llama4ViTModel.compute_rope_freqs input_tensorcCs|j|dS)zSets input tensor to the model's decoder block. Args: input_tensor (torch.Tensor): Input tensor to set. N)rset_input_tensor)rDrrErErFrszLlama4ViTModel.set_input_tensorrpattention_maskcCs||}|jr|j|jddd}tj||gdd}|jd|jks1J|jdd|j|||j }|j rA| |}| ddd}| }|j |||jd}| ddd}| }|jrh||}|S)aForward function of the ViT Model. This function passes the input tensors through the embedding layer and then the transformer. Args: x (torch.Tensor): input data of shape [batch, img_h, img_w] attention_mask (torch.Tensor with dtype=bool): Attention mask to use. Returns: x (torch.Tensor): output after final transformer block of shape [b, s, h]. rr^rUrz != rV)r)rr?rrrtrQrrrrrrrrrr)rDrprrrErErF_encodes( *  zLlama4ViTModel._encodeimagescCs2||}|ddddddf}||}|S)aProcesses input images through the ViT encoder and adapter. Args: images (torch.Tensor): Input image tensor of shape [batch, channels, height, width]. Returns: torch.Tensor: Projected embeddings after passing through the encoder and adapter, typically of shape [batch, num_output_patches, output_dim]. The CLS token output (if used) is removed before the adapter. Nr^)rr)rDr embeddingprojected_embeddingrErErFrzs  zLlama4ViTModel.forward)r)N)rHrIrJrK NORM_IMPLrrrrrOrNrLrmrQr{rrampautocastrrrrrzr|rErErnrFr:sX      k. "&r:)-rr dataclassesrtypingrrrrQ megatron.core.transformer.modulerrr(nemo.collections.vlm.mllama.model.visionr nemo.collections.vlm.vision.baser +megatron.core.extensions.transformer_enginer r ImportError nemo.utilsr r2warning7megatron.core.models.common.vision_module.vision_moduler megatron.core.transformer.enumsr$megatron.core.transformer.spec_utilsrr+megatron.core.transformer.transformer_blockr,megatron.core.transformer.transformer_configrrrrTModulerjr{rPrvrr:rErErErFs@           7'5