o }oim?@sddlmZddlmZmZmZddlZddlZddlm Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZmZdd lmZd eeejffd dZd ejfddZeGdddeZGdddeZ GdddeZ!dS)) dataclass)CallableDictOptionalNparallel_state)InferenceParams)PackedSeqParams)TransformerConfig)nn) TokenizerSpec)MODEL_CONFIG_ATTRMCoreNevaModel NevaConfig NevaModel)OptimizerModulereturncs ddlm}t|}t|trt|dkr|d}n|}td|r-d| r6d| dd}fd d | D}|durdd D]}t ||d}|durct |||jd d qM||d<tdkrd}d|vr|ddur|d}||d<|S)zLlama4 Omni Data Steprr)tokensattention_maskmedianum_media_tiles) position_ids)labels loss_maskpacked_seq_paramsNcs2i|]\}}||vr|dur|jddndqS)NT non_blocking)cuda).0keyval required_keysZ/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/llama4/model/base.py @s z$llama4_data_step..) cu_seqlens_q cu_seqlens_kvcu_seqlens_q_paddedcu_seqlens_kv_paddedTrrnum_valid_tokens_in_ub) megatron.corernext isinstancetuplelensetupdateis_pipeline_first_stageis_pipeline_last_stagegetitemsgetattrsetattrrpsget_context_parallel_world_sizesum)dataloader_iterrbatch_batchrattrvaluer,r$r"r%llama4_data_steps@        rBc Csb|d|d|d|dd|dd|dd|dd|d d|d dd }|d i|S) zLlama4 Omni Forward SteprrrrNrrrimage_token_maskr) images input_idsrrrrnum_image_tilesrCrr$)r6)modelr> forward_argsr$r$r%llama4_forward_stepSs       rIc@s2eZdZUdZdZeeed<dZeeed<dZ eeed<dZ e ed<dZ e ed <d Ze ed <d Ze ed <d Ze ed<dZe ed<dZeeed<dZeeed<dZeeed<dZe ed<dZe ed<dZe ed<dZe ed<ejZejed<eZe ed<e!Z"e ed<ddZ#d#dee dd fd!d"Z$dS)$Llama4OmniConfigzLlama4 Model Base ConfigNlanguage_transformer_configvision_transformer_configvision_projection_configTdrop_vision_class_tokenr$encoder_pipeline_model_parallel_sizer+"encoder_tensor_model_parallel_size num_layersnum_attention_headsi seq_lengthlanguage_model_from_pretrainedvision_model_from_pretrained!vision_projection_from_pretrainedFfreeze_language_modelfreeze_vision_modelfreeze_vision_projectionbf16 params_dtypeforward_step_fn data_step_fncCsD|jdur tD] }t||t|j|qt|jdt|ddSdS)Nuse_te_rng_tracker)rKr r9r8)selfr@r$r$r% __post_init__s zLlama4OmniConfig.__post_init__vp_stagerrc Cs d|j_|j|j_|j|j_|j|j_|j|j_|j|j_|j|j_|j|j_|j |j_ |j dkrX|j dks:Jd|j |j_|j |j_|j |j_ |j dkrX|j |j_|j |j_|j |j_ |j |j_ |ped}t||tjd|dtjd|dtjd|dtjd|dpt|j k|j|d}|S)NFrr+z&ViT can only live on 1 pipeline stage.ignore_virtualrb)config tokenizer pre_process post_process add_encoder add_decoderrNrb)rK#scatter_embedding_sequence_paralleltensor_model_parallel_sizesequence_parallelrLrMpipeline_model_parallel_sizecontext_parallel_sizeexpert_tensor_parallel_sizeexpert_model_parallel_sizerOrP moe_pad_expert_input_to_capacitymoe_expert_capacity_factorLlama4OmniBaseModelr:r4r5 get_pipeline_model_parallel_rankrN)r`rfrbrGr$r$r%configure_modelsB                     z Llama4OmniConfig.configure_model)N)%__name__ __module__ __qualname____doc__rKrr __annotations__rLrMrNboolrOintrPrQrSrTrUstrrVrWrXrYrZr[torchbfloat16r\dtyperIr]rrBr^rarvr$r$r$r%rJds.             rJc@s|eZdZdZ       ddejdejdeejdeejdeejdeejd eed eed ee d ejfd dZ dS)rtzHllama4 base model combining vision and text models with cross-attention.NrErrrrDrinference_paramsruntime_gather_outputrrc  Ksj|duod|jv} |duot|dk} | rd} n@|jr4| s4t|j}tjg|j|j d ddd} n$|jrU| rU||} | | } |durT| j d| j d|jd<n|j } |js]| Sd}|jr|}d||dk<|jj||d}|dd}| r|j }| d| d}||jdkd}||j }|d|d}|d  d}|}||dkrtd |d |dd |dd|d}|||}||}n|}| dur| jd ks|dd}n|}|||}}}|j dks|j!r|"|||| \}}}} |jdd|||||| d}t#j$d|j%ds(|S|dur/|S||fS)a)Forward function of the Llama4 model. Args: input_ids (torch.Tensor): Input text token IDs of shape [batch, text_seq_len]. position_ids (torch.Tensor): Positional IDs for the input text tokens of shape [batch, text_seq_len]. loss_mask (Optional[torch.Tensor]): Mask indicating which tokens should contribute to the loss, of shape [batch, text_seq_len]. attention_mask (Optional[torch.Tensor]): Attention mask for the model of shape [batch, 1, combined_seq_len, combined_seq_len]. images (Optional[torch.Tensor]): Input images represented as a list of image tile tensors per sample. Each tile tensor is of shape [C, H, W]. labels (Optional[torch.Tensor]): Target labels for language modeling, of shape [batch, combined_seq_len]. inference_params (Optional[InferenceParams]): Parameters for inference, such as KV cache. runtime_gather_output (Optional[bool]): Whether to gather outputs during runtime. If None, falls back to the `parallel_output` setting from the constructor. packed_seq_params (Optional[PackedSeqParams]): Parameters for handling packed sequences, including padding information (used for SP/CP). Returns: output (torch.Tensor): Loss of shape [b, s] if labels are provided, otherwise logits of shape [b, s, vocab_size]. loss_mask (torch.Tensor): Loss mask expanded to combined sequence length. Shape [b, s]. Nimage_tokens_countr)rdevicer+)rErz <|patch|>).rzMismatch: final_mask wants z& embeddings, but image_embeddings has z embeddings.thd)rErr decoder_inputrrrrFrc)&key_value_memory_dictr1rir. vision_model parametersrtensorrrreshapevision_projectionshapeencoder_hidden_staterjrgclonelanguage_model embedding transpose contiguousviewsizerf token_to_id unsqueezetor< ValueErrorexpandmasked_scatter qkv_formatcontext_parallel_lmsequence_parallel_lm!_process_embedding_token_parallelr:r5rb)r`rErrrrDrrrrkwargsuse_inference_kv_cache has_imagesimage_embeddings vision_paramlanguage_embeddingsinput_ids_textoriginal_inputs_embeds_shapeimage_embeddings_flattenedspecial_image_mask final_maskcombined_embeddings final_mask_1dnum_tokens_to_fill expanded_mask final_labelsfinal_loss_maskfinal_attention_maskoutputr$r$r%forwards'              zLlama4OmniBaseModel.forward)NNNNNNN) rwrxryrzrTensorrrr|r rr$r$r$r%rts<   rtc sTeZdZdZ   d dedeededdeeej gej fffdd Z Z S) Llama4OmniModelz&Lightning Module for the Llama4 model.Nreoptimrfr model_transformcstj||||ddS)N)rerrfr)super__init__)r`rerrfr __class__r$r%rUs  zLlama4OmniModel.__init__)NNN) rwrxryrzrJrrrr Moduler __classcell__r$r$rr%rRsr)" dataclassesrtypingrrrrtorch.distributedr-rr:megatron.core.inference_paramsrmegatron.core.packed_seq_paramsr ,megatron.core.transformer.transformer_configr r 1nemo.collections.common.tokenizers.tokenizer_specr $nemo.collections.vlm.neva.model.baser rrrnemo.lightning.pytorch.optimrr~rrBrIrJrtrr$r$r$r%s&        4R