o }oiD@sddlZddlmZmZddlmZmZmZmZddl Z ddl Z ddl m Z ddlmZddlmZddlmZddlmZd eee jffd d Zd e jfd d ZddlmZmZeGdddeZGdddeZdgZdS)N) dataclassfield)CallableDictListOptionalparallel_state)InferenceParams)PackedSeqParams)pack_image_features)IMAGE_TOKEN_INDEXreturncs ddlm}t|}t|trt|dkr|d}n|}td|r-d| r6d| dd}fd d | D}|durdd D]}t ||d}|durct |||jd d qM||d<tdkrd}d|vr|ddur|d}||d<|S)a Processes a batch of data from the dataloader for the LLaVA Next model. Args: dataloader_iter (Iterator): An iterator that provides batches of data from the dataloader. Returns: Dict[str, torch.Tensor]: A dictionary containing the processed batch, ready for input into the model. Notes: - Filters and moves required keys to the appropriate device. - Slices the batch along the sequence dimension for context parallelism. rr)tokensattention_maskmedianum_media_tiles image_sizes) position_idsr)labels loss_maskrpacked_seq_paramsNcs2i|]\}}||vr|dur|jddndqS)NT non_blocking)cuda).0keyval required_keys^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/llava_next/model/base.py Gs z(llava_next_data_step..) cu_seqlens_q cu_seqlens_kvcu_seqlens_q_paddedcu_seqlens_kv_paddedTrrnum_valid_tokens_in_ub) megatron.corer next isinstancetuplelensetupdateis_pipeline_first_stageis_pipeline_last_stagegetitemsgetattrsetattrrpsget_context_parallel_world_sizesum)dataloader_iterr batch_batchrattrvaluer)r!rr"llava_next_data_steps<         r?c Csb|d|d|d|dd|dd|dd|dd|d d|d dd }|d i|S) a Performs the forward step for the LLaVA Next model. Args: model (torch.nn.Module): The LLaVA Next model instance. batch (Dict[str, torch.Tensor]): A dictionary containing input tensors for the forward step. Returns: torch.Tensor: The output from the model's forward computation. Notes: - Constructs the forward arguments based on the provided batch. - Includes optional parameters like packed sequence parameters if available. rrrrNrrrrr) r input_idsrrrrrrrr!)r3)modelr; forward_argsr!r!r"llava_next_forward_step[s       rC)MCoreNevaModel NevaConfigc@sBeZdZUdZeedZeed<ee dZ eed<d d dd Z dS) LlavaNextConfigzx Configuration class for the LLaVA Next model. Overrides NevaConfig and modifies forward and data step fn. )defaultforward_step_fn data_step_fnNrMCoreLlavaNextModelc Csd|j_|j|j_|j|j_|j|j_|j|j_|j|j_|j|j_|jdkrN|jdks0Jd|j|j_|j|j_|j|j_|j dkrN|j |j_|j |j_|pQd}t ||t j d|dpbt |jkt jd|dt j d|dt jd|dp{t |jk|j|d}|S)z Configures the LLaVA Next model with the appropriate settings. Args: tokenizer: Tokenizer instance to be used with the model. Returns: MCoreLlavaNextModel: An instance of the LLaVA Next model. Frr(z&ViT can only live on 1 pipeline stage.)ignore_virtualvp_stageconfig tokenizer pre_process post_process add_encoder add_decoderdrop_vision_class_tokenrL)language_transformer_config#scatter_embedding_sequence_paralleltensor_model_parallel_sizesequence_parallelvision_transformer_configvision_projection_configpipeline_model_parallel_sizecontext_parallel_size$encoder_pipeline_model_parallel_size"encoder_tensor_model_parallel_sizerJr7r1 get_pipeline_model_parallel_rankr2rT)selfrOrLrAr!r!r"configure_models<                 zLlavaNextConfig.configure_model)N)rrJ) __name__ __module__ __qualname____doc__rrCrHr__annotations__r?rIrar!r!r!r"rF|s rFcseZdZdZ       ddedededed ed ed eed dffd d Zdddddde ddf de j de j de j dee j dee j dee j dee j dee dee edeedeedeed e j fddZZS)rJz The LLaVA Next model class, extending MCoreNevaModel. Attributes: image_newline (torch.nn.Parameter): A learnable parameter for handling image newlines. NTFrNrPrQrRrSrTrLrc sNtj||||||||ddt|jj} tjt |jj| |_ dS)a Initializes the LLaVA Next model. Calls the super class init and initialize image_newline parameter Args: config (LlavaNextConfig): Model configuration instance. tokenizer: Optional tokenizer instance. pre_process (bool): Whether to enable preprocessing. post_process (bool): Whether to enable postprocessing. add_encoder (bool): Whether to add the encoder module. add_decoder (bool): Whether to add the decoder module. drop_vision_class_token (bool): Whether to drop the vision class token. vp_stage (Optional[int]): Virtual pipeline stage. rMr(N) super__init__mathsqrtrZ hidden_sizetorchnn Parameterrandn image_newline) r`rNrOrPrQrRrSrTrL embed_std __class__r!r"rhs zMCoreLlavaNextModel.__init__r@rrrrrrinference_paramsrmedia_token_indexruntime_gather_outputrc  Cs|duod|jv} |jddk}| rd}n|jr*|s*tjg|j|jdddd}nj|jr|r|jrH|j |_ |j |dd}|d|j j }n| t|j j}|j ||j j dd }|jrvt|j d d}|dd|dddf}|}||}|dur|jd|jd|jd <n|j}|js|Sd}|jr|}d||dk<|jj||d }|dd}| durtj|jdtj|jd} nt| trtj| tj|jd} tj|| dd }t!||d|j"d\}}|| k#$}|jd}||kr t%d|d||| k&d}|'| |j}| |j|j}|(||}|}|}|)ddd}|}|j*dksA|j+rY|j*dkrM|dd}|,|||| \}}}} |jddd|||| | d}|dusp|durr|S||fS)atForward function of the LLaVA Next model. Args: images (torch.Tensor): input image of shape [num_tiles, img_h, img_w]. num_tiles means the number of image tiles in this batch. input_ids (torch.Tensor): input text ids [batch, text_seq_len]. position_ids (torch.Tensor): input text position ids [batch, text_seq_len]. image_sizes (torch.Tensor): Raw image sizes before tiling (N,2). attention_mask (torch.Tensor): Attention mask for the language model [batch, text seq length]. labels (torch.Tensor): Optional target text labels [batch, combined_seq_len]. loss_mask (torch.Tensor): Text loss mask [batch, text_seq_len]. inference_params (InferenceParams): Inference-time parameters including KV cache. num_media_tiles (list of int): Number of tiles per image. Default None assumes 1 tile per image. image_token_index (int): ID for input images. packed_seq_params (PackedSeqParams): Dict with padded token information. Required for using SP/CP with padding mask type. Returns: output (torch.Tensor): Loss ([b, s]) if labels are provided; logits ([b, s, vocab_size]) otherwise. loss_mask (torch.Tensor): Loss mask expanded to combined sequence length. Shape [b, s]. Nimage_tokens_countr)dtypedeviceT)output_hidden_statesr()num_unused_layersclass_token_lenmedia_tokens_count)r@r)dimrG)vision_feature_select_strategyrpz6Image features and image tokens do not match: tokens: z , features )r@rr decoder_inputrrtrvr)-key_value_memory_dictshaperRrltensorrxryreshapevision_model_from_hf vision_modelevalrNvision_feature_layertor+ parameters_drop_vision_class_tokenr5 contiguousvision_projectionencoder_hidden_staterSrPclonelanguage_model embedding transposeonesintr,listsplittolistr rpr9item ValueError unsqueeze expand_asmasked_scatterpermutecontext_parallel_lmsequence_parallel_lm!_process_embedding_token_parallel)r`r@rrrrrrrtrrurvruse_inference_kv_cache has_imagesmedia_embeddingsr}language_embeddingsinput_ids_text feature_lensn_image_tokensn_image_featuresspecial_image_maskcombined_embeddings final_labelsfinal_loss_maskoutputr!r!r"forwards$                zMCoreLlavaNextModel.forward)NTTTTFN)rbrcrdrerFboolrrrhr rlTensorr rr r __classcell__r!r!rrr"rJs|   ,     rJ) ri dataclassesrrtypingrrrrrltorch.distributedr*r r7megatron.core.inference_paramsr megatron.core.packed_seq_paramsr +nemo.collections.vlm.llava_next.model.utilsr 0nemo.collections.vlm.neva.data.multimodal_tokensr strrr?rC$nemo.collections.vlm.neva.model.baserDrErFrJ__all__r!r!r!r"s&     >;I