o }oiN @s,ddlZddlmZmZddlmZmZmZddlZddl m m Z ddl mZddlmZmZmZmZmZmZmZmZddlmZddlmZmZddlmZdd lm Z m!Z!m"Z"m#Z#dd l$m%Z%eGd d d eZ&eGd ddeZ'eGdddZ(Gdddeeeeeefe&e'e)fZ*dS)N) dataclassfield)ListOptionalUnion)PackedSeqParams)BatchCaptioningSampleDefaultTaskEncoderInterleavedSampleSampleSimilarityInterleavedSample VQASample generic_batch) stateless)AutoImageProcessor AutoProcessor) AutoTokenizer)_find_pattern_indicesconvert_to_packedgreedy_knapsackpredict_seq_len)loggingc@seZdZUdZedddZejed<edddZ ejed<edddZ ejed <ed ddZ ejed <ed ddZ ejed <dZ eeed<edddZeed<dS) DataSamplea>DataSample for multimodal data. This class represents a single data sample in a multimodal dataset, containing both image and text data along with their associated labels and masks. Attributes: images (torch.Tensor): Input images with shape (N, C, H, W), where N is typically 1 for a single sample, C is channels, H is height, and W is width. tokens (torch.Tensor): Input token IDs for text data. labels (torch.Tensor): Target labels for the tokens. loss_mask (torch.Tensor): Mask indicating which tokens should contribute to the loss. position_ids (torch.Tensor): Position embeddings for the tokens. packed_seq_params (PackedSeqParams, optional): Parameters for packed sequence processing. seqlen (int): Length of the sequence before padding. cC tdSNrtorchemptyrrZ/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/data/task_encoder.py: zDataSample.default_factoryimagescCtjdtjdSNrdtyperrlongrrrr r!;tokenscCr&r'r*rrrr r!<r,labelscCr&r'rrfloatrrrr r!=r, loss_maskcCr&r'r/rrrr r!>r, position_idsNpacked_seq_paramscCsdSrrrrrr r!@sseqlen)__name__ __module__ __qualname____doc__rr%rTensor__annotations__r-r.r1r2r3rrr4intrrrr r(s rc@seZdZUdZedddZejed<edddZ ejed<edddZ ejed <ed ddZ ejed <ed ddZ ejed <dZ eeed<dS) DataBatcha7DataBatch for multimodal data. This class represents a batch of data samples in a multimodal dataset. It maintains the same structure as DataSample but with an additional batch dimension. Attributes: images (torch.Tensor): Batched input images with shape (B, N, C, H, W), where B is batch size, N is typically 1, C is channels, H is height, and W is width. tokens (torch.Tensor): Batched input token IDs with shape (B, L), where L is sequence length. labels (torch.Tensor): Batched target labels with shape (B, L). loss_mask (torch.Tensor): Batched loss masks with shape (B, L). position_ids (torch.Tensor): Batched position embeddings with shape (B, L). packed_seq_params (PackedSeqParams, optional): Parameters for packed sequence processing. cCrrrrrrr r!Tr"zDataBatch.r#r%cCr&r'r*rrrr r!Ur,r-cCr&r'r*rrrr r!Vr,r.cCr&r'r/rrrr r!Wr,r1cCr&r'r/rrrr r!Xr,r2Nr3)r5r6r7r8rr%rr9r:r-r.r1r2r3rrrrrr r<Cs r<c@seZdZUdZdZeeed<dZee ed<dZ ee ed<dZ eed<dZ eed <d Zeed <d Zeeed <edddZeeed<dZeeed<dZeed<dZeed<dZeed<dZeed<dZeeed<ddZdS)TaskEncoderConfigaVConfiguration for multimodal processing. This class consolidates all configuration needed for multimodal processing, including model paths, tokenization, image processing, and sequence packing parameters. Args: hf_path (str, optional): HuggingFace model path used to load tokenizer and image processor. tokenizer (AutoTokenizer, optional): Pre-initialized tokenizer instance. image_processor (AutoImageProcessor, optional): Pre-initialized image processor instance. Note: Either hf_path or both tokenizer and image_processor must be provided. Nhf_path tokenizerimage_processorimage_token_stri8image_token_idiignore_place_holderz stop_stringcCsddgS)Nuser assistantrrrrr r!xszTaskEncoderConfig.r#roles chat_templateTimage_following_textFpacked_sequencei@num_image_embeddings_per_tilepacked_sequence_size@pad_to_multiple_ofcCs|js |jr |js td|jr`t|j|_|js)t|j|_t d|j|jsr?r@ ValueErrorrfrom_pretrained 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 startswithhasattrsetattrrErKrP)selfattrrrr __post_init__s0  zTaskEncoderConfig.__post_init__)r5r6r7r8r>rstrr:r?rr@rrBrCr;rDrErrHrrIrJboolrKrLrNrPr_rrrr r=\s"         r=c@seZdZdZdefddZedeee fde fddZ d e e de fd d Zd e defd dZddZeddZdede fddZdS) TaskEncoderaTaskEncoder for multimodal data processing. This class handles the processing of different types of multimodal samples, including Visual Question Answering (VQA), Captioning, and Interleaved samples. It provides functionality for encoding individual samples, batching them together, and handling packed sequences for efficient processing. The encoder supports: - VQA samples: Processing image-question pairs with corresponding answers - [In progress] Interleaved samples: Processing alternating image and text content - [In progress] Similarity interleaved samples: Processing image-text pairs for similarity tasks - [In progress] Packed sequences: Efficient processing of multiple samples in a single sequence Args: config (TaskEncoderConfig): Configuration object containing processing parameters Note: When using packed sequences, the micro batch size must be 1, and the global batch size and sequence length must be adjusted accordingly. configcCs*||_|jj|_|jj|_d|ji|_dS)zInitialize the multimodal processor. Args: config (TaskEncoderConfig): Configuration for processing rN)rcrUr?encode_vqa_sampleencoders)r]rcrrr __init__s    zTaskEncoder.__init__samplereturncCs6t|j}|j|}|std|||d}|S)zProcess a sample based on its type. Args: sample: Input sample to process. Can be a dictionary or an object. Returns: dict: Processed sample in a standardized format z'No encoder implemented for sample type ) input_sample)typer5regetrS)r]rg sample_typeencoderencoded_samplerrr encode_samples  zTaskEncoder.encode_samplesamplesc Cs|jjr-t|dkr(tdt|dt|dt|dt|d|jjd |d}nt|}t}|jD] }t||t ||q7|S) z* Batch a list of samples. z^Micro batch size should be 1 when training with packed sequence, but your micro batch size is z. The following config is equivalent to your current setting for a packed dataset. Please update your config to the following: Set micro batch size to 1 (currently z1) Set global batch size to `global_batch_size // z;` Set packed sequence length to `original_sample_seq_len * z ` (currently zr) For details please visit https://docs.nvidia.com/nemo-framework/user-guide/latest/sft_peft/packed_sequence.htmlr) rcrKlenrSrNrr<__dict__r\rY)r]rp batch_samplebatchr^rrr rus*   zTaskEncoder.batch batch_datacCs|j}d|vr|d|d<|d=|d\}}tj|tjd}|d|d}||d<d|vr5d |d<td d |d DrDd |d <|S) a Encode a batched set of samples for model input. This method transforms the raw batched data into a format ready for model input, including generating position IDs and other necessary fields. Parameters: batch_data (DataBatch): The raw batch of data to be encoded. Returns: dict: A dictionary containing the encoded batch data, ready for model input. r%mediar-r(rrqr2attention_maskNcss|]}|duVqdS)Nr).0paramrrr %sz+TaskEncoder.encode_batch..r3)rssizeraranger+ unsqueezerepeatall)r]rv batch_dictmicro_batch_size seq_lengthr2rrr encode_batch s  zTaskEncoder.encode_batchc spjjfdd|D}t||j}tt|t|}tdjjdt|dt|d||S)zSelects which samples will be packed together. NOTE: Energon dataloader calls this method internally if packing is used. Please see https://nvidia.github.io/Megatron-Energon/packing.html cs g|] }t|jjjdqS))media_token_indexrL)rr-rcrLryrgmedia_token_idr]rr 1sz6TaskEncoder.select_samples_to_pack..z&[Seq Packing Info] - Packing seq len: z, Buffered samples: z, Total number of bins: z, Average samples per bin: )rcrCrrNroundrrrrV)r]rplengthspacked_samplesavg_samples_per_binrrr select_samples_to_pack)s   z"TaskEncoder.select_samples_to_packc Cs|tdd|D}tdd|Ddd|Ddd|Dd\}}}}}tddd|Dd d |d j||||||d S) ay Function to pack a list of DataSample into a single DataBatch. NOTE: Energon dataloader calls this method internally if packing is used. Please see https://nvidia.github.io/Megatron-Energon/packing.html Args: samples: List of DataSample instances to pack into one sample. Returns: DataBatch instance. cSg|]}|jqSr)r%rrrr rQz5TaskEncoder.pack_selected_samples..cSrr)r-rrrr rSrcSrr)r.rrrr rTrcSrr)r4rrrr rUr)r-r.seqlens,cSrr)__key__)rysrrr rYrrNr) r__restore_key__ __subflavor____subflavors__r-r.r%r2r1r3)rstackrr<joinr)r]rp packed_images packed_tokens packed_labelspacked_position_idspacked_loss_maskr3rrr pack_selected_samplesBs$   z!TaskEncoder.pack_selected_samplesric Csg}|jjr|d|jjdt|jtr|jn|jg}t|jtr&|jn|jg}tt|t|}t |D]&}|| d|jj }||jj d|d||jj d||dq7|jj j|ddd}dt|jjd } t| |} g} | D]"} | |jjkr| |jjq}t| dkr| |jj | dd jq}tj| tjd } t| |jj}t|jtr|jn|jg}d}|D]>}||jjpd }|j j |dd d }tj|| jd}t| ||\}}|dkr| |||||<|}qtd|| ||| dd } |dd }t!| |jj"|jj}|jj#rO||jj#d|jj#|jj#}||}|dkrOt$%| d|fdd} t$%|d|fd|jj}tj|tj&d }d|||jjk<|jj'j(|j)ddddd}|*d*d}t+|j,|j-|j.|j/|| |||d S)aEncode a VQA sample into a DataSample format. Args: input_sample (VQASample): Input VQA sample containing image, context and answers Returns: DataSample: Encoded sample with processed image, tokens, labels and loss mask system)rolecontentrArrqF)tokenizeadd_generation_prompt())add_special_tokensr(rR input_ids)devicezUnable to find answer segment in the tokenized conversation. Skipping labeling for this and subsequent answers. Details: - Processed Text: %s - Tokens: %s - Target Answer Tokens: %s - Search Start Index: %dNrMconstantgpt)return_tensors do_rescale pixel_values) rrrrr%r-r.r1r4)0rc system_promptappend isinstancecontextlistanswersminrrrangereplace image_tokenrHr?apply_chat_templatereescaperBsplitrCextendrrtensorr+ ones_likerDrErrrwarning contiguousrrLrPFpadr0r@ preprocessimager~rrrrr)r]rimessagescontextsr min_lengthicontext_with_placeholderconversation_prompt regex_patternchunkstokenized_chunkschunkr-r.search_start_indexansweranswer_with_stop answer_tokensanswer_tokens_tensor answer_start answer_endr4 seqlen_paddedpad_lenr1rprocessed_imagerrr rdes        zTaskEncoder.encode_vqa_sampleN)r5r6r7r8r=rfrrdictobjectrrorr<rurrrrrdrrrr rbs "rb)+r dataclassesrrtypingrrrrtorch.nn.functionalnn functionalrmegatron.core.packed_seq_paramsrmegatron.energonrr r r r r rr"megatron.energon.task_encoder.baser transformersrr"nemo.collections.common.tokenizersrnemo.collections.vlm.data.utilsrrrr nemo.utilsrrr<r=rrbrrrr s6 (    M