o }oi8 @sddlmZmZmZddlZddlmZmZmZm Z m Z m Z m Z ddl mZddlmZmZmZmZddlmZmZmZmZddlmZGdd d eee eee feeefZdS) )DictListUnionN)CaptioningSampleDefaultTaskEncoderInterleavedSampleSimilarityInterleavedSample VQASample batch_listbatch_pad_stack) stateless)ImageTextRawBatchImageTextSamplePackedImageTextRawBatchPackedImageTextSample)InterleavedSampleEncoder SampleEncoderSimilarityInterleavedEncoderVQASampleEncoder)loggingc@seZdZdZ    dddZded ed dfd d Zed e e e e e fd efddZdee eefd e eeffddZded efddZddZeddZdS)MultiModalTaskEncodera+ A task encoder that handles multiple modalities including VQA, captioning, interleaved samples, and similarity interleaved samples. This class extends the DefaultTaskEncoder and provides a flexible mechanism to handle and encode different types of multimodal data. Support for VQA, captioning and interleaved samples is provided by default. It supports registering custom encoders for each sample type and provides methods for encoding individual samples, batching them, and further processing the batch for model input. F@Nc Csd||_||_||_||_||_||_tjt||||dt jt ||||dt jt ||||di|_ dS)a Initialize the MultiModalTaskEncoder with specific encoders for different sample types. Parameters: tokenizer (Tokenizer): The tokenizer used for processing textual components across sample types. image_processor (ImageProcessor): The image processor responsible for preprocessing image data. multimodal_sample_config (MultiModalSampleConfig): Configuration object defining properties and requirements for multimodal samples. packed_sequence (bool, optional): Flag indicating whether packed sequences are used. Default is False. packed_sequence_size (int, optional): The size of packed sequences, used when `packed_sequence` is True. Default is -1. num_image_embeddings_per_tile (int, optional): Number of image embeddings per image tile. Determines the granularity of image features. Default is 576. ) tokenizerimage_processormultimodal_sample_configimage_tag_typeN)r sample_configpacked_sequencenum_image_embeddings_per_tilerpacked_sequence_sizer __name__rrrrrencoders)selfrrrrr rrr$i/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/multimodal/data/energon/task_encoder.py__init__?s2 zMultiModalTaskEncoder.__init__ sample_typeencoderreturncCs||j|<dS)a} Registers a custom encoder for a specific sample type. This method allows adding or overriding encoders for specific sample types. Parameters: sample_type (str): The name of the sample type for which the encoder is being registered. encoder (SampleEncoder): The custom encoder instance that will handle the specified sample type. N)r")r#r'r(r$r$r%register_encoderrs z&MultiModalTaskEncoder.register_encodersamplecCs<t|j}|j|}|std||j|td}|S)aX Encode an individual sample based on its type. This method selects the appropriate encoder based on the sample type and encodes the sample into a format suitable for further processing. Parameters: sample (Union[VQASample, InterleavedSample, SimilarityInterleavedSample, CaptioningSample]): The sample to be encoded. The sample type is used to determine the appropriate encoder. Returns: ImageTextSample: The encoded sample. Raises: NotImplementedError: If no encoder is registered for the sample type. z'No encoder implemented for sample type ) input_sample output_sample)typer!r"getNotImplementedErrorencoder)r#r+r'r(encoded_sampler$r$r% encode_sample~s  z#MultiModalTaskEncoder.encode_samplesamplesc Csf|jr>t|dkr&tdt|dt|dt|dt|d|jd |d}t|j|j|j|j|j |j |j |j d Sgggggf\}}}}}g}|D]+}| |j| |j| |j| |j| |j |j d ury||j qNt|dkrd }t|} t|} | jd kr| jd g| jd d R} t|} t|} t|} t| | | | | |dS)a Batch a list of encoded samples into a single raw batch. This method collates individual encoded samples into a batch, preparing them for model input. Parameters: samples (List[ImageTextSample]): A list of encoded samples to be batched. Returns: ImageTextRawBatch: The batched data, including images, tokens, labels, and loss masks. z^Micro batch size should be 1 when training with packed sequence, but your micro batch size is z. The following config is equivalent to your current setting for a packed dataset. Please update your config to the following: Set micro batch size to 1 (currently z1) Set global batch size to `global_batch_size // z;` Set packed sequence length to `original_sample_seq_len * z ` (currently zr) For details please visit https://docs.nvidia.com/nemo-framework/user-guide/latest/sft_peft/packed_sequence.htmlr)__keys__imagestokenslabels loss_mask position_idspacked_seq_paramsnum_image_tilesNr)r6r7r8r9r:r=)rlen ValueErrorr r__key__r7r8r9r:r;r<r=appendextendr r ndimreshapeshaper )r#r4r+keysr7r8r9r:batch_num_image_tiles batch_keys batch_imagesbatch_prompt_tokens batch_labelsbatch_loss_maskr$r$r%batchsl            zMultiModalTaskEncoder.batch batch_datacCs|j}d|vr|d|d<|d=d|vo|dduo|ddk}d|vo/|dduo/|ddk}|r8|r8Jd|rC|d|d<|d=|d\}}tj|tjd }|d|d }||d <d |vrhd|d <|S) a Encode a batched set of samples for model input. This method transforms the raw batched data into a format ready for model input, including generating position IDs and other necessary fields. Parameters: batch_data (ImageTextRawBatch): The raw batch of data to be encoded. Returns: dict: A dictionary containing the encoded batch data, ready for model input. r7mediar=Nrnum_media_tileszJnum_image_tiles and num_media_tiles should not be present at the same timer8)dtyper5r;attention_mask)__dict__sizetorcharangelong unsqueezerepeat)r#rP batch_dictis_num_image_tiles_presentis_num_media_tiles_presentmicro_batch_size seq_lengthr;r$r$r% encode_batchs<       z"MultiModalTaskEncoder.encode_batchc sddlm}mjjjfdd|D}|||j}tt|t|}t djdt|dt|d||S) zSelects which samples will be packed together. NOTE: Energon dataloader calls this method internally if packing is used. Please see https://nvidia.github.io/Megatron-Energon/packing.html r)greedy_knapsackpredict_seq_lencsg|] }|jjdqS))media_token_indexr)r8r.0r+media_token_idrcr#r$r% sz@MultiModalTaskEncoder.select_samples_to_pack..z&[Seq Packing Info] - Packing seq len: z, Buffered samples: z, Total number of bins: z, Average samples per bin: ) /nemo.collections.vlm.neva.data.sequence_packingrbrcr image_tokentoken_idr roundr@rinfo)r#r4rblengthspacked_samplesavg_samples_per_binr$rgr%select_samples_to_pack s"  z,MultiModalTaskEncoder.select_samples_to_packc Csddlm}tjdd|Ddd}|jjj}|dd|Ddd|D|j||jjd\}}}}} g} |D] } | j d urC| | j q6t | dkrLd } t d d d|Dd |||||| | d S)a Function to pack a list of ImageTaskSample into a single ImageTaskSamplePacked. NOTE: Energon dataloader calls this method internally if packing is used. Please see https://nvidia.github.io/Megatron-Energon/packing.html Args: samples: List of ImageTaskSample instances to pack into one sample. Returns: ImageTaskSamplePacked instance. r)convert_to_packedcSg|]}|jqSr$)r7rer$r$r%ri5z?MultiModalTaskEncoder.pack_selected_samples..)dimcSrtr$)r8rer$r$r%ri8rucSrtr$)r9rer$r$r%ri9ru)r8r9rrd ignore_indexN,cSrtr$)rB)rfsr$r$r%riFrur$) rB__restore_key__r8r9r7r;r:r<r=)rjrsrWcatrrkrlrignore_place_holderr=rDr@rjoin) r#r4rs packed_imagesrh packed_tokens packed_labelspacked_position_idspacked_loss_maskr<rIr+r$r$r%pack_selected_samples%s8       z+MultiModalTaskEncoder.pack_selected_samples)FrrN)r! __module__ __qualname____doc__r&strrr*r rr rrrrr3rrr rrOdictrarrrr$r$r$r%r,s. 3    F,r)typingrrrrWmegatron.energonrrrrr r r "megatron.energon.task_encoder.baser /nemo.collections.multimodal.data.energon.configr rrr7nemo.collections.multimodal.data.energon.sample_encoderrrrr nemo.utilsrrrr$r$r$r%s $