o }oi#@sddlZddlmZmZddlmZddlZddlmm Z ddl m Z ddl mZmZddl mZddl mZddlmZeGd d d eZeGd d d eZGd ddeZdS)N) dataclassfield)Optional) VQASample) DataBatch DataSample) TaskEncoder)TaskEncoderConfig)_find_pattern_indicesc@s2eZdZUdZdZeeed<dZeeed<dS)r zConfiguration for Gemma 3 processing. This class consolidates all configuration needed for Gemma 3 processing, including model paths, tokenization, image processing, and sequence packing parameters. stop_stringN system_prompt) __name__ __module__ __qualname____doc__r rstr__annotations__r rrc/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/gemma3vl/data/task_encoder.pyr s r c@sZeZdZUdZedddZejed<edddZ ejed<edddZ ejed <d S) Gemma3DataSamplezData Sample for Gemma3cCtjdtjdSNrdtypetorchemptylongrrrr.zGemma3DataSample.)default_factory input_idscCrrrrrrrr/r position_idscCs tdS)Nr)rrrrrrr0s pixel_valuesN) rrrrrr"rTensorrr#r$rrrrr*s rcsJeZdZdZdefddZdedeffdd Zd e de fd d Z Z S) raTaskEncoder for Gemma 3 data processing. This class handles the processing of different types of Gemma 3 samples, including Visual Question Answering (VQA), Captioning, and Interleaved samples. It provides functionality for encoding individual samples, batching them together, and handling packed sequences for efficient processing. The encoder supports: - VQA samples: Processing image-question pairs with corresponding answers - [In progress] Interleaved samples: Processing alternating image and text content - [In progress] Similarity interleaved samples: Processing image-text pairs for similarity tasks - [In progress] Packed sequences: Efficient processing of multiple samples in a single sequence Args: config (TaskEncoderConfig): Configuration object containing processing parameters Note: When using packed sequences, the micro batch size must be 1, and the global batch size and sequence length must be adjusted accordingly. configcCs*||_|jj|_|jj|_d|ji|_dS)z~Initialize the Gemma 3 processor. Args: config (TaskEncoderConfig): Configuration for processing rN)r& hf_processor tokenizerencode_vqa_sampleencoders)selfr&rrr__init__Is    zTaskEncoder.__init__ batch_datareturncs8t|}|djdg|djddR|d<|S)aEncode a batched set of samples for model input. This method transforms the raw batched data into a format ready for model input, including generating position IDs and other necessary fields. Parameters: batch_data (DataBatch): The raw batch of data to be encoded. Returns: dict: A dictionary containing the encoded batch data, ready for model input. mediaN)super encode_batchreshapeshape)r+r- __class__rrr3Xs (zTaskEncoder.encode_batch input_samplec Csg}|jjr|d|jjdt|jtr|jn|jg}t|jtr&|jn|jg}tt|t|}t |D]&}|| d|jj }||jj d|d||jj d||dq7|j j|dd}|j |j|dd did } | d d} | d } t| |jj} d} |D]>}||jjpd }|jj|ddd }tj|| jd}t| || \}}|dkr| ||| ||<|} qtd|| || | dd} | dd} t| }|jjr||jjd|jj|jj}||}|dkrt| d|fdd} t| d|fd|jj} tj | tj!d}d|| |jjk<| dur9| "dkr9| j#| jtj$d} | }nd}t%|j&|j'|j(|j)|| | |dS)aEncode a VQA sample into a DataSample format. Args: input_sample (VQASample): Input VQA sample containing image, context and answers Returns: DataSample: Encoded sample with processed image, tokens, labels and loss mask system)rolecontentzrF)tokenizept do_rescale)imagestextreturn_tensors images_kwargsr"r$r )add_special_tokens)devicezUnable to find answer segment in the tokenized conversation. Skipping labeling for this and subsequent answers. Details: - Processed Text: %s - Tokens: %s - Target Answer Tokens: %s - Search Start Index: %dNr0constantrg)rEr)__key____restore_key__ __subflavor____subflavors__r$r"labels loss_mask)*r&r append isinstancecontextlistanswersminlenrangereplace image_tokenrolesr'apply_chat_templateimagesqueezegetr full_likeignore_place_holderr r(tensorrEr loggingwarning contiguouspad_to_multiple_ofFpad ones_likefloatnumeltobfloat16rrGrHrIrJ)r+r8messagescontextsrQ min_lengthicontext_with_placeholderconverted_messagesoutputstokensr@rKsearch_start_indexansweranswer_with_stop answer_tokensanswer_tokens_tensor answer_start answer_endseqlen seqlen_paddedpad_lenrLprocessed_imagerrrr)hs      zTaskEncoder.encode_vqa_sample) rrrrr r,rdictr3rrr) __classcell__rrr6rr3s r)r_ dataclassesrrtypingrrtorch.nn.functionalnn functionalrcmegatron.energonr&nemo.collections.vlm.data.task_encoderrrrBaseTaskEncoderr BaseTaskEncoderConfignemo.collections.vlm.data.utilsr rrrrrs