o ei6 @sdZddlZddlmZddlmZmZddlm Z m Z m Z ddl m Z mZddlmZGd d d e d d Zd eededeeefddZdeeeedeeedededejf ddZdedededefddZeGddde ZdgZdS)zProcessor class for Mllama.N) BatchFeature) ImageInputmake_nested_list_of_images)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringc@seZdZdddiiZdS)MllamaProcessorKwargs image_kwargsmax_image_tilesN)__name__ __module__ __qualname__ _defaultsrrj/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/mllama/processing_mllama.pyr s r F)total input_idsimage_token_idreturncsfddt|D}t|dkrgSt|dkr |ddggSddt|dd|ddD}||dt|g|dd}|dddD]}|d|ddkrZ||d<|d}qJ|S)a Generate a cross-attention token mask for image tokens in the input sequence. This function identifies the positions of image tokens in the input sequence and creates a mask that defines which subsequent tokens each image token should attend to. Args: input_ids (list[int]): A list of token ids representing the input sequence. image_token_id (int): The id of the token used to represent images in the sequence. Returns: list[list[int]]: A list of [start, end] pairs, where each pair represents the range of tokens an image token should attend to. Notes: - If no image tokens are present, an empty list is returned. - For a single image token, it attends to all subsequent tokens until the end of the sequence. - For multiple image tokens, each attends to tokens up to the next image token or the end of the sequence. - Consecutive image tokens are treated as a group and attend to all subsequent tokens together. csg|] \}}|kr|qSrr).0itokenrrr 8z2get_cross_attention_token_mask..rcSsg|]\}}||gqSrr)rloc1loc2rrrrAN) enumeratelenzipappend)rrimage_token_locations vision_masks last_mask_end vision_maskrrrget_cross_attention_token_mask"s  $  r-cross_attention_token_mask num_tiles max_num_tileslengthc Cst|}tdd|D}tj||||ftjd}tt||D]5\}\}} tt|| D]'\} \} } t| dkrT| \} }t||}|dkrH|}d||| || d| f<q-q |S)a Convert the cross attention mask indices to a cross attention mask 4D array. This function takes a sparse representation of cross attention masks and converts it to a dense 4D numpy array. The sparse representation is a nested list structure that defines attention ranges for each image in each batch item. Args: cross_attention_token_mask (list[list[list[int]]]): A nested list structure where: - The outer list represents the batch dimension. - The middle list represents different images within each batch item. - The inner list contains pairs of integers [start, end] representing token ranges for each image. num_tiles (list[list[int]]): A nested list structure specifying the number of tiles for each image in each batch item. max_num_tiles (int): The maximum possible number of tiles. length (int): The total sequence length of the input. Returns: np.ndarray: A 4D numpy array of shape (batch_size, length, max_num_images, max_num_tiles) The array contains `1` where attention is allowed and `0` where it is not. Note: - Special handling is done for cases where the end token is -1, which is interpreted as attending to the end of the sequence. cs|]}t|VqdSNr&)rmasksrrr pz?convert_sparse_cross_attention_mask_to_dense..)shapedtyper!r N)r&maxnpzerosint64r%r'min)r.r/r0r1 batch_sizemax_num_imagescross_attention_mask sample_idx sample_maskssample_num_tilesmask_idx locationsmask_num_tilesstartendrrr,convert_sparse_cross_attention_mask_to_denseRs"   rKprompt bos_token image_tokencCsP||vr|Sd}||r|t|d}|d7}||s ||||S)a\ Builds a string from the input prompt by adding `bos_token` if not already present. Args: prompt (`str`): The input prompt string. bos_token (`str`): The beginning of sentence token to be added. image_token (`str`): The image token used to identify the start of an image sequence. Returns: str: The modified prompt string with the `bos_token` added if necessary. Examples: >>> build_string_from_input("Hello world", "", "<|image|>") 'Hello world' >>> build_string_from_input("<|image|>Hello world", "", "<|image|>") '<|image|>Hello world' >>> build_string_from_input("Hello world", "", "<|image|>") 'Hello world' rNr ) startswithr&)rLrMrNnum_image_tokens_on_startrrrbuild_string_from_inputs  rQc szeZdZdfdd Ze  ddedBdeeBeeBeeBdBde e de fdd Z dd d Z eddZZS)MllamaProcessorNcsbt|dsd|_||j|_n|j|_|j|_d|_||j|_|j|_tj|||ddS)NrNz <|image|>z<|python_tag|>) chat_template) hasattrrNconvert_tokens_to_idsr python_tokenpython_token_idrMsuper__init__)selfimage_processor tokenizerrS __class__rrrYs zMllamaProcessor.__init__imagestextkwargsrc sb|dur |dur tdjtfdjji|}|ddd}i}|durzt|tr0|g}nt|tt fr@t dd|DsDtdfd d |D}fd d |D}j|fi|d}j ||d gd fdd |dD} | |dg} |durj |}t|}dd |D} |durtdd|Drt dd|Dstdt|dkr| |ks| | kr|durtdd} t| t|kr| |krd} n| | krd} td|d| d| |durj |fi|d} | d} | | |dur+|dur+fdd |dD}t|| j jtdd|dDd}||d <t||d!S)"a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. TODO: add aspect_ratio_ids and aspect_ratio_mask and cross_attention_mask Nz'You must specify either text or images.tokenizer_init_kwargs text_kwargsreturn_tensorscss|]}t|tVqdSr3) isinstancestrrtrrrr6sz+MllamaProcessor.__call__..zAInvalid input text. Please provide a string, or a list of stringscg|]}|jqSr)countrNrgrZrrrr$z,MllamaProcessor.__call__..csg|] }t|jjqSr)rQrMrN)r text_itemrkrrrrimage) modalitiescrir)rjrr token_idsrkrrrr$rrcSsg|]}t|qSrr4)rsamplerrrrscss|]}|dkVqdS)rNr)r batch_imgrrrr6r7zaIf a batch of text is provided, there should be either no images or at least one image per samplez@No image were provided, but there are image tokens in the promptzZMake sure to pass your images as a nested list, where each sub-list holds images per batchzhIf you activated truncation with `max_length`, increase the `max_length` so image tokens aren't cropped.z)The number of image tokens in each text (zA) should be the same as the number of provided images per batch (z). images_kwargsr/csg|]}t|jqSr)r-rrorkrrrs csr2r3r4)rrrrrr6 r7)r/r0r1rB)data tensor_type) ValueError _merge_kwargsr r\ init_kwargspoprerflisttupleall_check_special_mm_tokensupdater[ fetch_imagesranysumrKrr;r)rZr_r`ra output_kwargsrdrun_images_in_textencodingn_images_in_idsn_images_in_images add_messageimage_featuresr/r.rBrrkr__call__s         zMllamaProcessor.__call__TFcKs|jj|f||d|S)a Post-process the output of the model to decode the text. Args: generated_outputs (`torch.Tensor` or `np.ndarray`): The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)` or `(sequence_length,)`. skip_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to clean up the tokenization spaces. Argument passed to the tokenizer's `batch_decode` method. **kwargs: Additional arguments to be passed to the tokenizer's `batch_decode method`. Returns: `list[str]`: The decoded text. )skip_special_tokensclean_up_tokenization_spaces)r\ batch_decode)rZgenerated_outputsrrrarrrpost_process_image_text_to_textsz/MllamaProcessor.post_process_image_text_to_textcCs0|jj}|jj}dd|D}t||dgS)NcSsg|]}|dkr|qS)r/r)rnamerrrr3r$z5MllamaProcessor.model_input_names..rB)r\model_input_namesr[r{)rZtokenizer_input_namesimage_processor_input_namesrrrr,sz!MllamaProcessor.model_input_namesr3)NN)TF)rrrrYr rr r r{rr rrrpropertyr __classcell__rrr]rrRs$  [ rR)__doc__numpyr<feature_extraction_utilsr image_utilsrrprocessing_utilsrrrtokenization_utils_baser r utilsr r r{intr-ndarrayrKrfrQrR__all__rrrrs2  "0  0%