o ei:@sdZddlZddlmZddlmZmZddlZddl m Z ddl m Z m Z mZddlmZmZmZmZdd lmZmZmZdd lmZmZerNdd lmZeeZd efd dZ ddZ!ddZ"ddZ#ddZ$GdddeddZ%eGdddeZ&dgZ'dS)z Processor class for Idefics3. N) accumulate) TYPE_CHECKINGUnion) BatchFeature) ImageInputis_valid_image load_image)MultiModalDataProcessingKwargsProcessorMixinUnpack) AddedToken BatchEncoding TextInput)auto_docstringlogging)PreTokenizedInputreturncCst|to |dS)Nhttp) isinstancestr startswith)valrn/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/idefics3/processing_idefics3.pyis_url%srcCst|pt|SN)rr)elemrrris_image_or_image_url)src Cs~d}t|D]$}t|D]}||d|dd|dd||7}q |d7}q|d|||||7}|S)zKPrompt with expanded image tokens for when the image is split into patches. )range) image_seq_len image_rows image_colsfake_token_around_image image_tokenglobal_img_tokentext_split_imagesn_hn_wrrr_prompt_split_image-s"  ( r0cCs|||||S)z5Prompt with expanded image tokens for a single image.r)r'r*r+r,rrr_prompt_single_image@sr1cCs2|dkr|dkrt||||dSt||||||S)Nr)r*r+r,)r1r0)r(r)r'r*r+r,rrrget_image_prompt_stringJs r2c@s$eZdZdddddddidZdS)Idefics3ProcessorKwargsTF)add_special_tokenspaddingis_split_into_wordsreturn_mm_token_type_idsreturn_row_col_info) text_kwargs images_kwargsN)__name__ __module__ __qualname__ _defaultsrrrrr3Ys r3F)totalc seZdZ ddededBffdd ZddZe   dd ee eBe e eBd e e d e e e d fdedBd e e d ef ddZdddZZS)Idefics3ProcessorNr' chat_templatec stddddj|_tddddj|_tddddj|_d|_||_|j|_|j|_ |j|_ fdd t d D|_ t d |_d |j|j|jgi}||j|_tj|fd |i|dS)a image_seq_len (`int`, *optional*, defaults to 169): The length of the image sequence i.e. the number of tokens per image in the input. This parameter is used to build the string from the input prompt and image tokens and should match the value the model used. It is computed as: image_seq_len = int(((image_size // patch_size) ** 2) / (scale_factor**2)) zFT) normalizedspecialzzz c s:g|]}tdD]}d|dd|ddqqS)r!r"r#r$)r&convert_tokens_to_ids).0ij tokenizerrr zs  z.Idefics3Processor.__init__..rEz*(\n?\n?|\n?)+additional_special_tokensrBN)rcontentfake_image_tokenr+end_of_utterance_tokenglobal_image_tagr'rFimage_token_idfake_image_token_idglobal_image_token_idr& row_col_idsrecompile%_regex_to_remove_extra_special_tokensr4super__init__)selfimage_processorrKr'rBkwargs tokens_to_add __class__rJrrZis*    zIdefics3Processor.__init__cCsTg}|D]#}g}|D]}t|r||q t|r!|t|q ||q|Sr)rappendrr )r[prompts prompt_imagespromptimagesrrrr_extract_images_from_promptss  z.Idefics3Processor._extract_images_from_promptsretextrr]rc' sn|dur dur tdjtfdjji|}|dur|nj}|ddd}|ddd}g}g} i} |dur^t|trE|g}nt|t sUt|dtsUtd fd d |D}durt rlggnhtt t frt dr|durt |t krtd jd t |djdt d dgt t|fdd tt |Dn gntt t fstdt t fst ddstddd D} dd Djfi|d} | | |dur| |krtd|d| d| ddd |D} | ddd |D} j}j}j}g}g}t|| | D]k\}}}g}g}t||D])\}}t||||||d}jd|d}|jd ||||q>||||}t |dkr}td!|d}t|D]\}}||||d7}q||q0j|fi|d}j||d"gd#| |n)|durt|rtd$t |djd%jd*d&|i|d}| ||r1t| d'}t|} t|D]8\}}!t||j kd}"d}#|!D]"}$|#t |"krn|"|#}%|%|$}&d| ||%|&f<t!|"|&}#qq| "| d(<t#| |d)S)+a  image_seq_len (`int`, *optional*): The length of the image sequence. If not provided, the default value of self.image_seq_len is used. image_seq_len should be equal to int(((image_size // patch_size) ** 2) / (scale_factor**2)) Nz+You must provide either `text` or `images`.tokenizer_init_kwargsr9r7Freturn_tensorsrzAInvalid input text. Please provide a string, or a list of stringscsg|]}|jqSr)countr+rGsample)r[rrrLsz.Idefics3Processor.__call__..zThe total number of zP tokens in the prompts should be the same as the number of images passed. Found  z tokens and z images.cs$g|]}||dqS)r"r)rGrH)cumsum_images_in_textrerrrLszdInvalid input images. Please provide a single image or a list of images or a list of list of images.cSsg|]}t|qSr)lenrkrrrrLscSsg|] }dd|DqS)cSs g|] }t|r t|n|qSr)rr )rGimrrrrLs z9Idefics3Processor.__call__...rrkrrrrLsr:z!The number of images in the text z and images z should be the same.rowscSg|]}dg|qSrrrGn_imagesrrrrLcolscSrrrsrrtrrrrLrv)r+r*r,r"rz.The image token should be present in the text.image) modalitieszFound z. tokens in the text but no images were passed.rg input_idsmm_token_type_ids)data tensor_typer)$ ValueError _merge_kwargsr3rK init_kwargsr'poprrlistrtuplesumror+rr&r\updaterOrQzipr2rasplit enumerate_check_special_mm_tokensanynparray zeros_likewhererS searchsortedtolistr)'r[rergr'r] output_kwargsr7rin_images_in_textn_images_in_imagesinputs image_inputsr(r)rOr+r,prompt_stringsbatch_image_seq_lengthsrl sample_rows sample_colsimage_prompt_stringsimage_seq_lengthsn_rowsn_colsimage_prompt_string row_length split_samplerH text_inputs array_idsr| seq_lengthsimage_start_positionsrIseq_lenstartendr)rnrer[r__call__s                     zIdefics3Processor.__call__c  si}|durNtjdi|fdd|D}jd}jd}g}g}|D]\} } } || d} ||| | || q,|||dtd i|S) a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. Nr:cs"g|] }jjg|RqSr)r\get_number_of_image_patches)rG image_sizer:r[rrrL=sz@Idefics3Processor._get_num_multimodal_tokens..rrxr")num_image_tokensnum_image_patchesr)r3r>getrr'rar ) r[ image_sizesr] vision_datanum_image_row_colsbase_image_length col_lengthrr num_patchesnum_rowsnum_colsrrrr_get_num_multimodal_tokens+s"       z,Idefics3Processor._get_num_multimodal_tokens)NrAN)NNNr)r;r<r=intrrZrfrrrrrr r3rrr __classcell__rrr_rr@gs2%  r@)(__doc__rV itertoolsrtypingrrnumpyrfeature_extraction_utilsr image_utilsrrr processing_utilsr r r r tokenization_utils_baserrrutilsrrr get_loggerr;loggerboolrrr0r1r2r3r@__all__rrrrs.       j