o ei,@sddlZddlmZddlmZddlmZmZm Z m Z m Z ddl m Z mZddlmZmZddlmZer?d d lmZmZGd d d e d dZGddded dZeeddGddde ZdgZdS)N) BatchFeature) ImageInput)MultiModalDataProcessingKwargsProcessorMixin TextKwargsUnpack)PreTokenizedInput TextInput)auto_docstringis_vision_available)requires)Emu3ImageProcessorKwargs smart_resizec@seZdZUdZeed<dS)Emu3TextKwargsa return_for_image_generation (`bool`, *optional*, defaults to `False`): Whether the processed text is intended for image generation tasks. When `True`, the processor prepares inputs for image generation by appending image start tokens and size information to the prompt, and images should not be provided. When `False`, the processor prepares inputs for text generation from images and text, requiring both inputs to be provided. return_for_image_generationN)__name__ __module__ __qualname____doc__bool__annotations__rrf/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/emu3/processing_emu3.pyrs  rF)totalc@s4eZdZUeed<eed<dddddddZd S) Emu3ProcessorKwargs text_kwargs images_kwargsF)rreturn_mm_token_type_idsz1:1i)ratio image_area)rrN)rrrrrr _defaultsrrrrr+s  r)vision)backendsc seZdZ dfdd Ze  ddedBdeeBeeBeeBdBde e de fdd Z dd d Z d d ZdefddZ dddZZS) Emu3ProcessorNc sT|j|_|j|_|j|_|j|_|j|_|j|_|j |_ d|_ t j |||ddS)N) chat_template) image_tokenimage_token_id boi_tokenimage_start_token eoi_tokenimage_end_tokenimage_wrapper_tokenfake_token_around_image eof_token bos_tokendownsample_ratiosuper__init__)selfimage_processor tokenizerr(kwargs __class__rrr5=szEmu3Processor.__init__imagestextr9returnc st|tr |g}nt|tst|dtstdjtfdjji|}|ddd}|ddd }|dd d }|rH|d urHt d |sV|d urV|d urVt d i}j } j j } |s|d urӈj |fi|d}t|j} g} |D]J} j| vrt| }|\}}|j}|j}||d }| |d|jd|| }| j|d } j| } j| vs| | q~fdd| D}n.|r||j\}}| |d|jfdd|D}||ggt||d<|ddd }|ddd}j|fi|ddd i}j||dgd|rGt|d}t|d}d ||jk<||d<ti|||dS)aA Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. rzAInvalid input text. Please provide a string, or a list of stringstokenizer_init_kwargsrrFrr!Nr"zGYou should not provide `images` when `return_for_image_generation=True`zOYou must provide either text or images when `return_for_image_generation=False`r* csg|] }|djqS)rA)replacer).0sample)r6rr sz*Emu3Processor.__call__..csg|] }j|qSr)r2rC image_promptr6rrrFs image_sizesreturn_tensorsr image) modalities input_idsmm_token_type_ids)data tensor_type) isinstancestrlist TypeError _merge_kwargsrr8 init_kwargspop ValueErrorr,r1r.r7iterrIr)nextr3r0rBr2appendcalculate_generate_sizelen_check_special_mm_tokensnparray zeros_liker*tolistr)r6r<r=r9 output_kwargsrr!r"image_featuresimage_start_tokensimage_end_tokensrIprompt_stringsrE image_sizeheightwidthimage_seq_lengthimage_placeholderrJr text_inputs array_idsrNrrGr__call__Nsj        $  zEmu3Processor.__call__c Ksi}|durCg}|D])\}}t|||jj|jj|jj\}}||j}||j}||d}||q dgt|}|||dt di|S)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. Nr)num_image_tokensnum_image_patchesr) rr7spatial_factor min_pixels max_pixelsr3r[r]updater) r6rIr9 vision_datarprirjrkrqrrr_get_num_multimodal_tokenss$      z(Emu3Processor._get_num_multimodal_tokensc CsXtt|d\}}||}||d}tt|||}tt|||} || fS)N:g?)mapintsplitround) r6r!r"rrrjri current_area target_ratio token_height token_widthrrrr\s  z%Emu3Processor.calculate_generate_sizecKs|jj|fi|SN)r7 postprocess)r6r<r9rrrrszEmu3Processor.postprocessTcKs\|dus|dkr|j|fd|i|S|dkr"|j|dd}|dSt|jjd|d ) a  Post-process the output of a multimodal model to return the requested modality output. If the model cannot generated the requested modality, an error will be raised. Args: generated_outputs (`torch.Tensor` or `np.ndarray`): The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)` or `(sequence_length,)`. skip_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method. generation_mode (`str`, *optional*): Generation mode indicated which modality to output and can be one of `["text", "image", "audio"]`. **kwargs: Additional arguments to be passed to the tokenizer's `batch_decode method`. Returns: `list[Union[str, PIL.Image.Image]]`: The decoded text or generated image. Nr=skip_special_tokensrKzPIL.Image.Image)rJ pixel_valuesz# got an unexpected generation_mode=z.. Supported options are only `text` and `image)post_process_image_text_to_textrrXr;r)r6generated_outputsrgeneration_moder9r<rrrpost_process_multimodal_outputsz,Emu3Processor.post_process_multimodal_outputr)NN)TN)rrrr5r rr r rSr rrrorwr\rr __classcell__rrr:rr&:s( S" r&)numpyr_image_processing_utilsr image_utilsrprocessing_utilsrrrrr tokenization_utils_baser r utilsr r utils.import_utilsrimage_processing_emu3rrrrr&__all__rrrrs      9