o ॵi@shddlmZmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z Gd d d e ZdS) )AnyDictN)Image) transforms) load_image)ModeKeys)OfaBasePreprocessorcseZdZdZejffdd Zdeee fdeee ffddZ deee fdeee ffdd Z deee fdeee ffd d Z Z S) OfaVisualEntailmentPreprocessorz7 OFA preprocessor for visual entailment tasks. c sftt|j|||g|Ri|tddtj|j|jftjjdt tj |j |j dg|_ dS)zpreprocess the data Args: cfg(modelscope.utils.config.ConfigDict) : model config model_dir (str): model path, mode: preprocessor mode (model mode) cSs |dS)NRGB)convert)imagerb/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/preprocessors/ofa/visual_entailment.py#s z:OfaVisualEntailmentPreprocessor.__init__..) interpolation)meanstdN)superr __init__rComposeResizepatch_image_sizeInterpolationModeBICUBICToTensor Normalizerrpatch_resize_transform)selfcfg model_dirmodeargskwargs __class__rrrs$   z(OfaVisualEntailmentPreprocessor.__init__datareturncCs |jtjkr ||S||S)N)r!rTRAIN_build_train_sample_build_infer_sample)rr&rrr__call__+s   z(OfaVisualEntailmentPreprocessor.__call__c Cs||}d|d}|ddi|d<|j|ddd}|jdkr6t|j|g}t|dd |jg}n<|jd krQt|d |g}t|dd |jg}n!|jd krpt|d d d |g}t|dd |jg}nt|j j |d t | d<||d<||d<|j d urt t |t |jf}t |t |d}tt |t |dt |D]} |jg||| } |j | } d|| | <q||d<|S)aO Building training samples. step 1. Preprocess the data using the logic of `_build_infer_sample` and make sure the label data in the result. step 2. Preprocess the label data to generate the `target` and `prev_output_tokens`. - tokenize the label data. - calculate the target item. 1) if `promp_type` is `None`, using tokenized label data. 2) if `promp_type` is `src`, concatenating the `source` data and tokenized label data. 3) if `promp_type` is `prev_output`, concatenating the `source` data without eos token and tokenized label data step 3. Add constraint mask Args: data (`Dict[str, Any]`): Input data, should contains the key of `text` `text2` and `label` are optional. Return: A dict object, contains source text input, patch images, patch masks with `Tensor([True])` value, decoder prompt, label, target, previous output tokens and constraint mask. z {}labelg?ref_dictF)add_bosadd_eosnonerNsrcsource prev_outputtargetprev_output_tokensTconstraint_mask)r*format tokenize_text prompt_typetorchcatbos_itemeos_itemNotImplementedError tokenizer pad_token_idlenconstraint_triezerostgt_dictboolrangebostolistget_next_layer) rr&sampler5tgt_itemprev_output_item target_itemr7 start_idxiconstraint_prefix_tokenconstraint_nodesrrrr)1sJ     z3OfaVisualEntailmentPreprocessor._build_train_samplec CsN|||jd}||}d|vr-|||jd|j}|jjdd}||}n3d|vs:Jd| |||jd|j}|||jd|j}|jjdd}|||}| |}|j dkrpg} |j } n|j d kr|d d } |d d } nt ||td g| | d } d|jvr|jd|vr||jd| d<| S)aC Building inference samples. step 1. Preprocessing the image as model's image input. - get the pillow image input from `data` - do some transforms to the pillow image, such as resize, normalize etc. step 2. Building the instruction as model's source text input. - use text input to build instruction. so far, we support two kind of input form, we will take different examples to both of them to explain how to use them. 1) only `text` input in data. this setting can solve the tasks which judge whether or not the input `text` describe the input image. 2) both `text` and `text2` input in data. this setting can solve the tasks which judge whether or not the `text` together with input image can imply the `text2` - tokenize the instruction above. step 3. Calculate the decoder prompt input. step 4. Whether or not to add label data. Args: data (`Dict[str, Any]`): Input data, should contains the key of `text` `text2` and `label` are optional. Return: A dict object, contains source text input, patch images, patch masks with `Tensor([True])` value, decoder prompt and label. r text2textpromptz does the image describe " {} "?ztext must be in the input z/ can image and text1 " {} " imply text2 " {} "?r0r3Nr4T)r2 patch_image patch_mask prefix_tokendecoder_promptrelationr,) get_img_pil column_mapr pre_captionmax_src_lengthrmodelgetr8keysr9r:r=r?r;tensor) rr&r rV hypothesisrUrTcaptioninputsrXrYrKrrrr*osT         z3OfaVisualEntailmentPreprocessor._build_infer_sample)__name__ __module__ __qualname____doc__r INFERENCErrstrrr+r)r* __classcell__rrr$rr s""*>r )typingrrr;PILr torchvisionrmodelscope.preprocessors.imagermodelscope.utils.constantrbaser r rrrrs