o ॵi@shddlmZmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z Gd d d e ZdS) )AnyDictN)Image) transforms) load_image)ModeKeys)OfaBasePreprocessorcseZdZdZejffdd Zdeee fdeee ffddZ deee fdeee ffdd Z deee fdeee ffd d Z Z S) &OfaVisualQuestionAnsweringPreprocessorz5 OFA preprocessor for question answer tasks. c sftt|j|||g|Ri|tddtj|j|jftjjdt tj |j |j dg|_ dS)zpreprocess the data Args: cfg(modelscope.utils.config.ConfigDict) : model config model_dir (str): model path, mode: preprocessor mode (model mode) cSs |dS)NRGB)convert)imagerj/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/preprocessors/ofa/visual_question_answering.py#s zAOfaVisualQuestionAnsweringPreprocessor.__init__..) interpolation)meanstdN)superr __init__rComposeResizepatch_image_sizeInterpolationModeBICUBICToTensor Normalizerrpatch_resize_transform)selfcfg model_dirmodeargskwargs __class__rrrs$   z/OfaVisualQuestionAnsweringPreprocessor.__init__datareturncCs |jtjkr ||S||S)N)r!rTRAIN_build_train_sample_build_infer_sample)rr&rrr__call__+s   z/OfaVisualQuestionAnsweringPreprocessor.__call__c Cs||}|jd|dddd}|jdkr,t|j|g}t|dd|jg}n<|jdkrGt|d |g}t|dd|jg}n!|jd krft|d dd |g}t|dd|jg}nt|j j |dt | d<||d <||d <|j durt t |t |jf}t |t |d}tt |t |dt |D]}|jg|||} |j | } d||| <q||d<|S)a& Building training samples. step 1. Preprocess the data using the logic of `_build_infer_sample` and make sure the label data in the result. step 2. Preprocessing the label data to generate `target` and `prev_output_token`. - add blank in the front out label data and tokenize it as `target` item. - if `prompt_type` is `None`, add the bos token as previous output tokens, add eos tokens as target items. - if `prompt_type` is `src`, concatenate source text input with target item as previous output tokens, remove the bos token and add eos token as target items. - if `prompt_type` is `prev_output`, just like the `prompt_type` is src, the difference is that it will remove the eos token in source text input in this setting. - padding the source item as final target item. step 3. Add constraint mask. Args: data (`Dict[str, Any]`): Input data, should contains the key of `image` `text` and `label`. Return: A dict object, contains source text input, patch images, patch masks with `Tensor([True])`, decoder prompt, label, target previous output tokens and constraint mask. z {}labelF)add_bosadd_eosnonerNsrcsource prev_outputprev_output_tokenstargetTconstraint_mask)r* tokenize_textformat prompt_typetorchcatbos_itemeos_itemNotImplementedError tokenizer pad_token_idlenconstraint_triezerostgt_dictboolrangebostolistget_next_layer) rr&sampletgt_itemprev_output_item target_itemr6 start_idxiconstraint_prefix_tokenconstraint_nodesrrrr)1sJ     z:OfaVisualQuestionAnsweringPreprocessor._build_train_samplecCs|||jd}||}||jd}|||j}|ds&|dn|}|d|}|jdkr9|j}n|jdkrA|}n|jdkrM|dd }nt ||t d g|d }d |jvro|jd |vro||jd |d <|S)a< Building inference samples. step 1. Preprocessing image input for model's image input. - get pillow image from data. - do some transforms to the pillow image, such as resize, normalize etc. step 2. Preprocessing the text input for model's text input. - add blank in the front of input text. - tokenize the result above as source text input. step 3. Calculating the decoder prompt. - if `prompt_type` is `None`, using bos token. - if `prompt_type` is `src`, using source text input - if `prompt_type` is `prev_output`, using source text input without eos token. step 4. Whether or not to add label data which refer to an answer to the question in this task. Args: data (`Dict[str, Any]`): Input data, should contains the key of `image` `text`. Return: A dict object, contains source text input, patch images, patch masks with `Tensor([True])`, decoder prompt and label. r text? r/r0r2Nr3T)r1 patch_image patch_maskdecoder_promptanswerr,) get_img_pil column_mapr pre_questionmax_src_lengthendswithr7r9r<r>r:tensor)rr&r rUrRinputsrWrJrrrr*os*     z:OfaVisualQuestionAnsweringPreprocessor._build_infer_sample)__name__ __module__ __qualname____doc__r INFERENCErrstrrr+r)r* __classcell__rrr$rr s""*>r )typingrrr:PILr torchvisionrmodelscope.preprocessors.imagermodelscope.utils.constantrbaser r rrrrs