o ॵi$!@s|ddlmZmZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZddlm ZGd d d eZdS) )AnyDictN)Image) transforms) load_image)ModeKeys)OfaBasePreprocessorcseZdZdZejffdd Zdeee fdeee ffddZ deee fdeee ffdd Z deee fdeee ffd d Z Z S) OfaVisualGroundingPreprocessorz6 OFA preprocessor for visual grounding tasks. c stt|j|||g|Ri||jjdd|_|jtj kr>t t j |j g|j dt t j|j|j|jdg|_d St ddtj|j |j ftjjdttj|j|jdg|_d S) zpreprocess the data Args: cfg(modelscope.utils.config.ConfigDict) : model config model_dir (str): model path, mode: preprocessor mode (model mode) num_binsi)max_size)meanstdmax_image_sizecSs |dS)NRGB)convert)imagera/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/preprocessors/ofa/visual_grounding.py3s z9OfaVisualGroundingPreprocessor.__init__..) interpolation)r rN)superr __init__cfgmodelgetr moderTRAINTCompose RandomResizepatch_image_sizeToTensor Normalizer rrpositioning_transformrResizeInterpolationModeBICUBICpatch_resize_transform)selfr model_dirrargskwargs __class__rrrs>       z'OfaVisualGroundingPreprocessor.__init__datareturncCs |jtjkr ||S||S)N)rrr_build_train_sample_build_infer_sample)r)r/rrr__call__;s   z'OfaVisualGroundingPreprocessor.__call__c CsF|||jd}|j\}}gggt||gd}||jdd\}}}} tt|t|t|t| g} tt|t|t|t| gg|d<t dg|d<t|t|t| t|g} t| |d<| ||\} } | d d| d d }}d t | ddd|j d }d t | ddd |j d }d t | ddd |j d }d t | ddd |j d }d ||||}|||jd|j}|jjdd}| |}||}|j|dd}t|j|ddg}|| tdg||||||| d}|S)aI Building training samples. step 1. Preprocessing the image input for model's image input. - get the pillow image. - calculate the target boxes using for getting the exact area in the pillow image for input text by input `region_coord`. in training setting, `region_coord` will be a label data. - getting the target image as patch images and do some transforms such as resize, normalize etc. step 2. Preprocessing the text input for model's source text input. - do the str preprocessing to text input by function `pre_caption`. - build the instruction. the default instruction is ` which region does the text " {} " describe?`, `{}` refer to the text input. - tokenize the instruction as source text input. step 3. Preprocessing the patch image boxes for model's target text input. - quantize the coordinate of selected patch images - concatenate the quantization results by blank - tokenize the result above as target text input. step 4. Get the previous output tokens using target item without eos token. Args: data (`Dict[str, Any]`): Input data, should contains the key of `image` `text` and `region_coord`. Return: A dict object, contains source text input, patch images, patch masks with `Tensor([True])` value, target, previous output tokens, width scale ratio, height scale ratio and region coordinate. r)boxeslabelsareasize region_coord,r4rr5r6r7rzz {} {} {} {}textprompt, which region does the text " {} " describe?F)add_bosNT)source patch_image patch_masktargetprev_output_tokensw_resize_ratioh_resize_ratior8) get_img_pil column_mapr7torchtensorstripsplitfloatnparrayr$formatintr round pre_captionmax_src_lengthrrr tokenize_textcatbos_item)r)r/rwh boxes_targetx0y0x1y1regionr6rB patch_boxesresize_hresize_wquant_x0quant_y0quant_x1quant_y1r8 src_captionr=r<src_item target_itemprev_output_itemsamplerrrr1Asx   ""         z2OfaVisualGroundingPreprocessor._build_train_samplecCs|||jd}|j\}}||}t|j|}t|j|}|||jd|j}|j j dd} | |} | | } | |tdg||d} d|jvrz|jd|vrz||jdd\} }}}t| t|t|t|g| d <| S) a Building inference samples. step 1. Preprocessing image input for model's image input. - get pillow image from data. - do some transforms to the pillow image, such as resize, normalize etc. step 2. Preprocessing the text input for model's text input. - do the str preprocessing to text input by function `pre_caption`. - build the instruction. the default instruction is ` which region does the text " {} " describe?`, `{}` refer to the text input. - tokenize the instruction as source text input. step 3. Whether or not to add label data which refer to a region coordinate in this task. Args: data (`Dict[str, Any]`): Input data, should contains the key of `image` `text`. Return: A dict object, contains source text input, patch images, patch masks with `Tensor([True])` value, width scale ratio, height scale ratio and label. rr<r=r>T)rArBrCrFrGr8r9label)rHrIr7r(rJrKr!rTrUrrrrQrVrLrMrN)r)r/rrYrZrBrFrGrhr=r<rirlr\r]r^r_rrrr2s>       z2OfaVisualGroundingPreprocessor._build_infer_sample)__name__ __module__ __qualname____doc__r INFERENCErrstrrr3r1r2 __classcell__rrr-rr s"'"*Tr )typingrrnumpyrOrJPILr torchvisionrmodelscope.preprocessors.imagermodelscope.utils.constantrbaser utilsrr rrrrs