o ॵi.@sddlZddlZddlZddlZddlmZddlZddlZddl Z ddl Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZmZmZdd lmZdd lmZGd d d ZdS)N)path)Image)File) OFATokenizerOFATokenizerZH) load_image)Trie)_get_kaldi_fbank_get_torchaudio_fbankconvert_waveform)OFA_TASK_KEY_MAPPING)set_torch_seedc@sreZdZdZddZdddZeddd Zed d Zd d Z ddZ ddZ  dddZ de jfddZdS)OfaBasePreprocessorz# OFA base preprocessor for cOs.||_||_|jjdd|_tj|rtj|}|jdkr&t |}n |jdvr1t |}nt | ddtdD| ddtdD|jjd d d kr\| d d g||_t|jg|_t|jg|_t|jg|_dd|D|_|_|jdd|_|jdd|_|jdd|_|jjdd|_|jjdd|_|jjdd}t j!"|t#||jjdd}|rgd|_$gd|_%n gd|_$gd|_%|jjdd|_&d dt'|jj(D|_)t*|jd!r|jj+j)d"ur|jj+j)D] \} } | |j)| <q t,-d#dt.j/D|_0d"|_1|jjd$d"rt23||jjj4} t5| d%d&d' } t67| } Wd"n 1sPwY| |_8d(d|j8D|_9t:|j|_1t;| <D]\}}|j=d)|ddd*}|j1>|jg|?|jgqod"|_@d"|_Ad"S)+zpreprocess the data via the vocab.txt from the `model_dir` path Args: cfg(modelscope.utils.config.ConfigDict) : model config model_dir (str): model path languageen)zhcncSg|]}d|qS)z format.0irU/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/preprocessors/ofa/base.py 1z0OfaBasePreprocessor.__init__..i cSr)zrrrrrr2rimultimodal_typedefaulttext2sqlz>=z<=cSi|]\}}||qSrr)rkeyvaluerrr 9sz0OfaBasePreprocessor.__init__..max_src_lengthmax_tgt_lengthmax_image_sizei prompt_typenoneseedimagenet_default_mean_and_stdF)g ףp= ?gv/?gCl?)gZd;O?gy&1?g?)?r.r.patch_image_sizeicSsi|]}||qSrrrr"rrrr$NsdatasetNcSsi|]}|dqSNrr0rrrr$Ws answer2labelrzutf-8)encodingcSr!rr)rkvrrrr$_r )add_bosadd_eos)Bcfgmodemodelgetrosrexistsabspathrfrom_pretrainedrNotImplementedError add_tokensrange tokenizertorch LongTensor bos_token_idbos_item pad_token_idpad_item eos_token_ideos_item get_vocabitemstgt_dictsrc_dictr%r'r(r)nprandomr+rmeanstdr/r task column_maphasattrr1str maketransstring punctuationtranstabconstraint_trieospjoinr3openjsonload ans2label label2ansr enumeratekeys tokenize_textinserttolisttrain_audio_feature_transformstest_audio_feature_transforms)selfr; model_dirr<argskwargsrFr+r-r6r7ans2label_filereaderans2label_dictranswer answer_itemrrr__init__s                   zOfaBasePreprocessor.__init__TcCsX|durdS|j||jdddddd}|r t|j|g}|r*t||jg}|S)a Using `OFATokenizer` to tokenize text input. Args: text (`str`): Input text. add_bos ('bool', **optional**, default to `True`) Whether or not to add beginning of sentence token in the front of sentence. add_eos ('bool', **optional**, default to `True`) Whether or not to add ending of sentence token in the end of sentence. Returns: A list of tokens with the max length of `max_src_length + 2` NFTpt) max_lengthadd_special_tokens truncationreturn_tensors input_idsr)rFr%squeezerGcatrJrN)rntextr9r:inputsrrrriks$z!OfaBasePreprocessor.tokenize_textNcCs||ddddddd}tdd|}|d}|d}|d}|d ur` with `person`. step 4. Remove the `\n` in the end of the sentence. step 5. Split the sentence with token ` `, If `max_words` is not None, make a length truncation. Args: caption (`str`): Input text. max_words (`int`, **optional**, default `None`): The max length of input text. If None, do nothing, else make a truncation. Returns: A sequence of `str`. ,.!?*#:;~-r8/zperson\s{2,} N lowerlstripreplaceresubrstripstripsplitlenra)caption max_words caption_wordsrrr pre_captions   zOfaBasePreprocessor.pre_captioncCsl|ddddd}tdd|}|d}|d}|d}t||kr4d |d|}|S)aM Preprocessing for text sentence. Note that this function is very similar to `pre_caption`, should be merged in the future version. step 1. Get the lower case of input text. step 2. Remove the words within `,.!?*#:;~ ` in the beginning of the sentence. step 3. Replace the words within `-/` or pattern `\s{2,}` with word ` `. step 4. Remove the `\n` in the end of the sentence. step 5. Split the sentence with token ` `, If `max_words` is not None, make a length truncation. Args: question (`str`): Input text. max_ques_words (`int`, **optional**, default `None`): The max length of input text. If None, do nothing, else make a truncation. Returns: A sequence of `str`. rrr8rrrNr)questionmax_ques_wordsquestion_wordsrrr pre_questions$    z OfaBasePreprocessor.pre_questionc Cs|d}||jjdd}|jrPtt|t|jf }t||}t |t|D]}|j ||| }|j |}d|||<q.||d<dSdS)z& Add constraint mask. targetr)dimTconstraint_maskN)nerLsumitemr_rGzerosrrQboolrErJrkget_next_layer) rnsample target_itm len_label_itmr start_idxrconstraint_prefix_tokenconstraint_nodesrrradd_constraint_masks&  z'OfaBasePreprocessor.add_constraint_maskcCs t|tjr |}|St|}|S)av Get the pillow image. If the input is not a pillow image ,it will load image from a local path or an external url. Args: path_or_url_or_pil (`Union[str, Image]`): Can be: - A path or url reference to an image - A pillow image. Returns: A pillow image. ) isinstancerr)rnpath_or_url_or_pilimagerrr get_img_pils  zOfaBasePreprocessor.get_img_pilcCsNt|tr t|}|St|trt|}t|}|Stdt|d)NzUnsupported input type: .) rbytesioBytesIOrZrread TypeErrortype)rn path_or_url audio_bytes file_bytesrrrget_audio_bytess    z#OfaBasePreprocessor.get_audio_bytes>Fc Cstj||dt|gdt|gg\}}t||ddd\}}|d}|}t||d}|dur6t||d}|dur>td|rK|j durK| |}n|rZ| |durZ| |}t | }||}|S)NspeedrateT)to_mononormalize_volumeiPzGPlease install pyKaldi or torchaudio to enable fbank feature extraction) torchaudio sox_effectsapply_effects_tensorrZr numpyr r ImportErrorrlrmrG from_numpyfloat pack_frames) rnwaveform sample_ratertarget_sample_rateis_train _waveform_fbankrrr prepare_fbanks6       z!OfaBasePreprocessor.prepare_fbankfeaturecCsB|jjdkr|S|jd|jj}|d|jj|}||dS)Nr r)r;n_frames_per_stepshapereshape)rnrn_packed_framesrrrr's  zOfaBasePreprocessor.pack_frames)TTr2)rF)__name__ __module__ __qualname____doc__rwri staticmethodrrrrrrrGTensorrrrrrrs N ( ) r)rr?rr\rr`rcrrSrGrPILrmodelscope.fileior!modelscope.models.multi_modal.ofarrmodelscope.preprocessors.imagermodelscope.utils.trierutils.audio_helperr r r utils.constantr utils.random_helprrrrrrs$