o ॵi@sDddlmZmZddlZddlmZddlmZGdddeZdS))AnyDictN)ModeKeys)OfaBasePreprocessorcseZdZdZejffdd Zdeee fdeee ffddZ dd Z deee fdeee ffd d Z deee fdeee ffd d Z ZS)!OfaTextClassificationPreprocessorz9 OFA preprocessor for text classification tasks. cs&tt|j|||g|Ri|dS)zpreprocess the data Args: cfg(modelscope.utils.config.ConfigDict) : model config model_dir (str): model path, mode: preprocessor mode (model mode) N)superr__init__)selfcfg model_dirmodeargskwargs __class__d/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/preprocessors/ofa/text_classification.pyr s  z*OfaTextClassificationPreprocessor.__init__datareturncCs |jtjkr ||S||S)N)r rTRAIN_build_train_sample_build_infer_sample)r rrrr__call__s   z*OfaTextClassificationPreprocessor.__call__cCsfd|dd|j}d|dd|j}d}|||}||}|S)aE Building text classification task's instruction. The `data` should contains key `text` and `text2`, and the final instruction is like ` can text1 " {} " imply text2 " {} "?`, the first `{}` refer to the value of `text` and the latter refer to `text2` step 1. Preprocess for input text `text` and `text2` in `data`. - Do lower, stripe and restrict the maximum length as `max_src_length`. step 2. Using instruction template to generate the final instruction. step 3. Tokenize the instruction as result.  textNtext2z% can text1 " {} " imply text2 " {} "?)joinlowerstripsplitmax_src_lengthformat tokenize_text)r rtext1rpromptrinstruction_itmrrr_build_instruction%s   z4OfaTextClassificationPreprocessor._build_instructioncCs||}d|vs Jd|d}|jr|j|}|jd|dd}|jdkr+|}n|jdkrr<rrrrfs    z5OfaTextClassificationPreprocessor._build_infer_sample)__name__ __module__ __qualname____doc__r INFERENCEr rstrrrr'rr __classcell__rrrrr s""*+r) typingrrr2modelscope.utils.constantrbaserrrrrrs