o SiX @sddlmZddlmZmZddlmZgdZdefddZ d e de fd d Z dd e d ee de fddZ d e de fddZ d e de fddZd e de fddZd e de d e de fddZd dee efdeedeedee effddZdS)!sub)UnionOptional)T2S_DICT)filter_unusual_charactershandle_blank_characteruppercase_to_lowercasetraditional_to_simplifiedfull_angle_to_half_anglehandle_substitutereturncCstS)u获取所有的预处理管道) ALL_PIPELINESrrE/home/ubuntu/.local/lib/python3.10/site-packages/proces/preprocess.pyget_all_pipelinessrtextcCs d}d}td||dd|S)u过滤所有非常见字符，保留中文、英文、常见标点、空白字符 Attributes: text: input text z \u4E00-\u9FA5u{!\"#$%&\'()*+,\-./:;<=>?@\\\[\]^_`{|}~¥·—‘’“”…、。〈〉《》「」『』【】！（），：；？｜～z[^\w\sz]+r)rchinese punctuationrrrrsrrreplcCs td||S)uw处理空白字符，默认替换成空字符 Attributes: text: input text repl: replace text z\s+r)rrrrrr !s r cCs|S)u>大写转小写 Attributes: text: input text )lowerrrrrr +sr cCsddd|DS)u`繁体转简体 Attributes: text: input text convert data from mediawiki. rcSs$g|]}|tvrt|n|qSr)rkeys).0trrr <s$z-traditional_to_simplified..)joinrrrrr 4sr cCsTd}|D]#}t|}|dkrd}nd|krdkr!nn|d8}|t|7}q|S)u>全角转半角 Attributes: text: input text ri0 ii^i)ordchr)rresultuchar inside_coderrrr ?sr ptncCs t|||S)ut替换一些字符 Attributes: text: input text ptn: re pattern repl: replace text r)rr%rrrrr Qs r Ndata pipelinesparamscCsgd}|dur |}t|tkr|g}n|}g}|D]7}|D]-}|tvrD|dur0t||}q||vrCt||g||R}qtd|d||qt|tkr\|dS|S)u文本预处理 Attributes: data: input data pipelines: default is ["handle_blank_character", "uppercase_to_lowercase", "traditional_to_simplified", "full_angle_to_half_angle"] params: function parameters )r r r r Nz pipeline: z not support!r)typestrrglobalsr ValueErrorappend)r&r'r(default_pipelines data_listresultsrfuncrrr preprocess\s(     r2)r)NN)rertypingrrconfrrlistrr*rr r r r r dictr2rrrrs