o i @sZddlZddlmZddlmZmZddlZddlmZddl m Z ddl m Z de ded fd d Zd&de d ede fddZ d'de dedededede f ddZd&de deddedefddZde ded dedde fddZd&de ded deddede f dd Z d&de ded deddede f d!d"Zd(de d#eefd$d%ZdS))N)replace)LiteralOptional)NormalizerConfig)FSTS) TokenParsertextreturn)enzhjacCs>d}|D]}d|krdkrnqd}q|s|rdSdS)z Get the language of the text. Args: text: The text to get the language of. Returns: The language of the text. Fu一u鿿Tr r )isdigit)rcontains_chinesechr@/home/ubuntu/.local/lib/python3.10/site-packages/wetext/utils.pyget_langs  rFtraditional_to_simplecCs|r tdd|}|S)z Preprocess the text before normalization. Args: text: The text to preprocess. traditional_to_simple: Whether to convert traditional Chinese to simplified Chinese. Returns: The preprocessed text. preprocessrrstrip)rrrrrr*s r full_to_halfremove_interjections remove_punctstag_oovcCsX|r tdd|}|rtdd|}|rtdd|}|r(tdd|}|S)a Postprocess the text after normalization. Args: text: The text to postprocess. full_to_half: Whether to convert full-width characters to half-width. remove_interjections: Whether to remove interjections. remove_puncts: Whether to remove punctuations. tag_oov: Whether to tag out-of-vocabulary words. Returns: The postprocessed text. postprocessrrrrr)rrrrrrrrr9sroperator)tnitn remove_erhuacCs@|dkrttd|rdS|rtd|rdSdSt|dkS)a Check if the text should be normalized. Args: text: The text to check. operator: The operator to use. remove_erhua: Whether to remove erhua for TN. Returns: True if the text should be normalized, False otherwise. rz\dTu儿|兒Fr)boolresearchlen)rrrrrrshould_normalizeWs  r$langcCst|||S)z Reorder the text. Args: text: The text to reorder. lang: The language of the text. Returns: The reordered text. )rreorder)rr%rrrrr&ks r& enable_0_to_9cCs@t||d}|r|dkr|dkrt|dd}||S)z Tag the text. Args: text: The text to tag. lang: The language of the text. operator: The operator to use. enable_0_to_9: Whether to enable 0-to-9 conversion for ITN. Returns: The tagged text. taggerr rtagger_enable_0_to_9r)rr%rr'r(rrrtagxs  r*cCs@t||d}|r|dkr|dkrtddd}||S)z Verbalize the text. Args: text: The text to verbalize. lang: The language of the text. operator: The operator to use. remove_erhua: Whether to remove erhua for TN. Returns: The verbalized text. verbalizerr rverbalizer_remove_erhuar)rr%rrr+rrr verbalizes r-configcKst|ptfi|}|jrd|vrt|}t||j}|j}t||j |j rT|dkr0t |}|dkr;|j dkr;d}t |||j |j }t|||j }t|||j |j }t||j|j|j|j}|S)z Normalize the text. Args: text: The text to normalize. config: Optional normalization config object. Returns: The normalized text. 'autor rr )rrfix_contractions contractionsfixrrr%r$rrrr*r'r&r-rrrrr)rr.kwargsr%rrr normalizes   r5)F)FFFF)N)r! dataclassesrtypingrrr2 wetext.configrwetext.constantsrwetext.token_parserrstrrr rrr$r&r*r-r5rrrrsT     "(