o }oi @sddlmZGdddZdS))Listc@sPeZdZdZdefddZdeedefddZd efd d Zd efd d Z dS)IndicProcessorzj Tokenizer, Detokenizer and Normalizer utilities in Indic Languages. Currently supports: 'hi' lang_idcCsH|dkrtddlm}m}m}||d|_||d|_||d|_dS)Nhir)MosesDetokenizerMosesPunctNormalizerMosesTokenizer)lang)NotImplementedError sacremosesrrrmoses_tokenizermoses_detokenizer normalizer)selfrrrrrg/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/tokenizers/indic_tokenizers.py__init__s   zIndicProcessor.__init__tokensreturncCs |j|S)z Detokenizes a list of tokens Args: tokens: list of strings as tokens Returns: detokenized string )r detokenize)rrrrrr!s zIndicProcessor.detokenizetextcC|SNrrrrrrtokenize+zIndicProcessor.tokenizecCrrrrrrr normalize.rzIndicProcessor.normalizeN) __name__ __module__ __qualname____doc__strrrrrrrrrrrs   rN)typingrrrrrrs