o }oiE@sddlmZGdddZdS))Listc@sPeZdZdZdefddZdeedefddZd efd d Zd efd d Z dS)MosesProcessorzB Tokenizer, Detokenizer and Normalizer utilities in Moses lang_idcCs<ddlm}m}m}||d|_||d|_||d|_dS)Nr)MosesDetokenizerMosesPunctNormalizerMosesTokenizer)lang) sacremosesrrrmoses_tokenizermoses_detokenizer normalizer)selfrrrrrg/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/tokenizers/moses_tokenizers.py__init__s  zMosesProcessor.__init__tokensreturncC |j|S)z Detokenizes a list of tokens Args: tokens: list of strings as tokens Returns: detokenized string )r detokenize)r rrrrrs zMosesProcessor.detokenizetextcCs|jj|dddS)z> Tokenizes text using Moses -> Sentencepiece. FT)escape return_str)r tokenizer rrrrr(szMosesProcessor.tokenizecCr)N)r normalizerrrrr.s zMosesProcessor.normalizeN) __name__ __module__ __qualname____doc__strrrrrrrrrrrs  rN)typingrrrrrrs