o wiv@sdZddlZddlmZmZddlmZGdddZGdddeZGd d d eZ eee d Z dd eee de defddZ dS)z A collection of simple character based parsers. These parser handle cleaning and tokenization by default. We currently support English. N)ListOptional)cleanersc @seZdZdZdddddddeedededed ed ef d d Zd ede eefddZ d ede efddZ d edeefddZ ddZ dS) CharParserzFunctor for parsing raw strings into list of int tokens. Examples: >>> parser = CharParser(['a', 'b', 'c']) >>> parser('abc') [0, 1, 2] T)unk_idblank_id do_normalize do_lowercase do_tokenizelabelsrrr r r cCsP||_||_||_||_||_||_ddt|D|_tdd|D|_ dS)a,Creates simple mapping char parser. Args: labels: List of labels to allocate indexes for. Essentially, this is a id to str mapping. unk_id: Index to choose for OOV words (default: -1). blank_id: Index to filter out from final list of tokens (default: -1). do_normalize: True if apply normalization step before tokenizing (default: True). do_lowercase: True if apply lowercasing at normalizing step (default: True). cSsi|]\}}||qSr ).0indexlabelr r p/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/common/parts/preprocessing/parsers.py Bsz'CharParser.__init__..cSsg|] }t|dkr|qS))len)rrr r r Csz'CharParser.__init__..N) _labels_unk_id _blank_id _do_normalize _do_lowercase _do_tokenize enumerate _labels_mapset_special_labels)selfr rrr r r r r r__init__#szCharParser.__init__textreturncCs4|jr||}|durdS|js|S||}|SN)r _normalizer _tokenize)r r" text_tokensr r r__call__Es  zCharParser.__call__cCs|}|jr |}|Sr$)striprlowerr r"r r rr%QszCharParser._normalizecsg}t|dD]1\}}|dkr|jdj|jvr*|j|q |D] }|j|jq,q fdd|D}|S)N rcsg|] }|jkr|qSr )r)rtokenr r rrhsz(CharParser._tokenize..)rsplitappendrgetrr)r r"tokensword_idwordcharr r.rr&Ys zCharParser._tokenizecCsi}|jD]\}}|||<qd|t|j<d|t|jd<d|t|jd<g}|D]}||vr4q-|||q-d|S)Nzzrz

)ritemsrr0itemjoin)r str_inputr_mapkvoutir r rdecodels  zCharParser.decodeN)__name__ __module__ __qualname____doc__rstrintboolr!rr(r%r&rAr r r rrs0  "  rcsLeZdZdZddddZdfdd Zd d Zd ed eefddZ Z S) ENCharParserz,Incorporates english-specific parsing logic.plusandpercent)+&%NTcs0tj|i|d|_|r||_||_dS)aCreates english-specific mapping char parser. This class overrides normalizing implementation. Args: *args: Positional args to pass to `CharParser` constructor. **kwargs: Key-value args to pass to `CharParser` constructor. N)superr!_table_ENCharParser__make_trans_tableabbreviation_version)r rS make_tableargskwargs __class__r rr!s    zENCharParser.__init__cCNtj}|jD]}||d}q|jD]}||d}qt|dt|}|SNr7r,string punctuationPUNCTUATION_TO_REPLACEreplacerrF maketransrr r]r5rtabler r r__make_trans_table  zENCharParser.__make_trans_tabler"r#cCs4ztj||j|j|jd}W|StyYdSw)N)r\rbpunctuation_to_replacerS)r clean_textrQr^rS Exceptionr+r r rr%s zENCharParser._normalize)NT) rBrCrDrEr^r!rRrFrr% __classcell__r r rWrrI~s   rIcsHeZdZdZdddZfddZddZd ed eefd d Z Z S) RUCharParserz,Incorporates russian-specific parsing logic.uплюсuе)rMuёcs tj|i|||_dS)aCreates cyrillic-specific mapping char parser. This class overrides normalizing implementation. Args: *args: Positional args to pass to `CharParser` constructor. **kwargs: Key-value args to pass to `CharParser` constructor. N)rPr!_RUCharParser__make_trans_tablerQ)r rUrVrWr rr!szRUCharParser.__init__cCrYrZr[rar r rrcrdzRUCharParser.__make_trans_tabler"r#cCs0z tj||j|jd}W|StyYdSw)N)r\rbre)rrfrQr^rgr+r r rr%s  zRUCharParser._normalize) rBrCrDrEr^r!rjrFrr%rhr r rWrris     ri)baseenrurkr namer#cKs@|tvrtd|durttj}t|}|dd|i|}|S)a{Creates parser from labels, set of arguments and concise parser name. Args: labels: List of labels to allocate indexes for. If set to None then labels would be ascii table list. Essentially, this is an id to str mapping (default: None). name: Concise name of parser to create (default: 'base'). (default: -1). **kwargs: Other set of kwargs to pass to parser constructor. Returns: Instance of `CharParser`. Raises: ValueError: For invalid parser name. Examples: >>> type(make_parser(['a', 'b', 'c'], 'en')) ENCharParser zInvalid parser name.Nr r )NAME_TO_PARSER ValueErrorlistr\ printable)r rnrV parser_typeparserr r r make_parsers ru)Nrk) rEr\typingrr+nemo.collections.common.parts.preprocessingrrrIrirorFrur r r rs d2 *$