o oi$ @s<ddlZddlZddlmZddlmZddlZddlmZ ddZdZ dZ d Z d Z e ejeje d Zd ed eeefddZd ejfddZdejdedejfddZdeedefddZddZddZdeeded ed!efd"d#Zed$kreej eZ!ee!j"d%d&Z"ze"e!j#Z"Wn e$e!j#d'e!j%Z&eZ'ee"e&e'd(\Z(Z)e*d)e!j%d*e*d+e)d*e!j+duree&Z,ne!j+e!j-fgZ,e,D]AZ.ee.de.d,e(e!j/d-\Z+Z-e*d.e.dd/e.d,d0e+d/e-d1 e*d2e&e.de.d,d3e)e+e-d*qdSdS)4N)ArgumentParser)List)FarcCstd}|jddtdd|jddtdttgtd |jd d td dd |jddtdd|jddtdd|jddtdd|S)Nz map substring to output with FSTz--fstzFAR file containing FSTT)helptyperequiredz --grammarz tn or itnF)rrrchoicesdefaultz--rulez$rule name in FAR file containing FSTtokenize_and_classify)rrr rz--textz input stringzM2615 Forest Av, 90601 CA, Santa Clara. 10kg, 12/16/2018, $123.25. 1 Aug 2016.)rrr z--startz%start index of substring to be mappedz--endz#end index of substring to be mapped)r add_argumentstrITN_MODETN_MODEint parse_args)argsr`/home/ubuntu/.local/lib/python3.10/site-packages/nemo_text_processing/fst_alignment/alignment.pyrJs*rzu⎵itntnz$\:+-=textreturncCsg}dg}t|D]K\}}t|dkr|dkr||q |dkr6||t|dks.J||g}q |t|dkrT|d7}||t|dksOJ||q |S)zd Returns word segments from given text based on white space in form of list of index spans. r ) enumeratelenappend)rspanscur_spanidxchrrrget_word_segmentsis"     r"cCsDt}tddD] }|t||q |td|td|S)zg Creates and returns Pynini SymbolTable used to label alignment with ascii instead of integers "r )pynini SymbolTablerange add_symbolchrEPS WHITE_SPACE)tablenumrrrcreate_symbol_tables   r/fst input_text symbol_tablec st||}|jd}|}|}td|td|t t fdd|Dfdd|D}td|| | sOJd ttdd|D}||fS) z create alignment of input text based on shortest path in FST. Symbols used for alignment are from symbol_table Returns: output: list of tuples, each mapping input character to output )input_token_typeoutput_token_typezinput: zoutput: csg|]}|qSr)find.0xr2rr sz(get_string_alignment..z alignment: cSg|]}|dqS)rrr6rrrr:)r& shortestpathpathsilabelsolabelsloggingdebugistringostringlistzipnextdonejoinmapremove) r0r1r2latticer?r@rAoutput output_strrr9rget_string_alignments* rP alignmentindexcCsdd}d}||kr||dtkr|d7}|d7}||ks||dtkr0|d7}||dtks$|S)zj Given index in contracted input string computes corresponding index in alignment (which has EPS) rrr+)rQrR aligned_indexr rrr_get_aligned_indexsrUcCs<d}d}||kr||dtkr|d7}|d7}||ks|S)z` Given index in aligned output, returns corresponding index in contracted output string rrrS)rQrTog_indexr rrr_get_original_indexsrWcCs|tkrdS|tkr dS|S)Nr;r)r+r,)r8rrrsrXstartendmodec CsJt||}t||d}tdtttdd|||dDdtd|d|dd|ddkr||ddtkr||ddtvsW||ddtkr|d8}|ddkr||ddtkr||ddtvsW||ddtksW|dt|kr||ddtkr||ddtvs||ddtkr|d7}|dt|kr||ddtkr||ddtvs||ddtks|t kr|dt|kr||ddtvs||ddtkr|d7}|dt|kr||ddtvs||ddtkst ||d }t ||dd }||fS) a5 Given input start and end index of contracted substring return corresponding output start and end index Args: alignment: alignment generated by FST with shortestpath, is longer than original string since including eps transitions start: inclusive start position in input string end: exclusive end position in input string mode: grammar type for either tn or itn Returns: output_og_start_index: inclusive start position in output string output_og_end_index: exclusive end position in output string rz0: |cSr<)rrr6rrrr:r=z)indexed_map_to_output..|z1: |:r)rQrT) rUrBrCrFrKrLr+tn_itn_symbolsrrrW)rQrYrZr[ aligned_start aligned_endoutput_og_start_indexoutput_og_end_indexrrrindexed_map_to_outputs6 2 ( ((( *( rc__main__r)r[z1 not found. Please specify valid --rule argument.)r0r1r2z inp string: |r\z out string: |r)rYrZrQr[zinp indices: [r]z] out indices: []zin: |z| out: |)0rBstringargparsertypingrr&rrr+r,r rrF ascii_lettersdigitsr^r rr"r'r/FstrPtuplerUrWrLrc__name__ getLoggersetLevelINFOrr0rule ValueErrorrr1r-rQ output_textinforYindicesrZr8grammarrrrrsT   2  2  ,2