o 'N iã@s€dZddlZddlZddlZddlmZe dejdd¡Ze dejdd¡Z e d¡Z d d „Z d d „Z ddd„Z dS)z© Tokenizer for Indian languages. Currently, simple punctuation-based tokenizers are supported (see `trivial_tokenize`). Major Indian language punctuations are handled. éN)ÚIndicNlpExceptionz([zB\u0964\u0965\uAAF1\uAAF0\uABEB\uABEC\uABED\uABEE\uABEF\u1C7E\u1C7Fz])z6\u0609\u060A\u060C\u061E\u066A\u066B\u066C\u066D\u06D4z([0-9]+ [,.:/] )+[0-9]+cCs¢t d| dd¡¡}t dd|¡ d¡}d}d}t |¡D]$}| ¡}| ¡}||krA||||…}||||… dd¡}|}q|||d…}|}|  d¡S)a™tokenize string for Indian language scripts using Brahmi-derived scripts A trivial tokenizer which just tokenizes on the punctuation boundaries. This also includes punctuations for the Indian language scripts (the purna virama and the deergha virama). This is a language independent tokenizer Args: text (str): text to tokenize Returns: list: list of tokens ú \1 ú ú ú[ ]+ÚrN) Útriv_tokenizer_indic_patÚsubÚreplaceÚreÚstripÚ pat_num_seqÚfinditerÚstartÚendÚsplit)ÚtextÚtok_strÚsÚnew_sÚprevÚmrr©rúT/home/ubuntu/.local/lib/python3.10/site-packages/indicnlp/tokenize/indic_tokenize.pyÚtrivial_tokenize_indics€ rcCs.t d| dd¡¡}t dd|¡ d¡ d¡S)aƒtokenize Urdu string A trivial tokenizer which just tokenizes on the punctuation boundaries. This also includes punctuations for the Urdu script. These punctuations characters were identified from the Unicode database for Arabic script by looking for punctuation symbols. Args: text (str): text to tokenize Returns: list: list of tokens rrrr)Útriv_tokenizer_urdu_patr r r r r)rrrrrÚtrivial_tokenize_urduAsrÚhicCs|dkrt|ƒSt|ƒS)a°trivial tokenizer for Indian languages using Brahmi for Arabic scripts A trivial tokenizer which just tokenizes on the punctuation boundaries. Major punctuations specific to Indian langauges are handled. These punctuations characters were identified from the Unicode database. Args: text (str): text to tokenize lang (str): ISO 639-2 language code Returns: list: list of tokens Úur)rr)rÚlangrrrÚtrivial_tokenizeTsr )r)Ú__doc__Ústringr ÚsysÚindicnlp.commonrÚcompileÚ punctuationrrr rrr rrrrÚs   $