o 'Ni}@sXdZddlZddlmZddlmZedZedZedZ dd Z d d d Z dS)z Sentence splitter for Indian languages. Contains a rule-based sentence splitter that can understand common non-breaking phrases in many Indian languages. N)unicode_transliterate)langinfoz[\?!\u0964\u0965]zI[\.\?!\u0964\u0965\uAAF1\uAAF0\uABEB\uABEC\uABED\uABEE\uABEF\u1C7E\u1C7F]z[\u0964\u0965]cCshd}tj||d|vS)zIs the text a non-breaking phrase Args: text (str): text to check for non-breaking phrase lang (str): ISO 639-2 language code Returns: boolean: true if `text` is a non-breaking phrase >}ऄअआइईउऊऋऌऍऎएऐऑऒओऔकखगघङचछजझञटठडढणतथदधनऩपफबभमयरऱलळऴवशषसहॠॡआइआईआरऎचऎनऎफऎमऎलऎसएचएनएफएमएलएसकुकॆकेचिजिजीजॆजेटिटीडिडीडॉपिपीबिबीयुयूविवीसिसीसौ आर् ऎच् ऎन् ऎफ् ऎम् ऎल् ऎस् एच् एन् एफ् एम् एल् एस् जेड वाय ज़ेड ऎक्स एक्स क्यु क्यू जेड् वाय् व्हि व्ही श्री ज़ेड्ऎक्स्एक्स्डब्ल्युडब्ल्यूhi)rUnicodeIndicTransliterator transliterate)textlang ack_charsrW/home/ubuntu/.local/lib/python3.10/site-packages/indicnlp/tokenize/sentence_tokenize.pyis_acronym_abbvr#s srautocCs|dkrddlm}||}|S|dkr(t|r&t|dur#t}nt}nt}g}d}|}| |D]2}| }| } |dkrL||d rLq5|d} ||| } t | dkrc|| |d}q5||d} t | dkr{|| |ds|Sg} d} d }t|D]h\}}|d }t |dkr|d dkrd }| d |} q|d dkrt|d dd |rt | dkr|s| | d }|} q|r| d |} t | dkr| | d} d }qt | dkr| | |} d }qt | dkr| | | S) a split the text into sentences A rule-based sentence splitter for Indian languages written in Brahmi-derived scripts. The text is split at sentence delimiter boundaries. The delimiters can be configured by passing appropriate parameters. The sentence splitter can identify non-breaking phrases like single letter, common abbreviations/honorofics for some Indian languages. Args: text (str): text to split into sentence lang (str): ISO 639-2 language code delim_pat (str): regular expression to identify sentence delimiter characters. If set to 'auto', the delimiter pattern is chosen automatically based on the language and text. Returns: list: list of sentences identified from the input text urr)sentence_tokenizerrN.F T)urduhack.tokenizationrris_danda_delimCONTAINS_DANDAsearchDELIM_PAT_NO_DANDADELIM_PAT_DANDAstripfinditerstartend isnumericlenappend enumeratesplitr)rr delim_patr sentencescand_sentencesbeginmop1p2rsfinal_sentences sen_buffer bad_stateisentencewordsrrrsentence_splitsl          "       r)r) __doc__reindicnlp.transliteraterindicnlprcompilerrrrrrrrrs