o wi @s2dZddlZddlZddlmZejfddZdS)z Some useful functions for converting and disambiguating between common alternative orthographies (ways of writing) the same text. N) sanscriptcCsN|tjkr|}tdd|}tdd|}tdd|}tdd|}tdd|}tdd |}td d|}td d |}td d|}tdd|}tdd|}tdd|}tdd|}tdd|}tdd|}tdd|}tdd|}tdd |}td!d"|}tj||tjd#}|Std$||tdd|S)%a Given some (devanAgarI?) sanskrit text, this function produces a "key" so that 1] The key should be the same for different observed orthographical forms of the same text. For example: :: - "dharmma" vs "dharma" - "rAmaM gacChati" vs "rAma~N gacChati" vs "rAma~N gacChati" - "kurvan eva" vs "kurvanneva" 2] The key should be different for different for different texts. - "stamba" vs "stambha" This function attempts to succeed at [1] and [2] almostall the time. Longer the text, probability of failing at [2] decreases, while probability of failing at [1] increases (albeit very slightly). Sources of orthographically divergent forms: - Phonetically sensible grammar rules - Neglect of sandhi while writing - Punctuation, spaces, avagraha-s. - Regional-language-influenced mistakes (La instead of la.) Some example applications of this function: - Create a database of quotes or words with minimal duplication. - Search a database of quotes or words while being robust to optional forms. Also see equivalent function in the scala indic-transliteration package. z\P{IsDevanagari}z\sz\p{P}u[०-९।॥॰ऽ]|[॑-॔]u[यरल]्ँuम्u[ङञणन]uमuँ|ंuॐu ओम्u[ळऴ]uलu([क-हक़-य़])्\1+z\1u[कग]्ख्uख्u[कग]्घ्uघ्u च्छ्uछ्u ज्झ्uझ्u त्थ्uथ्u द्ध्uध्u ड्ढ्uढ्u प्फ्uफ्u ब्भ्uभ्)_from_tozgot script {} for '{}') r DEVANAGARIregexsub transliterate OPTITRANSloggingwarningformat)textencoding_schemekeyrf/home/ubuntu/maya3_transcribe/venv/lib/python3.10/site-packages/indic_transliteration/deduplication.pyget_approx_deduplicating_key s2 %r)__doc__r rindic_transliterationrrrrrrrs