o qi - @sUdZddlmZddlmZddlmZmZidddgd d gd d d gdgddgddgddgddgddgddgddgdddd gd!d"d#gd$d%d&gd'd(d)gd*gd+d,gd-id.gd/d0gd1d2d3d4gd5gd6d7gd8d9gd:d;gd<d=gd>d?gd@dAgdBdCgdDdEgdFdGdHdIgdJdKdLgdMdNgdOgdPdQgdRidSdTdUgdVgdWdXdYgdZgd[d\gd]d^gd_d`gdadbgdcgdddedfdggdhgdidjgdkdlgdmdngdodpgdqdrgdsdtgdugdvgdwgdxgdygggggdzd{gd|gd} Zeed~<iZ e D]\Z Z e eeee e q.dedefddZdd djdZeeefed<dedefddZdedefddZdedefddZdkgdmgdogdqgdsgdgdgdgdgdgd Zeed<iZe D]\Z Z e eeee e qdgdgdgdgdgdgdgdgdgdgd Zeed<iZe D]\Z Z e eeee e qddededefddZdedefddZdS)z Character Normalization functions provides functionality to put proper spaces before and after numeric digits, urdu digits and punctuations. )Dict)_DIACRITICS_RE)_SPACE_AFTER_PUNCTUATIONS_RE$_REMOVE_SPACE_BEFORE_PUNCTUATIONS_REuآuﺁuﺂuأuﺃuاuﺍuﺎuب)uﺏuﺐuﺑuﺒuپ)uﭖuﭘuﭙuت)uﺕuﺖuﺗuﺘuٹ)uﭦuﭧuﭨuﭩuث)uﺛuﺜuﺚuج)uﺝuﺞuﺟuﺠuح)uﺡuﺣuﺤuﺢuخ)uﺧuﺨuﺦuدuﺩuﺪuذuﺬuﺫuرuﺭuﺮuزuﺯuﺰuس)uﺱuﺲuﺳuﺴuش)uﺵuﺶuﺷuﺸuص)uﺹuﺺuﺻuﺼuض)uﺽuﺾuﺿuﻀuطuﻃuﻄuظ)uﻅuﻇuﻈuع)uﻉuﻊuﻋuﻌuغ)uﻍuﻏuﻐuف)uﻑuﻒuﻓuﻔuق)uﻕuﻖuﻗuﻘuل)uﻝuﻞuﻟuﻠuم)uﻡuﻢuﻣuﻤuن)uﻥuﻦuﻧuﻨuچ)uﭺuﭻuﭼuﭽuڈuﮈuﮉuڑuﮍuﮌuژuﮋuک)uﮎuﮏuﮐuﮑuﻛuكuگ)uﮒuﮓuﮔuﮕuںuﮞuﮟuو)ﻮuﻭruؤuﺅuھ)uﮪuﮬuﮭuﻬuﻫuﮫuہ)uﻩuﮦuﻪuﮧuﮩuﮨuهuۂuۃuةuءuﺀuی) uﯼuىuﯽuﻰuﻱuﻲuﯾuﯿuيuئuﺋuﺌuے)uﮮuﮯuﻳuﻴuۓ۰u٠۱u١۲u٢۳u٣۴u٤u٥u٦u٧u٨u٩uﻻuﻼuـ) ۵۶۷۸۹u۔u؟u٫u،uلاCORRECT_URDU_CHARACTERStextreturncCs |tS)uQ The most important module in the UrduHack is the :py:mod:`~urduhack.normalization.character` module, defined in the module with the same name. You can use this module separately to normalize a piece of text to a proper specified Urdu range (0600-06FF). To get an understanding of how this module works, one needs to understand unicode. Every character has a unicode. You can search for any character unicode from any language you will find it. No two characters can have the same unicode. This module works with reference to the unicodes. Now as urdu language has its roots in Arabic, Parsian and Turkish. So we have to deal with all those characters and convert them to a normal urdu character. To get a bit more of what the above explanation means is.:: >>> all_fes = ['ﻑ', 'ﻒ', 'ﻓ', 'ﻔ', ] >>> urdu_fe = 'ف' All the characters in all_fes are same but they come from different languages and they all have different unicodes. Now as computers deal with numbers, same character appearing in more than one place in a different language will have a different unicode and that will create confusion which will create problems in understanding the context of the data. :py:mod:`~character` module will eliminate this problem by replacing all the characters in all_fes by urdu_fe. This provides the functionality to replace wrong arabic characters with correct urdu characters and fixed the combine|join characters issue. Replace ``urdu`` text characters with correct ``unicode`` characters. Args: text (str): ``Urdu`` text Returns: str: Returns a ``str`` object containing normalized text. Examples: >>> from urduhack.normalization import normalize_characters >>> # Text containing characters from Arabic Unicode block >>> text = "مجھ کو جو توڑا ﮔیا تھا" >>> normalized_text = normalize_characters(text) >>> # Normalized text - Arabic characters are now replaced with Urdu characters >>> normalized_text مجھ کو جو توڑا گیا تھا ) translate _TRANSLATORrrT/home/ubuntu/.local/lib/python3.10/site-packages/urduhack/normalization/character.pynormalize_charactersSs %r)uآuأuۓCOMBINE_URDU_CHARACTERScCs"tD] \}}|||}q|S)uJ To normalize combine characters with single character unicode text, use the :py:func:`~urduhack.normalization.character.normalize_combine_characters` function in the :py:mod:`~urduhack.normalization.character` module. Replace combine|join ``urdu`` characters with single unicode character Args: text (str): ``Urdu`` text Returns: str: Returns a ``str`` object containing normalized text. Examples: >>> from urduhack.normalization import normalize_combine_characters >>> # In the following string, Alif ('ا') and Hamza ('ٔ ') are separate characters >>> text = "جرأت" >>> normalized_text = normalize_combine_characters(text) >>> # Now Alif and Hamza are replaced by a Single Urdu Unicode Character! >>> normalized_text جرأت )ritemsreplace)r_key_valuerrrnormalize_combine_characterssr!cCstd|}td|}|S)u Add spaces after punctuations used in ``urdu`` writing Args: text (str): ``Urdu`` text Returns: str: Returns a ``str`` object containing normalized text. Examples: >>> from urduhack.normalization.character import punctuations_space >>> text = "ہوتا ہے ۔ ٹائپ" >>> normalized_text = punctuations_space(text) >>> normalized_text ہوتا ہے۔ ٹائپ  z\1)rsubrrrrrpunctuations_spaces  r$cCs td|S)u- Remove ``urdu`` diacritics from text. It is an important step in pre-processing of the Urdu data. This function returns a String object which contains the original text minus Urdu diacritics. Args: text (str): ``Urdu`` text Returns: str: Returns a ``str`` object containing normalized text. Examples: >>> from urduhack.normalization import remove_diacritics >>> text = "شیرِ پنجاب" >>> normalized_text = remove_diacritics(text) >>> normalized_text شیر پنجاب r)rr#rrrrremove_diacriticss r%r rrrr) 0123456789ENG_URDU_DIGITS_MAPr&r'r(r)r*r+r,r-r.r/) rr r r r r rrrrURDU_ENG_DIGITS_MAPTwith_engcCs|r|tS|tS)z Replace urdu digits with English digits and vice versa Args: text (str): Urdu text string with_eng (bool): Boolean to convert digits from one language to other Returns: Text string with replaced digits )r_ENG_DIGITS_TRANSLATOR_URDU_DIGITS_TRANSLATOR)rr2rrrreplace_digitss  r5cCs.t|ts tdt|}t|}t|}|S)u To normalize some text, all you need to do pass ``unicode`` text. It will return a ``str`` with normalized characters both single and combined, proper spaces after digits and punctuations and diacritics removed. Args: text (str): ``Urdu`` text Returns: str: Normalized urdu text Raises: TypeError: If text param is not not str Type. Examples: >>> from urduhack import normalize >>> text = "اَباُوگل پاکستان ﻤﯿﮟ 20 سال ﺳﮯ ، وسائل کی کوئی کمی نہیں ﮨﮯ۔" >>> normalized_text = normalize(text) >>> # The text now contains proper spaces after digits and punctuations, >>> # normalized characters and no diacritics! >>> normalized_text اباوگل پاکستان ﻤﯿﮟ 20 سال ﺳﮯ ، وسائل کی کوئی کمی نہیں ﮨﮯ۔ ztext must be str type.) isinstancestr TypeErrorr%rr!rrrr normalizes r9N)T)__doc__typingrregexesrrrr__annotations__rrkeyvalueupdatedictfromkeysmapordr7rrr!r$r%r0r3r1r4boolr5r9rrrrs<         !"#$%&'()*+,-./0123A(