o iL @sdZddlZddlZddlmZgdZdZdZdZdd gZ d Z d Z gd Z gZ e e e ee ege ege e e ege e ge e gd de dZ ejdkrhe dZ ee ejejBZ dddZdS)uThis code inspired from underthesea package, edited by enamoria. What changed from the original version: PSG. consider to be abbreviation, but not with the dot. Just add a boundary for word at r"[A-ZĐ]+\s" N)Text)z==>z->z\.\.\.z>>z=\)\)z\d+([\.,_]\d+)+z.[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+u  # Capture 1: entire matched URL (?: (ftp|http)s?: # URL protocol and colon (?: /{1,3} # 1-3 slashes | # or [a-z0-9%] # Single letter or digit or '%' # (Trying not to match e.g. "URI::Escape") ) | # or # looks like domain name followed by a slash: [a-z0-9.\-]+[.] (?:[a-z]{2,13}) / ) (?: # One or more: [^\s()<>{}\[\]]+ # Run of non-space, non-()<>{}[] | # or \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) )+ (?: # End with: \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) | # or [^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars ) | # OR, the following to match naked domains: (?: (?csztokenize..text)rlowerreplacerefindallpatternsjoin)rformattokensr r r tokenizeZs    r)N)__doc__rsys$underthesea.feature_engineering.textrspecialsdigitemailurlsdatetimewordnon_word abbreviationsrextendr version_infodecodecompileVERBOSEUNICODErr r r r s8 +