o qi>A@sdZddlZddlZddlmZddlmZddlmZddl m Z dZ dZ d Z d ZGd d d ZGd ddeZGdddZdS)z Token, Word data structures N)Dict)CoNLL) Conllable) STOP_WORDSner start_charend_char span_typec@seZdZdZd&defddZddZedd Zej d d Zed d Z e j d d Z eddZ e j ddZ eddZ e j ddZ eddZ eddZeddZej ddZddZddZdd Zed!efd"d#Zed!efd$d%ZdS)'Tokenah A token class that stores attributes of a token and carries a list of words. A token corresponds to a unit in the raw text. In some languages such as English, a token has a one-to-one mapping to a word, while in other languages such as French, a (multi-word) token might be expanded into multiple words that carry syntactic annotations. N token_entrycCs|tjr |tjsJddgd\|_|_|_|_|_|_ |_ |tj|_ |tj|_ |tj d|_|td|_|durF|ng|_|jdurT|dSdS)zu Construct a token given a dictionary format token entry. Optionally link itself to the corresponding words. z,id and text should be included for the tokenN)getrIDTEXT_id_text_misc_words _start_char _end_char_neridtextMISCmiscNERrwordsinit_from_misc)selfr rrL/home/ubuntu/.local/lib/python3.10/site-packages/urduhack/core/unit/token.py__init__s &  zToken.__init__cCsl|jdD]-}|dd}t|dkrq|\}}|ttfvr#t|}d|}t||r3t|||qdS)z4 Create attributes by parsing from the `misc` field.|=_N)rsplitlen START_CHAREND_CHARinthasattrsetattrritem key_valuekeyvalueattrrrr r*s      zToken.init_from_misccC|jS)z! Access the index of this token. rrrrr r8zToken.idcC ||_dS)z Set the token's id value. Nr4rr1rrr r= cCr3)z/ Access the text of this token. Example: 'The' rr5rrr rBr6z Token.textcCr7)z, Set the token's text value. Example: 'The' Nr:r8rrr rGr9cCr3)z- Access the miscellaneousness of this token. rr5rrr rLr6z Token.misccC||rd}||_dS)z* Set the token's miscellaneousness value. N_is_nullrr8rrr rQ  cCr3)z; Access the list of syntactic words underlying this token. rr5rrr rZr6z Token.wordscCs||_|jD]}||_qdS)z6 Set this token's list of underlying syntactic words. N)rparent)rr1wordrrr r_s cCr3)zB Access the start character index for this token in the raw text. rr5rrr rfr6zToken.start_charcCr3)z@ Access the end character index for this token in the raw text. rr5rrr rkr6zToken.end_charcCr3)z3 Access the NER tag of this token. Example: 'B-ORG')rr5rrr rpr6z Token.nercCr<)z* Set the token's NER tag. Example: 'B-ORG'N)r>rr8rrr rur?cCtj|dddSNF)indent ensure_asciijsondumpsto_dictr5rrr __repr__zToken.__repr__cCs"g}|jD] }||q|S)z Dumps the token into a list of dictionary for this token with its extended words if the token is a multi-word token. )rappendrM)rretrBrrr rMs z Token.to_dictcC|dup|dkSNr%rr8rrr r>zToken._is_nullreturncCs |jtvS)zd Check the token is stop_word Returns: bool: Return true|False )rrr5rrr is_stops z Token.is_stopcCs&|jD] }t|dsdSqdS)z` Check the token is punct Returns: bool: Return true|False PFT)r unicodedatacategory startswith)rcharrrr is_puncts zToken.is_punct)N)__name__ __module__ __qualname____doc__rr!rpropertyrsetterrrrrrrrNrMr>boolrVr\rrrr r sF               r c@seZdZdZdefddZddZeddZej d dZed d Z e j d d Z ed dZ e j ddZ eddZ e j ddZ eddZ e j ddZ eddZej ddZeddZej ddZeddZej ddZedd Zej d!d Zed"d#Zej d$d#Zed%d&Zej d'd&Zed(d)Zej d*d)Zd+d,Zd-d.Zd/d0Zd1efd2d3Zd4S)5Wordz4 A word class that stores attributes of a word. word_entryc Cs|tjr |tjsJd|dgd\ |_|_|_|_|_ |_ |_ |_ |_ |_|_|tj|_|tj|_|tjd|_|tjd|_|tjd|_|tjd|_|tjd|_|tjd|_|tjd|_|tj d|_!|j!dur|"dSdS)z@ Construct a word given a dictionary format word entry. z/id and text should be included for the word. {}N )#r rrrformatrr_lemma_upos_xpos_feats_head_deprel_depsr_parentrrLEMMAlemmaUPOSuposXPOSxposFEATSfeatsHEADheadDEPRELdeprelDEPSdepsrrr)rrerrr r!s&&  z Word.__init__cCsX|jdD]#}|dd}t|dkrq|\}}d|}t||r)t|||qdS)z= Create attributes by parsing from the `misc` field. r"r#r$r%N)rr&r'r+r,r-rrr rs     zWord.init_from_misccCr3)z Access the index of this word. r4r5rrr rr6zWord.idcCr7)z Set the word's index value. Nr4r8rrr rr9cCr3)z- Access the text of this word. Example: 'The'r:r5rrr rr6z Word.textcCr7)z* Set the word's text value. Example: 'The'Nr:r8rrr rr9cCr3)z Access the lemma of this word. )rhr5rrr rqr6z Word.lemmacCs"||s |jdkr d}||_dS)z Set the word's lemma value. r%N)r>rrhr8rrr rqs cCr3zB Access the universal part-of-speech of this word. Example: 'NOUN'rir5rrr rsr6z Word.uposcCr<z? Set the word's universal part-of-speech value. Example: 'NOUN'Nr>rir8rrr rsr?cCr3)zI Access the treebank-specific part-of-speech of this word. Example: 'NNP')rjr5rrr rur6z Word.xposcCr<)zF Set the word's treebank-specific part-of-speech value. Example: 'NNP'N)r>rjr8rrr rur?cCr3)zF Access the morphological features of this word. Example: 'Gender=Fem')rkr5rrr rw r6z Word.featscCr<)z> Set this word's morphological features. Example: 'Gender=Fem'N)r>rkr8rrr rwr?cCr3)z- Access the id of the governer of this word. )rlr5rrr ryr6z Word.headcCs"||r d|_dSt||_dS)z# Set the word's governor id value. N)r>rlr*r8rrr rys  cCr3)z= Access the dependency relation of this word. Example: 'nmod')rmr5rrr r{'r6z Word.deprelcCr<)z: Set the word's dependency relation value. Example: 'nmod'N)r>rmr8rrr r{,r?cCr3)z' Access the dependencies of this word. )rnr5rrr r}5r6z Word.depscCr<)z$ Set the word's dependencies value. N)r>rnr8rrr r}:r?cCr3)z, Access the miscellaneousness of this word. r;r5rrr rCr6z Word.misccCr<)z) Set the word's miscellaneousness value. Nr=r8rrr rHr?cCr3)z Access the parent token of this word. In the case of a multi-word token, a token can be the parent of multiple words. Note that this should return a reference to the parent token object. ror5rrr rAQsz Word.parentcCr7)z Set this word's parent token. In the case of a multi-word token, a token can be the parent of multiple words. Note that value here should be a reference to the parent token object. Nrr8rrr rAXs cCr3r~rr5rrr pos_r6zWord.poscCr<rrr8rrr rdr?cCrErFrJr5rrr rNnrOz Word.__repr__cCs2i}tD]}t||durt||||<q|S)z+ Dumps the word into a dictionary. N)r get_fieldsgetattr)r word_dictfieldrrr rMqs  z Word.to_dictcCrRrSrr8rrr r>zrTz Word._is_nullrUcCsLg}tD]}t||}|dur|tjq|t|qtj|S)a Convert this Word to its CoNLL-U representation. A Token's CoNLL-U representation is a line. Note that this method does not include a newline at the end. Returns: str: A string representing the Word in CoNLL-U format. N)rrrrPEMPTYstrFIELD_DELIMITERjoin)ritemsrr1rrr conll}s   z Word.conllN)r]r^r_r`rr!rrarrbrrqrsrurwryr{r}rrArrNrMr>rrrrrr rdsp                          rdc@seZdZdZd"ddZddZddZed d Zej d d Zed d Z e j dd Z eddZ e j ddZ eddZ e j ddZ eddZ e j ddZ eddZej ddZeddZej ddZddZd d!ZdS)#Spanz A span class that stores attributes of a textual span. A span can be typed. A range of objects (e.g., entity mentions) can be represented as spans. NcCs|dus|dur |dusJd|dusJddgd\|_|_|_|_g|_g|_||_||_|dur:|||durF| ||dSdS)z Construct a span given a span entry or a list of tokens. A valid reference to a doc must be provided to construct a span (otherwise the text of the span cannot be initialized). NzMEither a span_entry or a token list needs to be provided to construct a span.z2A parent doc must be provided to construct a span.) r _span_typerr_tokensr_doc_sentinit_from_entryinit_from_tokens)r span_entrytokensr docsentrrr r!s z Span.__init__cCs>|tjd|_|td|_|td|_|td|_ dS)z init from entryN) r rrrTYPEr r(rr)r)rrrrr rszSpan.init_from_entrycCsxt|ts Jdt|dksJd||_||_|jdj|_|jdj|_|jj|j|j|_dd|D|_ dS)z init from tokensz6Tokens must be provided as a list to construct a span.rz)Tokens of a span cannot be an empty list.cSsg|] }|jD]}|qqSr)r).0twrrr sz)Span.init_from_tokens..N) isinstancelistr'rr rrrrr)rrr rrr rszSpan.init_from_tokenscCr3)z% Access the parent doc of this span. rr5rrr rr6zSpan.doccCr7)z" Set the parent doc of this span. Nrr8rrr rr9cCr3)z= Access the text of this span. Example: 'Stanford University'r:r5rrr rr6z Span.textcCr7)z: Set the span's text value. Example: 'Stanford University'Nr:r8rrr rr9cCr3)zD Access reference to a list of tokens that correspond to this span. rr5rrr rr6z Span.tokenscCr7)z Set the span's list of tokens. Nrr8rrr rr9cCr3)zC Access reference to a list of words that correspond to this span. r@r5rrr rr6z Span.wordscCr7)z Set the span's list of words. Nr@r8rrr rr9cCr3)z0 Access the type of this span. Example: 'PERSON'rr5rrr r r6zSpan.span_typecCr7)z Set the type of this span. Nrr8rrr r r9cCr3)z1 Access the start character offset of this span. rCr5rrr rr6zSpan.start_charcCr7)z. Set the start character offset of this span. NrCr8rrr rr9cCr3)z/ Access the end character offset of this span. rDr5rrr rr6z Span.end_charcCr7)z, Set the end character offset of this span. NrDr8rrr rr9cCs(gd}i}|D] }t||||<q|S)z# Dumps the span into a dictionary. )rr rr)r)rattrs span_dict attr_namerrr rMs z Span.to_dictcCrErFrJr5rrr rNrOz Span.__repr__)NNNNN)r]r^r_r`r!rrrarrbrrrr rrrMrNrrrr rsF                r)r`rKrXtypingrurduhack.conllrurduhack.conll.conllablerurduhack.stop_wordsrrr(r)rr rdrrrrr s     m