o qi@sdZddlmZmZmZmZmZmZddlZ dZ e dZ e dZ dZ ded efd d Zd ed eeeeefeeffd dZded eefddZded eefddZdS)zConll format parser)DictTupleAnyOptionalListIteratorN#z\s*([^=]+?)\s*=\s*(.+)z\s*(\S.*?)\s*$ linereturnc Cs~|ddkr |dd}|t}t|dkrtd||d|d|d|d |d |d |d |d |d|dd S)z Parse single conll line Args: line (str): A single conll-u token line Returns: dict: A dictionary containing conll-u token attributes Raises: ValueError: If the number of columns in line are not 10  N z@The number of columns per token line must be 10. Invalid token: r ) idtextlemmauposxposfeatsheaddepreldepsmisc)splitFIELD_DELIMITERlen ValueError)r fieldsr'I/home/ubuntu/.local/lib/python3.10/site-packages/urduhack/conll/parser.pyparse_conll_tokens   r)sentencec Cs|d}i}g}|D]<}|rG|dtkr>tt|}tt|}|r2|d}|d}|||<q |r=|d} d|| <q t|} || q ||fS)z Parse single conll sentence Args: sentence (str): A complete conll-u sentence Returns: tuple: Two dicts containing sentence metadata and token data r rrrN) r"COMMENT_MARKERrematchKEY_VALUE_COMMENT_PATTERNSINGLETON_COMMENT_PATTERNgroupr)append) r*lines sentence_metatokensr kv_matchsingleton_matchkeyvalktokenr'r'r(parse_conll_sentence.s(         r;r2ccspg}|D] }|}|r||q|r%d|}t|}||Vq|r6d|}t|}|VdSdS)a Iterate over the constructed sentences in the given lines. This method correctly takes into account newpar and newdoc comments as well. Args: lines: An iterator over the lines to parse. Yields: An iterator over the constructed Sentence objects found in the source. Raises: ValueError: If there is an error constructing the Sentence. r N)stripr1joinr;clear)r2 sent_linesr single_conll_sentence _sentencer'r'r( _iter_linesMs"     rB file_namecCsNg}t|dd}t|D]}||q Wd|S1s wY|S)a_ Load a CoNLL-U file given its location. Args: file_name (str): The location of the file. Returns: List[Tuple]: A Conll object equivalent to the provided file. Raises: IOError: If there is an error opening the given filename. ParseError: If there is an error parsing the input into a Conll object. utf8)encodingN)openrBr1)rC conll_datafiler*r'r'r( _load_filems    rI)__doc__typingrrrrrrregexr,r+r.r/r#strdictr)r;iterrBrIr'r'r'r(s *