o ic* @sddlmZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddl mZddlmZdd lmZmZmZmZeZdd ed eed eddfddZGdddeZGdddZdS)) OrderedDict)Path)AnyDictListOptionalUnionN) BloomFilter)Errors) get_string_id)SimpleFrozenDict ensure_pathload_language_dataregistryTlangtablesstrictreturnLookupscCst}|tjvr|rt|dkrttjjd||d|Stj |}|D]!}||vr<|r9ttjj||di}nt ||}| ||q'|S)aLoad the data from the spacy-lookups-data package for a given language, if available. Returns an empty `Lookups` container if there's no data or if the package is not installed. lang (str): The language code (corresponds to entry point exposed by the spacy-lookups-data package). tables (List[str]): Name of tables to load, e.g. ["lemma_lookup", "lemma_exc"] strict (bool): Whether to raise an error if a table doesn't exist. RETURNS (Lookups): The lookups container containing the loaded tables. rz, )tabler) rrlookupslen ValueErrorr E955formatjoingetr add_table)rrrrdatar language_datar!A/home/ubuntu/.local/lib/python3.10/site-packages/spacy/lookups.py load_lookupss   r#c@seZdZdZeddedeeddfddZddeedeeddfdd Z d e ee fd e ddfd d Z d e ee fd e ddfddZd e ee fde fddZdd e ee fdee de fddZd e ee fdefddZdefddZdeddfddZdS)TablezA table in the lookups. Subclass of builtin dict that implements a slightly more consistent and unified API. Includes a Bloom filter to speed up missed lookups. NrnamercCs||d}|||S)zInitialize a new table from a dict. data (dict): The dictionary. name (str): Optional table name for reference. DOCS: https://spacy.io/api/lookups#table.from_dict r%)update)clsrr%selfr!r!r" from_dict3s zTable.from_dictcCsTt|||_d|_|durtt|dn|j}t||_|r(| |dSdS)zInitialize a new table. name (str): Optional table name for reference. data (dict): Initial data, used to hint Bloom Filter. DOCS: https://spacy.io/api/lookups#table.init g.ANr ) r__init__r% default_sizemaxrr from_error_ratebloomr')r)r%rsizer!r!r"r+@s  zTable.__init__keyvaluecCs&t|}t||||j|dS)zSet new key/value pair. String keys will be hashed. key (str / int): The key to set. value: The value to set. N)r r __setitem__r/addr)r1r2r!r!r"r3QszTable.__setitem__cCs |||<dS)zSet new key/value pair. String keys will be hashed. Same as table[key] = value. key (str / int): The key to set. value: The value to set. Nr!r5r!r!r"set[ z Table.setcCst|}t||S)zGet the value for a given key. String keys will be hashed. key (str / int): The key to get. RETURNS: The value. )r r __getitem__r)r1r!r!r"r8ds zTable.__getitem__defaultcCst|}t|||S)zGet the value for a given key. String keys will be hashed. key (str / int): The key to get. default: The default value to return. RETURNS: The value. )r rr)r)r1r:r!r!r"rmsz Table.getcCs"t|}||jvr dSt||S)zCheck whether a key is in the table. String keys will be hashed. key (str / int): The key to check. RETURNS (bool): Whether the key is in the table. F)r r/r __contains__r9r!r!r"r;ws  zTable.__contains__cCs&|jt||jd}t|S)zSerialize table to a bytestring. RETURNS (bytes): The serialized table. DOCS: https://spacy.io/api/lookups#table.to_bytes )r%dictr/)r%r<itemsr/to_bytessrsly msgpack_dumps)r)rr!r!r"r>s   zTable.to_bytes bytes_datacCsHt|}|di}|d|_t|d|_||||S)zLoad a table from a bytestring. bytes_data (bytes): The data to load. RETURNS (Table): The loaded table. DOCS: https://spacy.io/api/lookups#table.from_bytes r<r%r/) r? msgpack_loadsrr%r from_bytesr/clearr')r)rAloadedrr!r!r"rCs    zTable.from_bytes)N)NN)__name__ __module__ __qualname____doc__ classmethodr<rstrr*r+rintrr3r6r8rboolr;bytesr>rCr!r!r!r"r$,s   $   r$c@s$eZdZdZd&ddZdedefddZdefd d Z e de efd d Z e fded edefddZdededdfddZefdededefddZdedefddZdedefddZdefddZdeddfddZ d'd eeefd!eddfd"d#Z d'd eeefd!eddfd$d%ZdS)(raSContainer for large lookup tables and dictionaries, e.g. lemmatization data or tokenizer exception lists. Lookups are available via vocab.lookups, so they can be accessed before the pipeline components are applied (e.g. in the tokenizer and lemmatizer), as well as within the pipeline components via doc.vocab.lookups. rNcCs i|_dS)zXInitialize the Lookups object. DOCS: https://spacy.io/api/lookups#init N_tablesr)r!r!r"r+s zLookups.__init__r%cCs ||S)zCheck if the lookups contain a table of a given name. Delegates to Lookups.has_table. name (str): Name of the table. RETURNS (bool): Whether a table of that name is in the lookups. ) has_tabler)r%r!r!r"r;s zLookups.__contains__cCs t|jS)z3RETURNS (int): The number of tables in the lookups.)rrPrQr!r!r"__len__s zLookups.__len__cCst|jS)z8RETURNS (List[str]): Names of all tables in the lookups.)listrPkeysrQr!r!r"rszLookups.tablesrcCs6||jvrttjj|dt||d}||j|<|S)aAdd a new table to the lookups. Raises an error if the table exists. name (str): Unique name of table. data (dict): Optional data to add to the table. RETURNS (Table): The newly added table. DOCS: https://spacy.io/api/lookups#add_table r&)r%r)rrr E158rr$rP)r)r%rrr!r!r"rs   zLookups.add_tablercCs||j|<dS)zSet a table. name (str): Name of the table to set. table (Table): The Table to set. DOCS: https://spacy.io/api/lookups#set_table NrO)r)r%rr!r!r" set_tableszLookups.set_tabler:cCs6||jvr|tkrttjj||jd|S|j|S)a<Get a table. Raises an error if the table doesn't exist and no default value is provided. name (str): Name of the table. default (Any): Optional default value to return if table doesn't exist. RETURNS (Table): The table. DOCS: https://spacy.io/api/lookups#get_table r%r)rPUNSETKeyErrorr E159rr)r)r%r:r!r!r" get_tables  zLookups.get_tablecCs,||jvrttjj||jd|j|S)zRemove a table. Raises an error if the table doesn't exist. name (str): Name of the table to remove. RETURNS (Table): The removed table. DOCS: https://spacy.io/api/lookups#remove_table rY)rPr[r r\rrpoprSr!r!r" remove_tables  zLookups.remove_tablecCs ||jvS)zCheck if the lookups contain a table of a given name. name (str): Name of the table. RETURNS (bool): Whether a table of that name exists. DOCS: https://spacy.io/api/lookups#has_table rOrSr!r!r"rRs zLookups.has_tablecKs t|jS)zSerialize the lookups to a bytestring. RETURNS (bytes): The serialized Lookups. DOCS: https://spacy.io/api/lookups#to_bytes )r?r@rP)r)kwargsr!r!r"r>r7zLookups.to_bytesrAcKs2i|_t|D] \}}t|||j|<q |S)zLoad the lookups from a bytestring. bytes_data (bytes): The data to load. RETURNS (Lookups): The loaded Lookups. DOCS: https://spacy.io/api/lookups#from_bytes )rPr?rBr=r$)r)rAr`r1r2r!r!r"rC szLookups.from_bytes lookups.binpathfilenamecKs^t|}|s |||}|d}||WddS1s(wYdS)zSave the lookups to a directory as lookups.bin. Expects a path to a directory, which will be created if it doesn't exist. path (str / Path): The file path. DOCS: https://spacy.io/api/lookups#to_disk wbN)rexistsmkdiropenwriter>)r)rbrcr`filepathfile_r!r!r"to_disks  "zLookups.to_diskcKsXt|}||}|r*|d }|}Wdn1s wY||S|S)a Load lookups from a directory containing a lookups.bin. Will skip loading if the file doesn't exist. path (str / Path): The directory path. RETURNS (Lookups): The loaded lookups. DOCS: https://spacy.io/api/lookups#from_disk rbN)rrergreadrC)r)rbrcr`rirjrr!r!r" from_disk)s    zLookups.from_disk)rN)ra)rFrGrHrIr+rKrMr;rLrTpropertyrrr r<r$rrXrZrr]r_rRrNr>rCrrrkrnr!r!r!r"rs<         )T) collectionsrpathlibrtypingrrrrrr? preshed.bloomr errorsr stringsr utilr rrrobjectrZrKrMr#r$rr!r!r!r"s     v