o i.@sddlZddlmZddlmZmZmZmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZddlmZmZmZmZddlmZdd lmZdd lmZmZdd lmZd d lmZd dl m!Z"d dl m#Z#GdddZ$ddZ%ddZ&ddZ'e(e$e&e'e$Z)dgZ*dS)N)Path)DictIterableIteratorListOptionalSetUnion)ndarray)NumpyOps)IDSORTHSPACY intify_attr)copy_reg)Errors)SimpleFrozenList ensure_path)Vocab) SpanGroups)DOCBIN_ALL_ATTRS)Docc @seZdZdZedefdeededee ddfdd Z de fd d Z d e ddfd dZ dedee fddZdddZdefddZdeddfddZdeeefddfddZdeeefddfddZdS) DocBinaPack Doc objects for binary serialization. The DocBin class lets you efficiently serialize the information from a collection of Doc objects. You can control which information is serialized by passing a list of attribute IDs, and optionally also specify whether the user data is serialized. The DocBin is faster and produces smaller data sizes than pickle, and allows you to deserialize without executing arbitrary Python code. The serialization format is gzipped msgpack, where the msgpack object has the following structure: { "attrs": List[uint64], # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE] "tokens": bytes, # Serialized numpy uint64 array with the token data "spans": List[Dict[str, bytes]], # SpanGroups data for each doc "spaces": bytes, # Serialized numpy boolean array with spaces data "lengths": bytes, # Serialized numpy int32 array with the doc lengths "strings": List[str] # List of unique strings in the token data "version": str, # DocBin version number } Strings for the words, tags, labels etc are represented by 64-bit hashes in the token data, and every string that occurs at least once is passed via the strings object. This means the storage is more efficient if you pack more documents together, because you have less duplication in the strings. A notable downside to this format is that you can't easily extract just one document from the DocBin. Fattrsstore_user_datadocsreturnNcCsdd|D}d|vr dd|D}ttjjd|tddt|}d|_dd|D|_|j d t g|_ g|_ g|_ g|_g|_g|_t|_||_|D]}||qQdS) a~Create a DocBin object to hold serialized annotations. attrs (Iterable[str]): List of attributes to serialize. 'orth' and 'spacy' are always serialized, so they're not required. store_user_data (bool): Whether to write the `Doc.user_data` to bytes/file. docs (Iterable[Doc]): Docs to add. DOCS: https://spacy.io/api/docbin#init cSg|]}t|qSr.0attrr r K/home/ubuntu/.local/lib/python3.10/site-packages/spacy/tokens/_serialize.py Cz#DocBin.__init__..NcSsg|] }t|dur|qSNr!r"r r r%r&Esr)dictkeykeysz0.1cSs g|] }|tkr|tkr|qSr )rrr"r r r%r&Ks r)KeyErrorrE983formatr r+sortedversionrinsertrtokensspacescats span_groups user_dataflagssetstringsradd)selfrrr int_attrs non_validdocr r r%__init__4s. zDocBin.__init__cCs t|jS)z7RETURNS: The number of Doc objects added to the DocBin.)lenr2)r;r r r%__len__Xs zDocBin.__len__r>cCs||j}t|jdkr||jddf}|j||t}|jd|jdks.J||jddf}|jt j |t d|j d|j i|D]C}|j|j|j|j|j|j|j|j|jt|j|j|j|j|j|j|j|j|jqN|j|j|jr|jt|j|j|j !|j "D].\}}|D]'}|j|j#|j$|j%j&jvr|j|j'|j(|j%j&jvr|j|j)qqdS)zAdd a Doc's annotations to the DocBin for serialization. doc (Doc): The Doc object to add. DOCS: https://spacy.io/api/docbin#add rrdtypehas_unknown_spacesN)*to_arrayrr@shapereshaper2appendrr3numpyasarrayboolr7rDr9r:texttag_lemma_norm_strmorphdep_ ent_type_ ent_kb_id_ent_id_r4rr6srsly msgpack_dumpsr5spansto_bytesitemslabel_kb_idr>vocabkb_id_idid_)r;r>arrayr3tokenr*groupspanr r r%r:\sB   z DocBin.addr]c cs|jD]}||q|jt}tt|jD]o}|j|}|j|}|j|}| dr0d}t ||dd|f|d}| |j|}|j ||_ |j |ra|j |tjkra|j|j |n|j|t|jkr|j|durtj|j|dd} |j| |VqdS)aaRecover Doc objects from the annotations, using the given vocab. Note that the user data of each doc will be read (if available) and returned, regardless of the setting of 'self.store_user_data'. vocab (Vocab): The shared vocab. YIELDS (Doc): The Doc objects. DOCS: https://spacy.io/api/docbin#get_docs rDN)wordsr3F)use_list)r9rindexrranger@r2r7r3getr from_arrayr4r5r _EMPTY_BYTESrX from_bytesclearr6rV msgpack_loadsupdate) r;r]stringorth_colir7r2r3r>r6r r r%get_docss*         zDocBin.get_docsothercCs|j|jkrttjjd|j|jd|j|jkr&ttjjd|j|jd|j|j|j|j|j |j |j |j |j |j |j |j |j|jdS)abExtend the annotations of this DocBin with the annotations from another. Will raise an error if the pre-defined attrs of the two DocBins don't match, or if they differ in whether or not to store user data. other (DocBin): The DocBin to merge into the current bin. DOCS: https://spacy.io/api/docbin#merge r)paramcurrentrtrN)r ValueErrorrE166r.rr2extendr3r9ror4r5r7r6)r;rtr r r%merges&  z DocBin.mergec Cs|jD]}t|jdksJ|jqdd|jD}|jr#t|jntg}|jr1t|jntg}|j|j| d| dtj|dd dt t |j |j |j|jd }|jrb|j|d<tt|S) zSerialize the DocBin's annotations to a bytestring. RETURNS (bytes): The serialized DocBin. DOCS: https://spacy.io/api/docbin#to_bytes r cSrr )r@)r#r2r r r%r&r'z#DocBin.to_bytes..Cint32rB) r0rr2r3lengthsr9r4r7r5r6)r2r@rFrIvstackrJr3r0rtobyteslistr/r9r4r7r5rr6zlibcompressrVrW)r;r2r}r3msgr r r%rYs$   zDocBin.to_bytes bytes_datacCs\z tt|}Wn tjyttjw|d|_t |d|_ t j |ddd}t j |dt d}t j |ddd}|jt|jt|jf}||}||jd f}t|||_t|||_|d |_|d d d |D|_|ddd |D|_d|vrt|d|_ndgt||_|jD]}t|jdksJ|jq|S)zDeserialize the DocBin's annotations from a bytestring. bytes_data (bytes): The data to load from. RETURNS (DocBin): The loaded DocBin. DOCS: https://spacy.io/api/docbin#from_bytes rr9r}r|rBr3r2uint64rr4r5cSsg|]}dqS)r r#_r r r%r&z%DocBin.from_bytes..r7cSsg|]}iqSr r rr r r%r&rr6Nr )rVrnr decompresserrorrwrE1014rr8r9rI frombufferrKsizer@rGr unflattenr2r3r4rir5r7rr6rF)r;rrr} flat_spaces flat_tokensrFr2r r r%rls0     zDocBin.from_bytespathc Csdt|}|d}z ||Wn tyttjwWddS1s+wYdS)zSave the DocBin to a file (typically called .spacy). path (str / Path): The file path. DOCS: https://spacy.io/api/docbin#to_disk wbN)ropenwriterYrwrE870r;rfile_r r r%to_disks   "zDocBin.to_diskcCsFt|}|d}||Wd|S1swY|S)zLoad the DocBin from a file (typically called .spacy). path (str / Path): The file path. RETURNS (DocBin): The loaded DocBin. DOCS: https://spacy.io/api/docbin#to_disk rbN)rrrlreadrr r r% from_disk s  zDocBin.from_disk)rtrrN)__name__ __module__ __qualname____doc__ ALL_ATTRSrrrPrKrr?intrAr:rrrsrzbytesrYrlr rrrr r r r%rs,! $& !rcCsRd}|D]}|durtdd|}|dur|}q||q|dur'|SdS)NT)rr)rrlrzrY)binsmerged byte_stringdoc_binr r r% merge_binss rcCst|ffSr() unpickle_binrY)rr r r% pickle_bin'srcCs t|Sr()rrl)rr r r%r+s r)+rpathlibrtypingrrrrrrr rIrVr thinc.apir rr rrrcompatrerrorsrutilrrr]r _dict_proxiesrr>rrrrrrrpickleBinder__all__r r r r%s0 $