o i @sddlmZddlmZmZmZddlZddlmZddl m Z m Z m Z m Z ddlmZmZmZmZe eded ed efd d Ze eded ed efd dZd efddZde deded eeeffddZdeded eeeeeegeffddZdS))Path)CallableOptionalTupleN)ByteBPEProcessor)ModelRaggeddeserialize_attrserialize_attr)Tok2PiecesBackpropT Tok2PiecesInTTok2PiecesModelTTok2PiecesOutTvaluenamereturncCs|j|jd}t|S)N)mergesvocab)rrsrsly msgpack_dumps_rrmodeldatarh/home/ubuntu/.local/lib/python3.10/site-packages/spacy_curated_transformers/tokenization/bbpe_encoder.pyserialize_byte_bpe_processor s rcCst|}t|d|dS)Nrr)r msgpack_loadsrrrrrdeserialize_byte_bpe_processors rcCstdttigdddddS)zConstruct a Byte-BPE piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers. This model must be separately initialized using an appropriate loader. byte_bpe_encoderzzz)byte_bpe_processor unk_piece bos_piece eos_piece)forwardattrs)rbyte_bpe_encoder_forwardrrrrrbuild_byte_bpe_encoder_v1sr(rXis_traincCs4|jd}|jd}|jd}|jd}||}|dur!td||}|dur.td||} | dur;tdg} |D]T} |g} dg} t| D]-\}}|jrUg}n|d krd| |dj|j}n|j}||}| || t |qK| || d| t |j | |j | q?| d d fS) Nr!r#r$r"z=Byte-BPE piece encoder vocabulary doesn't contain 'BOS' piecez=Byte-BPE piece encoder vocabulary doesn't contain 'EOS' piecez=Byte-BPE piece encoder vocabulary doesn't contain 'UNK' piecer rcSsgS)Nr)dYrrrdsz*byte_bpe_encoder_forward..)r&piece_id ValueError enumerateis_space whitespace_text encode_as_idsextendappendlenrops asarray1i)rr)r*bbpr#r$r"bos_ideos_idunk_idpiecesdoc doc_pieceslensidxtoken piece_idsr2rrrr'-sR              r' vocab_path merges_pathcsdfdd }|S)zConstruct a callback that initializes a Byte-BPE piece encoder model. vocab_path (Path): Path to the vocabulary file. merges_path (Path): Path to the merges file. Ncstjd|jd<|S)N)rrr!)rload_from_filesr&)rr)YrErDrrloadvs z.build_byte_bpe_encoder_loader_v1..load)NNr)rDrErIrrHr build_byte_bpe_encoder_loader_v1gsrJ)pathlibrtypingrrrrcurated_tokenizersr thinc.apirrr r typesr r rrregisterstrbytesrrr(boolr'rJrrrrsJ     :