o i*@sddlZddlmZddlmZddlmZddlmZm Z ddl m Z ddl m Z mZe r6d d lmZndZeeZd d d ZgdZGddde ZdgZdS)N)copyfile)Optional) processors) AddedToken BatchEncoding)PreTrainedTokenizerFast)is_sentencepiece_availablelogging)MBartTokenizerzsentencepiece.bpe.modelztokenizer.json) vocab_filetokenizer_file)ar_ARcs_CZde_DEen_XXes_XXet_EEfi_FIfr_XXgu_INhi_INit_ITja_XXkk_KZko_KRlt_LTlv_LVmy_MMne_NPnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CNc seZdZUdZeZddgZeZgZ e e e d<gZ e e e d<         d1fd d ZedefddZejdeddfddZ d2de e dee e de e fddZ d2de e dee e de e fddZdedeedeefddZ   d3d e eded!ee ededef fd"d# Zd$d%Zd&d'Zd4d(d)Zd*eddfd+d,Zd2d-ed.eedeefd/d0ZZS)5MBartTokenizerFastuO Construct a "fast" MBART tokenizer (backed by HuggingFace's *tokenizers* library). Based on [BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models). This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizerFast >>> tokenizer = MBartTokenizerFast.from_pretrained( ... "facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO" ... ) >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ``` input_idsattention_mask prefix_tokens suffix_tokensNc  st| tr t| dddn| } t| dur"fdd| Dtjd ||||||||| | | d | |_fddtD_ | durK| nd _  j _ | _ j dS) NTF)lstriprstripcsg|]}|vr|qSr4).0t)_additional_special_tokensr4e/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/mbart/tokenization_mbart_fast.py asz/MBartTokenizerFast.__init__..) r r bos_token eos_token sep_token cls_token unk_token pad_token mask_tokensrc_langtgt_langadditional_special_tokenscsi|]}||qSr4)convert_tokens_to_ids)r5 lang_codeselfr4r8 us z/MBartTokenizerFast.__init__..rr4) isinstancestrrFAIRSEQ_LANGUAGE_CODEScopyextendsuper__init__r lang_code_to_id _src_langrD cur_lang_coderBset_src_lang_special_tokens)rGr rr:r;r<r=r>r?r@rArBrCkwargs __class__)r7rGr8rOIs<  zMBartTokenizerFast.__init__returncCs|jSN)rQrFr4r4r8rA~szMBartTokenizerFast.src_lang new_src_langcCs||_||jdSrX)rQrS)rGrYr4r4r8rAs token_ids_0 token_ids_1cCs,|dur |j||jS|j|||jS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. The special tokens depend on calling set_lang. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`list[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: list of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)r+r,)rGrZr[r4r4r8 build_inputs_with_special_tokenssz3MBartTokenizerFast.build_inputs_with_special_tokenscCsP|jg}|jg}|durt|||dgSt||||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of zeros. Nr) sep_token_id cls_token_idlen)rGrZr[sepclsr4r4r8$create_token_type_ids_from_sequencess "z7MBartTokenizerFast.create_token_type_ids_from_sequencesreturn_tensorsrArBcKsJ|dus|dur td||_||fd|d|}||}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensrcforced_bos_token_id) ValueErrorrArD)rG raw_inputsrcrArB extra_kwargsinputs tgt_lang_idr4r4r8_build_translation_inputss z,MBartTokenizerFast._build_translation_inputsrr" src_texts tgt_textsc s"||_||_tj||fi|SrX)rArBrNprepare_seq2seq_batch)rGrlrArmrBrTrUr4r8rnsz(MBartTokenizerFast.prepare_seq2seq_batchcC ||jSrX)rSrArFr4r4r8_switch_to_input_mode z(MBartTokenizerFast._switch_to_input_modecCrorX)set_tgt_lang_special_tokensrBrFr4r4r8_switch_to_target_moderqz)MBartTokenizerFast._switch_to_target_modecC||||_g|_|j|jg|_||j}||j}tj|dg||ddg|tt |||j|jd|j _ dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].$A$Bsinglepairspecial_tokensN rDrRr+ eos_token_idr,convert_ids_to_tokensrTemplateProcessinglistzip _tokenizerpost_processor)rGrAprefix_tokens_strsuffix_tokens_strr4r4r8rS    z.MBartTokenizerFast.set_src_lang_special_tokenslangcCrt)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].rurvrwNr{)rGrrrr4r4r8rrrz.MBartTokenizerFast.set_tgt_lang_special_tokenssave_directoryfilename_prefixcCs~|jstdtj|std|ddStj||r"|dndtd}tj |j tj |krs        d