o eic7@sddlmZmZmZmZmZmZddlmZddl m Z m Z ddl m Z ddlmZeeZddd Zgd ZGd d d e Zd gZd S))Regex Tokenizerdecoders normalizerspre_tokenizers processors)BPE) AddedToken BatchEncoding)TokenizersBackend)loggingzsentencepiece.bpe.modelztokenizer.json) vocab_filetokenizer_file)ace_Arabace_Latnacm_Arabacq_Arabaeb_Arabafr_Latnajp_Arabaka_Latnamh_Ethiapc_Arabarb_Arabars_Arabary_Arabarz_Arabasm_Bengast_Latnawa_Devaayr_Latnazb_Arabazj_Latnbak_Cyrlbam_Latnban_Latnbel_Cyrlbem_Latnben_Bengbho_Devabjn_Arabbjn_Latnbod_Tibtbos_Latnbug_Latnbul_Cyrlcat_Latnceb_Latnces_Latncjk_Latnckb_Arabcrh_Latncym_Latndan_Latndeu_Latndik_Latndyu_Latndzo_Tibtell_Grekeng_Latnepo_Latnest_Latneus_Latnewe_Latnfao_Latnpes_Arabfij_Latnfin_Latnfon_Latnfra_Latnfur_Latnfuv_Latngla_Latngle_Latnglg_Latngrn_Latnguj_Gujrhat_Latnhau_Latnheb_Hebrhin_Devahne_Devahrv_Latnhun_Latnhye_Armnibo_Latnilo_Latnind_Latnisl_Latnita_Latnjav_Latnjpn_Jpankab_Latnkac_Latnkam_Latnkan_Kndakas_Arabkas_Devakat_Georknc_Arabknc_Latnkaz_Cyrlkbp_Latnkea_Latnkhm_Khmrkik_Latnkin_Latnkir_Cyrlkmb_Latnkon_Latnkor_Hangkmr_Latnlao_Laoolvs_Latnlij_Latnlim_Latnlin_Latnlit_Latnlmo_Latnltg_Latnltz_Latnlua_Latnlug_Latnluo_Latnlus_Latnmag_Devamai_Devamal_Mlymmar_Devamin_Latnmkd_Cyrlplt_Latnmlt_Latnmni_Bengkhk_Cyrlmos_Latnmri_Latnzsm_Latnmya_Mymrnld_Latnnno_Latnnob_Latnnpi_Devanso_Latnnus_Latnnya_Latnoci_Latngaz_Latnory_Oryapag_Latnpan_Gurupap_Latnpol_Latnpor_Latnprs_Arabpbt_Arabquy_Latnron_Latnrun_Latnrus_Cyrlsag_Latnsan_Devasat_Bengscn_Latnshn_Mymrsin_Sinhslk_Latnslv_Latnsmo_Latnsna_Latnsnd_Arabsom_Latnsot_Latnspa_Latnals_Latnsrd_Latnsrp_Cyrlssw_Latnsun_Latnswe_Latnswh_Latnszl_Latntam_Tamltat_Cyrltel_Telutgk_Cyrltgl_Latntha_Thaitir_Ethitaq_Latntaq_Tfngtpi_Latntsn_Latntso_Latntuk_Latntum_Latntur_Latntwi_Latntzm_Tfnguig_Arabukr_Cyrlumb_Latnurd_Arabuzn_Latnvec_Latnvie_Latnwar_Latnwol_Latnxho_Latnydd_Hebryor_Latnyue_Hantzho_Hanszho_Hantzul_LatncsfeZdZUdZeZddgZeZgZ e e e d<gZ e e e d<           d0d eeee fBdBdee eBdBffdd ZedefddZejdeddfddZdededBdedBfddZ        d1de eded e edBded!e dBd"e dBd#ededBd$edefd%d&Zd'd(Zd)d*Zd2d+d,Zd-eddfd.d/ZZS)3 NllbTokenizera Construct an NLLB tokenizer (backed by HuggingFace's *tokenizers* library). Based on [Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models). This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import NllbTokenizer >>> tokenizer = NllbTokenizer.from_pretrained( ... "facebook/nllb-200-distilled-600M", src_lang="eng_Latn", tgt_lang="fra_Latn" ... ) >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_french = "Le chef de l'ONU affirme qu'il n'y a pas de solution militaire en Syrie." >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_french, return_tensors="pt") ``` Args: vocab_file (`str`, *optional*): Path to the vocabulary file. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. sep_token (`str`, *optional*, defaults to `""`): The separator token. cls_token (`str`, *optional*, defaults to `""`): The classifier token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding. mask_token (`str`, *optional*, defaults to `""`): The token used for masking values. src_lang (`str`, *optional*): The language to use as source language for translation. tgt_lang (`str`, *optional*): The language to use as target language for translation. legacy_behaviour (`bool`, *optional*, defaults to `False`): Whether to use legacy behaviour (suffix pattern) or new behaviour (prefix pattern). input_idsattention_mask prefix_tokens suffix_tokensNFvocabmergesc s| dur| } n| dur t} t| trt| ddddn| } ||_|dur5t|dt|dt|dt|di}||_|p;g|_tt|j|jdt|ddd|_ t t t d d t t t d d g|j _tjd d dd|j _tjd d dd|j _tjd||||||| | | | |d |d|_ddddd|_dd|jD|_| dur| nd|_||j|_| |_||jdS)NT) normalizedlstripspecialrr F)rrdropout unk_tokenfuse_unk byte_fallbackz[\n\r\t] z {2,}u▁always) replacementprepend_schemesplit) bos_token eos_token sep_token cls_tokenr pad_tokensrc_langtgt_lang mask_tokenextra_special_tokenslegacy_behaviour)rrrrcSsi|]\}}||qSr).0kvrrh/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/nllb/tokenization_nllb.py sz*NllbTokenizer.__init__..r>r) FAIRSEQ_LANGUAGE_CODES isinstancestrr r_vocab_mergesrr _tokenizerrSequenceReplacerNFKC normalizerr Metaspace pre_tokenizerrdecodersuper__init__fairseq_offsetfairseq_tokens_to_idsitemsfairseq_ids_to_tokens _src_langconvert_tokens_to_ids cur_lang_coderset_src_lang_special_tokens)selfrrrrrrrrrrradditional_special_tokensrrkwargs __class__rrrYsz   zNllbTokenizer.__init__returncCs|jSN)rrrrrrszNllbTokenizer.src_lang new_src_langcCs||_||jdSr!)rr)rr#rrrrsreturn_tensorsrrcKsJ|dus|dur td||_||fd|d|}||}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensr$forced_bos_token_id) ValueErrorrr)r raw_inputsr$rr extra_kwargsinputs tgt_lang_idrrr_build_translation_inputss z'NllbTokenizer._build_translation_inputsr>rHlongestT src_texts tgt_texts max_lengthmax_target_lengthpadding truncationc Ks||_||_|dur |j}||fd|||| d| } |dur!| S|dur'|}|||fd|||| d| } | d| d<|| S)NT)r%r$r0r2r3)r%r$r2r0r3rlabels)rrmodel_max_length_switch_to_target_mode_switch_to_input_mode) rr.rr/rr0r1r2r$r3r model_inputsr4rrrprepare_seq2seq_batchsD   z#NllbTokenizer.prepare_seq2seq_batchcCs ||jSr!)rrr"rrrr7s z#NllbTokenizer._switch_to_input_modecCs|jdur |j|_||jSr!)rrset_tgt_lang_special_tokensr"rrrr6s  z$NllbTokenizer._switch_to_target_modecC|||_|jrg|_|j|jg|_n |jg|_|jg|_||j}||j}tj|dg||ddg|t t |||j|jd|j _ dS)zReset the special tokens to the source lang setting. - In legacy mode: No prefix and suffix=[eos, src_lang_code]. - In default mode: Prefix=[src_lang_code], suffix = [eos] $A$Bsinglepairspecial_tokensN rrrr eos_token_idrconvert_ids_to_tokensrTemplateProcessinglistzipr post_processor)rrprefix_tokens_strsuffix_tokens_strrrrr s      z)NllbTokenizer.set_src_lang_special_tokenslangcCr;)zReset the special tokens to the target lang setting. - In legacy mode: No prefix and suffix=[eos, tgt_lang_code]. - In default mode: Prefix=[tgt_lang_code], suffix = [eos] r<r=r>NrB)rrKrIrJrrrr:!s      z)NllbTokenizer.set_tgt_lang_special_tokens)NNrrrrrrrNNNNF)r>NrHNNr-NT)r N)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesrmodelrrFint__annotations__rrdictrpropertyrsetterr,boolr r9r7r6rr: __classcell__rrrrr!s 0^     6 rN) tokenizersrrrrrrtokenizers.modelsrtokenization_pythonr r tokenization_utils_tokenizersr utilsr get_loggerrLloggerrPrr__all__rrrrs