o 8wi7Z @sdZddlZddlZddlZddlmZddlmZddl Z ddl Z ddl m Z ddlmZddlmZddlmZeeZGd d d Zd d ZeGd ddZde jdeededeefddZdS)ziLibrary for Byte-pair-encoding (BPE) tokenization. Authors * Abdelwahab Heba 2020 * Loren Lugosch 2020 N) dataclass)List) merge_char) edit_distance) run_on_main) get_loggerc@sjeZdZdZ               dd d Zd dZddZddZgfddZdddZ dS) SentencePieceaBPE class call the SentencePiece unsupervised text tokenizer from Google. Reference: https://github.com/google/sentencepiece SentencePiece lib is an unsupervised text tokenizer and detokenizer. It implements subword units like Byte-pair-encoding (BPE), Unigram language model and char/word tokenizer. Arguments --------- model_dir : str The directory where the model will be saved (or already stored). vocab_size : int, None, optional Vocab size for the chosen tokenizer type (BPE, Unigram). The vocab_size is optional for char, and mandatory for BPE & unigram tokenization. annotation_train : str Path of the annotation file which is used to learn the tokenizer. It can be in JSON or csv format. annotation_read : str The data entry which contains the word sequence in the annotation file. model_type : str (bpe, char, unigram). If "bpe", train unsupervised tokenization of piece of words. see: https://www.aclweb.org/anthology/P16-1162/ If "word" take the vocabulary from the input text. If "unigram" do piece of word tokenization using unigram language model, see: https://arxiv.org/abs/1804.10959 char_format_input : bool Whether the read entry contains characters format input. (default: False) (e.g., a p p l e _ i s _ g o o d) character_coverage : int Amount of characters covered by the model, good defaults are: 0.9995 for languages with a rich character set like Japanese or Chinese and 1.0 for other languages with small character set. (default: 1.0) user_defined_symbols : string String contained a list of symbols separated by a comma. User-defined symbols are handled as one piece in any context. (default: None) max_sentencepiece_length : int Maximum number of characters for the tokens. (default: 10) bos_id : int If -1 the bos_id = unk_id = 0. otherwise, bos_id = int. (default: -1) eos_id : int If -1 the bos_id = unk_id = 0. otherwise, bos_id = int. (default: -1) pad_id : int If -1 the pad_id = unk_id = 0. otherwise, bos_id = int. (default: -1) unk_id : int The token corresponding to an unknown symbol (not in token set). split_by_whitespace : bool If False, allow the sentencepiece to extract piece crossing multiple words. This feature is important for : Chinese/Japanese/Korean. (default: True) num_sequences : int If not none, use at most this many sequences to train the tokenizer (for large datasets). (default: None) annotation_list_to_check : list, List of the annotation file which is used for checking the accuracy of recovering words from the tokenizer. annotation_format : str The format of the annotation file. JSON or csv are the formats supported. text_file: str An alternate path to the text file (needed when multiple models are trained on the same data file) add_dummy_prefix : bool If True the tokenizer adds dummy whitespace at the beginning of text. (default: True) Example ------- >>> import torch >>> dict_int2lab = {1: "HELLO", 2: "MORNING"} >>> model_dir = getfixture('tmpdir') / "tokenizer_data" >>> # Example with csv >>> annotation_train = "tests/samples/annotation/dev-clean.csv" >>> annotation_read = "wrd" >>> model_type = "bpe" >>> bpe = SentencePiece(str(model_dir), 100, annotation_train, annotation_read, model_type) >>> batch_seq = torch.Tensor([[1, 2, 2, 1],[1, 2, 1, 0]]) >>> batch_lens = torch.Tensor([1.0, 0.75]) >>> encoded_seq_ids, encoded_seq_pieces = bpe( ... batch_seq, batch_lens, dict_int2lab, task="encode" ... ) >>> # Example using JSON >>> annotation_train = str(model_dir + "/dev-clean.json") >>> annotation_read = "wrd" >>> bpe = SentencePiece(model_dir, 100, annotation_train, annotation_read, model_type, annotation_format = 'json') >>> encoded_seq_ids, encoded_seq_pieces = bpe( ... batch_seq, batch_lens, dict_int2lab, task="encode" ... ) NunigramF? rTcsvcCs"|dvrtdtj|st|t|tstd||_||_||_ |jdurJtj |jd}|durGtj |tj |j |d}||_tj |t|d||_t||_||_||_t||_t| |_t| |_t| |_t| |_t| |_||_||_||_t||_tj|jdst|j nt!"dt!"d t!"d |jdt!"d t|jt!"d |jt#$|_%|j%&|jdt|j|j%krd |jd|j%d}|jdkrt!'|dnt!'|d|durt|j(d|iddSdS)N)r bpecharz0model_type must be one of : [unigram, bpe, char]zvocab_size must be integer.z.txt_z.modelzTokenizer is already trained.z==== Loading Tokenizer ===zTokenizer path: zTokenizer vocab_size: zTokenizer type: zSentencePiece vocab size `z'` requested, but the loaded model has `zQ`! This can cause decoding errors or weird model training behavior in some cases.rz@ The model type is 'char', for which `vocab_size` has no impact.z7 Are you loading a tokenizer with the wrong parameters?list_annotation_files)kwargs)) ValueErrorospathisdirmakedirs isinstanceintannotation_trainannotation_readannotation_formatsplitextjoinbasenamereplace text_filestrprefix_model_file vocab_size model_typechar_format_inputcharacter_coveragemax_sentencepiece_lengthbos_ideos_idpad_idunk_id num_sequencessplit_by_whitespaceuser_defined_symbolsadd_dummy_prefixisfiler _train_BPEloggerinfospmSentencePieceProcessorsploadwarning_check_coverage_from_bpe)self model_dirr%rrr&r'r(r0r)r*r+r,r-r/r.annotation_list_to_checkrr"r1extbase_msgrAa/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/speechbrain/tokenizers/SentencePiece.py__init__ssv                   zSentencePiece.__init__c Cs2tjtj|jst|jdtd|jd|jt |jddd}t |}t |d}|j|vr@t|jd|j| |j}t |jd dd}d }|D]4}|jdurg||jkrgtd |jn |d 7}||}|jrt|g\}d |}||dqR||td|jdS)z?Read CSV file and convert specific data entries into text file.< is not a file. please provide annotation file for training.Extract  sequences from:rutf-8encodingN must exist in:w+r*Using %d sequences to train the tokenizer.r  Text file created at: )rrr2abspathrrr4r5ropenr readernextindexr"r.printr'rsplitrwriteclose) r<annotation_filerSheaders index_labelr"row_idxrowsentrArArB _csv2textsT     zSentencePiece._csv2textcCs tjtj|jst|jdtd|jd|jt |jddd }t |}Wdn1s7wYt |j ddd}d }| D]7}|jdur_||jkr_td |jn#|d 7}|||j}|jrzt|g\}d |}||d qJ|td|j dS)z@Read JSON file and convert specific data entries into text file.rDrErFrGrHrINrLrrMrrNrOrP)rrr2rQrrr4r5rrRjsonr9r"keysr.rVr'rrWrrXrY)r<fout_jsonr"r]snt_idr_rArArB _json2textsH   zSentencePiece._json2textcCs td|jtj|js*|jdkr|n|jdkr#| nt d|jd|jd|j d|jd|j d |j d |jd |jd |jd |jd|j}|jdvrj|dt|j7}|jdurv|d|j7}|js}|d7}tj|dS)zTrain tokenizer with unsupervised techniques (BPE, Unigram) using SentencePiece Library. If you use "char" mode, the SentencePiece creates a char dict so the vocab_size attribute is not needed. zTrain tokenizer with type:r razIAnnotation format not supported. Supported formats are csv and json. Got z--input=z --model_prefix=z --model_type=z --bos_id=z --eos_id=z --pad_id=z --unk_id=z --max_sentencepiece_length=z --character_coverage=z --add_dummy_prefix=)rz --vocab_size=Nz --user_defined_symbols=z --split_by_whitespace=false)r4r5r&rrr2r"rr`rfrr$r*r+r,r-r)r(r1r#r%r0r/r6SentencePieceTrainertrain)r<queryrArArBr3st           zSentencePiece._train_BPEc Cs^|D])}tjtj|r%td|jdkr@t|ddd}t |}t |d}|j |vr9t |j d|| |j }n t|jddd}t|}|j }Wdn1s[wYg}|D]n} |jdkrp| |} n|| |} |jrt| g\} d| } | d d } |j| } |j| } tjd g| dg| dgd d \} | dd kr| dD]} | d dkr| ddur| d|vr|| dqqd|jdkr|td|t|d krtdtt|tdt|jtdtdt t||jqtdtdtdqtd|qdS)aLogging the accuracy of the BPE model to recover words from the training text. Arguments --------- list_annotation_files : list, List of the annotation file which is used for checking the accuracy of recovering words from the tokenizer. z===== Accuracy checking for recovering text from tokenizer ===r rGrHrINrKrNrOrutt1T)compute_alignmentsWER alignment=rzrecover words from: zWrong recover words: zTokenizer vocab size: zaccuracy recovering words: zWrong recover words: 0r z No accuracy recover checking for)!rrr2rQr4r5rrRr rSrTrrrUrrar9r'rrWrr8 encode_as_ids decode_idsrwer_details_for_batchappendrYlenr:r#r%float)r<rrZfannotation_filerSr[r\rcwrong_recover_listr^ encoded_id decode_textdetailsalignrArArBr;Os                    z&SentencePiece._check_coverage_from_bpeencodecs|dkr dur td|dkrg}d}|jdt|D]<\}}fdd|d|D} jrEt| g\} d| } nd| } j | } | | t | |kr_t | }q#t j |jd|f|jd } t j |jd|jd }t|D]\}}t || |dt |f<t ||||<q|| |fS|d krfd d|DS|d krˆ|jdfd dt|DSdS)aoThis __call__ function implements the tokenizer encoder and decoder (restoring the string of word) for BPE, Regularized BPE (with unigram), and char (speechbrain/nnet/RNN.py). Arguments ---------- batch : tensor.IntTensor or list List if ( batch_lens = None and task = "decode_from_list") Contains the original labels. Shape: [batch_size, max_length] batch_lens : tensor.LongTensor Containing the relative length of each label sequences. Must be 1D tensor of shape: [batch_size]. (default: None) ind2lab : dict Dictionary which maps the index from label sequences (batch tensor) to string label. task : str ("encode", "decode", "decode_from_list) "encode": convert the batch tensor into sequence of tokens. the output contain a list of (tokens_seq, tokens_lens) "decode": convert a tensor of tokens to a list of word sequences. "decode_from_list": convert a list of token sequences to a list of word sequences. r{Nz0Tokenizer encoder must have the ind2lab functionrrcsg|]}t|qSrA)r).0rU)ind2labrArB s z*SentencePiece.__call__..rN)devicedecode_from_listcsg|] }j|dqS)rN)r8rprW)r|utt_seq)r<rArBr~sdecodecs6g|]\}}j|d|dqS)NrN)r8rprtolistrW)r|ir) batch_lensr<rArBr~s)rshaperoundr enumerater'rrr8rorrrstorchzerosrTensor)r<batchrr}taskr max_bpe_lenrrtokens words_listr_ bpe_encode bpe_tensorbpe_lensbpe_uttrA)rr}r<rB__call__sF        zSentencePiece.__call__)NNr Fr Nr r r r rTNNr NT)NNr{) __name__ __module__ __qualname____doc__rCr`rfr3r;rrArArArBrs2^ Z(( 2Trcs2t|fddtjD}|S)aFetch list of tokens, can be indexed by token id The resulting list can be used to map id to token. Arguments --------- model_path : str Path to SentencePiece model Returns ------- list Tokens in order by id (can be indexed by id) csg|]}j|qSrA)r8 id_to_piece)r|rmodelrArBr~sz"get_spm_tokens..)r6r7r9ranger8r%) model_pathmappingrArrBget_spm_tokenss rc@seZdZUdZdZeed<dS)$SentencePieceDecoderStreamingContextzGMutable streaming context for a single SentencePiece streaming session.remitted_symbol_countN)rrrrrr__annotations__rArArArBrs  r tokenizerhypscontextreturncCsh|j|gddd}|j}t|jdkr2|jdk}|r(|jdjdr(d|}|jt|j7_|S)uAssuming the tokenizer is sentencepiece, decodes the input hypothesis but avoids incorrectly stripping leading spaces when streaming. Operates on a single hypothesis, not a batch of hypotheses. Normally, the tokenizer always decodes full sentences at a time, with the consequence that the first space in decoding will get removed. However, when streaming, we might be decoding mid-utterance where spaces must not be removed mid-sentence. This function handles this case. e.g. if within the same streaming context, you decode `["▁how", "▁are"]` then `["▁you"]`, the decoder would normally return `"how areyou"` instead of `"how are you"` like this function does. Arguments --------- tokenizer : sentencepiece.SentencePieceProcessor The SentencePiece processor to use for decoding. hyps : list of output token hypotheses List of tokens to decode of any length `>=0`. context : SentencePieceDecoderStreamingContext Mutable streaming context for the sentencepiece decoder, which should be reused across calls for the same decoding stream. Returns ------- str Decoded text. Leading spaces are preserved, except at the start of a transcription. immutable_proto)out_typerru▁rN)rtextrspiecesrpiece startswith)rrrprotorshould_preserve_spacerArArB!spm_decode_preserve_leading_spaces" r)rr raos.pathr dataclassesrtypingr sentencepiecer6rspeechbrain.dataio.dataiorspeechbrain.utilsrspeechbrain.utils.distributedrspeechbrain.utils.loggerrrr4rrrr7rr#rrArArArBs:      U