o %ݫij@sdZddlZddlZddlZddlZddlZddlmZm Z m Z ddl m Z e e ZdZdZdZdZe Gd d d ZGd d d eZGd ddeZddZdS)z[Encoding categorical data as integers Authors * Samuele Cornell 2020 * Aku Rouhe 2020 N)mark_as_loader mark_as_saverregister_checkpoint_hooks) get_loggerzzc@sJeZdZdZdZdZdGddZddZd d Ze d d Z dHddZ dHddZ dIddZ ggd difddZddZddZddZddZefd d!Zd"d#Zd$d%ZdJd'd(ZdJd)d*ZdJd+d,ZdJd-d.Zd/d0Zd1d2Zed3d4Zd5d6Ze dHd7d8Z!d9d:Z"d;d<Z#d=d>Z$d?d@Z%dAdBZ&e'dCdDZ(e'dEdFZ)dS)KCategoricalEncoderaKEncode labels of a discrete set. Used for encoding, e.g., speaker identities in speaker recognition. Given a collection of hashables (e.g a strings) it encodes every unique item to an integer value: ["spk0", "spk1"] --> [0, 1] Internally the correspondence between each label to its index is handled by two dictionaries: lab2ind and ind2lab. The label integer encoding can be generated automatically from a SpeechBrain DynamicItemDataset by specifying the desired entry (e.g., spkid) in the annotation and calling update_from_didataset method: >>> from speechbrain.dataio.encoder import CategoricalEncoder >>> from speechbrain.dataio.dataset import DynamicItemDataset >>> dataset = {"ex_{}".format(x) : {"spkid" : "spk{}".format(x)} for x in range(20)} >>> dataset = DynamicItemDataset(dataset) >>> encoder = CategoricalEncoder() >>> encoder.update_from_didataset(dataset, "spkid") >>> assert len(encoder) == len(dataset) # different speaker for each utterance However can also be updated from an iterable: >>> from speechbrain.dataio.encoder import CategoricalEncoder >>> from speechbrain.dataio.dataset import DynamicItemDataset >>> dataset = ["spk{}".format(x) for x in range(20)] >>> encoder = CategoricalEncoder() >>> encoder.update_from_iterable(dataset) >>> assert len(encoder) == len(dataset) Note ---- In both methods it can be specified it the single element in the iterable or in the dataset should be treated as a sequence or not (default False). If it is a sequence each element in the sequence will be encoded. >>> from speechbrain.dataio.encoder import CategoricalEncoder >>> from speechbrain.dataio.dataset import DynamicItemDataset >>> dataset = [[x+1, x+2] for x in range(20)] >>> encoder = CategoricalEncoder() >>> encoder.ignore_len() >>> encoder.update_from_iterable(dataset, sequence_input=True) >>> assert len(encoder) == 21 # there are only 21 unique elements 1-21 This class offers 4 different methods to explicitly add a label in the internal dicts: add_label, ensure_label, insert_label, enforce_label. add_label and insert_label will raise an error if it is already present in the internal dicts. insert_label, enforce_label allow also to specify the integer value to which the desired label is encoded. Encoding can be performed using 4 different methods: encode_label, encode_sequence, encode_label_torch and encode_sequence_torch. encode_label operate on single labels and simply returns the corresponding integer encoding: >>> from speechbrain.dataio.encoder import CategoricalEncoder >>> from speechbrain.dataio.dataset import DynamicItemDataset >>> dataset = ["spk{}".format(x) for x in range(20)] >>> encoder.update_from_iterable(dataset) >>> 22 >>> encode_sequence on sequences of labels: >>> encoder.encode_sequence(["spk1", "spk19"]) [22, 40] >>> encode_label_torch and encode_sequence_torch return torch tensors >>> encoder.encode_sequence_torch(["spk1", "spk19"]) tensor([22, 40]) >>> Decoding can be performed using decode_torch and decode_ndim methods. >>> encoded = encoder.encode_sequence_torch(["spk1", "spk19"]) >>> encoder.decode_torch(encoded) ['spk1', 'spk19'] >>> decode_ndim is used for multidimensional list or pytorch tensors >>> encoded = encoded.unsqueeze(0).repeat(3, 1) >>> encoder.decode_torch(encoded) [['spk1', 'spk19'], ['spk1', 'spk19'], ['spk1', 'spk19']] >>> In some applications, it can happen that during testing a label which has not been encountered during training is encountered. To handle this out-of-vocabulary problem add_unk can be used. Every out-of-vocab label is mapped to this special label and its corresponding integer encoding. >>> import torch >>> try: ... encoder.encode_label("spk42") ... except KeyError: ... print("spk42 is not in the encoder this raises an error!") spk42 is not in the encoder this raises an error! >>> encoder.add_unk() 41 >>> encoder.encode_label("spk42") 41 >>> returns the encoding This class offers also methods to save and load the internal mappings between labels and tokens using: save and load methods as well as load_or_create. z => z================ rcKs i|_i|_||_||dSN)lab2indind2labstarting_indexhandle_special_labels)selfr special_labelsrN/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/dataio/encoder.py__init__szCategoricalEncoder.__init__cCsd|vr ||ddSdS)z)Handles special labels such as unk_label. unk_labelN)add_unkrrrrrr sz(CategoricalEncoder.handle_special_labelscCs t|jSr )lenr rrrr__len__s zCategoricalEncoder.__len__cCs|}|||S)z,Recreate a previously saved encoder directly)load)clspathobjrrr from_saveds zCategoricalEncoder.from_savedFcCs2|r tj|}nt|}|D]}||qdS)aUpdate from iterator Arguments --------- iterable : iterable Input sequence on which to operate. sequence_input : bool Whether iterable yields sequences of labels or individual labels directly. (default False) N) itertoolschain from_iterableiter ensure_label)riterablesequence_inputlabel_iteratorlabelrrrupdate_from_iterables   z'CategoricalEncoder.update_from_iterablecsN|g|jfdd|D|dWddS1s wYdS)aUpdate from DynamicItemDataset. Arguments --------- didataset : DynamicItemDataset Dataset on which to operate. output_key : str Key in the dataset (in data or a dynamic item) to encode. sequence_input : bool Whether the data yielded with the specified key consists of sequences of labels or individual labels directly. c3s|]}|VqdSr r).0 data_point output_keyrr sz;CategoricalEncoder.update_from_didataset..)r$N)output_keys_asr'r didatasetr+r$rr*rupdate_from_didatasets "z(CategoricalEncoder.update_from_didatasetNc Cst|jr|jj}td|d|rtj|}nt|}t |}| |D]\}} | |kr2|S| |q'|S)aProduce label mapping from iterable based on label counts Used to limit label set size. Arguments --------- iterable : iterable Input sequence on which to operate. sequence_input : bool Whether iterable yields sequences of labels or individual labels directly. False by default. n_most_common : int, None Take at most this many labels as the label set, keeping the most common ones. If None (as by default), take all. min_count : int Don't take labels if they appear less than this many times. Returns ------- collections.Counter The counts of the different labels (unfiltered). z+Limited_labelset_from_iterable called, but zx is not empty. The new labels will be added, i.e. won't overwrite. This is normal if there is e.g. an unk label already.) r __class____name__loggerinforrr r! collectionsCounter most_common add_label) rr#r$ n_most_common min_countclsnamer%countsr&countrrrlimited_labelset_from_iterables"  z1CategoricalEncoder.limited_labelset_from_iterablec Csz^tjjrC||sQ|D]}|||q|D]}|dur#td||||q||| |Wtjj | |dSWtjj | |dSWtjj | |dStjj | |w)zConvenient syntax for creating the encoder conditionally This pattern would be repeated in so many experiments that we decided to add a convenient shortcut for it here. The current version is multi-gpu (DDP) safe. Nz,Provide an output_key for DynamicItemDataset) sbutils distributedif_main_processload_if_possibler' ValueErrorr0r save ddp_barrierr) rrfrom_iterablesfrom_didatasetsr$r+rr#r/rrrload_or_creates2        z!CategoricalEncoder.load_or_createcCs@||jvr|jj}td||}||j|<||j|<|S)aAdd new label to the encoder, at the next free position. Arguments --------- label : hashable Most often labels are str, but anything that can act as dict key is supported. Note that default save/load only supports Python literals. Returns ------- int The index that was used to encode this label. Label already present in )r r2r3KeyError _next_indexr )rr&r<indexrrrr9s   zCategoricalEncoder.add_labelcCs||jvr |j|S||S)aAdd a label if it is not already present. Arguments --------- label : hashable Most often labels are str, but anything that can act as dict key is supported. Note that default save/load only supports Python literals. Returns ------- int The index that was used to encode this label. )r r9)rr&rrrr"4s   zCategoricalEncoder.ensure_labelcCs0||jvr|jj}td||||dS)aAdd a new label, forcing its index to a specific value. If a label already has the specified index, it is moved to the end of the mapping. Arguments --------- label : hashable Most often labels are str, but anything that can act as dict key is supported. Note that default save/load only supports Python literals. index : int The specific index to use. rKN)r r2r3rL enforce_label)rr&rNr<rrr insert_labelHs zCategoricalEncoder.insert_labelc Cst|}||jvr||j|krdS|j|j|=||jvr&|j|}d}nd}||j|<||j|<|rWtdt|d|dt|d|}||j|<||j|<dSdS)a&Make sure label is present and encoded to a particular index. If the label is present but encoded to some other index, it is moved to the given index. If there is already another label at the given index, that label is moved to the next free position. NTFz Moving label z from index z , because z was put at its place.)intr r r4r5reprrM)rr&rN saved_label moving_other new_indexrrrrO]s.       z CategoricalEncoder.enforce_labelcCs||_||S)aAdd label for unknown tokens (out-of-vocab). When asked to encode unknown labels, they can be mapped to this. Arguments --------- unk_label : hashable, optional Most often labels are str, but anything that can act as dict key is supported. Note that default save/load only supports Python literals. Default: . This can be None, as well! Returns ------- int The index that was used to encode this. )rr9)rrrrrrs zCategoricalEncoder.add_unkcCs&|j}||jvr|d7}||jvs|S)z'The index to use for the next new labelr1)r r )rrNrrrrMs   zCategoricalEncoder._next_indexcCs@t|j}|j|votddt|dd|ddDS)a4Check that the set of indices doesn't have gaps For example: If starting index = 1 Continuous: [1,2,3,4] Continuous: [0,1,2] Non-continuous: [2,3,4] Non-continuous: [1,2,4] Returns ------- bool True if continuous. css |] \}}||dkVqdS)r1Nr)r(ijrrrr,s z3CategoricalEncoder.is_continuous..Nr1)sortedr keysr allzip)rindicesrrr is_continuoussz CategoricalEncoder.is_continuousTcCs|z|j|WStyFt|dr |r |j|jYSt|dr/|s/td|dt|ds>|r>td|dtd|dw)aEncode label to int Arguments --------- label : hashable Label to encode, must exist in the mapping. allow_unk : bool If given, that label is not in the label set AND unk_label has been added with add_unk(), allows encoding to unk_label's index. Returns ------- int Corresponding encoded int value. rzUnknown label z=, and explicitly disallowed the use of the existing unk-labelzCannot encode unknown label zN. You have not called add_unk() to add a special unk-label for unknown labels.z+Couldn't and wouldn't encode unknown label .) _assert_lenr rLhasattrrrr& allow_unkrrr encode_labels$     zCategoricalEncoder.encode_labelcCst|||gS)aEncode label to torch.LongTensor. Arguments --------- label : hashable Label to encode, must exist in the mapping. allow_unk : bool If given, that label is not in the label set AND unk_label has been added with add_unk(), allows encoding to unk_label's index. Returns ------- torch.LongTensor Corresponding encoded int value. Tensor shape [1]. )torch LongTensorrdrbrrrencode_label_torchsz%CategoricalEncoder.encode_label_torchcsfdd|DS)aEncode a sequence of labels to list Arguments --------- sequence : iterable Labels to encode, must exist in the mapping. allow_unk : bool If given, that label is not in the label set AND unk_label has been added with add_unk(), allows encoding to unk_label's index. Returns ------- list Corresponding integer labels. cg|]}|qSrrdr(r&rcrrr z6CategoricalEncoder.encode_sequence..)r`rsequencercrrkrencode_sequencesz"CategoricalEncoder.encode_sequencecstfdd|DS)aEncode a sequence of labels to torch.LongTensor Arguments --------- sequence : iterable Labels to encode, must exist in the mapping. allow_unk : bool If given, that label is not in the label set AND unk_label has been added with add_unk(), allows encoding to unk_label's index. Returns ------- torch.LongTensor Corresponding integer labels. Tensor shape [len(sequence)]. crhrrirjrkrrrlrmz.)rerfrnrrkrencode_sequence_torchsz(CategoricalEncoder.encode_sequence_torchcCsV|g}|jdkr|D] }||jt|q |S|D] }|||q|S)aDecodes an arbitrarily nested torch.Tensor to a list of labels. Provided separately because Torch provides clearer introspection, and so doesn't require try-except. Arguments --------- x : torch.Tensor Torch tensor of some integer dtype (Long, int) and any shape to decode. Returns ------- list list of original labels r1)r`ndimappendr rQ decode_torch)rxdecodedelement subtensorrrrrts zCategoricalEncoder.decode_torchcCsN|zg}|D] }|||q |WSty&|jt|YSw)aDecodes an arbitrarily nested iterable to a list of labels. This works for essentially any pythonic iterable (including torch), and also single elements. Arguments --------- x : Any Python list or other iterable or torch.Tensor or a single integer element Returns ------- list, Any ndim list of original labels, or if input was single element, output will be, too. )r`rs decode_ndim TypeErrorr rQ)rrurvrxrrrry2s zCategoricalEncoder.decode_ndimcCs|}|||j|dS)a>Save the categorical encoding for later use and recovery Saving uses a Python literal format, which supports things like tuple labels, but is considered safe to load (unlike e.g. pickle). Arguments --------- path : str, Path Where to save. Will overwrite. N) _get_extras _save_literalr )rrextrasrrrrFMs zCategoricalEncoder.savecCsZ|jr|jj}td|d||\}}}||_||_||td|dS)a!Loads from the given path. CategoricalEncoder uses a Python literal format, which supports things like tuple labels, but is considered safe to load (unlike e.g. pickle). Arguments --------- path : str, Path Where to load from. zLoad called, but zs is not empty. Loaded data will overwrite everything. This is normal if there is e.g. an unk label defined at init.z!Loaded categorical encoding from N) r r2r3r4r5 _load_literalr _set_extrasdebug)rrr<r r r}rrrr\s   zCategoricalEncoder.loadc Csb~z||WdStytd|dYdSttfy0td|dYdSw)aRLoads if possible, returns a bool indicating if loaded or not. Arguments --------- path : str, Path Where to load from. end_of_epoch : bool Whether the checkpoint was end-of-epoch or not. Returns ------- bool : If load was successful. Example ------- >>> encoding_file = getfixture('tmpdir') / "encoding.txt" >>> encoder = CategoricalEncoder() >>> # The idea is in an experiment script to have something like this: >>> if not encoder.load_if_possible(encoding_file): ... encoder.update_from_iterable("abcd") ... encoder.save(encoding_file) >>> # So the first time you run the experiment, the encoding is created. >>> # However, later, the encoding exists: >>> encoder = CategoricalEncoder() >>> encoder.expect_len(4) >>> if not encoder.load_if_possible(encoding_file): ... assert False # We won't get here! >>> encoder.decode_ndim(range(4)) ['a', 'b', 'c', 'd'] z%Would load categorical encoding from z, but file doesn't exist yet.FzH, and file existed but seems to be corrupted or otherwise couldn't load.T)rFileNotFoundErrorr4rrE SyntaxError)rr end_of_epochrrrrDus!    z#CategoricalEncoder.load_if_possiblecCs ||_dS)aSpecify the expected category count. If the category count observed during encoding/decoding does NOT match this, an error will be raised. This can prove useful to detect bugs in scenarios where the encoder is dynamically built using a dataset, but downstream code expects a specific category count (and may silently break otherwise). This can be called anytime and the category count check will only be performed during an actual encoding/decoding task. Arguments --------- expected_len : int The expected final category count, i.e. `len(encoder)`. Example ------- >>> encoder = CategoricalEncoder() >>> encoder.update_from_iterable("abcd") >>> encoder.expect_len(3) >>> encoder.encode_label("a") Traceback (most recent call last): ... RuntimeError: .expect_len(3) was called, but 4 categories found >>> encoder.expect_len(4) >>> encoder.encode_label("a") 0 N expected_len)rrrrr expect_lens zCategoricalEncoder.expect_lencCs d|_dS)zSpecifies that category count shall be ignored at encoding/decoding time. Effectively inhibits the ".expect_len was never called" warning. Prefer :py:meth:`~CategoricalEncoder.expect_len` when the category count is known.Nrrrrr ignore_lens zCategoricalEncoder.ignore_lencCspt|dr#|jdur dSt|}||jkr!td|jd|ddSt|jjdt|d|dS)zIf `expect_len` was called, then check if len(self) matches the expected value. If it does not, raise a RuntimeError. If neither `expect_len` or `ignore_len` were ever called, warn once.rNz .expect_len(z) was called, but z categories foundz9.expect_len was never called: assuming category count of z to be correct! Sanity check your encoder using `.expect_len`. Ensure that downstream code also uses the correct size. If you are sure this does not apply to you, use `.ignore_len`.) rarr RuntimeErrorr4 warning_oncer2r3r)rreal_lenrrrr`s$     zCategoricalEncoder._assert_lencCs"d|ji}t|dr|j|d<|S)zzOverride this to provide any additional things to save Call super()._get_extras() to get the base extras r r)r rarrr}rrrr{s   zCategoricalEncoder._get_extrascCs d|vr |d|_|d|_dS)zvOverride this to e.g. load any extras needed Call super()._set_extras(extras) to set the base extras rr N)rr rrrrrs zCategoricalEncoder._set_extrascCst|dddE}|D]\}}|t|tjt|dq |tj|D]\}}|t|tjt|dq+|WddS1sOwYdS)z+Save which is compatible with _load_literalwutf-8encoding N) openitemswriterRrVALUE_SEPARATORstrEXTRAS_SEPARATORflush)rr r}fr&indkeyvaluerrrr|s0  "z CategoricalEncoder._save_literalc Csi}i}i}t|ddP}|D]&}|tjkrn|jtjdd\}}t|}t|}|||<|||<q|D]}|jtjdd\} } t| } t| } | || <q8Wdn1s_wY|||fS)zLoad which supports Python literals as keys. This is considered safe for user input, as well (unlike e.g. pickle). rrr1)maxsplitN) rrrstripsplitrrQast literal_eval) rr r r}rlineliteralrr& literal_key literal_valuerrrrrr~s0        z CategoricalEncoder._load_literal)r)F)FNr1T)*r3 __module__ __qualname____doc__rrrr r classmethodrr'r0r?rJr9r"rPrO DEFAULT_UNKrrMr^rdrgrprqrtryrrFrrrDrrr`r{r staticmethodr|r~rrrrr s^g      1 # #  '    2   rcseZdZdZfddZd"fdd Zd"fdd Z d#fd d Zee fd dZ ee dd fddZ ddZ ddZ ddZddZddZddZfddZfd d!ZZS)$ TextEncodera CategoricalEncoder subclass which offers specific methods for encoding text and handle special tokens for training of sequence to sequence models. In detail, aside special token already present in CategoricalEncoder for handling out-of-vocab tokens here special methods to handle beginning of sequence and tokens are defined. Note: update_from_iterable and update_from_didataset here have as default sequence_input=True because it is assumed that this encoder is used on iterables of strings: e.g. >>> from speechbrain.dataio.encoder import TextEncoder >>> dataset = [["encode", "this", "textencoder"], ["foo", "bar"]] >>> encoder = TextEncoder() >>> encoder.update_from_iterable(dataset) >>> encoder.expect_len(5) >>> encoder.encode_label("this") 1 >>> encoder.add_unk() 5 >>> encoder.expect_len(6) >>> encoder.encode_sequence(["this", "out-of-vocab"]) [1, 5] >>> Two methods can be used to add and to the internal dicts: insert_bos_eos, add_bos_eos. >>> encoder.add_bos_eos() >>> encoder.expect_len(8) >>> encoder.lab2ind[encoder.eos_label] 7 >>> add_bos_eos adds the special tokens at the end of the dict indexes >>> encoder = TextEncoder() >>> encoder.update_from_iterable(dataset) >>> encoder.insert_bos_eos(bos_index=0, eos_index=1) >>> encoder.expect_len(7) >>> encoder.lab2ind[encoder.eos_label] 1 >>> insert_bos_eos allows to specify whose index will correspond to each of them. Note that you can also specify the same integer encoding for both. Four methods can be used to prepend and append . prepend_bos_label and append_eos_label add respectively the and string tokens to the input sequence >>> words = ["foo", "bar"] >>> encoder.prepend_bos_label(words) ['', 'foo', 'bar'] >>> encoder.append_eos_label(words) ['foo', 'bar', ''] prepend_bos_index and append_eos_index add respectively the and indexes to the input encoded sequence. >>> words = ["foo", "bar"] >>> encoded = encoder.encode_sequence(words) >>> encoder.prepend_bos_index(encoded) [0, 3, 4] >>> encoder.append_eos_index(encoded) [3, 4, 1] csVt|d|vrd|vr|jdd|d|dddSd|vs%d|vr)tddS)z+Handles special labels such as bos and eos. bos_label eos_labelrr)rr bos_index eos_indexz.Only BOS or EOS specified. Need both for init.N)superr insert_bos_eosrzrr2rrr us  z!TextEncoder.handle_special_labelsTcst||S*Change default for sequence_input to True.)rr')rr#r$rrrr'sz TextEncoder.update_from_iterablecst|||Sr)rr0r.rrrr0sz!TextEncoder.update_from_didatasetNr1cstj|ddddS)rTNr1)r$r:r;)rr?)rr#r$r:r;rrrr?sz*TextEncoder.limited_labelset_from_iterablecCsB||krtd||n ||||||_||_dS)aHAdd sentence boundary markers in the label set. If the beginning-of-sentence and end-of-sentence markers are the same, will just use one sentence-boundary label. This method adds to the end of the index, rather than at the beginning, like insert_bos_eos. Arguments --------- bos_label : hashable Beginning-of-sentence label, any label. eos_label : hashable End-of-sentence label, any label. If set to the same label as bos_label, will just use one sentence-boundary label. IBOS and EOS labels are the same so using just one sentence boundary labelN)r4rr9rr)rrrrrr add_bos_eoss    zTextEncoder.add_bos_eosrcCsl||krtd|||n||||dur(td|||dn|||||_||_dS)aInsert sentence boundary markers in the label set. If the beginning-of-sentence and end-of-sentence markers are the same, will just use one sentence-boundary label. Arguments --------- bos_label : hashable Beginning-of-sentence label, any label eos_label : hashable End-of-sentence label, any label. If set to the same label as bos_label, will just use one sentence-boundary label. bos_index : int Where to insert bos_label. eos_index = bos_index + 1 eos_index : optional, int Where to insert eos_label. Default: eos_index = bos_index + 1 rNz,EOS label not specified, using BOS label + 1r1)r4rrPrr)rrrrrrrrrs    zTextEncoder.insert_bos_eoscCt|ds td||jS)(Returns the index to which blank encodesrzBOS label is not set!)rarrdrrrrr get_bos_index  zTextEncoder.get_bos_indexcCr)rrzEOS label is not set!)rarrdrrrrr get_eos_indexrzTextEncoder.get_eos_indexcCs"t|ds td|jgt|S)z/Returns a list version of x, with BOS prependedr*BOS label has not been added to label set!)rarLrlistrrurrrprepend_bos_label zTextEncoder.prepend_bos_labelcCsTt|ds tdt|rt|j|jg}t||gS|j|jgt|S)znReturns a list version of x, with BOS index prepended. If the input is a tensor, a tensor is returned.rr) rarLre is_tensorTensorr rcatr)rrubos_indrrrprepend_bos_index  zTextEncoder.prepend_bos_indexcCs"t|ds tdt||jgS)z/Returns a list version of x, with EOS appended.r*EOS label has not been added to label set!)rarLrrrrrrappend_eos_labelrzTextEncoder.append_eos_labelcCsTt|ds tdt|rt|j|jg}t||gSt||j|jgS)zmReturns a list version of x, with EOS index appended. If the input is a tensor, a tensor is returned.rr) rarLrerrr rrr)rrueos_indrrrappend_eos_indexrzTextEncoder.append_eos_indexcs6t}t|dr|j|d<t|dr|j|d<|SNrr)rr{rarrrrrrr{s     zTextEncoder._get_extrascs8t|d|vr|d|_d|vr|d|_dSdSr)rrrrrrrrrs  zTextEncoder._set_extrasr)TNr1)r3rrrr r'r0r? DEFAULT_BOS DEFAULT_EOSrrrrrrrrr{r __classcell__rrrrr3s. A  # (  rcsneZdZdZfddZefddZedfddZd d Zdd d Z dddZ fddZ fddZ Z S)CTCTextEncoderafSubclass of TextEncoder which also provides methods to handle CTC blank token. add_blank and insert_blank can be used to add special token to the encoder state. >>> from speechbrain.dataio.encoder import CTCTextEncoder >>> chars = ["a", "b", "c", "d"] >>> encoder = CTCTextEncoder() >>> encoder.update_from_iterable(chars) >>> encoder.add_blank() >>> encoder.expect_len(5) >>> encoder.encode_sequence(chars) [0, 1, 2, 3] >>> encoder.get_blank_index() 4 >>> encoder.decode_ndim([0, 1, 2, 3, 4]) ['a', 'b', 'c', 'd', ''] collapse_labels and collapse_indices_ndim can be used to apply CTC collapsing rules: >>> encoder.collapse_labels(["a", "a", "b", "c", "d"]) ['a', 'b', 'c', 'd'] >>> encoder.collapse_indices_ndim([4, 4, 0, 1, 2, 3, 4, 4]) # 4 is [0, 1, 2, 3] cs(d|vr |j|ddt|dS)z&Handles special labels such as blanks. blank_label)rNN) insert_blankrr rrrrr 9sz$CTCTextEncoder.handle_special_labelscCs||||_dS)zAdd blank symbol to labelset.N)r9r)rrrrr add_blankFs  zCTCTextEncoder.add_blankrcCs|||||_dS)z(Insert blank symbol at a given labelset.N)rPr)rrrNrrrrKs  zCTCTextEncoder.insert_blankcCr)z)Returns the index to which blank encodes.rzBlank label is not set!)rarrdrrrrrget_blank_indexPrzCTCTextEncoder.get_blank_indexTcs@tds td|rfddtDSfddDS)aApplies the CTC collapsing rules on one label sequence. Arguments --------- x : iterable Label sequence on which to operate. merge_repeats : bool Whether to merge repeated labels before removing blanks. In the basic CTC label topology, repeated labels are merged. However, in RNN-T, they are not. Returns ------- list List of labels with collapsing rules applied. rBlank label has not been addedcs6g|]\}}|dks||dkr|jkr|qSrr1r)r(rVr&rrrrlls "z2CTCTextEncoder.collapse_labels..csg|] }|jkr|qSrrrjrrrrlrs)rarL enumerate)rru merge_repeatsrrrcollapse_labelsVs  zCTCTextEncoder.collapse_labelsc st|ds tdg}D]}z ||||Wq ty$Ynw|S|j|j|r;fddtDSfddDS)aApplies the CTC collapsing rules on arbitrarily label sequence. Arguments --------- x : iterable Label sequence on which to operate. merge_repeats : bool Whether to merge repeated labels before removing blanks. In the basic CTC label topology, repeated labels are merged. However, in RNN-T, they are not. Returns ------- list List of labels with collapsing rules applied. rrcs4g|]\}}|dks||dkr|kr|qSrr)r(rVrN blank_indexrurrrls  z8CTCTextEncoder.collapse_indices_ndim..csg|]}|kr|qSrr)r(rN)rrrrlrm)rarLrscollapse_indices_ndimrzr rr)rrur collapsedrxrrrrts$     z$CTCTextEncoder.collapse_indices_ndimcs"t}t|dr|j|d<|SNr)rr{rarrrrrr{s   zCTCTextEncoder._get_extrascs&t|d|vr|d|_dSdSr)rrrrrrrrs zCTCTextEncoder._set_extrasr)r3rrrr DEFAULT_BLANKrrrrrr{rrrrrrrs     *rcCst}||t|jS)aLoads the encoder tokens from a pretrained model. This method is useful when you used with a pretrained HF model. It will load the tokens in the yaml and then you will be able to instantiate any CTCBaseSearcher directly in the YAML file. Arguments --------- model_path : str, Path Path to the pretrained model. Returns ------- list List of tokens. )rrrr rZ) model_path label_encoderrrrload_text_encoder_tokenss r)rrr6rre speechbrainr@speechbrain.utils.checkpointsrrrspeechbrain.utils.loggerrr3r4rrrrrrrrrrrrs4 l