o ei-@sdZddlZddlZddlmZmZmZddlZddlm Z ddl m Z ddl m Z e e ZGdd d ejZGd d d eZdS) aaGraph compiler class to create, store, and use k2 decoding graphs in speechbrain. Limits the output words to the ones in the lexicon. This code is an extension, and therefore heavily inspired or taken from icefall's (https://github.com/k2-fsa/icefall) graph compiler. Authors: * Pierre Champion 2023 * Zeyu Zhao 2023 * Georgios Karakasidis 2023 N)ListOptionalTuple) get_logger)k2)lexiconc @seZdZdZejdejfddZejde j fddZ ejddZ ej dd e ed edeejejffd d ZddeedefddZ ddeedefddZdS) GraphCompilerzR This abstract class is used to compile graphs for training and decoding. returncCdS)z@ Return the topology used to compile the graph. Nselfr r g/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/k2_integration/graph_compiler.pytopo zGraphCompiler.topocCr )z? Return the lexicon used to compile the graph. Nr r r r rr'rzGraphCompiler.lexiconcCr )z> Return the device used to compile the graph. Nr r r r rdevice.rzGraphCompiler.deviceTtexts is_trainingcCr )a Compile the graph for the given texts. Arguments --------- texts: List[str] A list of strings. Each string contains a sentence for an utterance. A sentence consists of spaces separated words. An example `texts` looks like: ['hello world', 'CTC training with k2'] is_training: bool Indictating whether this is for training or not (OOV warning in training). Returns ------- graph: GraphCompiler An FsaVec, the composition result of `self.ctc_topo` and the transcript FSA. target_lens: Torch.tensor It is an long tensor of shape (batch,). It contains lengths of each target sequence. Nr )rrrr r rcompile5szGraphCompiler.compileNF cache_dircachecCs0tdt|jjd}|jd}tt |j dtt |j d}|rQ|durQ|d|d}t j |rQtd|dtjtj|dd }|Std tj||d d }td t|}tdt|}td|j |dur|d|d}td|t|||S)a Compile the decoding graph by composing H with L. This is for decoding without language model. Arguments --------- cache_dir: str The path to store the composition in a .pt format. cache: bool Whether or not to load the composition from the .pt format (in the cache_dir dir). Returns ------- HL: k2.Fsa The HL composition Arc sorting LcpurNz/.HL_.ptz Loading HL '\' from its cached .pt format. Set 'caching: False' in the yaml if this is not what you want. map_locationzComposing H and Ltokens inner_labelsz Connecting HLzArc sorting HLz HL.shape: zCaching HL to: )loggerinforarc_sortrLtorstrhashshapeospathexistswarningFsa from_dicttorchloadcomposeconnectdebugsaveas_dict)rrrr$H file_hashr*HLr r r compile_HLSs0  $        zGraphCompiler.compile_HLc Cstdt|jjd}t|d}|jd}tt |j dtt |j dtt |j d}|rb|durb|d|d}t j |rbtd|dtjtj|dd }|Std t||} td t| } td t| } td t| } |j| } t| } t| } | jd| _tdt| } tdtj|| dd}tdt|}tdt|}td|j |dur|d|d}td|t|||S)a Compile the decoding graph by composing H with LG. This is for decoding with small language model. Arguments --------- G: k2.Fsa The language model FSA. cache_dir: str The path to store the composition in a .pt format. cache: bool Whether or not to load the composition from the .pt format (in the cache_dir dir). Returns ------- HL: k2.Fsa The HLG composition rrrNz/.HLG_rz Loading HLG 'rrzIntersecting L and Gz Connecting LGzDeterminizing LGz"Connecting LG after k2.determinizezArc sorting LGzComposing H and LGrrzConnecting HLGzArc sorting HLGz HLG.shape: zCaching HLG to: )r!r"rr#r L_disambigr%rr&r'r(r)r*r+r,r-r.r/r0r1r2 determinizeremove_LG_disambig_symbolsremove_epsilon aux_labelsremove_values_eqr3r4r5) rGrrr$r6r7r*HLGLGr r r compile_HLGsX                       zGraphCompiler.compile_HLGT)NF)__name__ __module__ __qualname____doc__abcabstractpropertyrr-rrLexiconrabstractmethodrr&boolrr/Tensorrrr9rCr r r rr s0  4r c @s|eZdZdZ ddejdejdefddZ e dd Z e d d Ze d d Z dde e dedeejejffddZdS)CtcGraphCompilera This class is used to compile decoding graphs for CTC training. Arguments --------- _lexicon: Lexicon It is built from `data/lang/lexicon.txt`. device: torch.device The device to use for operations compiling transcripts to FSAs. need_repeat_flag: bool If True, will add an attribute named `_is_repeat_token_` to ctc_topo indicating whether this token is a repeat token in ctc graph. This attribute is needed to implement delay-penalty for phone-based ctc loss. See https://github.com/k2-fsa/k2/pull/1086 for more details. Note: The above change MUST be included in k2 to enable this flag so make sure you have an up-to-date version. Example ------- >>> import torch >>> from speechbrain.k2_integration.losses import ctc_k2 >>> from speechbrain.k2_integration.graph_compiler import CtcGraphCompiler >>> from speechbrain.k2_integration.lexicon import Lexicon >>> from speechbrain.k2_integration.prepare_lang import prepare_lang >>> # Create a random batch of log-probs >>> batch_size = 4 >>> log_probs = torch.randn(batch_size, 100, 30) >>> log_probs.requires_grad = True >>> # Assume all utterances have the same length so no padding was needed. >>> input_lens = torch.ones(batch_size) >>> # Create a small lexicon containing only two words and write it to a file. >>> lang_tmpdir = getfixture('tmpdir') >>> lexicon_sample = "hello h e l l o\nworld w o r l d\n " >>> lexicon_file = lang_tmpdir.join("lexicon.txt") >>> lexicon_file.write(lexicon_sample) >>> # Create a lang directory with the lexicon and L.pt, L_inv.pt, L_disambig.pt >>> prepare_lang(lang_tmpdir) >>> # Create a lexicon object >>> lexicon = Lexicon(lang_tmpdir) >>> # Create a random decoding graph >>> graph = CtcGraphCompiler( ... lexicon, ... log_probs.device, ... ) >>> isinstance(graph.topo, k2.Fsa) True F_lexiconrneed_repeat_flagcCsz||_||_|j||jjjdusJ|jt|jj}t j |dd}|||_ |r;|j j |j j k|j _ dSdS)NF)modified)_devicerPrr%L_inv requires_gradr#maxrrctc_topolabelsr>_is_repeat_token_)rrPrrQ max_token_idrWr r r__init__ s     zCtcGraphCompiler.__init__cC|jS)z& Return the ctc_topo. )rWr r r rr"zCtcGraphCompiler.topocCr\)z% Return the lexicon. )rPr r r rr)r]zCtcGraphCompiler.lexiconcCr\)z,Return the device used for compiling graphs.)rSr r r rr0szCtcGraphCompiler.deviceTrrr c Cs|jj||d}|jj||d}dd|D}tjdd|Dtjd}tt||j }tj |jj |dd}| } t | } t| } t | } tj|j| dd} | jdus[J| |fS)a3 Build decoding graphs by composing ctc_topo with given transcripts. Arguments --------- texts: List[str] A list of strings. Each string contains a sentence for an utterance. A sentence consists of spaces separated words. An example `texts` looks like: ['hello world', 'CTC training with k2'] is_training: bool Indictating whether this is for training or not (OOV warning in training). Returns ------- graph: GraphCompiler An FsaVec, the composition result of `self.ctc_topo` and the transcript FSA. target_lens: Torch.tensor It is an long tensor of shape (batch,). It contains lengths of each target sequence. )log_unknown_warningcSsg|]}t|gqSr )sum).0innerr r r Zsz,CtcGraphCompiler.compile..cSsg|]}t|qSr )len)r`tr r rrb]s)dtypeF)treat_epsilons_specially)rtexts_to_word_idstexts_to_token_idsr/tensorlongradd_epsilon_self_loops linear_fsar intersectrTinvert_r#!remove_epsilon_and_add_self_loopsr1rWrU) rrrword_idx word2tids sentence_ids target_lensword_fsa_with_self_loopsfsaans_fsatranscript_fsafsa_with_self_loopsgraphr r rr5s:   zCtcGraphCompiler.compileN)FrD)rErFrGrHrrKr/rrMr[propertyrrr&rrr-rNrr r r rrOs07    rO)rHrIr)typingrrrr/speechbrain.utils.loggerrrrrEr!ABCr rOr r r rs    =