o ei@sdZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddl Z ddl m Z ddlZddlmZddlmZddlmZdd lmZdd lmZGd d d eZGd ddeZeGdddZGdddeZeGdddZGdddeZdS)al Specifies the inference interfaces for Automatic speech Recognition (ASR) modules. Authors: * Aku Rouhe 2021 * Peter Plantinga 2021 * Loren Lugosch 2020 * Mirco Ravanelli 2020 * Titouan Parcollet 2021 * Abdel Heba 2021 * Andreas Nautsch 2022, 2023 * Pooneh Mousavi 2023 * Sylvain de Langen 2023, 2024 * Adel Moumen 2023, 2024 * Pradnya Kandarkar 2023 N) dataclass)AnyListOptionalTuple)tqdm) Pretrained) split_path)DynChunkTrainConfig)fetch)split_fixed_chunkscsNeZdZdZdgZddgZfddZddZd d Zd d Z d dZ Z S)EncoderDecoderASRafA ready-to-use Encoder-Decoder ASR model The class can be used either to run only the encoder (encode()) to extract features or to run the entire encoder-decoder model (transcribe()) to transcribe speech. The given YAML must contain the fields specified in the *_NEEDED[] lists. Arguments --------- *args : tuple **kwargs : dict Arguments are forwarded to ``Pretrained`` parent class. Example ------- >>> from speechbrain.inference.ASR import EncoderDecoderASR >>> tmpdir = getfixture("tmpdir") >>> asr_model = EncoderDecoderASR.from_hparams( ... source="speechbrain/asr-crdnn-rnnlm-librispeech", ... savedir=tmpdir, ... ) # doctest: +SKIP >>> asr_model.transcribe_file("tests/samples/single-mic/example2.flac") # doctest: +SKIP "MY FATHER HAS REVEALED THE CULPRIT'S NAME" tokenizerencoderdecodercs\tj|i||jj|_d|_d|_t|jdr|jj|_t|jdr,|jj|_dSdS)NFtransducer_beam_searchtransformer_beam_search)super__init__hparamsrrrhasattrselfargskwargs __class__W/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/inference/ASR.pyr@s    zEncoderDecoderASR.__init__cKs@|j|fi|}|d}tdg}|||\}}|dSakTranscribes the given audiofile into a sequence of words. Arguments --------- path : str Path to audio file which to transcribe. **kwargs : dict Arguments forwarded to ``load_audio``. Returns ------- str The audiofile transcription produced by this ASR system. r?) load_audio unsqueezetorchtensortranscribe_batchrpathrwaveformbatch rel_lengthpredicted_wordspredicted_tokensrrrtranscribe_fileJs  z!EncoderDecoderASR.transcribe_filecCsJ|}||j||j}}|j||}|jr#|jj||}|S)aDEncodes the input audio into a sequence of hidden states The waveforms should already be in the model's desired format. You can call: ``normalized = EncoderDecoderASR.normalizer(signal, sample_rate)`` to get a correctly converted signal in most cases. Arguments --------- wavs : torch.Tensor Batch of waveforms [batch, time, channels] or [batch, time] depending on the model. wav_lens : torch.Tensor Lengths of the waveforms relative to the longest one in the batch, tensor of shape [batch]. The longest one should have relative length 1.0 and others len(waveform) / max_length. Used for ignoring padding. Returns ------- torch.Tensor The encoded batch )floattodevicemodsrr transformerencoderwavswav_lens encoder_outrrr encode_batchbs zEncoderDecoderASR.encode_batchcst5|j}||}jr|g}n||g}jj|\}}}}fdd|D}Wd||fS1s z6EncoderDecoderASR.transcribe_batch..N)r#no_gradr/r0r8rr1r)rr5r6r7inputsr,_r+rr?rr%s       z"EncoderDecoderASR.transcribe_batchcC |||Sz=Runs full transcription - note: no gradients through decodingr%rr5r6rrrforward zEncoderDecoderASR.forward) __name__ __module__ __qualname____doc__HPARAMS_NEEDEDMODULES_NEEDEDrr-r8r%rI __classcell__rrrrr #s  (r csVeZdZdZddgZdgZfddZddZd d Zd d Z d dZ ddZ Z S) EncoderASRa'A ready-to-use Encoder ASR model The class can be used either to run only the encoder (encode()) to extract features or to run the entire encoder + decoder function model (transcribe()) to transcribe speech. The given YAML must contain the fields specified in the *_NEEDED[] lists. Arguments --------- *args : tuple **kwargs : dict Arguments are forwarded to ``Pretrained`` parent class. Example ------- >>> from speechbrain.inference.ASR import EncoderASR >>> tmpdir = getfixture("tmpdir") >>> asr_model = EncoderASR.from_hparams( ... source="speechbrain/asr-wav2vec2-commonvoice-fr", ... savedir=tmpdir, ... ) # doctest: +SKIP >>> asr_model.transcribe_file("samples/audio_samples/example_fr.wav") # doctest: +SKIP rdecoding_functionrcs(tj|i||jj|_|dSN)rrrrset_decoding_functionrrrrrs  zEncoderASR.__init__cstjjtjrjj_d Stjjtjjj rtj tj j j r4j jfddttD}ntj tjrJfddtj D}ntdtjdrtjj}d|vrst|d\}}tt||jjd}||d<ni}jjd i|d|i_d Std ) aiSet the decoding function based on the parameters defined in the hyperparameter file. The decoding function is determined by the `decoding_function` specified in the hyperparameter file. It can be either a functools.partial object representing a decoding function or an instance of `speechbrain.decoders.ctc.CTCBaseSearcher` for beam search decoding. Raises: ValueError: If the decoding function is neither a functools.partial nor an instance of speechbrain.decoders.ctc.CTCBaseSearcher. Note: - For greedy decoding (functools.partial), the provided `decoding_function` is assigned directly. - For CTCBeamSearcher decoding, an instance of the specified `decoding_function` is created, and additional parameters are added based on the tokenizer type. csg|]}|qSrr)r=x)ind2labrrr@sz4EncoderASR.set_decoding_function..cr9r)r id_to_piecer=ir?rrr@rAz5The tokenizer must be sentencepiece or CTCTextEncodertest_beam_searchkenlm_model_path)sourcesavedir vocab_listzQThe decoding function must be an instance of speechbrain.decoders.CTCBaseSearcherNr) isinstancerrS functoolspartial issubclass speechbraindecodersctcCTCBaseSearcherrdataiorCTCTextEncoderrWrangelen sentencepieceSentencePieceProcessor vocab_size ValueErrorrr[r strr r^)rr_opt_beam_search_paramsr]flr\r)rWrrrUsV      z EncoderASR.set_decoding_functioncKsD|j|fi|}|d}tdg}|||\}}t|dSr)r!r"r#r$r%rpr&rrrr-s   zEncoderASR.transcribe_filecCs4|}||j||j}}|j||}|S)a=Encodes the input audio into a sequence of hidden states The waveforms should already be in the model's desired format. You can call: ``normalized = EncoderASR.normalizer(signal, sample_rate)`` to get a correctly converted signal in most cases. Arguments --------- wavs : torch.Tensor Batch of waveforms [batch, time, channels] or [batch, time] depending on the model. wav_lens : torch.Tensor Lengths of the waveforms relative to the longest one in the batch, tensor of shape [batch]. The longest one should have relative length 1.0 and others len(waveform) / max_length. Used for ignoring padding. Returns ------- torch.Tensor The encoded batch )r.r/r0r1rr4rrrr80szEncoderASR.encode_batchcst_|j}||}||}tjtj j j }tj jt jr>|r4fdd|D}n%fdd|D}ndd|D}Wd||fSWd||fSWd||fS1sfwY||fS)a{Transcribes the input audio into a sequence of words The waveforms should already be in the model's desired format. You can call: ``normalized = EncoderASR.normalizer(signal, sample_rate)`` to get a correctly converted signal in most cases. Arguments --------- wavs : torch.Tensor Batch of waveforms [batch, time, channels] or [batch, time] depending on the model. wav_lens : torch.Tensor Lengths of the waveforms relative to the longest one in the batch, tensor of shape [batch]. The longest one should have relative length 1.0 and others len(waveform) / max_length. Used for ignoring padding. Returns ------- list Each waveform in the batch transcribed. tensor Each predicted token id. csg|] }dj|qS))joinr decode_ndimr<r?rrr@pz/EncoderASR.transcribe_batch..cr9rr:r<r?rrr@urAcSsg|]}|djqS)r)text)r=hyprrrr@zsN)r#rBr/r0r8rSr`rrdrhrrirrarb)rr5r6r7 predictionsis_ctc_text_encoder_tokenizerr+rr?rr%Ms4          zEncoderASR.transcribe_batchcCrE)zRuns the encoder)r8rHrrrrI~rJzEncoderASR.forward) rKrLrMrNrOrPrrUr-r8r%rIrQrrrrrRs H1rRc@seZdZUdZeed<eed<ejed<dZe e ed<dZ e e ed<dZ e e e ed<dZe e e ed <dZe eed <dZe eed <dS) ASRWhisperSegmentaA single chunk of audio for Whisper ASR streaming. This object is intended to be mutated as streaming progresses and passed across calls to the lower-level APIs such as `encode_chunk`, `decode_chunk`, etc. Attributes ---------- start : float The start time of the audio chunk. end : float The end time of the audio chunk. chunk : torch.Tensor The audio chunk, shape [time, channels]. lang_id : str The language identifier associated with the audio chunk. words : str The predicted words for the audio chunk. tokens : List[int] The predicted tokens for the audio chunk. prompt : List[str] The prompt associated with the audio chunk. avg_log_probs : float The average log probability associated with the prediction. no_speech_prob : float The probability of no speech in the audio chunk. startendchunkNlang_idwordstokensprompt avg_log_probsno_speech_prob)rKrLrMrNr.__annotations__r#Tensorrrrprrrrrrrrrrr{s  r{cs\eZdZdZddgZddgZgdZfddZe d e fd d Z e d ej fd dZ e dej de fddZdddefddZe        d.d e dee dee deeded ed!ed"eefd#d$Z        d.d e dee dee deeded ed!ed"eed%eefd&d'Zd(d)Ze d*d+Zd,d-ZZS)/ WhisperASRaPA ready-to-use Whisper ASR model. The class can be used to run the entire encoder-decoder whisper model. The set of tasks supported are: ``transcribe``, ``translate``, and ``lang_id``. The given YAML must contains the fields specified in the *_NEEDED[] lists. Arguments --------- *args : tuple **kwargs : dict Arguments are forwarded to ``Pretrained`` parent class. Example ------- >>> from speechbrain.inference.ASR import WhisperASR >>> tmpdir = getfixture("tmpdir") >>> asr_model = WhisperASR.from_hparams(source="speechbrain/asr-whisper-medium-commonvoice-it", savedir=tmpdir,) # doctest: +SKIP >>> hyp = asr_model.transcribe_file("speechbrain/asr-whisper-medium-commonvoice-it/example-it.wav") # doctest: +SKIP >>> hyp # doctest: +SKIP buongiorno a tutti e benvenuti a bordo >>> _, probs = asr_model.detect_language_file("speechbrain/asr-whisper-medium-commonvoice-it/example-it.wav") # doctest: +SKIP >>> print(f"Detected language: {max(probs[0], key=probs[0].get)}") # doctest: +SKIP Detected language: it language sample_ratewhisperr) transcribe translatercs"tj|i||jjj|_dSrT)rrrrrrrrrrszWhisperASR.__init__r'cCsD|||jd}|jj|}|jj|\}}||fS)aDetects the language of the given audiofile. This method only works on input_file of 30 seconds or less. Arguments --------- path : str Path to audio file which to transcribe. Returns ------- language_tokens : torch.Tensor The detected language tokens. language_probs : dict The probabilities of the detected language tokens. Raises ------ ValueError If the model doesn't have language tokens. r) r!r.r/r0r"r1r_get_meldetect_language)rr'r5mellanguage_tokenslanguage_probsrrrdetect_language_fileszWhisperASR.detect_language_filewavcCs(|jj|}|jj|\}}||fS)aDetects the language of the given wav Tensor. This method only works on wav files of 30 seconds or less. Arguments --------- wav : torch.tensor Batch of waveforms [batch, time, channels]. Returns ------- language_tokens : torch.Tensor of shape (batch_size,) ids of the most probable language tokens, which appears after the startoftranscript token. language_probs : List[Dict[str, float]] list of dictionaries containing the probability distribution over all languages. Raises ------ ValueError If the model doesn't have language tokens. Example ------- >>> from speechbrain.inference.ASR import WhisperASR >>> import torchaudio >>> tmpdir = getfixture("tmpdir") >>> asr_model = WhisperASR.from_hparams( ... source="speechbrain/asr-whisper-medium-commonvoice-it", ... savedir=tmpdir, ... ) # doctest: +SKIP >>> wav, _ = torchaudio.load("your_audio") # doctest: +SKIP >>> language_tokens, language_probs = asr_model.detect_language(wav) # doctest: +SKIP )r1rrr)rrrrrrrrdetect_language_batchs"z WhisperASR.detect_language_batchrtaskcCsf|jjjg|jd}d}|jjjdus|dkr/|jj|\}}dd|D}|jj|||fS)aJDetects the language of the given mel spectrogram. Arguments --------- mel : torch.tensor Batch of mel spectrograms [batch, time, channels]. task : str The task to perform. Returns ------- language_tokens : Tensor, shape = (n_audio,) ids of the most probable language tokens, which appears after the startoftranscript token. language_probs : List[Dict[str, float]], length = n_audio list of dictionaries containing the probability distribution over all languages. rNrcSsg|] }t||jdqS))key)maxget)r=probsrrrr@%sz/WhisperASR._detect_language..)r1rrshaperrset_lang_tokens)rrr languages lang_probs lang_tokensrrr_detect_languageszWhisperASR._detect_languagestreamertorchaudio.io.StreamReaderframes_per_chunkc#fddtjD}ddt|D}t|dkr)tdt|d|d|dd}j|||jjd dd D]\}| d }| d}|Vq?d S) RFrom a :class:`torchaudio.io.StreamReader`, identifies the audio stream and returns an iterable stream of chunks (after resampling and downmixing to mono). Arguments --------- streamer : torchaudio.io.StreamReader The stream object. Must hold exactly one source stream of an audio type. frames_per_chunk : int The number of frames per chunk. For a streaming model, this should be determined from the DynChunkTrain configuration. Yields ------ chunks from streamer cg|]}|qSrget_src_stream_inforYrrrr@>z0WhisperASR._get_audio_stream..cS"g|] \}}|jdkr||fqSaudio media_typer=rZ stream_inforrrr@C  IExpected stream to have only 1 stream (with any number of channels), got  (with streams: )rfltpr stream_indexrformat num_channelsN rjnum_src_streams enumeraterkroadd_basic_audio_streamaudio_normalizerrstreamsqueezer"rrr stream_infosaudio_stream_infosaudio_stream_indexr~rrr_get_audio_stream)0     zWhisperASR._get_audio_streamN333333?Finitial_promptlogprob_thresholdcondition_on_previous_textverboseuse_torchaudio_streaming chunk_sizec " +s|dur!|jvr|dkrjj|n td|dj| jj} |r6tj |} | | } nj |fi| }| d}t || } tdg}g}d}|durhjjd|}||ng}tt| |dD]\}}|j}jj|}|| }|d | }jj|}||\}}|dkrt||||dd Vqr||d}jj|j||\}}}}|t|dd }|durjjj d|k} |dur||krd } | rt||||dd g||!jjj dd Vqrfdd|D}!t||||d|!d|d||!jjj dd V||d|r9jjj"dkr=t|}qrdS)atTranscribes the given audiofile into a sequence of words. This method supports the following tasks: ``transcribe``, ``translate``, and ``lang_id``. It can process an input audio file longer than 30 seconds by splitting it into chunk_size-second segments. Arguments --------- path : str URI/path to the audio to transcribe. When ``use_torchaudio_streaming`` is ``False``, uses SB fetching to allow fetching from HF or a local file. When ``True``, resolves the URI through ffmpeg, as documented in :class:`torchaudio.io.StreamReader`. task : Optional[str] The task to perform. If None, the default task is the one passed in the Whisper model. initial_prompt : Optional[str] The initial prompt to condition the model on. logprob_threshold : Optional[float] The log probability threshold to continue decoding the current segment. no_speech_threshold : float The threshold to skip decoding segment if the no_speech_prob is higher than this value. condition_on_previous_text : bool If True, the model will be condition on the last 224 tokens. verbose : bool If True, print the transcription of each segment. use_torchaudio_streaming : bool Whether the audio file can be loaded in a streaming fashion. If not, transcription is still performed through chunks of audio, but the entire audio file is fetched and loaded at once. This skips the usual fetching method and instead resolves the URI using torchaudio (via ffmpeg). chunk_size : Optional[int] The size of the chunks to split the audio into. The default chunk size is 30 seconds which corresponds to the maximal length that the model can process in one go. **kwargs : dict Arguments forwarded to ``load_audio`` Yields ------ ASRWhisperSegment A new ASRWhisperSegment instance initialized with the provided parameters. NrzTask z$ not supported. Supported tasks are rr  )disabler)r|r}r~rFrs) r|r}r~rrrrrrc g|] }jj|ddqST)skip_special_tokensrdecodestripr=tr?rrr@z8WhisperASR.transcribe_file_streaming..g?)#TASKSr1rset_taskrorr torchaudioio StreamReaderrr!r"r r#r$rrr3rextendrrr/r0rforward_encoderrr{ set_promptsumrkno_speech_probsitem temperature)"rr'rrrno_speech_thresholdrrrrrnum_frames_per_chunkrsegmentsr(r)r* all_tokensprompt_reset_sinceinitial_prompt_tokensrZsegment mel_segmentr|r}r7rrDrr,scoresr should_skipr+rr?rtranscribe_file_streaming_s8                 z$WhisperASR.transcribe_file_streamingreturnc  Ksrg} |j|f|||||||| d| D]"} | | |r6|dkr$| jn| j} td| jd| jd| q| S)a Run the Whisper model using the specified task on the given audio file and return the ``ASRWhisperSegment`` objects for each segment. This method supports the following tasks: ``transcribe``, ``translate``, and ``lang_id``. It can process an input audio file longer than 30 seconds by splitting it into chunk_size-second segments. Arguments --------- path : str URI/path to the audio to transcribe. When ``use_torchaudio_streaming`` is ``False``, uses SB fetching to allow fetching from HF or a local file. When ``True``, resolves the URI through ffmpeg, as documented in :class:`torchaudio.io.StreamReader`. task : Optional[str] The task to perform. If None, the default task is the one passed in the Whisper model. It can be one of the following: ``transcribe``, ``translate``, ``lang_id``. initial_prompt : Optional[str] The initial prompt to condition the model on. logprob_threshold : Optional[float] The log probability threshold to continue decoding the current segment. no_speech_threshold : float The threshold to skip decoding segment if the no_speech_prob is higher than this value. condition_on_previous_text : bool If True, the model will be condition on the last 224 tokens. verbose : bool If True, print the details of each segment. use_torchaudio_streaming : bool Whether the audio file can be loaded in a streaming fashion. If not, transcription is still performed through chunks of audio, but the entire audio file is fetched and loaded at once. This skips the usual fetching method and instead resolves the URI using torchaudio (via ffmpeg). chunk_size : Optional[int] The size of the chunks to split the audio into. The default chunk size is 30 seconds which corresponds to the maximal length that the model can process in one go. **kwargs : dict Arguments forwarded to ``load_audio`` Returns ------- results : list A list of ``WhisperASRChunk`` objects, each containing the task result. )rrrrrrrrr[zs --> zs] )rappendrrprintr|r})rr'rrrrrrrrrresultswhisper_segmentpredrrrr-s4: zWhisperASR.transcribe_filecCs2|j|jtjd}|jj|}|jj|}|S)aEncodes the input audio into a sequence of hidden states The waveforms should already be in the model's desired format. You can call: ``normalized = EncoderDecoderASR.normalizer(signal, sample_rate)`` to get a correctly converted signal in most cases. Arguments --------- wavs : torch.tensor Batch of waveforms [batch, time, channels]. wav_lens : torch.tensor Lengths of the waveforms relative to the longest one in the batch, tensor of shape [batch]. The longest one should have relative length 1.0 and others len(waveform) / max_length. Used for ignoring padding. Returns ------- torch.tensor The encoded batch )r0dtype)r/r0r#float32r1rrr)rr5r6rr7rrrr8YszWhisperASR.encode_batchcsf|j}||}j||\}}}}fdd|D}jjr/fdd|D}||fS)aNTranscribes the input audio into a sequence of words The waveforms should already be in the model's desired format. You can call: ``normalized = EncoderDecoderASR.normalizer(signal, sample_rate)`` to get a correctly converted signal in most cases. Arguments --------- wavs : torch.tensor Batch of waveforms [batch, time, channels]. wav_lens : torch.tensor Lengths of the waveforms relative to the longest one in the batch, tensor of shape [batch]. The longest one should have relative length 1.0 and others len(waveform) / max_length. Used for ignoring padding. Returns ------- list Each waveform in the batch transcribed. tensor Each predicted token id. crrrrr?rrr@rz/WhisperASR.transcribe_batch..csg|] }j|dqS)r)r normalizesplit)r=rwr?rrr@rv)r.r/r0r8r1rrnormalized_transcripts)rr5r6r7r,rDr+rr?rr%us   zWhisperASR.transcribe_batchcCrErFrGrHrrrrIrJzWhisperASR.forward)NNrrFFFr)rKrLrMrNrOrPrrr#rBrprrrrintrrr.boolrrr{r-r8r%rIrQrrrrrs % 6  *   S (rc@sJeZdZUdZeed< eed< eed< eed< eeeed<dS)ASRStreamingContextaStreaming metadata, initialized by :meth:`~StreamingASR.make_streaming_context` (see there for details on initialization of fields here). This object is intended to be mutate: the same object should be passed across calls as streaming progresses (namely when using the lower-level :meth:`~StreamingASR.encode_chunk`, etc. APIs). Holds some references to opaque streaming contexts, so the context is model-agnostic to an extent.configfea_extractor_contextencoder_contextdecoder_contexttokenizer_contextN) rKrLrMrNr rrrrrrrrrs  rc seZdZdZgdZddgZfddZddd efd d Z d#d e de fddZ d#d e de fddZ d e fddZ d e defddZe d$dedejdeejfddZededejdeeeeeeffdd Z d$dedejdeejfd!d"ZZS)% StreamingASRaA ready-to-use, streaming-capable ASR model. Arguments --------- *args : tuple **kwargs : dict Arguments are forwarded to ``Pretrained`` parent class. Example ------- >>> from speechbrain.inference.ASR import StreamingASR >>> from speechbrain.utils.dynamic_chunk_training import DynChunkTrainConfig >>> tmpdir = getfixture("tmpdir") >>> asr_model = StreamingASR.from_hparams(source="speechbrain/asr-conformer-streaming-librispeech", savedir=tmpdir,) # doctest: +SKIP >>> asr_model.transcribe_file("speechbrain/asr-conformer-streaming-librispeech/test-en.wav", DynChunkTrainConfig(24, 8)) # doctest: +SKIP )fea_streaming_extractormake_decoder_streaming_contextrS make_tokenizer_streaming_contexttokenizer_decode_streamingencproj_enccs"tj|i||jjj|_dSrT)rrrr  properties filter_propsrrrrrszStreamingASR.__init__rrrc#r) rcrrrrYrrrr@rz2StreamingASR._get_audio_stream..cSrrrrrrrr@rrrrrrrrrNrrrrrrrzStreamingASR._get_audio_streamTdynchunktrain_configrcks||}|rtj|}|||}n|j|fi|}|d} t| |}t dg} | |} tj d|f|j dg|j j|} t|| D]} || | | }|dVqKdS)aTranscribes the given audio file into a sequence of words, in a streaming fashion, meaning that text is being yield from this generator, in the form of strings to concatenate. Arguments --------- path : str URI/path to the audio to transcribe. When ``use_torchaudio_streaming`` is ``False``, uses SB fetching to allow fetching from HF or a local file. When ``True``, resolves the URI through ffmpeg, as documented in :class:`torchaudio.io.StreamReader`. dynchunktrain_config : DynChunkTrainConfig Streaming configuration. Sane values and how much time chunks actually represent is model-dependent. use_torchaudio_streaming : bool Whether the audio file can be loaded in a streaming fashion. If not, transcription is still performed through chunks of audio, but the entire audio file is fetched and loaded at once. This skips the usual fetching method and instead resolves the URI using torchaudio (via ffmpeg). **kwargs : dict Arguments forwarded to ``load_audio`` Yields ------ generator of str An iterator yielding transcribed chunks (strings). There is a yield for every chunk, even if the transcribed string for that chunk is an empty string. rr r)r0N)get_chunk_size_framesrrrrr!r"r r#r$make_streaming_contextzerosr0rr !get_recommended_final_chunk_count itertoolschaintranscribe_chunk)rr'rrrrrchunksr(r)r*context final_chunksr~r+rrrrs( '      z&StreamingASR.transcribe_file_streamingcCs$d}||||D]}||7}q |S)aNTranscribes the given audio file into a sequence of words. Arguments --------- path : str URI/path to the audio to transcribe. When ``use_torchaudio_streaming`` is ``False``, uses SB fetching to allow fetching from HF or a local file. When ``True``, resolves the URI through ffmpeg, as documented in :class:`torchaudio.io.StreamReader`. dynchunktrain_config : DynChunkTrainConfig Streaming configuration. Sane values and how much time chunks actually represent is model-dependent. use_torchaudio_streaming : bool Whether the audio file can be loaded in a streaming fashion. If not, transcription is still performed through chunks of audio, but the entire audio file is fetched and loaded at once. This skips the usual fetching method and instead resolves the URI using torchaudio (via ffmpeg). Returns ------- str The audio file transcription produced by this ASR system. rs)r)rr'rrr text_chunkrrrr-Vs   zStreamingASR.transcribe_filecCs*t||jj|jj||jddS)ayCreate a blank streaming context to be passed around for chunk encoding/transcription. Arguments --------- dynchunktrain_config : DynChunkTrainConfig Streaming configuration. Sane values and how much time chunks actually represent is model-dependent. Returns ------- ASRStreamingContext N)rrrrr)rrr rr1r r rrrrrrs z#StreamingASR.make_streaming_contextrcCs|jjd|jS)aReturns the chunk size in actual audio samples, i.e. the exact expected length along the time dimension of an input chunk tensor (as passed to :meth:`~StreamingASR.encode_chunk` and similar low-level streaming functions). Arguments --------- dynchunktrain_config : DynChunkTrainConfig The streaming configuration to determine the chunk frame count of. Returns ------- chunk size r)rstriderrrrrrsz"StreamingASR.get_chunk_size_framesNrr~ chunk_lencCs|dur t|df}|}||j||j}}|jd||jks+J|j j ||j |d}|j j ||j}|j |}|S)a3Encoding of a batch of audio chunks into a batch of encoded sequences. For full speech-to-text offline transcription, use `transcribe_batch` or `transcribe_file`. Must be called over a given context in the correct order of chunks over time. Arguments --------- context : ASRStreamingContext Mutable streaming context object, which must be specified and reused across calls when streaming. You can obtain an initial context by calling `asr.make_streaming_context(config)`. chunk : torch.Tensor The tensor for an audio chunk of shape `[batch size, time]`. The time dimension must strictly match `asr.get_chunk_size_frames(config)`. The waveform is expected to be in the model's expected format (i.e. the sampling rate must be correct). chunk_len : torch.Tensor, optional The relative chunk length tensor of shape `[batch size]`. This is to be used when the audio in one of the chunks of the batch is ending within this chunk. If unspecified, equivalent to `torch.ones((batch_size,))`. Returns ------- torch.Tensor Encoded output, of a model-dependent shape.Nrr)rlengths)r#onessizer.r/r0rrrrr rr1r forward_streamingrr)rrr~rrVrrr encode_chunks( zStreamingASR.encode_chunkrVcsVj|j}jdurfddtt|D_fddt|D}||fS)aDecodes the output of the encoder into tokens and the associated transcription. Must be called over a given context in the correct order of chunks over time. Arguments --------- context : ASRStreamingContext Mutable streaming context object, which should be the same object that was passed to `encode_chunk`. x : torch.Tensor The output of `encode_chunk` for a given chunk. Returns ------- list of str Decoded tokens of length `batch_size`. The decoded strings can be of 0-length. list of list of output token hypotheses List of length `batch_size`, each holding a list of tokens of any length `>=0`. Ncsg|]}jqSr)rr )r=rDr?rrr@rz-StreamingASR.decode_chunk..cs*g|]\}}jjj|j|qSr)rr rr)r=rZ cur_tokensrrrrr@s )rrSrrrjrkr)rrrVrrrr&r decode_chunks    zStreamingASR.decode_chunkcCs^|dur t|df}|}||j||j}}||||}|||\}}|S)aTranscription of a batch of audio chunks into transcribed text. Must be called over a given context in the correct order of chunks over time. Arguments --------- context : ASRStreamingContext Mutable streaming context object, which must be specified and reused across calls when streaming. You can obtain an initial context by calling `asr.make_streaming_context(config)`. chunk : torch.Tensor The tensor for an audio chunk of shape `[batch size, time]`. The time dimension must strictly match `asr.get_chunk_size_frames(config)`. The waveform is expected to be in the model's expected format (i.e. the sampling rate must be correct). chunk_len : torch.Tensor, optional The relative chunk length tensor of shape `[batch size]`. This is to be used when the audio in one of the chunks of the batch is ending within this chunk. If unspecified, equivalent to `torch.ones((batch_size,))`. Returns ------- str Transcribed string for this chunk, might be of length zero. Nr)r#r!r"r.r/r0r$r')rrr~rrVrrDrrrrs#zStreamingASR.transcribe_chunk)TrT)rKrLrMrNrOrPrrrr rrr-rrr#rBrrrr$rrrpr'rrQrrrrrsj  : B )  6 0r)rNrar dataclassesrtypingrrrrrlr#rrrd speechbrain.inference.interfacesrspeechbrain.utils.data_utilsr (speechbrain.utils.dynamic_chunk_trainingr speechbrain.utils.fetchingr speechbrain.utils.streamingr r rRr{rrrrrrrs8        V'{