o }oi_ @shddlZddlZddlmZddlmZddlmZmZddl Z ddl Z ddl Z ddl mZddlmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddl m!Z!ddl"m#Z#m$Z$m%Z%dZ&ddZ'dJddZ(dJddZ)dKde*de+fddZ,ddZ-GdddZ.Gd d!d!e!Z/d"d#Z0Gd$d%d%e!Z1Gd&d'd'Z2Gd(d)d)Z3Gd*d+d+e2Z4Gd,d-d-e3Z5Gd.d/d/e5Z6Gd0d1d1e5Z7Gd2d3d3Z8Gd4d5d5e3Z9Gd6d7d7e3Z:Gd8d9d9e3Z;eGd:d;d;Zd?d?Z>dLdAe+eBdBe*dCe?e j@e*ffdDdEZAGdFdGdGeZBGdHdIdIeZCdS)MN) dataclass)Path) NamedTupleOptional) OmegaConf) pad_sequence) DataLoaderDataset)PromptedAudioToTextMiniBatch)ASRModel)StreamingEncoder)normalize_batch) get_samples) rnnt_utils)IterableDataset) LengthsTypeMelSpectrogramType NeuralTypecCsxt|}|dkr8t|d}t|D]'}t|D]}|dkr(t|ddddt|||ddqtqdSdS)z Print an alignment matrix of the shape (m + 1, n + 1) Args: alignment: An integer alignment matrix of shape (m + 1, n + 1) r4dz | )endN)lenrangeprint) alignmentmnijr d/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/streaming_utils.pyprint_alignment*s   r"cCs$|duri}||d<t||dS)z Writes out the LCS alignment to a file, along with any extras provided. Args: alignment: An alignment matrix of shape [m + 1, n + 1] filepath: str filepath extras: Optional dictionary of items to preserve. Nr)torchsave)rfilepathextrasr r r!write_lcs_alignment_to_pickle<s r'cst|}t|fddt|dD}d}gd}t|dD]P}tdD]G}|dks3|dkr:d|||<q)||d||dkrj||d|dd|||<||||kri|||}|||g}q)d|||<q)q!|dd\}}||k} | r|d} | dkr|dkr|dkr||d|ddkr| d8} |d|d}}n|d|d| d}}} n | dkr|dkr|dksnd} } |} d}d}d}t|ddD]$}tddD]}|||| kr|| kr|||} |} |} |}qq| tkr | }d}d|d<n| d}|d}d}d}t||dD]4}d}t|||dD]}|dkrG|||dkrAd}q+d||}q+||7}|d7}qd}||7}| dkr|dkr|| |dkrt|d8}|d7}|dkr|d7}| d8} |d8}| dkr|dksctd| }td|}|||d<||d<||d<|dur| |||d }t|||d td |||fS) a Longest Common Subsequence merge algorithm for aligning two consecutive buffers. Base alignment construction algorithm is Longest Common Subsequence (reffered to as LCS hear after) LCS Merge algorithm looks at two chunks i-1 and i, determins the aligned overlap at the end of i-1 and beginning of ith chunk, and then clips the subsegment of the ith chunk. Assumption is that the two chunks are consecutive chunks, and there exists at least small overlap acoustically. It is a sub-word token merge algorithm, operating on the abstract notion of integer ids representing the subword ids. It is independent of text or character encoding. Since the algorithm is merge based, and depends on consecutive buffers, the very first buffer is processes using the "middle tokens" algorithm. It requires a delay of some number of tokens such that: lcs_delay = math.floor(((total_buffer_in_secs - chunk_len_in_sec)) / model_stride_in_secs) Total cost of the model is O(m_{i-1} * n_{i}) where (m, n) represents the number of subword ids of the buffer. Args: X: The subset of the previous chunk i-1, sliced such X = X[-(lcs_delay * max_steps_per_timestep):] Therefore there can be at most lcs_delay * max_steps_per_timestep symbols for X, preserving computation. Y: The entire current chunk i. filepath: Optional filepath to save the LCS alignment matrix for later introspection. Returns: A tuple containing - - i: Start index of alignment along the i-1 chunk. - j: Start index of alignment along the ith chunk. - slice_len: number of tokens to slice off from the ith chunk. The LCS alignment matrix itself (shape m + 1, n + 1) cs"g|] }ddtdDqS)cSg|]}dqSrr .0_r r r! sz?longest_common_subsequence_merge...r)rr*rr r!r-ss"z4longest_common_subsequence_merge..rr)rrrN)is_complete_mergeXY slice_idx)r%r&zWrote alignemnt to :)rrMIN_MERGE_SUBSEQUENCE_LENmaxr'r)r3r4r%rLCSuffresult result_idxrrr2lengthmax_j max_j_idx i_partial j_partialj_skip slice_counti_idxj_idxi_tempj_tempj_exp j_any_skipr&r r/r! longest_common_subsequence_mergeLs%               rHmax_steps_per_timestepr%c Cs|dkr ||7}|St|dkr||7}|St||}|| d}t|||d\}} |d|d} || d}||7}|S)a Merges the new text from the current frame with the previous text contained in the buffer. The alignment is based on a Longest Common Subsequence algorithm, with some additional heuristics leveraging the notion that the chunk size is >= the context window. In case this assumptio is violated, the results of the merge will be incorrect (or at least obtain worse WER overall). rrN)r%r1)rintrH) bufferdatadelaymodelrJr% search_size buffer_slicelcs_idx lcs_alignmentr5r r r!lcs_alignment_merge_buffer's    rTcCs8|dkr ||7}|St|dkr||7}|S||7}|S)a Merges the new text from the current frame with the previous text contained in the buffer. The alignment is based on a Longest Common Subsequence algorithm, with some additional heuristics leveraging the notion that the chunk size is >= the context window. In case this assumptio is violated, the results of the merge will be incorrect (or at least obtain worse WER overall). rr)r)rLrM timestepsrOr r r!inplace_buffer_mergeJs  rVc@sPeZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ dS)StreamingFeatureBuffererz Class to append each feature frame to a buffer and return an array of buffers. This class is designed to perform a real-life streaming decoding where only a single chunk is provided at each step of a streaming pipeline. cCs6d|_t|jdr|jjrd|_nd|_||_|jj|_|jjj |_ ||_ |jjj }t ||j|_t ||j|_||_t ||}|jjj|_tt |j|j|_tj|j|gtjd|j|_t |||_||_|t|j}t|jdd|j_ d|j_!d|j_ t"#|j|_$|j$%|j&d S) aL Args: asr_model: Reference to the asr model instance for which the feature needs to be created chunk_size (float): Duration of the new chunk of audio buffer_size (float): Size of the total audio in seconds maintained in the buffer h㈵>log(\0dtypeFrNoneN)' NORM_CONSTANThasattr preprocessorrYZERO_LEVEL_SPEC_DB_VAL asr_modelcfg sample_ratesr normalizemodel_normalize_type chunk_size window_striderKn_chunk_look_backn_chunk_samples buffer_sizefeaturesn_featr#zeros sample_bufferonesfloat32rLfeature_chunk_lenfeature_buffer_lenresetcopydeepcopyr set_structditherpad_tor from_config_dictraw_preprocessortodevice)selfrcrirmtimestep_durationtotal_buffer_lenrdr r r!__init__hs4       z!StreamingFeatureBufferer.__init__cCs^tj|jjtjd|j|_g|_tt|j |j |_ tj|j |j gtjd|j|_dS)9 Reset frame_history and decoder's state r\N)r#rrrLshapersrb frame_buffersrprKrmrfrqrorufeature_bufferrr r r!rvs zStreamingFeatureBufferer.resetcCs<|j|jd|jd|j <||j|j d<dS)z Add time-series audio signal to `sample_buffer` Args: chunk (Tensor): Tensor filled with time-series audio signal N)rqrlclone)rchunkr r r!_add_chunk_to_buffers"z-StreamingFeatureBufferer._add_chunk_to_buffercCsT|jdd|jdf|jddd|j f<||jdd|j df<dS)z> Add an extracted feature to `feature_buffer` N)rrtr)r feat_chunkr r r!_update_feature_buffers2"z/StreamingFeatureBufferer._update_feature_buffercCs|jSN)rrr r r!get_raw_feature_buffersz/StreamingFeatureBufferer.get_raw_feature_buffercCs6t|jdtt|jg|jd\}}}|dS)Nrxseq_lennormalize_type)r r unsqueezer#tensorrrhsqueeze)rnormalized_bufferr,r r r!get_normalized_feature_buffers   z6StreamingFeatureBufferer.get_normalized_feature_buffercCs|jd|j|j }|jj}|d|}t|j dg|}|j ||d\}}| }| |dd|j dfdS)zU Extract features from the time-series audio buffer `sample_buffer`. Nrr input_signalr;)rqrlrkrcr unsqueeze_r~r#Tensorrr}rrrt)rsamplesr audio_signalaudio_signal_lenrn features_lenr r r!_convert_buffer_to_featuress "z4StreamingFeatureBufferer._convert_buffer_to_featurescCslt||jkrtd|jdt||jkr+tj|jtjd}||d|jd<|}|||dS)z Update time-series signal `chunk` to the buffer then generate features out of the signal in the audio buffer. Args: chunk (Tensor): Tensor filled with time-series audio signal zchunk should be of length z or lessr\Nr) rrl ValueErrorr#rprsrrr)rr temp_chunkr r r!update_feature_buffers   z.StreamingFeatureBufferer.update_feature_bufferN) __name__ __module__ __qualname____doc__rrvrrrrrrr r r r!rWas*   rWc@s&eZdZd ddZddZddZdS) AudioFeatureIteratorTc Cs||_||_d|_d|_d|_||_|jd}|||_t |j d |}t |jj dg |}|||d\|_|_|j|_dS)NrTrjr)_samples _frame_len_startoutputcountpad_to_frame_len_cfg_feature_frame_lenr# from_numpyrr~rr _features _features_lenr) rr frame_lenrarrrrrr r r!rs  zAudioFeatureIterator.__init__cC|Srr rr r r!__iter__zAudioFeatureIterator.__iter__cCs|jstt|j|j}||jdkr&|jdd|j|f}||_nF|js;|jdd|j|jdf}n.t j |jj dt|jgdd}|jdd|j|jdf}||ddd|j df<d|_|j d7_ |S)Nrrsr\rF) r StopIterationrKrrrrcpurnprprr)rlastframesegmentr r r!__next__s$ "zAudioFeatureIterator.__next__N)T)rrrrrrr r r r!rs  rc Cst|\}}d}|ddu}|rt|}g}|D]!\}}|r;|}||kr6d||f}tjj||}||q|rLt|}t|}||fSd\}}||fS)arcollate batch of audio sig, audio len, tokens, tokens len Args: batch (Optional[FloatTensor], Optional[LongTensor], LongTensor, LongTensor): A tuple of tuples of signal, signal lengths, encoded tokens, and encoded tokens length. This collate func assumes the signals are 1d torch tensors (i.e. mono audio). rN)NN) zipr7itemr#nn functionalpadappendstack) batchr, audio_lengths max_audio_len has_audiorsigsig_lenrr r r!speech_collate_fns(        rcsHeZdZeddZfddZddZddZd d Zd d Z Z S) AudioBuffersDataLayercCstdtttdtdS)N)BDTr)processed_signalprocessed_length)rrtuplerrr r r! output_types(s z"AudioBuffersDataLayer.output_typescstdSrsuperrr __class__r r!r/szAudioBuffersDataLayer.__init__cCrrr rr r r!r2rzAudioBuffersDataLayer.__iter__cCs`|jt|jkr t|jd7_tj|j|jdtjdtj|j|jdjdtjdfS)Nrr\) _buf_countrsignalrr# as_tensorrsrint64rr r r!r5s  zAudioBuffersDataLayer.__next__cCs||_|jdj|_d|_dSNr)rr signal_shaper)rsignalsr r r! set_signal>s z AudioBuffersDataLayer.set_signalcCsdSNrr rr r r!__len__CrzAudioBuffersDataLayer.__len__) rrrpropertyrrrrrr __classcell__r r rr!r's   rc@sZeZdZdZdddZdd Zd d Zd d ZddZddZ ddZ ddZ ddZ dS)FeatureFrameBuffererz\ Class to append each feature frame to a buffer and return an array of buffers. 皙?@TcCst|jdr|jjrd|_nd|_||_|jj|_||_|jjj }t |||_ t ||}|jjj |_ tj|j |gtjd|j|_||_||_d|_d|_||_tj|j |jgtjd|j|_g|_d|_|d|_dS) Args: frame_len: frame's duration, seconds frame_overlap: duration of overlaps before and after current frame, seconds offset: number of symbols to drop for smooth streaming rYrZr[r\FNr)r`rarYrbrcrrerfrrjrK n_frame_lenrnrorrrrsrLpad_to_buffer_len batch_size signal_end frame_readerrurrbuffered_features_sizerv buffered_len)rrcrr total_bufferrrrr r r!rMs,     zFeatureFrameBufferer.__init__cCsXtj|jjtjd|j|_d|_g|_g|_d|_ tj|j |j gtjd|j|_ dS)r)rr]rr\N) rrrrLrrsrb prev_charunmergedrrrorurrr r r!rvpszFeatureFrameBufferer.resetcCsJ|jrgSg}|jD]}|t|t||jkr|Sq d|_|S)NT)rrrrrwrr)r batch_framesrr r r!get_batch_frames}s z%FeatureFrameBufferer.get_batch_framescCsg|_|D]L}|jd}|j|7_||jkr%|js%|jt|q|jdd|df|jddd| f<||jdd|j df<|jt|jq|jSr) rrrrurrrrwrLr)rframesrcurr_frame_lenr r r!get_frame_bufferss *z&FeatureFrameBufferer.get_frame_bufferscCs||_d|_dSNF)rrrrr r r!set_frame_reader z%FeatureFrameBufferer.set_frame_readercCs|jd}||jkr|jst||_n*|jdd|jddf|jddd|jd f<||jdd|jd df<|j|jd7_dSr)rrurrrwrr)r feat_framerr r r!rs 6z+FeatureFrameBufferer._update_feature_buffercCsfg}t|D]*\}}||tj|jdd}tj|jdd}|||jd||jdfq|S)Nr)axis) enumeraterrmeanrstdrreshapero)rr norm_constsrrmean_from_bufferstdev_from_bufferr r r!get_norm_consts_per_frames $z.FeatureFrameBufferer.get_norm_consts_per_framecCs>d}t|D]\}}|||d||d|||<qdS)NrXrr)r)rrrCONSTANTr frame_bufferr r r!normalize_frame_bufferss&z,FeatureFrameBufferer.normalize_frame_bufferscCsJ|}t|dkr#||}||}t|dkrq||||SgSrrrrrr rrrrr r r!get_buffers_batchs     z&FeatureFrameBufferer.get_buffers_batchN)rrrT) rrrrrrvrrrrrr rr r r r!rGs #    rc@seZdZdZ    dddZdd Zd efd d Zd dZe dddZ e dddZ dde de defddZddZdS) FrameBatchASRz class for streaming frame-based ASR use reset() method to reset FrameASR's state call transcribe(frame) to do ASR on contiguous signal's frames rrrTcCst|||||d|_||_t|dd|_||_g|_g|_g|_|jdur-t |j j |_ nt |jdr;t |jj |_ nt |jj |_ |j |_ g|_g|_|t|j}||_||_t|jdd|j_d|j_d|j_t|j|_|j |j!|j|_dS) r)rcrrrrdecoderN vocabularyFr[rr^)"rframe_buffererrcgetattrrr all_logits all_predsrr tokenizerrblank_idr`joint toks_unmergedrrvrwrxrrdrrryrarzr{rgr r|r}r~r)rrcrrrrrdr r r!rs@    zFrameBatchASR.__init__cCsNd|_g|_t|_t|j|jtd|_g|_g|_ g|_ g|_ |j dS)rrr collate_fnN)rrr data_layerrrr data_loaderrrrrrrvrr r r!rvszFrameBatchASR.resetaudio_filepathcCsNt|}t|dt|||jjjf}t||j|j |jj }| |dSr rrrrKrcrrerrr}rrrrrNmodel_stride_in_secsrrr r r!read_audio_files"zFrameBatchASR.read_audio_filecCs|j|dSrrrrr r r!rszFrameBatchASR.set_frame_readerFcCsh|j}t|dkr2|j|dd7_|j|dd|||j}t|dks dSdSr)rrrrrr_get_batch_preds)r keep_logitsrr r r! infer_logitss    zFrameBatchASR.infer_logitscCs|jj}t|jD]b}|\}}||||}}|j||d}t|dkr:|\}}|jj|d} | jddd} n|\} }} t | } | D] } |j | qF|rht | } | D] } |j | q\n~ ~~ q dS)Nrprocessed_signal_lengthr0encoder_outputr1F)dimkeepdim)rcriterrr~r ctc_decoderargmaxr#unbindrrrnumpyr)rr%rr feat_signalfeat_signal_len forward_outsencoded encoded_len log_probs predictionspredspredlog_probr r r!r$!s,    zFrameBatchASR._get_batch_predstokens_per_chunkrNr%c Cs||g|_|jD]}|}|j|t|d|t|d||7_q ||j}|s5|Sg}|jD] }|jd} || d|| d||ddf}||q:t |d}||fS)Nrr) r&rrtolistr greedy_mergerrrr#concat) rr<rNr%r:decoded hypothesisrr;rr r r! transcribe=s  4   (  zFrameBatchASR.transcribecCsNg}|j}|D]}||ks||jkr||jkr|||}q|j|}|Sr)rrr ids_to_text)rr9decoded_predictionpreviousprAr r r!r>Os  zFrameBatchASR.greedy_mergeN)rrrTF)rrrrrrvstrr"rr#no_gradr&r$rKboolrBr>r r r r!rs  6   rcsfeZdZdZdfdd ZfddZd d Zd d Zd dZddZ ddZ ddZ ddZ Z S)BatchedFeatureFrameBuffererzi Batched variant of FeatureFrameBufferer where batch dimension is the independent audio samples. rrrcstj||||d|jjj}t||}tj||j|gtj d|j |_ ddt |j D|_ddt |j D|_ddt |j D|_d|_||`|`dS) r)rrrr\cSr(rr r*r r r!r-pr.z8BatchedFeatureFrameBufferer.__init__..cSr(rGr r*r r r!r-qr.cSr(rr r*r r r!r-rr.rN)rrrrarjrKrrrrorsrbrLrrall_frame_readerrsignal_end_index buffer_numberrvrr)rrcrrrrrrr r!r_s  z$BatchedFeatureFrameBufferer.__init__cszttj|j|j|jgtjd|j|_ ddt |jD|_ ddt |jD|_ ddt |jD|_ d|_dS)rr\cSr(rr r*r r r!r-r.z5BatchedFeatureFrameBufferer.reset..cSr(rGr r*r r r!r-r.cSr(rr r*r r r!r-r.rN)rrvrrrrrorursrbrrrLrrMrNrrr r!rvys  z!BatchedFeatureFrameBufferer.resetc Cst|jrgSg}t|jD]4\}}zt|}t|}||WqtyB|dd|j|<|j |dur@|j |j |<Yqw|j d7_ |S)NTr) allrrrLnextrrwrrrMrN)rridxrrr r r!rs"       z,BatchedFeatureFrameBufferer.get_batch_framescCsg|_t|jD]\}||}|durG|j|dd|jdf|j|ddd|j f<||j|dd|j df<|jt|j|gq|j|ddddfd9<|jt|j|gq|jS)Nr[)rrrrLrrrrw)rrrQrr r r!rs2 z-BatchedFeatureFrameBufferer.get_frame_bufferscCs"||j|<d|j|<d|j|<dSr)rLrrMrrrQr r r!rs  z,BatchedFeatureFrameBufferer.set_frame_readercCs|dur3|j|dd|jddf|j|ddd|jd f<||j|dd|jd df<dS|j|ddddfd9<dS)Nrr[)rr)rrrQr r r!rs:$$z2BatchedFeatureFrameBufferer._update_feature_buffercCsJt|D] \}}|||qtj|jddd}tj|jddd}||fS)Nr0T)rkeepdims)rrrrrr)rrrQrrrr r r!rs z5BatchedFeatureFrameBufferer.get_norm_consts_per_framecCsBd}tt|D]}|||d||d||||<qdS)Ng:0yE>rr)rr)rrrr rr r r!r s*z3BatchedFeatureFrameBufferer.normalize_frame_bufferscCs<|}t|dkr||}||}||||SgSrr r r r r!rs    z-BatchedFeatureFrameBufferer.get_buffers_batch)rrr)rrrrrrvrrrrrr rrr r rr!rKZs   rKcseZdZdZ     ddedeffd d Zfd d Zd efddZ ddZ e ddZ e ddZdedefddZddZddZZS)BatchedFrameASRRNNTzz Batched implementation of FrameBatchASR for RNNT models, where the batch dimension is independent audio samples. rr rIFrJstateful_decodingcstj||||d||_||_ddt|jD|_ddt|jD|_ddt|jD|_d|_ ddt|jD|_ z|j j j |_ Wn tySd |_ Ynwtd |jt||||d |_|dS) a Args: asr_model: An RNNT model. frame_len: frame's duration, seconds. total_buffer: duration of total audio chunk size, in seconds. batch_size: Number of independent audio samples to process at each step. max_steps_per_timestep: Maximum number of tokens (u) to process per acoustic timestep (t). stateful_decoding: Boolean whether to enable stateful decoding for preservation of state across buffers. rrrcSg|]}gqSr r r*r r r!r-r.z0BatchedFrameASRRNNT.__init__..cSrXr r r*r r r!r-r.cSrXr r r*r r r!r-r.NcSi|]}||qSr r r+rQr r r! sz0BatchedFrameASRRNNT.__init__..r1zPerforming Stateful decoding :)rcrrr)rrrJrVrrall_alignmentsrall_timestampsprevious_hypothesesbatch_index_maprcreos_id ExceptionrrKrrv)rrcrrrrJrVrr r!rs(    zBatchedFrameASRRNNT.__init__cstddtjD_ddtjD_ddtjD_d_ddtjD_ddtjD_ fd dtjD_ dS) rcSrXr r r*r r r!r-r.z-BatchedFrameASRRNNT.reset..cSrXr r r*r r r!r-r.cSrXr r r*r r r!r-r.NcSrYr r rZr r r!r[z-BatchedFrameASRRNNT.reset..cSsg|]}tqSr )rr*r r r!r- rbcs g|] }tj|dtdqS)rr)rrrrZrr r!r-!s) rrvrrr\rr]r^r_rrrrrr!rvs   zBatchedFrameASRRNNT.resetrcCsvt||jks Jt|jD]*}t||}t|dt|||jjj f}t ||j |j |jj }|||qdSr)rrrrrrrKrcrrerrr}rr)rrrNr!rQrrr r r!r"&s "z#BatchedFrameASRRNNT.read_audio_filecCs|j||dSrr#rRr r r!r0sz$BatchedFrameASRRNNT.set_frame_readercCs||j}t|dkr<|j|dd7_t|D]\}}|j||ddq||j}t|dks dSdSr)rrrrrrrr$)rrrQrLr r r!r&3s   z BatchedFrameASRRNNT.infer_logitscCs|jj}dd|jD}g}g}g}t|jD]-}|jj|r qt||}|\}} ||| |}} | || | | |qt |dkrMdSt |d}t |d} ~~|j|| d\} } |j r|jdurg} t|D]\} }|j|}| |j|qr| |_|jjj| | d|jd}|j r||_t|D]\}}||}|jj|}|s|j| |jqdd|D}t|D]\}}||}|jj|}|s|j| |qd d|D}t|D]\}}||}|jj|}|s|j| |q|j rB|jj| }t|D].\}}t |dkr@|d |jkr@|j|jdd |j|_|jj|||j|_qt |t |jkr[t|D] \} }| |j|<qP~ ~ ~~dS) a Perform dynamic batch size decoding of frame buffers of all samples. Steps: - Load all data loaders of every sample - For all samples, determine if signal has finished. - If so, skip calculation of mel-specs. - If not, compute mel spec and length - Perform Encoder forward over this sub-batch of samples. Maintain the indices of samples that were processed. - If performing stateful decoding, prior to decoder forward, remove the states of samples that were not processed. - Perform Decoder + Joint forward for samples that were processed. - For all output RNNT alignment matrix of the joint do: - If signal has ended previously (this was last buffer of padding), skip alignment - Otherwise, recalculate global index of this sample from the sub-batch index, and preserve alignment. - Same for preds - Update indices of sub-batch with global index map. - Redo steps until all samples were processed (sub-batch size == 0). cSsg|]}t|qSr )r-)r+rr r r!r-Zsz8BatchedFrameASRRNNT._get_batch_preds..rNr'T)return_hypothesespartial_hypothesescSg|]}|jqSr  y_sequencer+hypr r r!r-rbcSrer ) timestamprhr r r!r-rbr1)rcrrrrrrrPr~rrr#catrVr^rr_decodingrnnt_decoder_predictions_tensorr\ alignmentsrrr1r]rinitialize_stater`rgbatch_select_state dec_state)rr data_iters feat_signalsfeat_signal_lensnew_batch_keysrQrr2r3r5r6new_prev_hypothesis new_batch_idxglobal_index_keyold_posbest_hyprihas_signal_endedr9r: timestampsrj reset_statesr r r!r$As              z$BatchedFrameASRRNNT._get_batch_predsr<rNc Cs|ddt|jD|_t|jD]_\}}|jj|}|dur&tdt|D]H\}}|t |kr7d}nd}|t |||t ||||}| ||j j |j \} } t | dkrr||krrt|j|| ||j d|j|<q*qg} t|jD] }| ||j|q{| S)^ Performs "middle token" alignment prediction using the buffered audio chunk. cSrXr r r*r r r!r-r.z2BatchedFrameASRRNNT.transcribe..NSignal did not endrrrO)r&rrrrr\rrMrr_alignment_decoderrcrrrVrr>) rr<rNrQrnsignal_end_idxa_idxroffsetidstoksrr r r!rBs6  " zBatchedFrameASRRNNT.transcribec Csg}g}tt|D]1}tt||D]&}|||\}} t| } | |kr9|| gd} || || q qq ||fSr)rrrK ids_to_tokensr) rrnrrsrtur,token_idtokenr r r!rs   z&BatchedFrameASRRNNT._alignment_decodercCs dd|D}|jj|}|S)NcSg|]}|qSr r )r+rFr r r!r-r.z4BatchedFrameASRRNNT.greedy_merge..)rcrrC)rr9rDrAr r r!r>sz BatchedFrameASRRNNT.greedy_merge)rrrUrIF)rrrrrKrJrrvlistr"rr#rIr&r$rBrr>rr r rr!rTs4 .   r *rTcsNeZdZdZ      dded ed effd d Zd edefddZZS)BatchedFrameASRTDTa Batched implementation of FrameBatchASR for TDT models, where the batch dimension is independent audio samples. It's mostly similar to BatchedFrameASRRNNT with special handling of boundary cases due to the frame-skipping resulted by TDT models. rrrUrIFrrJrVtdt_search_boundarycstj||||d||_dS)ab Args: asr_model: An RNNT model. frame_len: frame's duration, seconds. total_buffer: duration of total audio chunk size, in seconds. batch_size: Number of independent audio samples to process at each step. max_steps_per_timestep: Maximum number of tokens (u) to process per acoustic timestep (t). stateful_decoding: Boolean whether to enable stateful decoding for preservation of state across buffers. tdt_search_boundary: The max number of frames that we search between chunks to match the token at boundary. rWN)rrr)rrcrrrrJrVrrr r!rs zBatchedFrameASRTDT.__init__r<rNcCs|ddt|jD|_t|jD]\}}|jj|}|dur&tdt|D]\}}|t |kr7d}nd}|t ||||j t ||||} |t |||t ||||}| | |j j |j\} } | ||j j |j\} } t | dkr||kr|dkst |j|dkrt|j|| ||j d|j|<q*t |j|dkrt | dkr|j|d}tt | t | t | d}d}t||dD]}| ||kr| |dd} nqt|j|| ||j d|j|<q*qg}t|jD] }|||j|q|S) r~cSrXr r r*r r r!r-r.z1BatchedFrameASRTDT.transcribe..Nrrrrr1)r&rrrrr\rrMrrrrrcrrrVminrr>)rr<rNrQrnrrrrlonger_alignment longer_ids longer_toksrr, id_to_matchstartrrrr r r!rBs~   "   1zBatchedFrameASRTDT.transcribe)rrrUrIFr) rrrrrKrJrrBrr r rr!rs( rcsNeZdZdZ      dded ed effd d Zd edefddZZ S)+LongestCommonSubsequenceBatchedFrameASRRNNTz Implements a token alignment algorithm for text alignment instead of middle token alignment. For more detail, read the docstring of longest_common_subsequence_merge(). rrrrIFNrJrValignment_basepathcs,t||||||d|_d|_||_dS)ab Args: asr_model: An RNNT model. frame_len: frame's duration, seconds. total_buffer: duration of total audio chunk size, in seconds. batch_size: Number of independent audio samples to process at each step. max_steps_per_timestep: Maximum number of tokens (u) to process per acoustic timestep (t). stateful_decoding: Boolean whether to enable stateful decoding for preservation of state across buffers. alignment_basepath: Str path to a directory where alignments from LCS will be preserved for later analysis. rr1N)rr sample_offset lcs_delayr)rrcrrrrJrVrrr r!rds z4LongestCommonSubsequenceBatchedFrameASRRNNT.__init__r<rNc Cs|jdkr td|ddt|jD|_t|jD]\}}|jj |}|dur/tdt|D]\}}|dkri|t |d|d}| ||j j |j\}} t |dkrht|j||||j d|j|<q3| ||j j |j\}} t |dkr||kr|jdur|j} |j|} |} tj| t| } tj| dd tj| d t| d } | }nd}t|j|||j|j |j|d |j|<q3qg}t|jD] }|||j|q|S) NrzYPlease set LCS Delay valus as `(buffer_duration - chunk_duration) / model_stride_in_secs`cSrXr r r*r r r!r-r.zJLongestCommonSubsequenceBatchedFrameASRRNNT.transcribe..rrrT)exist_ok alignment_z.pt)rOrJr%)rrr&rrrrr\rrMrrrcrrrVrrospathjoinrHmakedirsrTrJrr>)rr<rNrQrnrrrrrbasepathralignment_offsetrr%rr r r!rB~s^       *z6LongestCommonSubsequenceBatchedFrameASRRNNT.transcribe)rrrrIFN) rrrrrKrJrHrrBrr r rr!r]s( rc@seZdZdZdddZddZdd Zd d Zd d ZddZ ddZ d ddZ d ddZ d ddZ ddZd!ddZddZdS)"CacheAwareStreamingAudioBufferz A buffer to be used for cache-aware streaming. It can load a single or multiple audio files/processed signals, split them in chunks and return one on one. It can be used to simulate streaming audio or audios. NFcCs||_d|_d|_d|_d|_||_||_t|jt st d|jj dur*|j |jj |_ |jj |_||_t|jdrOt|jjdrO|jj|_dSd|_dS)a1 Args: model: An ASR model. online_normalization (bool): whether to perform online normalization per chunk or normalize the whole audio before chunking pad_and_drop_preencoded (bool): if true pad first audio chunk and always drop preencoded Nrz`The model's encoder is not inherited from StreamingEncoder, and likely not to support streaming! pre_encodeget_sampling_frames)rOrL buffer_idxstreams_lengthsteppad_and_drop_preencodedonline_normalization isinstanceencoderr r streaming_cfgsetup_streaming_params_feat_ininput_featuresextract_preprocessorrar`rrsampling_frames)rrOrrr r r!rs&       z'CacheAwareStreamingAudioBuffer.__init__ccs |j|jdkr dS|jdkr*t|jjtr*|jr#|jjd}n|jjd}nt|jjtr7|jjdn|jj}|jdkrXt|jjtrX|jrQ|jjd}n|jjd}nt|jjtre|jjdn|jj}|jdddd|j|j|f}|j dur|jdkrt|j tr|j d}nt|j tr|j dn|j }|d|krdSd}|jdkrt|jj tr|jr|jj d}n|jj d}t j |d|j |f|j|jd}nLt|jj tr|jj d}n|jj }|j|} | dkrd} |jdddd| |jf}|d|kr$t j |d|d||df|j|jd}|d} t j||fdd}|jrMt|t |dg|d|jd\}} } |durbt j||fdd}| |d7} |j|j} | | } t j| d|dd }|j|7_|jd7_||fVq) NTr1rr)rr]r+r)rr7)rrLsizerrrirr shift_sizerpre_encode_cache_sizer#rprrr]rkrr rrhrclampr)rrir audio_chunkcur_sampling_frames zeros_padscache_pre_encode_num_framescache_pre_encoderstart_pre_encode_cache added_lenx_meanx_stdmax_chunk_lengths chunk_lengthsr r r!rs  $         z'CacheAwareStreamingAudioBuffer.__iter__cCs|j|jdkr dSdS)Nr1TF)rrLrrr r r!is_buffer_emptyMsz.CacheAwareStreamingAudioBuffer.is_buffer_emptycC t|jSr)rrLrr r r!rS z&CacheAwareStreamingAudioBuffer.__len__cCsd|_d|_d|_d|_dSr)rLrrrrr r r! reset_bufferVs z+CacheAwareStreamingAudioBuffer.reset_buffercCsd|_d|_dSr)rrrr r r!reset_buffer_pointer\rz3CacheAwareStreamingAudioBuffer.reset_buffer_pointercCs`t|jj}|jj|_t|jdd|j_ d|j_ |j r"d|j_|j |j}| |S)NFr[rr^)rwrxrOrrargrhrryrzr{rr|r~get_model_device)rrdrar r r!r`s z3CacheAwareStreamingAudioBuffer.extract_preprocessorr1cCs$t|}|||\}}}|||fSr)r append_audio)rr stream_idaudiorr(r r r!append_audio_filels z0CacheAwareStreamingAudioBuffer.append_audio_filecCs*||\}}|||\}}}|||fSr)preprocess_audioappend_processed_signal)rrrrr(r r r!rqs  z+CacheAwareStreamingAudioBuffer.append_audiocCstj|d|jd}|dkr|jdur|t|jkrtd|jdur:|dkr,td||_tj|g|jd|_n{|jd|dkrItd|dkrrtjj j |jdd |_tj |jtjdg|jjdfdd |_t|jd}|j||}||jdkrtjj j |jd||jdfd |_||j|dd|j||j||f<|j||d|j|<|j rt |t|g|jd \}}}|||fS) Nr1rrzNot valid stream_id!z7stream_id can not be specified when there is no stream.rz:Buffer and the processed signal have different dimensions!)rrrrrr)rrr)r#rrrrrrrLrrrrkrr rh)rrrr( needed_lenrrr r r!rxs>  &   z6CacheAwareStreamingAudioBuffer.append_processed_signalcCs|jjSr)rOrrr r r!rsz/CacheAwareStreamingAudioBuffer.get_model_devicecCsX|dur|}t|d|}t|jdg|}|j||d\}}||fS)Nrr)rr#rrr~rrra)rrrrrrr(r r r!rs z/CacheAwareStreamingAudioBuffer.preprocess_audiocCs4|j}|jrt|t|j|jd\}}}||jfS)Nr)rLrr r#rrrh)rrrrr r r!get_all_audioss  z-CacheAwareStreamingAudioBuffer.get_all_audiosr)r1r)rrrrrrrrrrrrrrrrrr r r r!rs !d  #  rcseZdZdfdd ZfddZdefddZd efd d Ze dd dZ dde e de e defddZddZddZddZddZZS) FrameBatchMultiTaskAEDrcs:tj||||dd|jjj|_|jjj|_dg|_dS)NFrr)rrrrarjrsubsampling_factor chunk_offsetsrrcrrrrr r!rs    zFrameBatchMultiTaskAED.__init__cstdg|_dSr)rrvrrrr r!rvs  zFrameBatchMultiTaskAED.resetsamplecs|jjdkr hd}i}n|jjdkr!ddh}dddd d d d }n td |jjfdd|D}|r?td|d|D] \}}|||<qC|jjjdi|jjjdidgdd}t j |t j |jj d dS)Ncanary>pnctaskname source_lang target_langcanary2rrrz<|emo:undefined|>z <|noitn|>z<|notimestamp|>z <|nodiarize|>z<|pnc|>)decodercontextemotionitnrjdiarizerzUnknown prompt format: csg|]}|vr|qSr r )r+krr r!r-sz;FrameBatchMultiTaskAED.get_input_tokens..z4We found sample that is missing the following keys: zWPlease ensure that every utterance in the input manifests contains these keys. Sample: user spl_tokens)roleslots)turns context_idsr]rr)rc prompt_formatr RuntimeErroritemsgetprompt encode_dialogPROMPT_LANGUAGE_SLOTr#rlongrr)rrexpected_slotsdefault_slot_values missing_keysrvtokensr rr!get_input_tokenssF      z'FrameBatchMultiTaskAED.get_input_tokensrcCs^|||_t|}t|dt|||jjjf}t ||j |j |jj dd}| |dSNrF)r)r input_tokensrrrrKrcrrerrr}rr)rrrNr! meta_datarrr r r!r"s "z&FrameBatchMultiTaskAED.read_audio_fileFc Cs|jj}t|jD]S}|\}}|j|||||}}|j| | dd}t j | dg| d|d }t||dd||ddd}|jj|dd} |j| ~ q dS)Nrrr)r audio_lens transcripttranscript_lensr prompt_lensprompted_transcriptprompted_transcript_lensT)has_processed_signal)rcrr-rrextendr=r~rrepeatrr#rrr predict_stepr) rr%rrr2r3r tokens_len batch_inputr8r r r!r$s*$ z'FrameBatchMultiTaskAED._get_batch_predsNr<rNr%cCs.||||j}|s|Std|gfS)V unsued params are for keeping the same signature as the parent class zZkeep_logits=True is not supported for MultiTaskAEDFrameBatchInfer. Returning empty logits.)r&_join_hypothesesrrrr<rNr%rAr r r!rB s  z!FrameBatchMultiTaskAED.transcribecCsZt|dkr |dStjdtggggdd}|||}|||}|||}|S)Nrrr[)charwordr)scorergrj)rr Hypothesisr#r _join_text_join_y_sequence_join_timestamp)r hypothesesmerged_hypthesisr r r!r s   z'FrameBatchMultiTaskAED._join_hypothesescCsddd|D|_|S)NrcSrer )textr+hr r r!r-5rbz5FrameBatchMultiTaskAED._join_text..)rrrmerged_hypothesisrr r r!r4z!FrameBatchMultiTaskAED._join_textcCstdd|D|_|S)NcSrer rfrr r r!r-9rbz;FrameBatchMultiTaskAED._join_y_sequence..)r#rkrgrr r r!r8rz'FrameBatchMultiTaskAED._join_y_sequencecsdt|D])\}}j|7fdd|jdD}fdd|D}|jd|qdt|D])\}}j|7fdd|jdD}fdd|D}|jd|q6|S) Nrc:g|]}i||dj|djdqS start_offset end_offset)rrrr+r cumulative_offsetrr r!r-Csz:FrameBatchMultiTaskAED._join_timestamp..r c>g|]}i||djj|djjdqSrr)rrrjrr rr r!r-Ocrrrr+rr!r r!r-`srcr#r$r%r'rr r!r-jr&)rrrjr)rrrrrupdated_timestampsr r!r!r<s*      z&FrameBatchMultiTaskAED._join_timestamprrrrGNNF)rrrrrvdictrrHr"r#rIr$rrKrJrBr rrrrr r rr!rs& )  rc`eZdZdfdd ZdefddZeddd Z dd e e d e e d e fddZ Z S)FrameBatchChunkedRNNTrctj||||dddSNFrrrrr r!ryzFrameBatchChunkedRNNT.__init__rcCRt|}t|dt|||jjjf}t||j|j |jj dd}| |dSrrr r r r!r"| "z%FrameBatchChunkedRNNT.read_audio_fileFc Csx|jj}t|jD]0}|\}}||||}}|j||d\}}|jjj||dd\}} |j|~~ ~~q dS)Nr'F)r*encoded_lengthsrc) rcrr-rr~rlrmrr) rr%rrr2r3r5r6 best_hyp_text all_hyp_textr r r!r$s   z&FrameBatchChunkedRNNT._get_batch_predsNr<rNr%cC.||d|j}|s|Std|gfS)r rzTkeep_logits=True is not supported for FrameBatchChunkedRNNT. Returning empty logits.r&rrrr r r r!rB  z FrameBatchChunkedRNNT.transcriber)rGr*rrrrrHr"r#rIr$rrKrJrBrr r rr!r-xs r-cr,)FrameBatchChunkedCTCrcr.r/rrrr r!rr0zFrameBatchChunkedCTC.__init__rcCr1rrr r r r!r"r2z$FrameBatchChunkedCTC.read_audio_fileFc Cs|jj}t|jD]P}|\}}||||}}|j||d}t|dkr?|\}}|jj|d} |jjj| |dd\} } n|\} }} |jj j| |dd\} } |j | ~ ~~ q dS)Nr'r0r)F)decoder_outputsdecoder_lengthsrc) rcrr-rr~rr. ctc_decodingctc_decoder_predictions_tensorrlrr) rr%rrr2r3resultsr5r6r7transcribed_textsr,r8r r r!r$s0   z%FrameBatchChunkedCTC._get_batch_predsNr<rNr%cCr6)r rzSkeep_logits=True is not supported for FrameBatchChunkedCTC. Returning empty logits.r7r r r r!rBr8zFrameBatchChunkedCTC.transcriber)rGr*r9r r rr!r:s r:c@sheZdZUeed<eed<eed<defddZdeddfdd Zd ed ed ddefd dZddZ dS) ContextSizeleftrrightreturncC|j|j|jSzTotal context sizerBrrCrr r r!totalzContextSize.totalfactorcCs t|j||j||j|dS)h Subsample context size by factor Args: factor: subsampling factor rG)rArBrrCrrJr r r! subsamples zContextSize.subsample num_frames is_last_chunkexpected_contextcCs||j|jkrtd|d||j|j7_d|_|j|7_|r.|j|_d|_n |j|_|j|j8_t||d}|j|8_|sV|j|jksVJ|S)z Add frames to context size Args: num_frames: number of frames to add is_last_chunk: if last chunk Returns: number of frames removed from the left side zAdded chunk length z2 is larger than expected chunk with right context r)rrCrrBr7rH)rrNrOrP extra_samplesr r r!add_frames_get_removed_s& z#ContextSize.add_frames_get_removed_cCsd|jd|jd|jS)NzLeft z - Chunk z - Right rGrr r r!__str__szContextSize.__str__N) rrrrK__annotations__rHrMrJrRrSr r r r!rAs  rAc@sleZdZUdZejed<ejed<ejed<dejfddZdeddfd d Z d ejd ejd dfddZ dS)ContextSizeBatchzBatched context sizerBrrCrDcCrErFrGrr r r!rH$rIzContextSizeBatch.totalrJcCs8ttj|j|ddtj|j|ddtj|j|dddS)rKfloor) rounding_moderG)rUr#divrBrrCrLr r r!rM(s zContextSizeBatch.subsamplenum_frames_batchis_last_chunk_batchrPrAcCs|j|j7_|jd|j|7_t||j|j|_t|d|j|j|_t|jdk|jd|_t||t|j}|j|8_t|jdkt|j|j|_dS)z Add frames to context size Args: num_frames_batch: number of frames to add is_last_chunk_batch: if last chunk Returns: number of frames removed from the left side rN) rBrfill_rCr#wheremaximumrH zeros_like)rrYrZrPrQr r r! add_frames_5s  $zContextSizeBatch.add_frames_N) rrrrr#rrTrHrKrMr_r r r r!rUs     rUc @sReZdZdZdededejdeje BfddZ dej d ej d e d ej fd d Z dS)StreamingBatchedAudioBufferzaBatched audio buffer with strict context management for streaming inference without left padding.rcontext_samplesr]rcCsv||_||_tj|dg||d|_tdddd|_ttj|gtj|dtj|gtj|dtj|gtj|dd|_ dS)z Init batched audio buffer for streaming inference Args: batch_size: batch size context_samples: context size dtype: buffer dtype device: device for buffer rrrGN) rrPr#rprrA context_sizerUrcontext_size_batch)rrrar]rr r r!rSs  z$StreamingBatchedAudioBuffer.__init__ audio_batchrrOrZcCst|jd}tj|j|fdd|_|jj|||jd}|jj|||jd|dkr8|jdd|df |_dSdS)a Add audio batch to buffer Args: audio_batch: chunk with audio audio_lengths: length of audio is_last_chunk: if last chunk is_last_chunk_batch: if last chunk for each audio utterance rr)rOrP)rYrZrPrN) rr#rkrrbrRrPrcr_r)rrdrrOrZadded_chunk_lengthextra_samples_in_bufferr r r!add_audio_batch_fs  z,StreamingBatchedAudioBuffer.add_audio_batch_N)rrrrrKrAr#r]rrHrrrJrgr r r r!r`Ps"r`> file_pathrerDcCs&tj||d\}}tj|tjd|fS)zLoad audio from file)rfr\)librosaloadr#rrs)rirerrfr r r! load_audiosrlc@s>eZdZUejed<ejed<edeejddfddZdS) AudioBatch audio_signalsaudio_signal_lengthsrdrDcCs<tdd|Dddd}tdd|D}t||dS)z0 Collate audio signals to batch cSrr r r+ audio_tensorr r r!r-r.z)AudioBatch.collate_fn..Tr[) batch_first padding_valuecSsg|]}|jdqSr))rrpr r r!r-s)rnro)rr#rrrm)rdrnror r r!rszAudioBatch.collate_fnN) rrrr#rrT staticmethodrrr r r r!rms  rmcsPeZdZdZd deeeBdeffdd Zdede j fd d Z d d Z Z S)SimpleAudioDatasetz;Dataset constructed from audio filenames. Each item - audiorhaudio_filenamesrecst||_||_dSr)rrrvre)rrvrerr r!rs  zSimpleAudioDataset.__init__rrDcCst|j|\}}|Sr)rlrv)rrrr,r r r! __getitem__szSimpleAudioDataset.__getitem__cCrr)rrvrr r r!rrzSimpleAudioDataset.__len__rh)rrrrrrHrrKrr#rrwrrr r rr!rus  rur)rINrx)Drwr dataclassesrpathlibrtypingrrrjr1rr# omegaconfrtorch.nn.utils.rnnrtorch.utils.datarr 7nemo.collections.asr.data.audio_to_text_lhotse_promptedr nemo.collections.asr.modelsr +nemo.collections.asr.parts.mixins.streamingr 1nemo.collections.asr.parts.preprocessing.featuresr 0nemo.collections.asr.parts.preprocessing.segmentr nemo.collections.asr.parts.utilsrnemo.core.classesrnemo.core.neural_typesrrrr6r"r'rHrKrHrTrVrWrrrrrrKrTrrrrr-r:rArUr`rrrlrmrur r r r!sl             \#}(! }fdsF09:3&7