o }oi@sddlZddlZddlmZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl mZmZmZddl mZddlmZddlmZdd lmZdd lmZdd lmZmZmZmZmZm Z dd l!m"Z"m#Z#m$Z$dd l%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:dgZ;Gddde1ee,e.ZGddde1e,e.Z?e/@e<dure/jAedeGdddZBdS)N) OrderedDict) dataclass)ListOptionalSetTuple) DictConfig ListConfig open_dict)nn)CacheAwareStreamingConfig)StreamingEncoder) CausalConv1D)ConformerLayer)LocalAttRelPositionalEncodingMultiHeadAttentionPositionalEncodingRelPositionalEncodingRelPositionMultiHeadAttention'RelPositionMultiHeadAttentionLongformer)ConvSubsamplingStackingSubsamplingSubsamplingReductionModule) adapter_utils)#compute_stochastic_depth_drop_probs) typecheck) Exportable) AccessMixinadapter_mixins) NeuralModule)AcousticEncodedRepresentationBoolType ChannelType LengthsType NeuralTypeSpectrogramType)loggingConformerEncodercseZdZdZdRddZeddZedd Zed d Zed d Z eddZ eddZ                                  dSd!e d"e d#ed$ed%ed&ed'ed(effd)d* Z dTd+d,ZdUd-d.Ze    dVd/d0Z    dVd1d2Zd3efd4d5Zd6d7Zd8d9ZdUd:d;Zdd?Z     @dWdAedBedCedDedEef dFdGZdejdd fdHdIZ    dXdJe dDe!edKedLej"fdMdNZ#dOefdPdQZ$Z%S)Yr'a The encoder for ASR model of Conformer. Based on this paper: 'Conformer: Convolution-augmented Transformer for Speech Recognition' by Anmol Gulati et al. https://arxiv.org/abs/2005.08100 Args: feat_in (int): the size of feature channels n_layers (int): number of layers of ConformerBlock d_model (int): the hidden size of the model feat_out (int): the size of the output features Defaults to -1 (means feat_out is d_model) subsampling (str): the method of subsampling: choices = ['vggnet', 'striding', 'dw-striding', 'stacking', 'stacking_norm'] Defaults to striding. subsampling_factor (int): the subsampling factor which should be power of 2 Defaults to 4. subsampling_conv_chunking_factor(int): optionally, force chunk inputs (helpful for large inputs) Should be power of 2, 1 (auto-chunking, default), or -1 (no chunking) subsampling_conv_channels (int): the size of the convolutions in the subsampling module Defaults to -1 which would set it to d_model. reduction (str, Optional): the method of reduction, choices=['pooling', 'striding']. If no value is passed, then no reduction is performed and the models runs with the original 4x subsampling. reduction_position (int, Optional): the index of the layer to apply reduction. If -1, apply reduction at the end. reduction_factor (int): the reduction factor which should be either 1 or a power of 2 Defaults to 1. ff_expansion_factor (int): the expansion factor in feed forward layers Defaults to 4. self_attention_model (str): the type of the attention layer and positional encoding. 'rel_pos': relative positional embedding and Transformer-XL 'rel_pos_local_attn': relative positional embedding and Transformer-XL with local attention using overlapping chunks. Attention context is determined by att_context_size parameter. 'abs_pos': absolute positional embedding and Transformer Default is rel_pos. pos_emb_max_len (int): the maximum length of positional embeddings Defaults to 5000 n_heads (int): number of heads in multi-headed attention layers Defaults to 4. att_context_size (List[Union[List[int],int]]): specifies the context sizes on each side. Each context size should be a list of two integers like `[100, 100]`. A list of context sizes like `[[100,100]`, `[100,50]]` can also be passed. -1 means unlimited context. Defaults to `[-1, -1]` att_context_probs (List[float]): a list of probabilities of each one of the att_context_size when a list of them is passed. If not specified, uniform distribution is being used. Defaults to None att_context_style (str): 'regular' or 'chunked_limited'. Defaults to 'regular' xscaling (bool): enables scaling the inputs to the multi-headed attention layers by `sqrt(d_model)`. Defaults to True. untie_biases (bool): whether to not share (untie) the bias weights between layers of Transformer-XL Defaults to True. conv_kernel_size (int): the size of the convolutions in the convolutional modules Defaults to 31. conv_norm_type (str): the type of the normalization in the convolutional modules Defaults to 'batch_norm'. conv_context_size (list): it can be"causal" or a list of two integers while `conv_context_size[0]+conv_context_size[1]+1==conv_kernel_size`. `None` means `[(conv_kernel_size-1)//2`, `(conv_kernel_size-1)//2]`, and 'causal' means `[(conv_kernel_size-1), 0]`. Defaults to None. conv_dual_mode (bool): specifies if convolution should be dual mode when dual_offline mode is being used. When enables, the left half of the convolution kernel would get masked in streaming cases. Defaults to False. use_bias (bool): Use bias in all Linear and Conv1d layers from each ConformerLayer to improve activation flow and stabilize training of huge models. Defaults to True. dropout (float): the dropout rate used in all layers except the attention layers Defaults to 0.1. dropout_pre_encoder (float): the dropout rate used before the encoder Defaults to 0.1. dropout_emb (float): the dropout rate used for the positional embeddings Defaults to 0.1. dropout_att (float): the dropout rate used for the attention layer Defaults to 0.0. stochastic_depth_drop_prob (float): if non-zero, will randomly drop layers during training. The higher this value, the more often layers are dropped. Defaults to 0.0. stochastic_depth_mode (str): can be either "linear" or "uniform". If set to "uniform", all layers have the same probability of drop. If set to "linear", the drop probability grows linearly from 0 for the first layer to the desired value for the final layer. Defaults to "linear". stochastic_depth_start_layer (int): starting layer for stochastic depth. All layers before this will never be dropped. Note that drop probability will be adjusted accordingly if mode is "linear" when start layer is > 1. Defaults to 1. global_tokens (int): number of tokens to be used for global attention. Only relevant if self_attention_model is 'rel_pos_local_attn'. Defaults to 0. global_tokens_spacing (int): how far apart the global tokens are Defaults to 1. global_attn_separate (bool): whether the q, k, v layers used for global tokens should be separate. Defaults to False. use_pytorch_sdpa (bool): use torch sdpa instead of manual attention. Defaults to False. use_pytorch_sdpa_backends (list[str]): list of backend names to use in sdpa. None or empty list means all backends. e.g. ["MATH"] Defaults to None. bypass_pre_encode: if True, skip the pre-encoder module and the `audio_signal` should be pre-encoded embeddings. The `audio_signal` input supports two formats depending on the `bypass_pre_encode` boolean flag. This determines the required format of the input variable `audio_signal`. Defaults to `bypass_pre_encode=False`. `bypass_pre_encode=True` is used for the cases where frame-level, context-independent embeddings are needed to be saved or reused. (e.g., speaker cache in streaming speaker diarization) sync_max_audio_length (bool): when true, performs NCCL all_reduce to allocate the same amount of memory for positional encoding buffers on all GPUs. Disabling this setting may help with deadlocks in certain scenarios such as model parallelism, or generally when this module is not being ran on some GPUs as a part of the training step. c Cst|j}|jro|}|jdur9t|jjtr|jjd}n|jj}t|jjtr1|jjd}n|jj}||}t j ||j ||d}t j |d||f|t j d}|j|||d\} } } t||| dd| dd| g} | St j ||j ||d}t j |d||f|t j d}t||g} | S)zs Generates input examples for tracing etc. Returns: A tuple of input examples. Nr(devicer+dtype) batch_sizer+max_dimr)next parametersr+export_cache_support streaming_cfg isinstance chunk_sizelistpre_encode_cache_sizetorchrandn_feat_inrandintint64get_initial_cache_statetuple transpose) self max_batchr0dev window_sizer6r8 input_exampleinput_example_lengthcache_last_channelcache_last_timecache_last_channel_lenall_input_examplerKb/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/modules/conformer_encoder.pyrEs>     zConformerEncoder.input_examplec C`ttdtttdttdtddtdtddttdtddtttdddS)*Returns definitions of module input ports.BDTrPrQrPrRrQToptionalrQrPrQrR audio_signallengthrGrHrIbypass_pre_encoderr$r%r?r#r"r!rArKrKrL input_types zConformerEncoder.input_typesc CrM)rNrOrPrPrQrRrQTrTrPrQrQrRrWr[r\rKrKrLinput_types_for_exportr^z'ConformerEncoder.input_types_for_exportc CPttdtttdttdtddtdtddttdtdddS)+Returns definitions of module output ports.rOrPrSTrTrVoutputsencoded_lengthscache_last_channel_nextcache_last_time_nextcache_last_channel_next_lenrr$r r?r#r"r\rKrKrL output_types zConformerEncoder.output_typesc Crb)rcrOrPr_TrTr`rdrjr\rKrKrLoutput_types_for_exportrlz(ConformerEncoder.output_types_for_exportcC|js tgdStS)NrGrHrIr3setr\rKrKrLdisabled_deployment_input_names z0ConformerEncoder.disabled_deployment_input_namescCrn)N)rgrhrirpr\rKrKrL disabled_deployment_output_namesrsz1ConformerEncoder.disabled_deployment_output_namesFstridingr,Nrel_posregularT batch_norm皙?linearrstochastic_depth_drop_probstochastic_depth_modestochastic_depth_start_layer global_tokensglobal_tokens_spacingglobal_attn_separateuse_pytorch_sdpasync_max_audio_lengthc'- srt|| }'||_||_||_||_||_||_||_|!|_ |#|_ |"|_ |$|_ |%dur0g}%|%|_ |&|_|j|||||d\|_|_|_|_|rQt||_nd|_| dkrZ|} |r|dkr|dvrtt||||dkrndndd|_nt||||| |td|d |_nt|||_| r| dkr| dkr| |ksJt| || d |_| |_nd|_d|_||_ |s|d kr||}(t!t"#||(})t!t"#||(}*tj$%|)tj$%|*nd})d}*||_&|d krt'||||j|d |_(n9|d krt)|dkrt*dt+|||||j|d|_(n|dkr"d})d}*t,||||jd|_(nt*d|dt-|_.t/|D]H}+t0d'id|d|'d|d|!d|"d|#d|d|d|d|jd|d |d!|)d"|*d#|jd$|d%|j d&|j },|j.1|,q3|dkr||j krt|j ||_2||_ nd|_2||_ |3|j&d|_4|5d|_6t7t8|j.||| |_9d|_:dS)(N)att_context_styleatt_context_sizeatt_context_probsconv_context_sizeconv_kernel_sizerur()stacking stacking_normrTF)subsampling_factorfeat_infeat_outnorm) subsamplingrrr conv_channels subsampling_conv_chunking_factor activation is_causal) reductiond_modelreduction_factorrwr dropout_ratemax_lenxscaledropout_rate_embrel_pos_local_attnr8When using local attention, context size must be set > 0rrrrrrabs_posrrrr!Not valid self_attention_model: ''!rd_ffself_attention_modelrrrn_headsrconv_norm_typerdropout dropout_att pos_bias_u pos_bias_vruse_biasruse_pytorch_sdpa_backendsrK);super__init__rn_layersr;rrrrrrrrrr_calc_context_sizesatt_context_size_allrrrmathsqrtrr pre_encoderr ReLULinearrreduction_subsamplingreduction_position _feat_out Parameterr9Tensorinitzeros_pos_emb_max_lenrpos_encmax ValueErrorrr ModuleListlayersrangerappendout_projset_max_audio_length use_pad_masksetup_streaming_paramsr3rlenlayer_drop_probsinterctc_capture_at_layers)-rArrrrcausal_downsamplingrrrsubsampling_conv_channelsrrrff_expansion_factorrrrrrxscaling untie_biasesrrrrrrdropout_pre_encoder dropout_embrrrrrrrrrrrd_headrrilayer __class__rKrLr!s0 )                    zConformerEncoder.__init__cCs|dur|dd}|dd}|j|||||d}|j|dd}t|dkr)|S|ddurC|ddurC|d durC|d|dfS|d|d|ddd|ddd|d fS) z] Forward function for model export. Please see `forward()` for more details. Nrr(roF)keep_all_outputsr,)r@forward_internalstreaming_post_processr)rArXrYrGrHrIretsrKrKrLforward_for_exports*   $z#ConformerEncoder.forward_for_exportcCst|dkr|d|ddddfS|\}}}}}|dur;|jjdkr;|jjdkr;|dddd|jj dddf}|jjdkr`|rH|jdkr`|ddddd|jjf}tj||jjd}|||||fS)z Post-process the output of the forward function for streaming. Args: rets: The output of the forward function. keep_all_outputs: Whether to keep all outputs. rrr(Nrxr)rr4last_channel_cache_size valid_out_lenrr9clamp)rArrencoded encoded_lenrgrhrirKrKrLr s   z'ConformerEncoder.streaming_post_processcCs|s|jd|jkrtd|jd|jdd|r2|jd|jkr2td|jd|jdd|rC|j|d|j|jd n |j|d|jd |j||||||d S) a Forward function for the ConformerEncoder accepting an audio signal and its corresponding length. The `audio_signal` input supports two formats depending on the `bypass_pre_encode` boolean flag. This determines the required format of the input variable `audio_signal`: (1) bypass_pre_encode = False (default): `audio_signal` must be a tensor containing audio features. Shape: (batch, self._feat_in, n_frames) (2) bypass_pre_encode = True: `audio_signal` must be a tensor containing pre-encoded embeddings. Shape: (batch, n_frame, self.d_model) zFIf bypass_pre_encode is False, audio_signal should have shape (batch, z", n_frame) but got last dimension .ruzNIf bypass_pre_encode is True, audio_signal should have shape (batch, n_frame, z) but got last dimension r) seq_lengthr+)rGrHrIrZ) shaper;rrupdate_max_seq_lengthsizerr+r)rArXrYrGrHrIrZrKrKrLforward$s:zConformerEncoder.forwardc Cs|dur|j|df|dtj|jd}|jr,t|jdkr,tj |j|j dd}n|j }|st |dd}t |jtjrE||}n2|j||d\}}|tj}|jjdkrw|durw|dd|jjdddf}||jjjdd }|jdur|durtd |d}|dur|jj} ||jj} || }|| } t|| } n|} d} d} d} |j|| d \}}|j|| || |jd \}}|dur|dd| df}|dur|dd| df}g}g} tt|j|jD]\}\}}|}|dur||}||}nd}d}|||||||d }|dur&|\}}}| || ||jrJ|dkrJt!d|k}|r@|d|}n ||d||}|j|krt|j"||d\}}|d}|j|| d \}}|j|||| |jd \}}|#t$|ddr|j%dur|j&'di'dg|_%||j%vr|}|j(dur|(|}|j)d|t |ddd|j)d||dq|j(dur|(|}|jdkr|j"||d\}}t |dd}|jtjd}|durtj*| dd} tj*|dd}||| |tj|| | dfS||fS)a The `audio_signal` input supports two formats depending on the `bypass_pre_encode` boolean flag. This determines the required format of the input variable `audio_signal`: (1) bypass_pre_encode = False (default): `audio_signal` must be a tensor containing audio features. Shape: (batch, self._feat_in, n_frames) (2) bypass_pre_encode = True: `audio_signal` must be a tensor containing pre-encoded embeddings. Shape: (batch, n_frame, self.d_model) `bypass_pre_encode=True` is used in cases where frame-level, context-independent embeddings are needed to be saved or reused (e.g., speaker cache in streaming speaker diarization). Nrrur.r+r()weightsr)xlengths)minz4Caching with reduction feature is not supported yet!)r cache_len)rpadding_lengthmax_audio_lengthoffsetr+)ratt_maskpos_embpad_maskrGrHr}? model_guidinterctccapture_layersinterctc/layer_output_)nametensorinterctc/layer_length_)r.)dimr)+new_fullrr9r=r+trainingrrrandomchoicesrrr@r5rr rtor4drop_extra_pre_encodedrrrrcache_drop_sizenegr _create_masks enumerateziprrrrandris_access_enabledgetattrr access_cfggetrregister_accessible_tensorstack)rArXrYrGrHrIrZcur_att_context_sizerrcache_keep_sizerrrgrrrrhlth drop_probroriginal_signalcache_last_channel_curcache_last_time_cur should_drop_lth_audio_signalrKrKrLrSs                     z!ConformerEncoder.forward_internalrcCsb|jr#tjr#tj|gtj|d}tjj|tjjjd| }||j kr/| |dSdS)z Updates the maximum sequence length for the model. Args: seq_length (int): New maximum sequence length. device (torch.device): Device to use for computations. r)opN) rr9 distributedis_initializedrfloat32 all_reduceReduceOpMAXintitemrr)rArr+global_max_lenrKrKrLrs   z&ConformerEncoder.update_max_seq_lengthcCs6||_t|j}t|j}|j|||dS)z Sets maximum input length. Pre-calculates internal seq_range mask. Args: max_audio_length (int): New maximum sequence length. N)rr1r2r+r.r extend_pe)rArr+r.rKrKrLrsz%ConformerEncoder.set_max_audio_lengthcCs |jdkrtjd||tj|d}|jdkr3|ddkr$|j|d d}|ddkr2|j|dd}ng|jdkr|ddkrN|ddkrM|j|d d}nL|dd}|ddkra|d|}nd }tjd|tj|j d} tj | |d d } | d| d} t t | |t| d} t || d}nd}tjd||d |dd| dk} |durtjd||d |dd| dk} | | } |dur| dd|dg}t ||dd }|ddd|d|f}t |||j }|}| } | |fS)Nrr(rrxr)diagonalchunked_limitedru'trunc) rounding_moder*r)rr9onesboolrtriutrilaranger(r+div unsqueeze logical_andlegeexpandrrepeatr@r )rArrrrr+rr6left_chunks_num chunk_idx diff_chunkschunked_limited_maskr pad_mask_offpad_mask_for_att_maskrKrKrLr s\            zConformerEncoder._create_maskscCs|j}||_|S)z Enables or disables the pad mask and assign the boolean state `on`. Returns: mask (bool): The current state of the pad mask. )r)rAonmaskrKrKrLenable_pad_maskTsz ConformerEncoder.enable_pad_maskc Cs|rZt|}t|dtr|g}t|D]D\}}t|tr#t|||<|dkrX|ddkrD|d|dddkrDtd|d|d|ddkrXt|dkrXtd|dqnd d gg}|r|t|t|krmtd t|}t|dkr{td n d t|gt|}|durt|trt|}t|tst|tstd |dkr|ddg}n"|d|dd|krtd|j dn |dd|ddg}||d||fS)Nrr-r(zatt_context_size[z][0] % (att_context_size[z][1] + 1) should be zero!z Right context (att_context_size[z5][1]) can not be unlimited for chunked_limited style!ruzIThe size of the att_context_probs should be the same as att_context_size.zTThe sum of numbers in att_context_probs should be equal to one to be a distribution.rzVInvalid conv_context_size! It should be the string 'causal' or a list of two integers.causalzInvalid conv_context_size: !r) r7r5r(rr rrsumstrr) rArrrrrrratt_csrKrKrLr`sR  $    z$ConformerEncoder._calc_context_sizescCs<||jvrtd|d|j|dur||_|dS)z Sets the default attention context size from `att_context_size` argument. Args: att_context_size (list): The attention context size to be set. zatt_context_size=z5 is not among the list of the supported look-aheads: N)rr&warningrr)rArrKrKrLset_default_att_context_sizes  z-ConformerEncoder.set_default_att_context_sizer.r6 shift_size left_chunksr max_contextc Cst}|dur |j}|dur |dkrtd|d}|||_n*|jdkr-|d}d|_n|jdkrE|d|j|jd|j}||_nd|_d}|dur\|ddkrX|dn||_n!|durx|ddkrj|dn||_t d|jdn|||_t |j d r|j }nd}t |tr|d|j||d|j|g|_n|d||_t |tr|d|d||j|d|d||jg|_n |d||j|_t |jtr|jd|d|jd|_n|j|j|_t |j d r|j |_nd|_t |jtr%|jddkr!d|jdd|j|_n d|_n|j|j|_|jD]} t | d rMt | trC|j| _t | trM|j| _q1||_dS) a This function sets the needed values and parameters to perform streaming. The configuration would be stored in self.streaming_cfg. The streaming configuration is needed to simulate streaming inference. Args: chunk_size (int): overrides the chunk size shift_size (int): overrides the shift size for chunks left_chunks (int): overrides the number of left chunks visible to each chunk max_context (int): the value used for the cache size of last_channel layers if left context is set to infinity (-1) Defaults to -1 (means feat_out is d_model) Nr(z7chunk_size needs to be a number larger or equal to one.r-rrxz/left_chunks is not set. Setting it to default: rget_sampling_framesget_streaming_cache_size_max_cache_len)r rrr rrrrr&rKhasattrrrPr5r7rr6rMrrQr8r rmodulesrrr4) rAr6rMrNrrOr4lookahead_stepssampling_framesmrKrKrLrs|              z'ConformerEncoder.setup_streaming_paramsc Cs|dur t|j}|dkrtj}ntj}|jd}|t|j||j j |j f||d}|t|j||j |f||d}|dkr|tj dt ||j j |f|tjd} t|D]&} d|dd| | | dddf<| | dkrzd|dd| ddddf<qTn tj||tjd} ||| fS)Nrr-)r1r2r+r9r:zerosrrrr4rrr<rr=r) rAr/r.r+r0 create_tensorlast_time_cache_sizerGrHrIrrKrKrLr> sH       z(ConformerEncoder.get_initial_cache_stater update_configr+c Cs^|rt|}n|j}|dur|j}|dkrt|dkrtd|dkr6t|jj|jj|jj |j |jj d}n4|dkrNt ||jj|jj|jj |j |jj d}n|dkrbt |jj|jj|jj |j d }ntd |d |durt|j|d }|`||_||_||_||j |D]\}}t|tkr|dkrt|jj|jj|jj|ddd|j|jd }n=|dkrt|jj|jj|jj|d|dd|j|jd }n!|dkrt|jj|jj|jj|d|j|jd}ntd|d|dur|j|d }|j|jdd|`||_||_q|r-t|j||j_||j_WddS1s&wYdSdS)a Update the self_attention_model which changes the positional encoding and attention layers. Args: self_attention_model (str): type of the attention layer and positional encoding 'rel_pos': relative positional embedding and Transformer-XL 'rel_pos_local_attn': relative positional embedding and Transformer-XL with local attention using overlapping windows. Attention context is determined by att_context_size parameter. 'abs_pos': absolute positional embedding and Transformer If None is provided, the self_attention_model isn't changed. Defaults to None. att_context_size (List[int]): List of 2 ints corresponding to left and right attention context sizes, or None to keep as it is. Defaults to None. update_config (bool): Whether to update the config or not with the new attention model. Defaults to True. device (torch.device): If provided, new layers will be moved to the device. Defaults to None. Nrrrrwrrrrrrr*)n_headn_featr max_cache_lenrrrr) r\r]rr^rrrrr)r\r]rr^rr'z|' is not not a valid value for 'self_attention_model', valid values can be from ['rel_pos', 'rel_pos_local_attn', 'abs_pos']F)strict)r7rrrrr_cfgrrrrrrrr rr named_modulestyperrrrrrrrload_state_dict self_attn state_dictr ) rArrr[r+ new_pos_encrrWnew_attnrKrKrLchange_attention_model4s         $z'ConformerEncoder.change_attention_modelrcCs,t|jds tddS|jj|ddS)a Update the conv_chunking_factor (int) Default is 1 (auto) Set it to -1 (disabled) or to a specific value (power of 2) if you OOM in the conv subsampling layers Args: subsampling_conv_chunking_factor (int) 'change_subsampling_conv_chunking_factorzPModel pre_encoder doesn't have a change_subsampling_conv_chunking_factor method N)r)rSrr&inforj)rArrKrKrLrjs  z8ConformerEncoder.change_subsampling_conv_chunking_factor)r(r))#ruFrvr,r(ruNNr(r,rwr,NNrxTTryrzr{NTr|r|r|r}r}r~r(rr(FFNTNNN)T)NNNF)NNNNr.)NNTN)&__name__ __module__ __qualname____doc__rEpropertyr]rarkrmrrrtfloatrIr(r2rrrrrrrrr rErrLr7rr9r$r>rr+rirj __classcell__rKrKrrLr'>s t*        !"#$%'M  2 ) 8 0 k* cseZdZdZdedefddZdefddZdde ed efd d Z de efddZ de fddZdeeffdd ZZS)ConformerEncoderAdapterzLThis class inherits from ConformerEncoder and wraps the adapter mixin class.rcfgcCs&||}|jD]}|||qdSN)_update_adapter_cfg_input_dimr add_adapter)rArruconformer_layerrKrKrLrxs  z#ConformerEncoderAdapter.add_adapterreturncCstdd|jDS)NcSsg|]}|qSrK)is_adapter_available).0ryrKrKrL z@ConformerEncoderAdapter.is_adapter_available..)anyrr\rKrKrLr{sz,ConformerEncoderAdapter.is_adapter_availableNTenabledcCs|jD] }|j||dqdS)N)rr)rset_enabled_adapters)rArrryrKrKrLrs z,ConformerEncoderAdapter.set_enabled_adapterscCs2tg}|jD] }||qtt|}|Srv)rqrupdateget_enabled_adapterssortedr7)rAnamesryrKrKrLrs   z,ConformerEncoderAdapter.get_enabled_adapterscCstj|||jd}|S)N) module_dim)rupdate_adapter_cfg_input_dimr)rArurKrKrLrwsz5ConformerEncoderAdapter._update_adapter_cfg_input_dimcs8t}t|dkr|tjtjtjg|}|S)Nr)rget_accepted_adapter_typesrset_accepted_adapter_typesrLINEAR_ADAPTER_CLASSPATHMHA_ADAPTER_CLASSPATHRELMHA_ADAPTER_CLASSPATH)rAtypesrrKrLrs  z2ConformerEncoderAdapter.get_accepted_adapter_types)NT)rmrnrorprIdictrxr2r{rrrrrrwrrcrrsrKrKrrLrtsrtc s`eZdZdZ   ddedeedededef fd d Z dd e e j e j ffd d Z ZS)#ConformerMultiLayerFeatureExtractorai A wrapper module that extracts features from multiple layers of a ConformerEncoder, by reusing existing mechanisim for interctc loss. To use it, set `layer_idx_list` to specify the indices of layers to extract from. Also, you can specify an `aggretator` module to aggregate the features from different layers, default not aggregating. NFencoderlayer_idx_list aggregatordetachconvert_to_cpucs|t||_dd|D|_|jD]}|dks |t|jkr.td|dt|jdqd|ji||d|_||_dS) NcSsg|]}t|qSrK)r()r|lyr_idxrKrKrLr} r~z@ConformerMultiLayerFeatureExtractor.__init__..rz layer index z out of range [0, )r)rrr) rrrrrrrenc_access_cfgr)rArrrrrrrrKrLrs   z,ConformerMultiLayerFeatureExtractor.__init__rzc Cst|jt|ddd}|j|jt|ddd|jdt|ddd|j|||||d}i}||jD]} | D]} | drL| |vrLt d| dq9| | q5g} g} |j D]@} z|d | }|d | }Wnt y{t d | d wt |d kst |d krt d| |d| |dqZ|j|j|t|ddd|jdur|| | S| | fS)Nr)guidT)access_enabledr)rXrYrGrHrIz interctc/zlayer z has been logged multiple times!rrzIntermediate layer zS was not captured! Check the layer index and the number of ConformerEncoder layers.r(z4Make sure encoder.forward is called exactly one timer)rrupdate_access_cfgrset_access_enabledrget_module_registryvalues startswith RuntimeErrorrrKeyErrorrrreset_registryr)rArXrYrGrHrIold_access_flagrtotal_registrymodule_registrykey encoded_listencoded_len_list layer_idx layer_outputs layer_lengthsrKrKrLrsJ        z+ConformerMultiLayerFeatureExtractor.forward)NFFrl)rmrnrorpr'rr(rr2rrr9rrrsrKrKrrLrs( r) base_class adapter_classc@s6eZdZUdZdZeeed<dZee e ed<dS)ConformerChangeConfiga Change self_attention_model for Conformer. Options: 'rel_pos': relative positional embedding and Transformer-XL 'rel_pos_local_attn': relative positional embedding and Transformer-XL with local attention using overlapping chunks. Attention context is determined by att_context_size parameter. 'abs_pos': absolute positional embedding and Transformer Nrr) rmrnrorprrrI__annotations__rrr(rKrKrKrLrMs  r)Crr collectionsr dataclassesrtypingrrrrr9torch.distributed omegaconfrr r r #nemo.collections.asr.models.configsr +nemo.collections.asr.parts.mixins.streamingr 2nemo.collections.asr.parts.submodules.causal_convsr7nemo.collections.asr.parts.submodules.conformer_modulesr:nemo.collections.asr.parts.submodules.multi_head_attentionrrrrrr1nemo.collections.asr.parts.submodules.subsamplingrrr nemo.collections.asr.parts.utilsr5nemo.collections.asr.parts.utils.regularization_utilsrnemo.core.classes.commonrnemo.core.classes.exportablernemo.core.classes.mixinsrrnemo.core.classes.modulernemo.core.neural_typesr r!r"r#r$r% nemo.utilsr&__all__r'AdapterModuleMixinrtrget_registered_adapterregister_adapterrrKrKrKrLsP               -S