o }oiQ6@sddlZddlZddlmZmZmZddlZddlmZddl mZddlm Z ddl m Z ddlmZmZddlmZddlmZdd lmZmZmZmZmZGd d d ej jZGd d d ej jZGdddeZGdddeZGdddej j!Z"dS)N)DictListTuple) DictConfig)nn) functional)form_attention_masktransformer_weights_init)TransformerEncoder) NeuralModule)AcousticEncodedRepresentation AudioSignal LengthsType NeuralTypeSpectrogramTypec@seZdZdZddZdS) TransposeLastzM Transposes last dimension. Useful for adding to a sequential block. cCs |ddS)N) transposeselfxr`/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/modules/wav2vec_modules.pyforward)s zTransposeLast.forwardN)__name__ __module__ __qualname____doc__rrrrrr$s rcs$eZdZfddZddZZS)SamePadcst|ddk|_dS)Nr)super__init__remove)r kernel_size __class__rrr".s zSamePad.__init__cCs$|jr|ddddddf}|S)Nr)r#rrrrr2szSamePad.forward)rrrr"r __classcell__rrr%rr-s rcs~eZdZdZeddZeddZ    dd eee e fd e d e ffdd Z ddZ ddZddZddZZS)ConvFeatureEncodera9 Encoder used to isolate features in raw audio for Wav2Vec style training. Treated as preprocessor module in NeMo ASR training. Defaults values are for base model found in Baeski et al (https://arxiv.org/abs/2006.11477), save for use of layer normalization as default schema. (Chosen for stability.) cCs$tdt|jdttdtdS)aReturns definitions of module input ports. input_signal: 0: AxisType(BatchTag) 1: AxisType(TimeTag) input_signal_length: 0: AxisType(BatchTag) Note: length is in number of samples, not seconds )BT)freqr)) input_signallength)rr _sample_ratetuplerrrrr input_types@s zConvFeatureEncoder.input_typescCtdtttdtdS)aQReturns definitions of module output ports. For compatibility, processed features are treated as Spectrogram types processed_signal: 0: AxisType(BatchTag) 1: AxisType(ChannelTag) 2: AxisType(ProcessedTimeTag) processed_signal_length: 0: AxisType(BatchTag) r)Cr*r))processed_signalprocessed_signal_lengthrrr/rr0rrr output_typesO zConvFeatureEncoder.output_types layer_normF?T conv_layersextractor_mode conv_biasc st||_||_ dfdd }d}||_t|_||_t |D];\} } t | dks6Jdt | | d| d| d } } |j ||| | |jd k|jd koX| d k|d }q$|jdd} | |krqt | |nd|_t||_dS)NFc sfdd}|o |dusJd|r,t|tttjddttS|r>t|tjddtSt|tS)Ncs&tjd}tj|j|S)N)stridebias)rConv1dinitkaiming_normal_weight)convr?kn_inn_outr@rr make_convpsz=ConvFeatureEncoder.__init__..block..make_convFz'layer norm and group norm are exclusiveT)elementwise_affine)affine)r Sequentialr LayerNormGELU GroupNorm)rIrJrHr@ is_layer_norm is_group_normr?rKdimrGrblockms z*ConvFeatureEncoder.__init__..blockzinvalid conv definition: emb_dimr$r@r: group_normr)rRrSr?r)FFF)r!r" grad_multnormalize_input layer_cfgr ModuleListr=mode enumeratelenstrappendLinearpost_extract_projrOr:)rr=r>r?feature_grad_multnormalize_audio embedding_dimrVin_diclrHr@final_conv_dimr%rTrr"_s8   zConvFeatureEncoder.__init__cCs|jD]}||}q|SN)r=)rrrFrrr apply_layerss  zConvFeatureEncoder.apply_layerscCsxt.t|dD]}||d||f}t||j}|||d||f<q Wd|S1s5wY|S)Nr)torchno_gradrangesizeFr:shape)rsourcelengthsrjorignormrrr normalizes  zConvFeatureEncoder.normalizecCs|jr |||}|d}|jdkr%||}|jdkr$t||j}nt ||}Wdn1s9wY| dd}|j durN| |}|j dkrX| |}| dd}|j |d}||fS)NrWrr;r r:) audio_lengths)r\ry unsqueezer[rn GradMultiplyapplyrorprrer_r: get_lengths)rr,r-r5r6rrrrs&              zConvFeatureEncoder.forwardcCs8|jD]}|d}|d}tj|||ddd}q|S)Nr$r@floor) rounding_moderW)r]rodiv)rrzrFkernelr@rrrr~s zConvFeatureEncoder.get_lengths)r:Fr;Tr<)rrrrpropertyr1r8rrrbintboolr"rnryrr~r'rrr%rr(8s,  ? r(csbeZdZdZddededeffdd Zedd Zed d Z d d Z dddZ ddZ Z S)Wav2VecTransformerEncoderad Encoder module following Transformer encoder paradigm as described in Vaswani et al. (https://arxiv.org/abs/1706.03762). Used for Wav2Vec style encoding of context vectors as described by in Baeski et al (https://arxiv.org/abs/2006.11477). Takes convolutional encodings of all time steps and adds to features before applying series of self-attention layers. Example configs may be found at: https://github.com/NVIDIA/NeMo/tree/main/examples/asr/conf/wav2vec Args: layer_drop: Floating point value specifying proportion of module for layer dropout (See Fan et al. https://arxiv.org/pdf/1909.11556.pdf). If non-zero, each layer will draw from uniform probability to determine if applied in current forward call. Occurs only during training step pos_embed: Config specifying parameters for contextual embedding convolutions. Module configures convolutional padding to maintain number of time steps Must contain following: embedding_dim: Depth/number of channels of each time step from feature encoding conv_pos: Kernel size for convolution conv_pos_groups: Number of groups for convolution transformer: Config for transformer encoder. Uses self-attention layers found in: nemo.collections.nlp.modules.common.transformer Must contain followign: num_layers: Number of attention layers hidden_size: Expected input depth (embedding size between model layers) inner_size: Depth of embeddings within feed-forward sections of encoder layers num_attention_heads: Number of attention heads attn_score_dropout: Probability of dropout applied to attention scores attn_layer_dropout: Probability of dropout applied to the output of the attention layers (prior to normalization) ffn_dropout: Probability of dropout applied to feed-forward modules hidden_act: Activation function for hidden layers pos_embed transformer layer_dropcstjd i||j}tj|||j|jd|jd|_||_|j |_ t dd|j |j|j}tj j|jjd|dtj |jjdtjj|jddd|_t|jt|jt|_t||_|d d dS) Nr )r$paddinggroupsr;r)meanstdrE)namerUcSs t|ddS)NF)xavier)r )rrrrs z4Wav2VecTransformerEncoder.__init__..r)r!r"rhrrBconv_posconv_pos_groupspos_convrattn_layer_dropoutdropoutmathsqrtrCnormal_rE constant_rAutils weight_normrNrrPrOr:r})rrrrrYrr%rrr"s$  z"Wav2VecTransformerEncoder.__init__cCr2)a1Returns definitions of module output ports. We treat features as SpectrogramType for Nemo compatibility audio_signal: 0: AxisType(BatchTag) 1: AxisType(ChannelTag) 2: AxisType(ProcessedTimeTag) length: 0: AxisType(BatchTag) r3r)) audio_signalr-r7r0rrrr1r9z%Wav2VecTransformerEncoder.input_typescCr2)a@Returns definitions of module output ports. We're using SpectrogramType for now to keep things Nemo safe processed_signal: 0: AxisType(BatchTag) 1: AxisType(ChannelTag) 2: AxisType(ProcessedTimeTag) processed_length: 0: AxisType(BatchTag) r3r))r5processed_length)rr r/rr0rrrr8 r9z&Wav2VecTransformerEncoder.output_typescCs|||}t|D]\}}d||dd|df<q ||}||}|dd}||}|j||d}|dd}||fS)NrrWr ) padding_mask)create_padding_maskr`rrr:apply_transformer)rrr-ridxra signal_conv context_embrrrr0s     z!Wav2VecTransformerEncoder.forwardNcCsnt|}|jr%|jr%t|jD]\}}t}||jkr"||||}q|St|jD] \}}||||}q*|Srm)rrtrainingr`layersrandom)rrrencoder_attn_mask_layerprrrrEs  z+Wav2VecTransformerEncoder.apply_transformercCs>t|}tj||jd}|t|||dktj}|S)N)devicerW) maxroarangerexpandrar{typeuint8)rr-max_lenrrrrrSs"z-Wav2VecTransformerEncoder.create_padding_mask)rrm)rrrrrfloatr"rr1r8rrrr'rrr%rrs   rc@s$eZdZeddZeddZdS)r|cCs||_||}|Srm)scalenew)ctxrrresrrrr_s zGradMultiply.forwardcCs||jdfSrm)r)rgradrrrbackwardeszGradMultiply.backwardN)rrr staticmethodrrrrrrr|^s  r|)#rrtypingrrrro omegaconfromegaconf.dictconfigrtorch.nnrrsnemo.collections.common.partsrr /nemo.collections.nlp.modules.common.transformerr nemo.core.classes.moduler nemo.core.neural_typesr r rrrModulerrr(rautogradFunctionr|rrrrs&