o i?!@sdZddlmZmZmZmZmZddlZddlm Z ddl m Z ddl mZddlmZmZddlmZdd lmZdd lmZmZdd lmZmZmZddlm Z dd lmZdd l m!Z!ddl"m#Z#de$dee$efde%dee%e%ffddZ&de$de%deee$efde'de'dee$eff ddZ(de$deee$efdee%fddZ)de$de$de$deeeeefeeefffd d!Z*dee$efd"ej+j,d#e%deee#eej+j-fe%ffd$d%Z.de$dee$efd&e$d'e$dee eff d(d)Z/dee$efd*e$d&e$d'e$d+e$de f d,d-Z0dee$efd.e$defd/d0Z1  1 2 3 4 4 5 5 6d@de$d7e%de$deee$efd8e%de$d9e$d:e$d;e$d+e$dd?Z2dS)Az-Set of methods to create custom architecture.)AnyDictListTupleUnionN)ConvolutionModule) EncoderLayer)get_activation) CausalConv1dConv1d)TransformerDecoderLayer)VGG2L)MultiHeadedAttentionRelPositionMultiHeadedAttention)PositionalEncodingRelPositionalEncodingScaledPositionalEncoding)PositionwiseFeedForward)MultiSequential)Conv2dSubsamplingnet_partblock num_blockreturncCs|d}|durtd||f|dkrhd}nI|dkrd_ffheadsd_hidden conformer>rrr use_conv_mod macaron_styledecoderz%Decoder does not support 'conformer'.r Tconv_mod_kernelzHBlock %d: 'use_conv_mod' is True but 'conv_mod_kernel' is not specified causal-conv1d>idimodim kernel_sizeencoderz)Encoder does not support 'causal-conv1d'.conv1dz"Decoder does not support 'conv1d.'zRWrong type. Currently supported: causal-conv1d, conformer, conv-nd or transformer.z_%s in %s in position %d: Expected block arguments : %s. See tutorial page for more information.rrrr%r&)get ValueErrorNotImplementedErrorissubset)rrr block_type argumentsblock_ior2a/home/ubuntu/.local/lib/python3.10/site-packages/espnet/nets/pytorch_backend/transducer/blocks.pyverify_block_arguments"sT    r4input_layer_type feats_dimblocks dropout_ratepos_enc_dropout_ratecCs~i}|ddd}|dkrd|d<n||d<||d<||d<||d<|d vr3|dd d|d <|S|ddd|d <|S) arPrepare input layer arguments. Args: input_layer_type: Input layer type. feats_dim: Dimension of input features. blocks: Blocks parameters for network part. dropout_rate: Dropout rate for input layer. pos_enc_dropout_rate: Dropout rate for input layer pos. enc. Return: input_block: Input block parameters. rrNr$c-embed dropout-ratepos-dropout-rater%r*rr&)r+)r5r6r7r8r9 input_blockfirst_block_typer2r2r3prepare_input_layerls r?csfddt|D}ddhdd|Dkrtdtdt|D]}||dd||d kr>td ||dfq#|d dS) zPrepare model body blocks. Args: net_part: Network part, either 'encoder' or 'decoder'. blocks: Blocks parameters for network part. Return: : Network output dimension. cs g|] \}}t||dqS))r4).0ibrr2r3 sz&prepare_body_model..rrcSsh|]}|dqS)rr2)rArCr2r2r3 sz%prepare_body_model..zO: transformer and conformer blocks can't be used together in the same net part.r@rz4Output/Input mismatch between blocks %d and %d in %s) enumerater-rangelenr,)rr7cmp_iorBr2rDr3prepare_body_models   rL pos_enc_typeself_attn_typecCsj|dkrt}n|dkrt}n|dkr!|dkr|dkrtdt}ntd|dkr/t}||fSt}||fS)aSGet positional encoding and self attention module class. Args: net_part: Network part, either 'encoder' or 'decoder'. pos_enc_type: Positional encoding type. self_attn_type: Self-attention type. Return: pos_enc_class: Positional encoding class. self_attn_class: Self-attention class. abs_posscaled_abs_posrel_posr( rel_self_attnz1'rel_pos' is only compatible with 'rel_self_attn'zFpos_enc_type should be either 'abs_pos', 'scaled_abs_pos' or 'rel_pos')rrr,rr-rr)rrMrN pos_enc_classself_attn_classr2r2r3get_pos_enc_and_att_classs rUrS padding_idxc Cs$|d}|d}|d}|d}|d}|jdkr|||}nd}|dkrDtjtj||tj|tj|tj|||d fS|d krQt||||d fS|d kr]t |||d fS|d krttjtjj |||d|||d fS|dkrtjtjj |||dtj|d fSt d|)a2Build input layer. Args: block: Architecture definition of input layer. pos_enc_class: Positional encoding class. padding_idx: Padding symbol ID for embedding layer (if provided). Returns: : Input layer module. subsampling_factor: Subsampling factor. rr%r&r;r<rNlinearr@conv2dvgg2lembed)rVr:zCInvalid input layer: %s. Supported: linear, conv2d, vgg2l and embed) __name__torchnn SequentialLinear LayerNormDropoutReLUrr Embeddingr-) rrSrV input_typer%r&r8pos_dropout_ratepos_enc_class_subsamplingr2r2r3build_input_layersR       rh pw_layer_typepw_activation_typecsnddddddd|dkrtd|dkr%tn|d kr+tfd d S) a_Build function for transformer block. Args: net_part: Network part, either 'encoder' or 'decoder'. block: Transformer block parameters. pw_layer_type: Positionwise layer type. pw_activation_type: Positionwise activation type. Returns: : Function to create transformer (encoder or decoder) block. rr;r<att-dropout-raterWz7Transformer block only supports linear pointwise layer.r(r"c s,tdtdtS)Nrr)rrr r2att_dropout_raterrr8rfrjtransformer_layer_classr2r3Dsz)build_transformer_block..)r+r-rr )rrrirjr2rmr3build_transformer_block"s   rqrTconv_mod_activation_typec sdd}dddd}dddd |d kr1t ||t|f ntd rAt||t|f rNtd t|f f d d S)aBuild function for conformer block. Args: block: Conformer block parameters. self_attn_type: Self-attention module type. pw_layer_type: Positionwise layer type. pw_activation_type: Positionwise activation type. conv_mod_activation_type: Convolutional module activation type. Returns: : Function to create conformer (encoder) block. rrr;rkr<rlr!r rWz)Conformer block only supports linear yet.r#cs>t d rnd rSdS)Nr)ConformerEncoderLayerr2 rnrconv_mod conv_mod_argsrr8 macaron_netmacaron_net_argsr!pw_layer pw_layer_argsrTr r2r3rps z'build_conformer_block..)r+rr r-r)rrTrirjrrrrfr2rtr3build_conformer_blockQs:   $r{r/c s|dkrtntdddddddddd d d d d f d dS)zBuild function for causal conv1d block. Args: block: CausalConv1d or Conv1D block parameters. Returns: : Function to create conv1d (encoder) or causal conv1d (decoder) block. r)strider@dilationgroupsbiasTzuse-batch-normFzuse-relur;rkc s(dddd S)Nr%r&r')r|r}r~rrelu batch_normr8r2r2 rr conv_classr}r8r~r|use_batch_normuse_relur2r3rpsz$build_conv1d_block..)r r r+)rr/r2rr3build_conv1d_blocks        r self_attnrOrWrrkrGr% repeat_blockpositional_encoding_typepositionwise_layer_typepositionwise_activation_typeinput_layer_dropout_rate input_layer_pos_enc_dropout_ratec Csg} t|||\}}t|||| | }t||}t||| \}}tt|D]5}||d}|dvr9t|||}n|dkrHt|||||| }n |dkrUt|||||}| |q%|dkrc| |} |t dd| D||fS)aBuild custom model blocks. Args: net_part: Network part, either 'encoder' or 'decoder'. idim: Input dimension. input_layer: Input layer type. blocks: Blocks parameters for network part. repeat_block: Number of times provided blocks are repeated. positional_encoding_type: Positional encoding layer type. positionwise_layer_type: Positionwise layer type. positionwise_activation_type: Positionwise activation type. conv_mod_activation_type: Convolutional module activation type. input_layer_dropout_rate: Dropout rate for input layer. input_layer_pos_enc_dropout_rate: Dropout rate for input layer pos. enc. padding_idx: Padding symbol ID for embedding layer. Returns: in_layer: Input layer all_blocks: Encoder/Decoder network. out_dim: Network output dimension. conv_subsampling_factor: Subsampling factor in frontend CNN. r)r$r)rrr@cSsg|]}|qSr2r2)rAfnr2r2r3rEsz build_blocks..) rUr?rLrhrIrJrr{rqappendr)rr%r5r7rrNrrrrrrrrV fn_modulesrSrTr=out_dim input_layerconv_subsampling_factorrBr/moduler2r2r3 build_blockssX(   r) rrrOrWrrrkrkrG)3__doc__typingrrrrrr]1espnet.nets.pytorch_backend.conformer.convolutionr3espnet.nets.pytorch_backend.conformer.encoder_layerrrs&espnet.nets.pytorch_backend.nets_utilsr 2espnet.nets.pytorch_backend.transducer.conv1d_netsr r @espnet.nets.pytorch_backend.transducer.transformer_decoder_layerr ,espnet.nets.pytorch_backend.transducer.vgg2lr 1espnet.nets.pytorch_backend.transformer.attentionrr1espnet.nets.pytorch_backend.transformer.embeddingrrr5espnet.nets.pytorch_backend.transformer.encoder_layerAespnet.nets.pytorch_backend.transformer.positionwise_feed_forwardr.espnet.nets.pytorch_backend.transformer.repeatr3espnet.nets.pytorch_backend.transformer.subsamplingrstrintr4floatr?rLrUr^Moduler_rhrqr{rrr2r2r2r3s            J  ) "   '  D   /  D+