o ¡¿¯ig7ã@sÂddlZddlZddlZddlZddlmmZddl m Z m Z ddl m Z ddlmZmZGdd„dejjƒZGdd„dejjƒZd d „ZGd d „d ejjƒZGd d„dejjƒZdd„ZdS)éN)Úpack_padded_sequenceÚpad_packed_sequence)Úget_vgg2l_odim)Ú make_pad_maskÚ to_devicecó,eZdZdZd‡fdd„ Zd dd„Z‡ZS) ÚRNNPa†RNN with projection layer module :param int idim: dimension of inputs :param int elayers: number of encoder layers :param int cdim: number of rnn units (resulted in cdim * 2 if bidirectional) :param int hdim: number of projection units :param np.ndarray subsample: list of subsampling numbers :param float dropout: dropout rate :param str typ: The RNN type Úblstmc sêtt|ƒ ¡|ddk}tj |¡D]M} | dkr|} n|} d|vr&tjjntjj } | | |d|dd} t |d|r:dnd | f| ƒ|rSt |d | tj  d ||¡ƒqt |d | tj  ||¡ƒq||_ ||_ ||_||_||_||_dS) NrÚbÚlstméT)Ú num_layersÚ bidirectionalÚ batch_firstz%s%dÚbirnnÚrnnúbt%dé)ÚsuperrÚ__init__ÚsixÚmovesÚrangeÚtorchÚnnÚLSTMÚGRUÚsetattrÚLinearÚelayersÚcdimÚ subsampleÚtypÚbidirÚdropout) ÚselfÚidimrr Úhdimr!r$r"r#ÚiÚinputdimÚRNNr©Ú __class__©ú\/home/ubuntu/.local/lib/python3.10/site-packages/espnet/nets/pytorch_backend/rnn/encoders.pyrs(  ÿ  z RNNP.__init__Nc sˆt |jjdt|ƒ¡g}tj |j¡D]¨}t |t j ƒs#t   |¡}t || ¡dd}t||jr3dndt|ƒƒ}|jrA| ¡|durL|jrLt|ƒ}|||durTdn||d\}} | | ¡t|dd\} }|j|d‰ˆdkr‹| dd…ddˆ…f} t   ‡fd d „|Dƒ¡}t|d |ƒ} | |  ¡ d |  d ¡¡ƒ} |  |  d¡|  d¡d ¡}||jdkr¾t  tj||jd¡}q|||fS)aXRNNP forward :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, idim) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor prev_state: batch of previous RNN states :return: batch of hidden state sequences (B, Tmax, hdim) :rtype: torch.Tensor ú input lengths: T©rrrN©Úhxr csg|] }t|dƒˆ‘qS©r )Úint)Ú.0r(©Úsubr-r.Ú Ssz RNNP.forward..réÿÿÿÿrr)Úp)ÚloggingÚdebugr,Ú__name__ÚstrrrrrÚ isinstancerÚTensorÚtensorrÚcpuÚgetattrr#ÚtrainingÚflatten_parametersrÚreset_backward_rnn_stateÚappendrr!Ú contiguousÚviewÚsizeÚtanhÚFr$) r%Úxs_padÚilensÚ prev_stateÚ elayer_statesÚlayerÚxs_packrÚysÚstatesÚys_padÚprojection_layerÚ projectedr-r6r.Úforward6s6    ÿ € z RNNP.forward©r ©N©r=Ú __module__Ú __qualname__Ú__doc__rrXÚ __classcell__r-r-r+r.r s rcr) r*a9RNN module :param int idim: dimension of inputs :param int elayers: number of encoder layers :param int cdim: number of rnn units (resulted in cdim * 2 if bidirectional) :param int hdim: number of final projection units :param float dropout: dropout rate :param str typ: The RNN type r csˆtt|ƒ ¡|ddk}d|vrtjj|||d||dn tjj|||d||d|_|r7tj |d|¡|_ ntj ||¡|_ ||_ dS)Nrr r T)rr$rr) rr*rrrrrÚnbrnnrÚl_lastr")r%r&rr r'r$r"r#r+r-r.ris0  øú úö z RNN.__init__Nc CsÒt |jjdt|ƒ¡t|tjƒst |¡}t ||  ¡dd}|j r)|j   ¡|dur5|j jr5t|ƒ}|j ||d\}}t|dd\}}t | | ¡ d| d¡¡¡¡}| | d¡| d ¡d¡}|||fS) aVRNN forward :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, D) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor prev_state: batch of previous RNN states :return: batch of hidden state sequences (B, Tmax, eprojs) :rtype: torch.Tensor r/Tr0Nr1r9rrr )r;r<r,r=r>r?rr@rArrBrDr`rErrFrrKrarHrIrJ) r%rMrNrOrRrSrTrUrWr-r-r.rX…s   ÿ z RNN.forwardrYrZr[r-r-r+r.r*^s r*cCs<t|ttfƒr|D] }d|ddd…<q |Sd|ddd…<|S)zeSets backward BRNN states to zeroes Useful in processing of sliding windows over the inputs çr Nr)r?ÚlistÚtuple)rTÚstater-r-r.rF¥s ÿrFcs*eZdZdZd‡fdd„ Zdd„Z‡ZS)ÚVGG2LzIVGG-like module :param int in_channel: number of input channels r csxtt|ƒ ¡tjj|ddddd|_tjjdddddd|_tjjdddddd|_tjjdddddd|_ ||_ dS)Né@ér )ÚstrideÚpaddingé€) rrfrrrÚConv2dÚconv1_1Úconv1_2Úconv2_1Úconv2_2Ú in_channel)r%rqr+r-r.r¸s  zVGG2L.__init__cKs`t |jjdt|ƒ¡| | d¡| d¡|j| d¡|j¡ dd¡}t   |  |¡¡}t   |  |¡¡}t j |dddd}t   | |¡¡}t   | |¡¡}t j |dddd}t |¡rd| ¡ ¡}ntj|tjd}tjt |d¡tjd}tjt tj|tjdd¡tjd ¡}| dd¡}| ¡ | d¡| d¡| d¡| d¡¡}||d fS) a%VGG2L forward :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, D) :param torch.Tensor ilens: batch of lengths of input sequences (B) :return: batch of padded hidden state sequences (B, Tmax // 4, 128 * D // 4) :rtype: torch.Tensor r/rr rT)riÚ ceil_mode)ÚdtyperhN)r;r<r,r=r>rIrJrqÚ transposerLÚrelurmrnÚ max_pool2drorprÚ is_tensorrBÚnumpyÚnpÚarrayÚfloat32ÚceilÚint64ÚtolistrH)r%rMrNÚkwargsr-r-r.rXÂs:üû ÿþ "ÿ z VGG2L.forwardr3r[r-r-r+r.rf²s rfcs.eZdZdZ d‡fdd„ Zd dd„Z‡ZS) ÚEncodera×Encoder module :param str etype: type of encoder network :param int idim: number of dimensions of encoder network :param int elayers: number of layers of encoder network :param int eunits: number of lstm units of encoder network :param int eprojs: number of projection units of encoder network :param np.ndarray subsample: list of subsampling numbers :param float dropout: dropout rate :param int in_channel: number of input channels r c sZtt|ƒ ¡| d¡ d¡} | dvrt d¡| d¡rn|ddkrGtj   t |ƒt t ||d|||||| dg¡|_t d|  ¡d ¡n"tj   t |ƒtt ||d||||| dg¡|_t d|  ¡d ¡d |_dS|ddkrtj   t ||||||| dg¡|_t |  ¡d ¡ntj   t|||||| dg¡|_t |  ¡d ¡d|_dS)NÚvggr:)r Úgrur Úbgruz:Error: need to specify an appropriate encoder architecturer9)rq)r"zUse CNN-VGG + z P for encoderz for encoderéz( with every-layer projection for encoderz without projection for encoderr )rr€rÚlstripÚrstripr;ÚerrorÚ startswithrrÚ ModuleListrfrrÚencÚinfoÚupperr*Úconv_subsampling_factor) r%Úetyper&rÚeunitsÚeprojsr!r$rqr"r+r-r.rýsX    ùþÿ úþÿ  ÿÿ zEncoder.__init__Nc CsŒ|dur dgt|jƒ}t|ƒt|jƒksJ‚g}t|j|ƒD]\}}||||d\}}}| |¡qt|t|ƒ d¡ƒ}| |d¡||fS)anEncoder forward :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, D) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor prev_state: batch of previous encoder hidden states (?, ...) :return: batch of hidden state sequences (B, Tmax, eprojs) :rtype: torch.Tensor N)rOr9rb)ÚlenrŠÚziprGrrÚ unsqueezeÚ masked_fill) r%rMrNÚ prev_statesÚcurrent_statesÚmodulerOrTÚmaskr-r-r.rX3s  zEncoder.forwardr3rZr[r-r-r+r.r€ðs  ÿ6r€c Cs¤t|ddƒ}|dkrt|j||j|j|j||jƒS|dkrKtj  ¡}t |ƒD]"}t|j||||j||j||j|||j|ƒ}|  |¡q&|St d  |¡ƒ‚)a.Instantiates an encoder module given the program arguments :param Namespace args: The arguments :param int or List of integer idim: dimension of input, e.g. 83, or List of dimensions of inputs, e.g. [83,83] :param List or List of List subsample: subsample factors, e.g. [1,2,2,1,1], or List of subsample factors of each encoder. e.g. [[1,2,2,1,1], [1,2,2,1,1]] :rtype torch.nn.Module :return: The encoder module Únum_encsr z0Number of encoders needs to be more than one. {})rCr€rŽrrrÚ dropout_raterrr‰rrGÚ ValueErrorÚformat)Úargsr&r!r™Úenc_listÚidxrŠr-r-r.Ú encoder_forKs8 ù  ù ÿr )r;rxryrrÚtorch.nn.functionalrÚ functionalrLÚtorch.nn.utils.rnnrrÚespnet.nets.e2e_asr_commonrÚ&espnet.nets.pytorch_backend.nets_utilsrrÚModulerr*rFrfr€r r-r-r-r.Ús QG > [