o ´©iî6ã@sÒddlZddlZddlZddlZddlmmZddl m Z ddl m Z ddl m Z ddlmZddlmZGdd„dejjƒZGd d „d ejjƒZd d „ZGd d„dejjƒZGdd„dejjƒZdd„ZdS)éN)Úpack_padded_sequence)Úpad_packed_sequence)Úget_vgg2l_odim)Ú make_pad_mask)Ú to_devicecó,eZdZdZd‡fdd„ Zd dd„Z‡ZS) ÚRNNPa†RNN with projection layer module :param int idim: dimension of inputs :param int elayers: number of encoder layers :param int cdim: number of rnn units (resulted in cdim * 2 if bidirectional) :param int hdim: number of projection units :param np.ndarray subsample: list of subsampling numbers :param float dropout: dropout rate :param str typ: The RNN type Úblstmc sêtt|ƒ ¡|ddk}tj |¡D]M} | dkr|} n|} d|vr&tjjntjj } | | |d|dd} t |d|r:dnd | f| ƒ|rSt |d | tj  d ||¡ƒqt |d | tj  ||¡ƒq||_ ||_ ||_||_||_||_dS) NrÚbÚlstméT)Ú num_layersÚ bidirectionalÚ batch_firstz%s%dÚbirnnÚrnnúbt%dé)ÚsuperrÚ__init__ÚsixÚmovesÚrangeÚtorchÚnnÚLSTMÚGRUÚsetattrÚLinearÚelayersÚcdimÚ subsampleÚtypÚbidirÚdropout) ÚselfÚidimrr Úhdimr!r$r"r#ÚiÚinputdimÚRNNr©Ú __class__©ú]/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/language_model/rnn/encoders.pyrs$   z RNNP.__init__Nc s‚t |jjdt|ƒ¡g}tj |j¡D]¥}t |t j ƒs#t   |¡}t || ¡dd}t||jr3dndt|ƒƒ}| ¡|durI|jrIt|ƒ}|||durQdn||d\}} | | ¡t|dd\} }|j|d‰ˆdkrˆ| dd…ddˆ…f} t   ‡fd d „|Dƒ¡}t|d |ƒ} | |  ¡ d |  d ¡¡ƒ} |  |  d¡|  d¡d ¡}||jdkr»t  tj||jd¡}q|||fS)aXRNNP forward :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, idim) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor prev_state: batch of previous RNN states :return: batch of hidden state sequences (B, Tmax, hdim) :rtype: torch.Tensor ú input lengths: T©rrrN©Úhxr csg|] }t|dƒˆ‘qS©r )Úint)Ú.0r(©Úsubr-r.Ú Psz RNNP.forward..réÿÿÿÿrr)Úp)ÚloggingÚdebugr,Ú__name__ÚstrrrrrÚ isinstancerÚTensorÚtensorrÚcpuÚgetattrr#Úflatten_parametersrÚreset_backward_rnn_stateÚappendrr!Ú contiguousÚviewÚsizeÚtanhÚFr$) r%Úxs_padÚilensÚ prev_stateÚ elayer_statesÚlayerÚxs_packrÚysÚstatesÚys_padÚprojection_layerÚ projectedr-r6r.Úforward6s0     € z RNNP.forward©r ©N©r=Ú __module__Ú __qualname__Ú__doc__rrWÚ __classcell__r-r-r+r.rs rcr) r*a9RNN module :param int idim: dimension of inputs :param int elayers: number of encoder layers :param int cdim: number of rnn units (resulted in cdim * 2 if bidirectional) :param int hdim: number of final projection units :param float dropout: dropout rate :param str typ: The RNN type r csˆtt|ƒ ¡|ddk}d|vrtjj|||d||dn tjj|||d||d|_|r7tj |d|¡|_ ntj ||¡|_ ||_ dS)Nrr r T)rr$rr) rr*rrrrrÚnbrnnrÚl_lastr")r%r&rr r'r$r"r#r+r-r.rfs0  øú úö z RNN.__init__Nc CsÌt |jjdt|ƒ¡t|tjƒst |¡}t ||  ¡dd}|j   ¡|dur2|j j r2t|ƒ}|j ||d\}}t|dd\}}t | | ¡ d| d¡¡¡¡}| | d¡| d ¡d¡}|||fS) aVRNN forward :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, D) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor prev_state: batch of previous RNN states :return: batch of hidden state sequences (B, Tmax, eprojs) :rtype: torch.Tensor r/Tr0Nr1r9rrr )r;r<r,r=r>r?rr@rArrBr_rDrrErrJr`rGrHrI) r%rLrMrNrQrRrSrTrVr-r-r.rW‚s   " z RNN.forwardrXrYrZr-r-r+r.r*[s r*cCs<t|ttfƒr|D] }d|ddd…<q |Sd|ddd…<|S)zeSets backward BRNN states to zeroes Useful in processing of sliding windows over the inputs çr Nr)r?ÚlistÚtuple)rSÚstater-r-r.rEŸs ÿrEcs*eZdZdZd‡fdd„ Zdd„Z‡ZS)ÚVGG2LzIVGG-like module :param int in_channel: number of input channels r csxtt|ƒ ¡tjj|ddddd|_tjjdddddd|_tjjdddddd|_tjjdddddd|_ ||_ dS)Né@ér )ÚstrideÚpaddingé€) rrerrrÚConv2dÚconv1_1Úconv1_2Úconv2_1Úconv2_2Ú in_channel)r%rpr+r-r.r²s  zVGG2L.__init__cKs`t |jjdt|ƒ¡| | d¡| d¡|j| d¡|j¡ dd¡}t   |  |¡¡}t   |  |¡¡}t j |dddd}t   | |¡¡}t   | |¡¡}t j |dddd}t |¡rd| ¡ ¡}ntj|tjd}tjt |d¡tjd}tjt tj|tjdd¡tjd ¡}| dd¡}| ¡ | d¡| d¡| d¡| d¡¡}||d fS) a%VGG2L forward :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, D) :param torch.Tensor ilens: batch of lengths of input sequences (B) :return: batch of padded hidden state sequences (B, Tmax // 4, 128 * D // 4) :rtype: torch.Tensor r/rr rT)rhÚ ceil_mode)ÚdtypergN)r;r<r,r=r>rHrIrpÚ transposerKÚrelurlrmÚ max_pool2drnrorÚ is_tensorrBÚnumpyÚnpÚarrayÚfloat32ÚceilÚint64ÚtolistrG)r%rLrMÚkwargsr-r-r.rW¼s2üû * "ÿ z VGG2L.forwardr3rZr-r-r+r.re¬s recr) ÚEncodera×Encoder module :param str etype: type of encoder network :param int idim: number of dimensions of encoder network :param int elayers: number of layers of encoder network :param int eunits: number of lstm units of encoder network :param int eprojs: number of projection units of encoder network :param np.ndarray subsample: list of subsampling numbers :param float dropout: dropout rate :param int in_channel: number of input channels r c sZtt|ƒ ¡| d¡ d¡} | dvrt d¡| d¡rn|ddkrGtj   t |ƒt t ||d|||||| dg¡|_t d|  ¡d ¡n"tj   t |ƒtt ||d||||| dg¡|_t d|  ¡d ¡d |_dS|ddkrtj   t ||||||| dg¡|_t |  ¡d ¡ntj   t|||||| dg¡|_t |  ¡d ¡d|_dS)NÚvggr:)r Úgrur Úbgruz:Error: need to specify an appropriate encoder architecturer9)rp)r"zUse CNN-VGG + z P for encoderz for encoderéz( with every-layer projection for encoderz without projection for encoderr )rrrÚlstripÚrstripr;ÚerrorÚ startswithrrÚ ModuleListrerrÚencÚinfoÚupperr*Úconv_subsampling_factor) r%Úetyper&rÚeunitsÚeprojsr!r$rpr"r+r-r.rõsX    ùþÿ úþÿ  ÿÿ zEncoder.__init__Nc CsŒ|dur dgt|jƒ}t|ƒt|jƒksJ‚g}t|j|ƒD]\}}||||d\}}}| |¡qt|t|ƒ d¡ƒ}| |d¡||fS)anEncoder forward :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, D) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor prev_state: batch of previous encoder hidden states (?, ...) :return: batch of hidden state sequences (B, Tmax, eprojs) :rtype: torch.Tensor N)rNr9ra)Úlenr‰ÚziprFrrÚ unsqueezeÚ masked_fill) r%rLrMÚ prev_statesÚcurrent_statesÚmodulerNrSÚmaskr-r-r.rW)s  zEncoder.forwardr3rYrZr-r-r+r.rès 4rc Cs¤t|ddƒ}|dkrt|j||j|j|j||jƒS|dkrKtj  ¡}t |ƒD]"}t|j||||j||j||j|||j|ƒ}|  |¡q&|St d  |¡ƒ‚)a.Instantiates an encoder module given the program arguments :param Namespace args: The arguments :param int or List of integer idim: dimension of input, e.g. 83, or List of dimensions of inputs, e.g. [83,83] :param List or List of List subsample: subsample factors, e.g. [1,2,2,1,1], or List of subsample factors of each encoder. e.g. [[1,2,2,1,1], [1,2,2,1,1]] :rtype torch.nn.Module :return: The encoder module Únum_encsr z0Number of encoders needs to be more than one. {})rCrrrrŽrÚ dropout_raterrrˆrrFÚ ValueErrorÚformat)Úargsr&r!r˜Úenc_listÚidxr‰r-r-r.Ú encoder_forAs4 ù  ù rŸ)r;rwrxrrÚtorch.nn.functionalrÚ functionalrKÚtorch.nn.utils.rnnrrÚfunasr.metrics.commonrÚ*funasr.models.transformer.utils.nets_utilsrrÚModulerr*rErerrŸr-r-r-r.Ús      LD < Y