o }oi P@sddlZddlmZmZmZmZddlZddlZddl m Z         d-de de de d ee d ee d ee d eed ee de de de dejjfddZGdddejjZGdddejjZGdddejjZGdddejjZGdddejjZ  d.de de de d ee d ee d ee dee dee dejjfddZGd d!d!ejjZGd"d#d#ejjZde d$ejjd%ed&edejjf d'd(ZGd)d*d*ejjZd/d+d,ZdS)0N)ListOptionalTupleUnion)logging? input_size hidden_size num_layersnormforget_gate_biasdropoutnorm_first_rnnt_maxweights_init_scalehidden_hidden_bias_scale proj_sizereturnc Cs|dvr td||durt|||||||| | d S|dkr/t|||d|||||| | d S|dkrCtjt|||||||| d SdS) a Utility function to provide unified interface to common LSTM RNN modules. Args: input_size: Input dimension. hidden_size: Hidden dimension of the RNN. num_layers: Number of RNN layers. norm: Optional string representing type of normalization to apply to the RNN. Supported values are None, batch and layer. forget_gate_bias: float, set by default to 1.0, which constructs a forget gate initialized to 1.0. Reference: [An Empirical Exploration of Recurrent Network Architectures](http://proceedings.mlr.press/v37/jozefowicz15.pdf) dropout: Optional dropout to apply to end of multi-layered RNN. norm_first_rnn: Whether to normalize the first RNN layer. t_max: int value, set to None by default. If an int is specified, performs Chrono Initialization of the LSTM network, based on the maximum number of timesteps `t_max` expected during the course of training. Reference: [Can recurrent neural networks warp time?](https://openreview.net/forum?id=SJcKhk-Ab) weights_init_scale: Float scale of the weights after initialization. Setting to lower than one sometimes helps reduce variance between runs. hidden_hidden_bias_scale: Float scale for the hidden-to-hidden bias scale. Set to 0.0 for the default behaviour. Returns: A RNN module )Nbatchlayerz unknown norm=N r r r rr rrrrrT) r r rnn_layers batch_normrr rrrrrrr r r rr rrr) ValueError LSTMDropoutBNRNNSumtorchjitscriptln_lstm) r r r r r rrrrrrr"U/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/parts/rnn.pyrnnsT2 r$cs@eZdZdZdejjffdd ZdejdejfddZ Z S) OverLastDimaaCollapses a tensor to 2D, applies a module, and (re-)expands the tensor. An n-dimensional tensor of shape (s_1, s_2, ..., s_n) is first collapsed to a tensor with shape (s_1*s_2*...*s_n-1, s_n). The module is called with this as input producing (s_1*s_2*...*s_n-1, s_n') --- note that the final dimension can change. This is expanded to (s_1, s_2, ..., s_n-1, s_n') and returned. Args: module (torch.nn.Module): Module to apply. Must accept a 2D tensor as input and produce a 2D tensor as output, optionally changing the size of the last dimension. modulecst||_dSN)super__init__r&)selfr& __class__r"r#r)s  zOverLastDim.__init__xrcCsR|^}}d}|D]}||9}q ||d}||}|jg|dR}|S)N)sizeviewr&)r*r-dims_ reduced_dimsdimr"r"r#forwards   zOverLastDim.forward) __name__ __module__ __qualname____doc__rnnModuler)Tensorr6 __classcell__r"r"r+r#r%ys r%cseZdZ    ddedededeed eed eed ed ed effdd Z ddejdee ejejfde eje ejejfffddZ Z S)rNrrrr r r rr rrrrc  stt|tjj||||| d|_|durc|jD]E\} } d| vrat|j| } | } | d}| j dt tjj | j d|d|d| j |d|<| j |d| | j d|<qn=|dur|jD]3\} } d| vrt|j| }|j |d| |d | vrt|j| }|j |d|t|9<ql|rtj|nd|_|D]\} } d | vsd| vr| j t|9_ qdS) a Returns an LSTM with forget gate bias init to `forget_gate_bias`. Args: input_size: See `torch.nn.LSTM`. hidden_size: See `torch.nn.LSTM`. num_layers: See `torch.nn.LSTM`. dropout: See `torch.nn.LSTM`. forget_gate_bias: float, set by default to 1.0, which constructs a forget gate initialized to 1.0. Reference: [An Empirical Exploration of Recurrent Network Architectures](http://proceedings.mlr.press/v37/jozefowicz15.pdf) t_max: int value, set to None by default. If an int is specified, performs Chrono Initialization of the LSTM network, based on the maximum number of timesteps `t_max` expected during the course of training. Reference: [Can recurrent neural networks warp time?](https://openreview.net/forum?id=SJcKhk-Ab) weights_init_scale: Float scale of the weights after initialization. Setting to lower than one sometimes helps reduce variance between runs. hidden_hidden_bias_scale: Float scale for the hidden-to-hidden bias scale. Set to 0.0 for the default behaviour. Returns: A `torch.nn.LSTM`. )r r r rrNbiasrr.bias_ihbias_hhweight)r(rr)rr;LSTMlstmnamed_parametersgetattrnelementdatafill_loginituniform_floatDropoutr)r*r r r rr rrrrnamevpnr?r+r"r#r)s@'      zLSTMDropout.__init__r-hrcCs(|||\}}|jr||}||fSr')rFr)r*r-rUr"r"r#r6s zLSTMDropout.forward)Nrrrr') r7r8r9intrrOr)rr=rr6r>r"r"r+r#rs@  LrcseZdZdZejjddddddfdeded ejjd e d e e d e ed e de deffdd Z ddej de eeej ej fdeej eej ej fffddZddZZS)RNNLayerz+A single RNNLayer with optional batch norm.TrNrrr r rnn_typerr rrrrc sht|rttj||_t|tjjr)|s)t ||dd||||| d |_ dS|||| d|_ dS)Nr.rr)r r r?) r(r)r%rr; BatchNorm1dbn isinstancerErr$) r*r r rXrr rrrrr+r"r#r)s   zRNNLayer.__init__r-hxrcCs6t|dr|}||}|j||d\}}||fS)NrZr\)hasattr contiguousrZr$)r*r-r\rUr"r"r#r6s  zRNNLayer.forwardcCs|jdSr')r$flatten_parameters)r*r"r"r#_flatten_parametersszRNNLayer._flatten_parametersr')r7r8r9r:rr;rErVr<boolrrOr)r=rrr6rar>r"r"r+r#rWsJ  "  rWcseZdZdZejjdddddddddf d ed ed ejjd ed e de e de e de de ede de deffdd Z d dej de eeej ej fdeej eej ej fffddZde eej ej fdeedeeej ej fffddZddZZS)!raNRNN wrapper with optional batch norm. Instantiates an RNN. If it is an LSTM it initialises the forget gate bias =`lstm_gate_bias`. Optionally applies a batch normalisation layer to the input with the statistics computed over all time steps. If dropout > 0 then it is applied to all layer outputs except the last. r.TrrFNrr r rXrrrr rrrrrc st||_tj|_t|D]5} |d| k}|jt ||||o(|p(| dk|| | | | d |durE|dkrE|sE|jtj ||}qdS)Nr.r)rXrr rrrrr) r(r)rrr; ModuleListlayersrangeappendrWrP)r*r r rXrrrr rrrrri final_layerr+r"r#r)'s,    zBNRNNSum.__init__r-r\rc Cs||}g}g}d}|jD]+}t|tjjr||}q||||d\}}||d||d|d7}~qtj|dd}tj|dd} ||| ffS)Nrr]r.r5)_parse_hidden_staterdr[rr;rPrfstack) r*r-r\hscsrnn_idxrh_outh_0c_0r"r"r#r6Ps    zBNRNNSum.forwardcsR|dur dg|jS|\jd|jkrtdfddtjdDS)z Dealing w. hidden state: Typically in pytorch: (h_0, c_0) h_0 = ``[num_layers * num_directions, batch, hidden_size]`` c_0 = ``[num_layers * num_directions, batch, hidden_size]`` NrzgProvided initial state value `h_0` must be of shape : [num_layers * num_directions, batch, hidden_size]csg|] }||fqSr"r").0rgrqrpr"r# zsz0BNRNNSum._parse_hidden_state..)rshaperre)r*r\r"rsr#rjfs zBNRNNSum._parse_hidden_statecCs4|jD]}t|tjjtjjtjjfr|qdSr')rdr[rr;rEGRURNNra)r*rr"r"r#ra|s zBNRNNSum._flatten_parametersr')r7r8r9r:rr;rErVr<rbrrOr)r=rrr6rrjrar>r"r"r+r#rsf      *  rcsNeZdZdZdeffdd Zdeeej deej ej ffddZ Z S) StackTimezZ Stacks time within the feature dim, so as to behave as a downsampling operation. factorcstt||_dSr')r(r)rVry)r*ryr+r"r#r)s zStackTime.__init__r-rcCs|\}}|g}td|jD]%}t|}||dddddf|d| ddddf<||q t||j}tj|dddd|jddddf|fS)Nr.rAri) reryr zeros_likerfceilrOrVcat)r*r-x_lensseqrgtmpr"r"r#r6s 2 ,zStackTime.forward) r7r8r9r:rVr)rrrr=r6r>r"r"r+r#rxs0rxcCsn|dur |dkr td|durtd|durtd|dur'tdt|tt|||gt|||gdS)z9Returns a ScriptModule that mimics a PyTorch native LSTM.Nrz*`dropout` not supported with LayerNormLSTMz6LayerNormLSTM does not support chrono init via `t_max`z1`weights_init_scale` is ignored for LayerNormLSTMz7`hidden_hidden_bias_scale` is ignored for LayerNormLSTM)first_layer_argsother_layer_args)rrwarning StackedLSTM LSTMLayerLayerNormLSTMCellrr"r"r#r!s      r!c sVeZdZfddZdejdeejejfdeejeejejfffddZZS)rcstt||||_dSr')r(rr)cell)r*r cell_argsr+r"r#r)szLSTMLayer.__init__inputstatercCsL|d}g}tt|D]}||||\}}||g7}q t||fS)Nr)unbindrelenrrrk)r*rrinputsoutputsrgoutr"r"r#r6s  zLSTMLayer.forward) r7r8r9r)rr=rr6r>r"r"r+r#rs rc s^eZdZfddZddZdejdeejejfdeejeejejfffdd ZZ S) rcst||_||_tjtd|||_tjtd|||_ tj d||_ tj d||_ tj ||_ ||j jj|d|d|j jj|d||dS)Nr@rAr)r(r)r r rr; Parameterrandn weight_ih weight_hh LayerNorm layernorm_i layernorm_h layernorm_creset_parametersr?rJrK)r*r r r r+r"r#r)s  zLayerNormLSTMCell.__init__cCs6dt|j}|D] }tjj|| |q dS)Nr)mathsqrtr parametersrr;rMrN)r*stdvrDr"r"r#rs z"LayerNormLSTMCell.reset_parametersrrrcCs|\}}|t||j}|t||j}||}|dd\}} } } t|}t| } t | } t| } | | ||| } | t | } | | | ffS)Nr@r.) rrmmrtrrchunksigmoidtanhr)r*rrr\cxigateshgatesgatesingate forgetgatecellgateoutgatecyhyr"r"r#r6s     zLayerNormLSTMCell.forward) r7r8r9r)rrr=rr6r>r"r"r+r#rs rrrrcs2|gfddt|dD}tj|S)Ncsg|]}qSr"r")rrr3rrr"r#rtsz%init_stacked_lstm..r.)rerr;rc)r rrrrdr"rr#init_stacked_lstms& rc sxeZdZdedejjdedeffdd Zdej de ee ej ej fd e ej ee ej ej fffd d Z Z S) rr rrrcs"tt|t|||||_dSr')r(rr)rrd)r*r rrrr+r"r#r)szStackedLSTM.__init__rstatesrc Cs|dur0g}|d}|jD]}|tj||jj|j|jdtj||jj|j|jdfq|}g}|}t |jD]\}} ||} | || \}} || |d7}q9||fS)Nr.dtypedevice) r0rdrfrzerosrr rr enumerate) r*rr temp_statesrr output_statesoutputrg rnn_layerr out_stater"r"r#r6s$    zStackedLSTM.forward)r7r8r9rVrr;r<rr)r=rrr6r>r"r"r+r#rs"rcCst|tjr |tjSt|ttfstdt|t|}t dd|D}t j ||fdt j d}t |D]\}}|||dt|f<q8tj|tj|d}|S)a\Collates the label inputs for the rnn-t prediction network. If `labels` is already in torch.Tensor form this is a no-op. Args: labels: A torch.Tensor List of label indexes or a torch.Tensor. device: Optional torch device to place the label on. Returns: A padded torch.Tensor of shape (batch, max_seq_len). z(`labels` should be a list or tensor not css|]}t|VqdSr')r)rrlabelr"r"r# *sz label_collate..r) fill_valuerNr)r[rr=typeint64listtuplerrmaxnpfullint32rtensor)labelsr batch_sizemax_len cat_labelselr"r"r# label_collates r)NrrNNrrr)NNr')rtypingrrrrnumpyrr nemo.utilsrrVstrrOrbr;r<r$r%rrWrrxr!rrrcrrrr"r"r"r#s     `W1c  +