o %ݫi%@sdZddlmZddlZddlmZddlmZddlm Z e e Z Gdddej j ZGd d d ej j ZGd d d ej j ZGd ddej j ZGdddej j ZGdddej j ZdS)zeLibrary implementing quaternion-valued recurrent neural networks. Authors * Titouan Parcollet 2020 )OptionalN)QLinear) QBatchNorm) get_loggercsbeZdZdZ       dfdd Zd d Zdd eejfddZ d eejfddZ Z S)QLSTMaThis function implements a quaternion-valued LSTM as first introduced in : "Quaternion Recurrent Neural Networks", Parcollet T. et al. Input format is (batch, time, fea) or (batch, time, fea, channel). In the latter shape, the two last dimensions will be merged: (batch, time, fea * channel) Arguments --------- hidden_size : int Number of output neurons (i.e, the dimensionality of the output). Specified value is in terms of quaternion-valued neurons. Thus, the output is 4*hidden_size. input_shape : tuple The expected shape of the input tensor. num_layers : int, optional Number of layers to employ in the RNN architecture (default 1). bias : bool, optional If True, the additive bias b is adopted (default True). dropout : float, optional It is the dropout factor (must be between 0 and 1) (default 0.0). bidirectional : bool, optional If True, a bidirectional model that scans the sequence both right-to-left and left-to-right is used (default False). init_criterion : str , optional (glorot, he). This parameter controls the initialization criterion of the weights. It is combined with weights_init to build the initialization method of the quaternion-valued weights (default "glorot"). weight_init : str, optional (quaternion, unitary). This parameter defines the initialization procedure of the quaternion-valued weights. "quaternion" will generate random quaternion weights following the init_criterion and the quaternion polar form. "unitary" will normalize the weights to lie on the unit circle (default "quaternion"). More details in: "Quaternion Recurrent Neural Networks", Parcollet T. et al. autograd : bool, optional When True, the default PyTorch autograd will be used. When False, a custom backpropagation will be used, reducing by a factor 3 to 4 the memory consumption. It is also 2x slower (default True). Example ------- >>> inp_tensor = torch.rand([10, 16, 40]) >>> rnn = QLSTM(hidden_size=16, input_shape=inp_tensor.shape) >>> out_tensor = rnn(inp_tensor) >>> torch.Size([10, 16, 64]) TFglorot quaternionc st|d|_||_||_||_||_d|_||_||_ | |_ t |dkr+d|_t t |dd|_|d|_||_dSNFTr)super__init__ hidden_size num_layersbiasdropout bidirectionalreshapeinit_criterion weight_initautogradlentorchprodtensorfea_dim batch_size _init_layersrnn) selfr input_shaperrrrrrr __class__^/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/nnet/quaternion_networks/q_RNN.pyrGs   zQLSTM.__init__c Csrtjg}|j}t|jD](}t||j|j|j|j |j |j |j |j d }|||j r3|jd}q|j}q|S)zInitializes the layers of the quaternionLSTM. Returns ------- rnn : ModuleList The initialized QLSTM_Layers )rrrrrr)rnn ModuleListrranger QLSTM_Layerrrrrrrrappendr"r! current_dimirnn_layr&r&r'r gs&   zQLSTM._init_layersNhxcCT|jr|jdkr||jd|jd|jd|jd}|j||d\}}||fS)a]Returns the output of the vanilla QuaternionRNN. Arguments --------- x : torch.Tensor Input tensor. hx : torch.Tensor Hidden layer. Returns ------- output : torch.Tensor Output of Quaternion RNN hh : torch.Tensor Hidden states r rrrr r1rndimshape _forward_rnnr"xr1outputhhr&r&r'forwards  *z QLSTM.forwardcCg}|dur|jr||j|jd|j}t|jD]%\}}|dur+||||d}n||dd}||dddddfqtj |dd}|jr^||j dd|j d|j}||fS| dd}||fS)alReturns the output of the vanilla QuaternionRNN. Arguments --------- x : torch.Tensor Input tensor. hx : torch.Tensor Hidden layer. Returns ------- x : torch.Tensor The output of the Quaternion RNN layer. h : torch.Tensor The hiddens states. Nrr3rdimr rrrrr enumerater!r,rstackr6 transposer"r9r1hr/r0r&r&r'r7s"   zQLSTM._forward_rnn)rTrFr r TN __name__ __module__ __qualname____doc__rr rrTensorr<r7 __classcell__r&r&r$r'rs8  rcsbeZdZdZ     dfdd Zdd eejfd d Zd dZ ddZ ddZ ddZ Z S)r+aThis function implements quaternion-valued LSTM layer. Arguments --------- input_size : int Feature dimensionality of the input tensors (in term of real values). hidden_size : int Number of output values (in term of real values). num_layers : int, optional Number of layers to employ in the RNN architecture (default 1). batch_size : int Batch size of the input tensors. dropout : float, optional It is the dropout factor (must be between 0 and 1) (default 0.0). bidirectional : bool, optional If True, a bidirectional model that scans the sequence both right-to-left and left-to-right is used (default False). init_criterion : str , optional (glorot, he). This parameter controls the initialization criterion of the weights. It is combined with weights_init to build the initialization method of the quaternion-valued weights (default "glorot"). weight_init : str, optional (quaternion, unitary). This parameter defines the initialization procedure of the quaternion-valued weights. "quaternion" will generate random quaternion weights following the init_criterion and the quaternion polar form. "unitary" will normalize the weights to lie on the unit circle (default "quaternion"). More details in: "Quaternion Recurrent Neural Networks", Parcollet T. et al. autograd : bool, optional When True, the default PyTorch autograd will be used. When False, a custom backpropagation will be used, reducing by a factor 3 to 4 the memory consumption. It is also 2x slower (default True). rFr r truec stt|d|_t||_||_||_||_||_||_ | |_ t |j|jdd|j |j|j d|_ t |jd|jdd|j |j|j d|_ |jrR|jd|_|dtd|jd||jtjj|jdd|_td g|_dS) Nr Tr# n_neuronsrrrrrh_initrFpinplace?)rrintr input_sizerrrrrrrwuregister_bufferrzeros _init_dropr(Dropoutdroprfloat drop_mask_te) r"rXrrrrrrrrr$r&r'rs>     zQLSTM_Layer.__init__Nr1cC|jr|d}tj||gdd}||||}|dur&|||}n|||j}|jrG|jddd\}}|d}tj||gdd}|S)a1Returns the output of the QuaternionRNN_layer. Arguments --------- x : torch.Tensor Input tensor. hx : torch.Tensor Hidden layer. Returns ------- h : torch.Tensor The output of the Quaternion RNN layer. rrr?Nr) rfliprcat_change_batch_sizerY_quaternionlstm_cellrRchunkr"r9r1x_fliprYrFh_fh_br&r&r'r<(    zQLSTM_Layer.forwardcCsg}|j}||}t|jdD]q}|dd|f||}|dd\}} } } } } }}}}}}}}}}ttj|| | | gdd}ttj| | ||gdd}ttj||||gdd}|t tj||||gdd|||}|t |}| |qtj |dd}|S)a7Returns the hidden states for each time step. Arguments --------- w : torch.Tensor Linearly transformed input. ht : torch.Tensor Hidden layer. Returns ------- h : torch.Tensor The hidden states for all steps. rNr>r?) rR_sample_drop_maskr*r6rZrgrsigmoidrdtanhr,rC)r"rYhthiddensct drop_maskkgatesitritiitjitkftrftiftjftkotrotiotjotkctrctictjctkitftotrFr&r&r'rfOsL   z QLSTM_Layer._quaternionlstm_cellcCTtjj|jdd|_tdg|_d|_d|_ |t |j|j dj |_ dSwInitializes the recurrent dropout operation. To speed it up, the dropout masks are sampled in advance. FrSrV>rr Nrr(r^rr_rr`ra N_drop_masks drop_mask_cntonesrdata drop_masksr"rr&r&r'r]szQLSTM_Layer._init_dropcCv|jr6|j|j|jkr!d|_|tj|j|jd|jdj |_ |j |j|j|j}|j|j|_|S|j }|Sz-Selects one of the pre-defined dropout masks.rr device trainingrrrr_rrrrrrrar"rYrtr&r&r'rns"zQLSTM_Layer._sample_drop_maskcCR|j|jdkr%|jd|_|jr'|tj|j|jd|jdj |_ dSdSdSThis function changes the batch size when it is different from the one detected in the initialization method. This might happen in the case of multi-gpu or when we have different batch sizes in train and test. We also update the h_int and drop masks. rr rN rr6rr_rrrrrrrr"r9r&r&r're zQLSTM_Layer._change_batch_size)rFr r rOrG)rIrJrKrLrrrrMr<rfr]rnrerNr&r&r$r'r+s*7'>r+cdeZdZdZ        dfd d Zd d ZddeejfddZ deejfddZ Z S)QRNNaThis function implements a vanilla quaternion-valued RNN. Input format is (batch, time, fea) or (batch, time, fea, channel). In the latter shape, the two last dimensions will be merged: (batch, time, fea * channel) Arguments --------- hidden_size : int Number of output neurons (i.e, the dimensionality of the output). Specified value is in term of quaternion-valued neurons. Thus, the output is 4*hidden_size. input_shape : tuple Expected shape of the input tensor. nonlinearity : str, optional Type of nonlinearity (tanh, relu) (default "tanh"). num_layers : int, optional Number of layers to employ in the RNN architecture (default 1). bias : bool, optional If True, the additive bias b is adopted (default True). dropout : float, optional It is the dropout factor (must be between 0 and 1) (default 0.0). bidirectional : bool, optional If True, a bidirectional model that scans the sequence both right-to-left and left-to-right is used (default False). init_criterion : str , optional (glorot, he). This parameter controls the initialization criterion of the weights. It is combined with weights_init to build the initialization method of the quaternion-valued weights (default "glorot"). weight_init : str, optional (quaternion, unitary). This parameter defines the initialization procedure of the quaternion-valued weights. "quaternion" will generate random quaternion weights following the init_criterion and the quaternion polar form. "unitary" will normalize the weights to lie on the unit circle (default "quaternion"). More details in: "Quaternion Recurrent Neural Networks", Parcollet T. et al. autograd : bool, optional When True, the default PyTorch autograd will be used. When False, a custom backpropagation will be used, reducing by a factor 3 to 4 the memory consumption. It is also 2x slower (default True). Example ------- >>> inp_tensor = torch.rand([10, 16, 40]) >>> rnn = QRNN(hidden_size=16, input_shape=inp_tensor.shape) >>> out_tensor = rnn(inp_tensor) >>> torch.Size([10, 16, 64]) rprTrFr r c t|d|_||_||_||_||_||_d|_||_ | |_ | |_ t |dkr.d|_t t |dd|_|d|_||_dSr rrr nonlinearityrrrrrrrrrrrrrrr r! r"rr#rrrrrrrrr$r&r'rs   z QRNN.__init__c Cvtjg}|j}t|jD]*}t||j|j|j|j |j |j |j |j |jd }|||j r5|jd}q|j}q|S)z Initializes the layers of the quaternionRNN. Returns ------- rnn : ModuleList The initialized QRNN_Layers. rrrrrrr)rr(r)rr*r QRNN_Layerrrrrrrrrr,r-r&r&r'r s(   zQRNN._init_layersNr1cCr2)aReturns the output of the vanilla QuaternionRNN. Arguments --------- x : torch.Tensor Input tensor. hx : torch.Tensor Hidden layer. Returns ------- output : torch.Tensor hh : torch.Tensor r rrrr r3r4r8r&r&r'r<?s  *z QRNN.forwardcCr=)aGReturns the output of the vanilla QuaternionRNN. Arguments --------- x : torch.Tensor Input tensor. hx : torch.Tensor Hidden layer. Returns ------- x : torch.Tensor Outputs h : torch.Tensor Hidden states. Nrr3r>rr?rrArEr&r&r'r7Ws"   zQRNN._forward_rnn)rprTrFr r TrGrHr&r&r$r'rs9"#rcsdeZdZdZ      dfdd Zdd eejfd d ZddZ ddZ ddZ ddZ Z S)raThis function implements quaternion-valued recurrent layer. Arguments --------- input_size : int Feature dimensionality of the input tensors (in term of real values). hidden_size : int Number of output values (in term of real values). num_layers : int, optional Number of layers to employ in the RNN architecture (default 1). batch_size : int Batch size of the input tensors. dropout : float, optional It is the dropout factor (must be between 0 and 1) (default 0.0). nonlinearity : str, optional Type of nonlinearity (tanh, relu) (default "tanh"). bidirectional : bool, optional If True, a bidirectional model that scans the sequence both right-to-left and left-to-right is used (default False). init_criterion : str , optional (glorot, he). This parameter controls the initialization criterion of the weights. It is combined with weights_init to build the initialization method of the quaternion-valued weights (default "glorot"). weight_init : str, optional (quaternion, unitary). This parameter defines the initialization procedure of the quaternion-valued weights. "quaternion" will generate random quaternion weights following the init_criterion and the quaternion polar form. "unitary" will normalize the weights to lie on the unit circle (default "quaternion"). More details in: "Quaternion Recurrent Neural Networks", Parcollet T. et al. autograd : bool, optional When True, the default PyTorch autograd will be used. When False, a custom backpropagation will be used, reducing by a factor 3 to 4 the memory consumption. It is also 2x slower (default True). rrpFr r rOc stt|d|_t||_||_||_||_||_| |_ | |_ t |j|jd|j |j|j d|_ t |jd|jd|j |j|j d|_ |jrN|jd|_|dtd|jd||jtjj|jdd|_td g|_|d krtj|_dStj|_dS) Nr TrPrrRrFrSrVrp)rrrWrrXrrrrrrrrYrZr[rr\r]r(r^r_rr`raTanhactReLU) r"rXrrrrrrrrrr$r&r'rsD     zQRNN_Layer.__init__Nr1cCrb)a&Returns the output of the QuaternionRNN_layer. Arguments --------- x : torch.Tensor Input tensor. hx : torch.Tensor Hidden layer. Returns ------- h : torch.Tensor Output of the Quaternion RNN rrr?Nr) rrcrrdrerY_quaternionrnn_cellrRrgrhr&r&r'r<rlzQRNN_Layer.forwardcCsfg}||}t|jdD]}|dd|f||}|||}||qtj|dd}|S)a7Returns the hidden states for each time step. Arguments --------- w : torch.Tensor Linearly transformed input. ht : torch.Tensor The hidden layer. Returns ------- h : torch.Tensor Hidden states for each step. rNr?)rnr*r6rZrr,rrC)r"rYrqrrrtruatrFr&r&r'r s  zQRNN_Layer._quaternionrnn_cellcCrrrrr&r&r'r]*szQRNN_Layer._init_dropcCrrrrr&r&r'rn9s"zQRNN_Layer._sample_drop_maskcCr)rrrrNrrr&r&r'reQrzQRNN_Layer._change_batch_size)rrpFr r rOrG)rIrJrKrLrrrrMr<rr]rnrerNr&r&r$r'rs,>'rcr)QLiGRUaB This function implements a quaternion-valued Light GRU (liGRU). Ligru is single-gate GRU model based on batch-norm + relu activations + recurrent dropout. For more info see: "M. Ravanelli, P. Brakel, M. Omologo, Y. Bengio, Light Gated Recurrent Units for Speech Recognition, in IEEE Transactions on Emerging Topics in Computational Intelligence, 2018" (https://arxiv.org/abs/1803.10225) To speed it up, it is compiled with the torch just-in-time compiler (jit) right before using it. It accepts in input tensors formatted as (batch, time, fea). In the case of 4d inputs like (batch, time, fea, channel) the tensor is flattened as (batch, time, fea*channel). Arguments --------- hidden_size : int Number of output neurons (i.e, the dimensionality of the output). Specified value is in term of quaternion-valued neurons. Thus, the output is 2*hidden_size. input_shape : tuple Expected shape of the input. nonlinearity : str Type of nonlinearity (tanh, relu). num_layers : int Number of layers to employ in the RNN architecture. bias : bool If True, the additive bias b is adopted. dropout: float It is the dropout factor (must be between 0 and 1). bidirectional : bool If True, a bidirectional model that scans the sequence both right-to-left and left-to-right is used. init_criterion : str, optional (glorot, he). This parameter controls the initialization criterion of the weights. It is combined with weights_init to build the initialization method of the quaternion-valued weights (default "glorot"). weight_init : str, optional (quaternion, unitary). This parameter defines the initialization procedure of the quaternion-valued weights. "quaternion" will generate random quaternion-valued weights following the init_criterion and the quaternion polar form. "unitary" will normalize the weights to lie on the unit circle (default "quaternion"). More details in: "Deep quaternion Networks", Trabelsi C. et al. autograd : bool, optional When True, the default PyTorch autograd will be used. When False, a custom backpropagation will be used, reducing by a factor 3 to 4 the memory consumption. It is also 2x slower (default True). Example ------- >>> inp_tensor = torch.rand([10, 16, 40]) >>> rnn = QLiGRU(input_shape=inp_tensor.shape, hidden_size=16) >>> out_tensor = rnn(inp_tensor) >>> torch.Size([4, 10, 5]) leaky_relurTrFr r c rr rrr$r&r'rs   zQLiGRU.__init__c Cr)z Initializes the layers of the liGRU. Returns ------- rnn : ModuleList The initialized QLiGRU_Layers. rr)rr(r)rr*r QLiGRU_Layerrrrrrrrrr,r-r&r&r'r s(   zQLiGRU._init_layersNr1cCr2)aReturns the output of the QuaternionliGRU. Arguments --------- x : torch.Tensor Input tensor. hx : torch.Tensor Hidden layer. Returns ------- output : torch.Tensor hh : torch.Tensor r rrrr r3)rr5r6_forward_ligrur8r&r&r'r<s  *zQLiGRU.forwardcCr=)a?Returns the output of the quaternionliGRU. Arguments --------- x : torch.Tensor Input tensor. hx : torch.Tensor Hidden layer. Returns ------- x : torch.Tensor Output h : torch.Tensor Hidden states Nrr3r>rr?rrA)r"r9r1rFr/ ligru_layr&r&r'rs"   zQLiGRU._forward_ligru)rrTrFr r TrG) rIrJrKrLrr rrrMr<rrNr&r&r$r'rcsB !rcsfeZdZdZ       dfd d Zdd eejfd dZddZ ddZ ddZ ddZ Z S)raThis function implements quaternion-valued Light-Gated Recurrent Units (ligru) layer. Arguments --------- input_size: int Feature dimensionality of the input tensors. hidden_size: int Number of output values. num_layers: int Number of layers to employ in the RNN architecture. batch_size: int Batch size of the input tensors. dropout: float It is the dropout factor (must be between 0 and 1). nonlinearity: str Type of nonlinearity (tanh, relu). normalization: str The type of normalization to use (batchnorm or none) bidirectional: bool If True, a bidirectional model that scans the sequence both right-to-left and left-to-right is used. init_criterion: str , optional (glorot, he). This parameter controls the initialization criterion of the weights. It is combined with weights_init to build the initialization method of the quaternion-valued weights (default "glorot"). weight_init: str, optional (quaternion, unitary). This parameter defines the initialization procedure of the quaternion-valued weights. "quaternion" will generate random quaternion weights following the init_criterion and the quaternion polar form. "unitary" will normalize the weights to lie on the unit circle (default "quaternion"). More details in: "Deep quaternion Networks", Trabelsi C. et al. autograd: bool, optional When True, the default PyTorch autograd will be used. When False, a custom backpropagation will be used, reducing by a factor 3 to 4 the memory consumption. It is also 2x slower (default True). rr batchnormFr r Tc stt|d|_t||_||_||_||_| |_| |_ ||_ ||_ | |_ t |j|jdd|j |j|j d|_t |jd|jdd|j |j|j d|_|jrX|jd|_d|_|j dkrmt|ddd|_d|_n t|ddd|_d|_|d td |jd||jtjj|jdd |_td g|_|j d krtj|_dS|j dkrtj|_dStj |_dS)Nr rFrPrr>)rXr@TrRrrSrVrpr)!rrrWrrXrrrrr normalizationrrrrYrZ normalizernormr[rr\r]r(r^r_rr`rarr LeakyReLUr) r"rXrrrrrrrrrrr$r&r'rMsX        zQLiGRU_Layer.__init__Nr1c Cs|jr|d}tj||gdd}||||}|jrB|||j d|j d|j d}||j d|j d|j d}|durM| ||}n| ||j }|jrn|j ddd\}}|d}tj||gdd}|S)zReturns the output of the quaternion liGRU layer. Arguments --------- x : torch.Tensor Input tensor. hx : torch.Tensor Returns ------- Output of quaternion liGRU layer. rrr?rN) rrcrrdrerYrrrr6_quaternion_ligru_cellrRrg) r"r9r1rirYw_bnrFrjrkr&r&r'r<s    (  zQLiGRU_Layer.forwardc Csg}||}t|jdD]N}|dd|f||}|dd\}}} } } } } }tj||| | gdd}tj| | | |gdd}t|}|||}||d||}| |qtj |dd}|S)aReturns the hidden states for each time step. Arguments --------- w : torch.Tensor Linearly transformed input. ht : torch.Tensor Returns ------- h : torch.Tensor Hidden states for all steps. rNr>r?) rnr*r6rZrgrrdrorr,rC)r"rYrqrrrtrurvatratiatjatkztrztiztjztkrzthcandrFr&r&r'rs   z#QLiGRU_Layer._quaternion_ligru_cellc CsZtjj|jdd|_tdg|_d|_d|_ | d|t |j|j dj dS) rFrSrVrrrr N)rr(r^rr_rr`rarrr[rrrrr&r&r'r]szQLiGRU_Layer._init_dropcCs|jr6|j|j|jkr!d|_|tj|j|jd|jdj |_ |j |j|j|j}|j|j|_|S|j |j|_ |j }|S)z,Selects one of the pre-defined dropout masksrr r) rrrrr_rrrrrrratorr&r&r'rns$zQLiGRU_Layer._sample_drop_maskcCrrrrr&r&r'rerzQLiGRU_Layer._change_batch_size)rrrFr r TrG)rIrJrKrLrrrrMr<rr]rnrerNr&r&r$r'r$s.N+#r)rLtypingrr-speechbrain.nnet.quaternion_networks.q_linearr4speechbrain.nnet.quaternion_networks.q_normalizationrspeechbrain.utils.loggerrrIloggerr(Modulerr+rrrrr&r&r&r's$    ;y=dB