o %ݫis @sdZddlZddlZddlZddlmmZddl m Z ddl m Z Gdddej jZddZd d Zd ed ed ededef ddZd ed ed ededef ddZ d"ddZd#ddZddZddZddZd d!ZdS)$aThis library implements different operations needed by quaternion- valued architectures. This work is inspired by: "Quaternion neural networks" - Parcollet T. "Quaternion recurrent neural networks" - Parcollet T. et al. "Quaternion convolutional neural networks for end-to-end automatic speech recognition" - Parcollet T. et al. "Deep quaternion networks" - Gaudet Chase J. et al. Authors * Titouan Parcollet 2020 N)chi)Variablec@s(eZdZdZeddZeddZdS)QuaternionLinearCustomBackwarda"This class redefine the backpropagation of a quaternion linear layer (not a spinor layer). By doing so, we can save up to 4x memory, but it is also 2x slower than 'quaternion_linear_op'. It should be used within speechbrain.nnet.quaternion_networks.linear.QuaternionLinear. c Cs|||||||tj|| | | gdd}tj||| |gdd}tj|||| gdd} tj|| ||gdd} tj||| | gdd} |jrQt||| St|| S)a Applies a quaternion linear transformation to the incoming data: It is important to notice that the forward phase of a QNN is defined as W * Inputs (with * equal to the Hamilton product). The constructed cat_kernels_4_quaternion is a modified version of the quaternion representation so when we do torch.mm(Input,W) it's equivalent to W * Inputs. Arguments --------- ctx : PyTorch context object Used to save the context necessary to perform a backwards pass. input : torch.Tensor Quaternion input tensor to be transformed. Shape: [batch*time, X]. r_weight : torch.Parameter Real part of the quaternion weight matrix of this layer. i_weight : torch.Parameter First imaginary part of the quaternion weight matrix of this layer. j_weight : torch.Parameter Second imaginary part of the quaternion weight matrix of this layer. k_weight : torch.Parameter Third imaginary part of the quaternion weight matrix of this layer. bias : torch.Parameter Returns ------- The linearly transformed quaternions rdim)save_for_backwardtorchcat requires_gradaddmmmm) ctxinputr_weighti_weightj_weightk_weightbiascat_kernels_4_rcat_kernels_4_icat_kernels_4_jcat_kernels_4_kcat_kernels_4_quaternionr^/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/nnet/quaternion_networks/q_ops.pyforwards4   z&QuaternionLinearCustomBackward.forwardcCs\|j\}}}}}}d}} } } } } tj|| | | gdd}tj||| |gdd}tj|||| gdd}tj|| ||gdd}ttj||||gdddddd}|d}|dd|d}|d|d|d}|d|d |d}|d||d|d}tj|| | | gdd}tj||| |gdd}tj|||| gdd}tj|| ||gdd}ttj||||gdddd}|d}|dd|d}|d|d|d}|d|d |d}|d||d|d}tj||||gdd}tj| ||| gdd}tj| | ||gdd}tj| || |gdd}tj||||gdd}|jdrG||}|jdr|dd|dd}|d}|d}|dd|dd|} |dd|d||} |dd|d|d |} |dd|d|d |} |jd r| d d} || | | | | fS) a Run the backward phase of the forward call defined above. This implementation follows the quaternion backpropagation of a quaternion layer that can be found in "Quaternion neural networks" - Parcollet T. Page 48. Arguments --------- ctx : Pytorch context object Contains saved weights and bias grad_output : torch.Tensor The output of the forward part Returns ------- The corresponding gradients of this op NrrrF)r ) saved_tensorsr r rpermutesizenarrowneeds_input_gradr sumsqueeze)r grad_outputrrrrrr grad_input grad_weight_r grad_weight_i grad_weight_j grad_weight_k grad_biasinput_rinput_iinput_jinput_kcat_kernels_4_quaternion_T nb_hiddenrijk input_matgrad_mat grad_weight unit_size_x unit_size_yrrrbackward\s          z'QuaternionLinearCustomBackward.backwardN)__name__ __module__ __qualname____doc__ staticmethodrr?rrrrrs  =rc Cstj|| | | gdd}tj||| |gdd}tj|||| gdd}tj|| ||gdd} tj|||| gdd} |dkrS|jrMt||| St|| St|| } |jr`| |S| S)a Applies a quaternion linear transformation to the incoming data: It is important to notice that the forward phase of a QNN is defined as W * Inputs (with * equal to the Hamilton product). The constructed cat_kernels_4_quaternion is a modified version of the quaternion representation so when we do torch.mm(Input,W) it's equivalent to W * Inputs. Arguments --------- input : torch.Tensor Quaternion input tensor to be transformed. r_weight : torch.Parameter Real part of the quaternion weight matrix of this layer. i_weight : torch.Parameter First imaginary part of the quaternion weight matrix of this layer. j_weight : torch.Parameter Second imaginary part of the quaternion weight matrix of this layer. k_weight : torch.Parameter Third imaginary part of the quaternion weight matrix of this layer. bias : torch.Parameter Returns ------- The linearly transformed quaternions rrrr)r r rr r r matmul) rrrrrrrrrrroutputrrrquaternion_linear_ops0    rGcCsV||}||} ||} ||} t|| | | d} || } || }|| }|| }d}|||} |||} |||} || |}|| |}|| |}|||}|||}|||}|jrtj||d| | ||||||gdd}tj|||||d| | |||gdd}tj||||||||d| | gdd}n9tj|d| | ||||gdd}tj|||d| | ||gdd}tj|||||d| | gdd}tj||||gdd}tj||||gdd}|dkr|jrt|||St||St||}|jr)||S|S)a6 Applies a quaternion rotation transformation to the incoming data: The rotation W*x*W^t can be replaced by R*x following: https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation Works for unitary and non-unitary weights (they will be normalized). The initial size of the input must be a multiple of 4 with the real part equal to zero. Rotations only affect the vector part of a quaternion. Arguments --------- input : torch.Tensor Quaternion input tensor to be transformed. r_weight : torch.Parameter Real part of the quaternion weight matrix of this layer. i_weight : torch.Parameter First imaginary part of the quaternion weight matrix of this layer. j_weight : torch.Parameter Second imaginary part of the quaternion weight matrix of this layer. k_weight : torch.Parameter Third imaginary part of the quaternion weight matrix of this layer. bias : torch.Parameter scale : torch.Parameter In the context of a spinor neural network, multiple rotations of the input vector x are performed and summed. Hence, the norm of the output vector always increases with the number of layers, making the neural network instable with deep configurations. The scale parameters are learnable parameters that acts like gates by multiplying the output vector with a small trainable parameter. zero_kernel : torch.Parameter The zero kernel is simply a tensor of zeros with require grad = False. Its shape is equivalent to a quaternion component shape. In fact, it is only needed to make the dimensions match when using the rotation matrix : https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation Returns ------- The linearly rotated quaternions -C6?@?rrrr)r sqrtr r rr r rE)rrrrrrscale zero_kernelsquare_rsquare_isquare_jsquare_knorm r_n_weight i_n_weight j_n_weight k_n_weight norm_factorrirjrkijikjk rot_kernel_1 rot_kernel_2 rot_kernel_3 zero_kernel2global_rot_kernelrFrrrquaternion_linear_rotation_ops,                      rcstridepaddinggroupsdilationconv1dc " Cs<||} ||}||}||}t| |||d}||}||}||}||}d}|||}|||}|||}|||}|||}|||}|||}|||}|||}|jrtj||d||||||||gdd}tj|||||d|||||gdd}tj||||||||d||gdd}n9tj|d||||||gdd}tj|||d||||gdd}tj|||||d||gdd}tj||||gdd} tj| |||gdd}!| rtj||!||| | | dStj||!||| | | dS)a Applies a quaternion rotation transformation to the incoming data: The rotation W*x*W^t can be replaced by R*x following: https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation Works for unitary and non-unitary weights (they will be normalized). The initial size of the input must be a multiple of 4 with the real part equal to zero. Rotations only affect the vector part of a quaternion. Arguments --------- input : torch.Tensor Quaternion input tensor to be transformed. r_weight : torch.Parameter Real part of the quaternion weight matrix of this layer. i_weight : torch.Parameter First imaginary part of the quaternion weight matrix of this layer. j_weight : torch.Parameter Second imaginary part of the quaternion weight matrix of this layer. k_weight : torch.Parameter Third imaginary part of the quaternion weight matrix of this layer. bias : torch.Parameter scale : torch.Parameter In the context of a spinor neural network, multiple rotations of the input vector x are performed and summed. Hence, the norm of the output vector always increases with the number of layers, making the neural network instable with deep configurations. The scale parameters are learnable parameters that acts like gates by multiplying the output vector with a small trainable parameter. zero_kernel : torch.Parameter The zero kernel is simply a tensor of zeros with require grad = False. Its shape is equivalent to a quaternion component shape. In fact, it is only needed to make the dimensions match when using the rotation matrix : https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation stride : int Stride factor of the convolutional filters. padding : int Amount of padding. See torch.nn documentation for more information. groups : int This option specifies the convolutional groups. See torch.nn documentation for more information. dilation : int Dilation factor of the convolutional filters. conv1d : bool If true, a 1D convolution operation will be applied. Otherwise, a 2D convolution is called. Returns ------- The rotated quaternion inputs rHrIrJrrrrweightrrdrergrf)r rKr r Frhconv2d)"rrrrrrrLrMrdrerfrgrhrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrrrquaternion_conv_rotation_opysB                     rmc  Cstj|| | | gdd} tj||| |gdd} tj|||| gdd} tj|| ||gdd}tj| | | |gdd}| rKtj|||||| |dStj|||||| |dS)a Applies a quaternion convolution transformation to the incoming data: It is important to notice that the forward phase of a QCNN is defined as W * Inputs (with * equal to the Hamilton product). The constructed cat_kernels_4_quaternion is a modified version of the quaternion representation so when we do torch.mm(Input,W) it's equivalent to W * Inputs. Arguments --------- input : torch.Tensor Quaternion input tensor to be transformed. r_weight : torch.Parameter Real part of the quaternion weight matrix of this layer. i_weight : torch.Parameter First imaginary part of the quaternion weight matrix of this layer. j_weight : torch.Parameter Second imaginary part of the quaternion weight matrix of this layer. k_weight : torch.Parameter Third imaginary part of the quaternion weight matrix of this layer. bias : torch.Parameter stride : int Stride factor of the convolutional filters. padding : int Amount of padding. See torch.nn documentation for more information. groups : int This option specifies the convolutional groups. See torch.nn documentation for more information. dilation : int Dilation factor of the convolutional filters. conv1d : bool If true, a 1D convolution operation will be applied. Otherwise, a 2D convolution is called. Returns ------- The convolved quaternion inputs rrrri)r r rkrhrl)rrrrrrrdrerfrgrhrrrrrrrrquaternion_conv_opsF4  rnglorotcCstjjtdd|durt|}||}||}n|}|}|dkr1dtd||}n dtd|}|durC||f}nt|turS||ft |f}n ||fg|R}t t j dd||d } t|} t | d d } t | d d } t | d d } td| D]1}t| |d| |d| |dd }| ||<| ||<| ||<q| |} | |} | |} t|tj tj}| t|}| | t|}| | t|}| | t|}||||fS) a Returns a matrix of quaternion numbers initialized with the method described in "Quaternion Recurrent Neural Network " - Parcollet T. Arguments --------- in_features : int Number of real values of the input layer (quaternion // 4). out_features : int Number of real values of the output layer (quaternion // 4). kernel_size : int Kernel_size for convolutional layers (ex: (3,3)). criterion : str (glorot, he) Returns ------- Matrix of initialized quaternion numbers i)seedNrorJrrr)locrLr$rrrH)nprandomrpr initial_seedprodrKtypeinttuple from_numpyrrvs FloatTensoruniform_rangereshaperandmathpicossin) in_features out_features kernel_size criterionreceptive_fieldfan_infan_outs kernel_shapemodulusnumber_of_weightsv_iv_jv_kr7rRphaseweight_rweight_iweight_jweight_krrrquaternion_initysD     .    rhec CsT|dur ||f}nt|tur||ft|f}n ||fg|R}t|}t|dd}t|dd}t|dd}t|dd} td|D]?} t || d|| d|| d| | dd} || | <|| | <|| | <| | | <qP| |}| |}| |}| |} |||| fS)aReturns a matrix of unitary quaternion numbers. Arguments --------- in_features : int Number of real values of the input layer (quaternion // 4). out_features : int Number of real values of the output layer (quaternion // 4). kernel_size : int Kernel_size for convolutional layers (ex: (3,3)). criterion : str (glorot, he) Returns ------- Matrix of unitary quaternion numbers. NrrrrrH) rvrwrxrrrur r{r|r}rKr~) rrrrrrv_rrrrr7rRrrr unitary_inits0   4     rc Cs^||d|dd|\}}}} ||j|_||j|_||j|_| |j|_dS)a Applies the weight initialization function given to the parameters. Arguments --------- r_weight : torch.Parameters (nb_quaternion_in, nb_quaternion_out) i_weight : torch.Parameters (nb_quaternion_in, nb_quaternion_out) j_weight : torch.Parameters (nb_quaternion_in, nb_quaternion_out) k_weight : torch.Parameters (nb_quaternion_in, nb_quaternion_out) init_func : function (unitary_init, quaternion_init) init_criterion : str (glorot, he) rrNr$type_asdata) rrrr init_funcinit_criterionr6r7r8r9rrr affect_inits rc Csh|d}|d}|||||d\} } } } | |j|_| |j|_| |j|_| |j|_dS)arApplies the weight initialization function given to the parameters. This is specifically written for convolutional layers. Arguments --------- r_weight : torch.Parameters (nb_quaternion_in, nb_quaternion_out) i_weight : torch.Parameters (nb_quaternion_in, nb_quaternion_out) j_weight : torch.Parameters (nb_quaternion_in, nb_quaternion_out) k_weight : torch.Parameters (nb_quaternion_in, nb_quaternion_out) kernel_size : int Kernel size. init_func : function (unitary_init, quaternion_init) init_criterion : str (glorot, he) rr)rrNr) rrrrrrr in_channels out_channelsr6r7r8r9rrraffect_conv_inits  rcCsHt|dvrtdtt|d}|ddkr"tdt|dS)zCheck the quaternion-valued shape for a linear layer. Arguments --------- input_shape : tuple Expected shape of the input. >rrr zFQuaternion linear accepts only input of dimension 2 or 3. input.dim = rrrzPQuaternion torch.Tensors must have dimensions divisible by 4. input.size()[1] = N)len Exceptionstrrr) input_shaper5rrrcheck_quaternion_inputBs   rcCst|jd|jd|jd|jd}tj|dd|d}||}|j|9_|j|9_|j|9_|j|9_dS)a.Renorms the magnitude of the quaternion-valued weights. Arguments --------- r_weight : torch.Parameter i_weight : torch.Parameter j_weight : torch.Parameter k_weight : torch.Parameter max_norm : float The maximum norm of the magnitude of the quaternion weights rr)prmaxnormN)r rKrrenorm)rrrrmax_normweight_magnituderenormed_weight_magnitudefactorrrr!renorm_quaternion_weights_inplaceZs"r)Nro)Nr)rCrnumpyrrr torch.nn.functionalnn functionalrk scipy.statsrtorch.autogradrautogradFunctionrrGrcrwboolrmrnrrrrrrrrrrsT   !;     %   ] L3+