o %ݫiY@sdZddlmZddlZddlmZddlmmZddl m Z ddl m Z m Z mZmZmZmZddlmZeeZGdddejjZGd d d ejjZdS) z}Library implementing quaternion-valued convolutional neural networks. Authors * Titouan Parcollet 2020 * Drew Wagner 2024 )TupleN)get_padding_elem)affect_conv_initquaternion_conv_opquaternion_conv_rotation_opquaternion_init!renorm_quaternion_weights_inplace unitary_init) get_loggercsheZdZdZ          dfd d Zd d ZddZdededefddZddZ Z S)QConv1da This function implements quaternion-valued 1d convolution. Arguments --------- out_channels : int Number of output channels. Please note that these are quaternion-valued neurons. If 256 channels are specified, the output dimension will be 1024. kernel_size : int Kernel size of the convolutional filters. input_shape : tuple The shape of the input. stride : int, optional Stride factor of the convolutional filters (default 1). dilation : int, optional Dilation factor of the convolutional filters (default 1). padding : str, optional (same, valid, causal). If "valid", no padding is performed. If "same" and stride is 1, output shape is same as input shape. "causal" results in causal (dilated) convolutions (default "same"). groups : int, optional Default: 1 This option specifies the convolutional groups. See torch.nn documentation for more information (default 1). bias : bool, optional If True, the additive bias b is adopted (default True). padding_mode : str, optional This flag specifies the type of padding. See torch.nn documentation for more information (default "reflect"). init_criterion : str , optional (glorot, he). This parameter controls the initialization criterion of the weights. It is combined with weights_init to build the initialization method of the quaternion-valued weights (default "glorot"). weight_init : str, optional (quaternion, unitary). This parameter defines the initialization procedure of the quaternion-valued weights. "quaternion" will generate random quaternion weights following the init_criterion and the quaternion polar form. "unitary" will normalize the weights to lie on the unit circle (default "quaternion"). More details in: "Quaternion Recurrent Neural Networks", Parcollet T. et al. spinor : bool, optional When True, the layer will be turned into a spinor layer. More precisely W*x will be turned into W*x*W-1. The input x will be rotated by W such as in a spinor neural network. However, x MUST be a quaternion with the real part equal to zero. (0 + xi + yj + zk). Indeed, the rotation operation only acts on the vector part. Note that W will always be normalized before the rotation to ensure the quaternion algebra (default False). More details in: "Quaternion neural networks", Parcollet T. vector_scale : bool, optional The vector_scale is only used when spinor = True. In the context of a spinor neural network, multiple rotations of the input vector x are performed and summed. Hence, the norm of the output vector always increases with the number of layers, making the neural network instable with deep configurations. The vector_scale parameters are learnable parameters that acts like gates by multiplying the output vector with a small trainable parameter (default False). max_norm: float kernel max-norm. Example ------- >>> inp_tensor = torch.rand([10, 16, 40]) >>> cnn_1d = QConv1d( ... input_shape=inp_tensor.shape, out_channels=12, kernel_size=3 ... ) >>> out_tensor = cnn_1d(inp_tensor) >>> out_tensor.shape torch.Size([10, 16, 48]) NsameTreflectglorot quaternionFcst||_||_||_||_||_||_||_| |_ d|_ | |_ | |_ | |_ | |_||_||d|_|\|_|_tjtj|j|_tjtj|j|_tjtj|j|_tjtj|j|_|j r}tjjt|jjdd|_n t|jj d|_|j r|jrtjt|jj|_!tjj"#|j!j$n t|jj d|_!|rtjtd|j|_%n td|j d|_%|j%j$&dt't(d|j |_)t*|j|j|j|j|j|j)|j dS)NF requires_gradrrunitary)+super__init__ input_shape out_channels kernel_sizestridedilationpaddinggroups padding_mode unsqueezeinit_criterion weight_initspinor vector_scalemax_norm _check_input in_channels_get_kernel_and_weight_shapek_shapew_shapetorchnn ParameterTensorr_weighti_weightj_weightk_weightzerosshape zero_kernelrequires_grad_ scale_paraminitxavier_uniform_databiasfill_rr winitr)selfrrrrrrrr;rr!r"r#r$r% __class__^/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/nnet/quaternion_networks/q_CNN.pyrfsl    zQConv1d.__init__cCs|dd}|jdurt|j|j|j|j|jd|jdkr*|||j |j |j }n#|jdkr@|j d|j }t ||df}n |jdkrFntd |j|jrlt||j|j|j|j|j|j|j|j |j d|jd d }nt||j|j|j|j|j|j |j d|jd d }|dd}|S) Returns the output of the convolution. Arguments --------- x : torch.Tensor (batch, time, channel) Input to convolve. 3d or 4d tensors are expected. Returns ------- x : torch.Tensor The convolved outputs. r Nr%r causalrvalid1Padding must be 'same', 'valid' or 'causal'. Got Tscaler5rrrrconv1drrrrrK) transposer%rr/r0r1r2r_manage_paddingrrrFpad ValueErrorr#rr;r7r5rr)r>xnum_padoutrArArBforwardsl      zQConv1d.forwardcCsZ|j|jdkr td|j|jdkrtd|j}|j|j|jft|f}||fS)BReturns the kernel size and weight shape for convolutional layers.r'in_channels must be divisible by groups(out_channels must be divisible by groups)r'rrQrrtupler>ksr*rArArBr( sz$QConv1d._get_kernel_and_weight_shaperrrcCs.|jd}t||||}tj|||jd}|S)aThis function performs zero-padding on the time axis such that their lengths is unchanged after the convolution. Arguments --------- x : torch.Tensor Input tensor. kernel_size : int Kernel size. dilation : int Dilation. stride: int Stride. Returns ------- x : torch.Tensor The padded input. rDmode)r4rrOrPr)r>rRrrrL_inrrArArBrNs zQConv1d._manage_paddingcCsft|dkr |d}ntdt||jddkr#tdt|j|ddkr1tdt||S):Checks the input and returns the number of input channels.z(QuaternionConv1d expects 3d inputs. Got r1The field kernel size must be an odd number. Got rzPQuaternion torch.Tensors must have dimensions divisible by 4. input.size()[3] = lenrQstrrr>rr'rArArBr&9s$    zQConv1d._check_input) Nr r r r TrrrFFN) __name__ __module__ __qualname____doc__rrUr(intrNr& __classcell__rArAr?rBr s&MXN  r cseZdZdZ           dfd d Zd d ZddZddZdee e fdee e fdee e ffddZ Z S)QConv2daThis function implements quaternion-valued 1d convolution. Arguments --------- out_channels : int Number of output channels. Please note that these are quaternion-valued neurons. If 256 channels are specified, the output dimension will be 1024. kernel_size : int Kernel size of the convolutional filters. input_shape : tuple The shape of the input. stride : int, optional Stride factor of the convolutional filters (default 1). dilation : int, optional Dilation factor of the convolutional filters (default 1). padding : str, optional (same, causal). If "valid", no padding is performed. If "same" and stride is 1, output shape is same as input shape (default "same"). groups : int, optional This option specifies the convolutional groups. See torch.nn documentation for more information. (default 1). bias : bool, optional If True, the additive bias b is adopted (default True). padding_mode : str, optional This flag specifies the type of padding. See torch.nn documentation for more information. (default "reflect") init_criterion : str , optional (glorot, he). This parameter controls the initialization criterion of the weights. It is combined with weights_init to build the initialization method of the quaternion-valued weights (default "glorot"). weight_init : str, optional (quaternion, unitary). This parameter defines the initialization procedure of the quaternion-valued weights. "quaternion" will generate random quaternion weights following the init_criterion and the quaternion polar form. "unitary" will normalize the weights to lie on the unit circle (default "quaternion"). More details in: "Quaternion Recurrent Neural Networks", Parcollet T. et al. spinor : bool, optional When True, the layer will be turned into a spinor layer. More precisely W*x will be turned into W*x*W-1. The input x will be rotated by W such as in a spinor neural network. However, x MUST be a quaternion with the real part equal to zero. (0 + xi + yj + zk). Indeed, the rotation operation only acts on the vector part. Note that W will always be normalized before the rotation to ensure the quaternion algebra (default False). More details in: "Quaternion neural networks", Parcollet T. vector_scale : bool, optional The vector_scale is only used when spinor = True. In the context of a spinor neural network, multiple rotations of the input vector x are performed and summed. Hence, the norm of the output vector always increases with the number of layers, making the neural network instable with deep configurations. The vector_scale parameters are learnable parameters that acts like gates by multiplying the output vector with a small trainable parameter (default False). max_norm: float kernel max-norm. swap: bool If True, the convolution is done with the format (B, C, W, H). If False, the convolution is done with (B, H, W, C). Active only if skip_transpose is False. skip_transpose : bool If False, uses batch x spatial.dim2 x spatial.dim1 x channel convention of speechbrain. If True, uses batch x channel x spatial.dim1 x spatial.dim2 convention. Example ------- >>> inp_tensor = torch.rand([10, 4, 16, 40]) >>> cnn_1d = QConv2d( ... input_shape=inp_tensor.shape, out_channels=12, kernel_size=3 ... ) >>> out_tensor = cnn_1d(inp_tensor) >>> out_tensor.shape torch.Size([10, 4, 16, 48]) Nr r TrrrFcs$t||_||_||_||_||_||_||_| |_ | |_ | |_ | |_ | |_ ||_||_||_t|tr<||f|_t|trF||f|_t|trP||f|_||d|_|\|_|_tjtj|j|_tjtj|j|_tjtj|j|_tjtj|j|_|j rtjjt |jj!dd|_"n t|jj!#d|_"|j r|j rtjt|jj!|_$tjj%&|j$j'n t|jj!#d|_$|rtjtd|j|_(n|)dtd|j#d|j(j'*dt+t,d|j |_-t.|j|j|j|j|j|j-|j dS)NrFrr;rr)/rrrrrrrrrrr!r"r#r$r%swapskip_transpose isinstancerkr&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;register_bufferr<rr r=r)r>rrrrrrrr;rr!r"r#r$r%rnror?rArBrs|          zQConv2d.__init__cCs.|js|dd}|jr|dd}|jdur%t|j|j|j|j|jd|j dkr6| ||j |j |j }n |j dkrrRrTrArArBrUst       zQConv2d.forwardcCst|dkr |d}ntdt||jdddks%|jdddkr.tdt|j|ddkrBtdtdd t||S) r_rrDz(QuaternionConv1d expects 4d inputs. Got rrar rbzKQuaternion torch.Tensors must have dimensions divisible by 4. input.size()[z] = rcrfrArArBr&Ys,   $ zQConv2d._check_inputcCsh|j|jdkr td|j|jdkrtd|jd|jdf}|j|j|jfg|R}||fS)rVrrWrXr )r'rrQrrrZrArArBr(ssz$QConv2d._get_kernel_and_weight_shaperrrc Cs^|jd}t||d|d|d}t||d|d|d}||}tjj|||jd}|S)aThis function performs zero-padding on the time and frequency axes such that their lengths is unchanged after the convolution. Arguments --------- x : torch.Tensor Input tensor. kernel_size : int Kernel size. dilation : int Dilation. stride: int Stride. Returns ------- x : torch.Tensor The padded inputs. rDrrr\)r4rr, functionalrPr) r>rRrrrr^ padding_time padding_freqrrArArBrN~s zQConv2d._manage_padding)Nr r r r TrrrFFNFF) rgrhrirjrrUr&r(rrkrNrlrArAr?rBrmTs6SdQ   rm)rjtypingrr+torch.nnr,torch.nn.functionalrsrOspeechbrain.nnet.CNNr*speechbrain.nnet.quaternion_networks.q_opsrrrrrr speechbrain.utils.loggerr rgloggerModuler rmrArArArBs     :