o i3@s ddlZddlmZddZdS)Ncsdkr|St|tjtjtjfsJ|jjdks*|jddks)Jdn%|jdkr;|j dks:Jdn|jd|jd}|dksOJdfdd }| ||S) aN Wraps modules and applies quantization noise to the weights for subsequent quantization with Iterative Product Quantization as described in "Training with Quantization Noise for Extreme Model Compression" Args: - module: nn.Module - p: amount of Quantization Noise - block_size: size of the blocks for subsequent quantization with iPQ Remarks: - Module weights must have the right sizes wrt the block size - Only Linear, Embedding and Conv2d modules are supported for the moment - For more detail on how to quantize by blocks with convolutional weights, see "And the Bit Goes Down: Revisiting the Quantization of Neural Networks" - We implement the simplest form of noise here as stated in the paper which consists in randomly dropping blocks rz0Input features must be a multiple of block sizesrrz0Input channels must be a multiple of block sizesz,Kernel size must be a multiple of block sizec s8|jrs.|j}|d}|d}tj|||jd}||dd|}nT|j}|j }|j }|j dkrZtjt |||jd}||dd|}n(tj|d|d|jd}|| d ddd|j d|j d}|tj}dd}|||d|j_dSdS)Nrr)devicer)trainingweightsizetorchzerosr bernoulli_repeat_interleaveview in_channels out_channels kernel_sizeint unsqueezerepeattobool masked_filldata) modinputr in_features out_featuresmaskrrs block_sizeis_convpV/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/data2vec/quant_noise.py_forward_pre_hook:s8        z&quant_noise.._forward_pre_hook) isinstancennLinear EmbeddingConv2dr ndimr rrregister_forward_pre_hook)moduler$r"kr'r%r!r& quant_noise s"   *r1)r torch.nnr)r1r%r%r%r&s