o ۷i|u@sddlZddlZddlZddlmZddlmZddlmZe e Z Gdddej Z Gdd d ej ZGd d d ej ZGd d d ej ZGdddej ZGdddej Zd$ddZd$ddZd$ddZGdddeZGdddeZGdddeZd%d d!ZGd"d#d#eZdS)&N)nn)Function)loggingcs>eZdZdZ         d fdd Zd d d ZZS) QuantEmbeddinga Quantized version of `torch.nn.Embedding`. Adds quantization-specific arguments on top of `torch.nn.Embedding`. Args: weight_bit (`int`, *optional*, defaults to `8`): Bitwidth for the quantized weight. momentum (`float`, *optional*, defaults to `0.95`): Momentum for updating the activation quantization range. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. N@Fffffff?c st||_||_||_||_||_||_||_t t ||g|_ |dt d|dt |j | |_| |_| |_d|_tj|_dS)Nweight_scaling_factorweight_integerF)super__init__num_dim padding_idxmax_norm norm_typescale_grad_by_freqsparser Parametertorchzerosweightregister_buffer zeros_like weight_bitmomentum quant_modepercentile_modeSymmetricQuantFunctionapplyweight_function) selfnum_embeddings embedding_dimrrrrr_weightrrr __class__]/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/ibert/quant_modules.pyr,s  zQuantEmbedding.__init__c Cs|jstj||j|j|j|j|j|j dfS|j}|j }| d}| d}t|j||d|_||j|j|j|j|_tj||j|j|j|j|j|j }||j|jfS)Nr F)rr functional embeddingrrrrrrdatadetachminexpandmax$symmetric_linear_quantization_paramsrr r"rr ) r#x positionsincremental_statew w_transformw_minw_maxemb_intr)r)r*forwardMs<   zQuantEmbedding.forward) NNrFFNrr FNN)__name__ __module__ __qualname____doc__rr; __classcell__r)r)r'r*rs!rcs>eZdZdZd fdd ZddZ     d d d ZZS) QuantActap Quantizes the given activation. Args: activation_bit (`int`): Bitwidth for the quantized activation. act_range_momentum (`float`, *optional*, defaults to `0.95`): Momentum for updating the activation quantization range. per_channel (`bool`, *optional*, defaults to `False`): Whether to or not use channel-wise quantization. channel_len (`int`, *optional*): Specify the channel length when set the *per_channel* True. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. r FNcst||_||_||_||_d|_tj|_ |jsF| dt d| dt d| dt d|j d8_ |jd7_dStd)NFx_minr x_maxact_scaling_factorgh㈵>;per-channel mode is not currently supported for activation.)r ractivation_bitact_range_momentumr per_channel percentiler r! act_functionrrrrCrDNotImplementedError)r#rGrHrI channel_lenrr'r)r*rs zQuantAct.__init__c Cs:|jjd|jd|jd|jdd|jdd S)Nz(activation_bit=z, quant_mode: z , Act_min: z.2fz , Act_max: ))r(r=rGrrCitemrD)r#r)r)r*__repr__s  zQuantAct.__repr__c Cs|dur|n||}|jr|jrJd|jrJd|j}|j} | dkr5|dks9Jd|jdkrT|j dkrT|j||_|j | |_ n2|j dkrjt |j||_t |j | |_ n|j|j |d|j |_|j |j | d|j |_ |j s|dfS|dur|jn|}|dur|j n|} t |j|| |jd |_|dur|||j|j|j} n t|||j|j||} |jd} | | |jfS) Nz:percentile mode is not currently supported for activation.rFrz5NaN detected when computing min/max of the activationg&|g&|>r )rI)trainingrJrIr-r/r1isnansumrCrDrHrrr2rGrErK FixedPointMulr!view) r#r3pre_act_scaling_factoridentityidentity_scaling_factor specified_min specified_maxx_actrCrD quant_act_intcorrect_output_scaler)r)r*r;sH   "    zQuantAct.forward)r FNF)NNNNNr=r>r?r@rrPr;rAr)r)r'r*rBrs rBcs:eZdZdZ d fdd Zfdd Zdd d ZZS) QuantLineara8 Quantized version of `torch.nn.Linear`. Adds quantization-specific arguments on top of `torch.nn.Linear`. Args: weight_bit (`int`, *optional*, defaults to `8`): Bitwidth for the quantized weight. bias_bit (`int`, *optional*, defaults to `32`): Bitwidth for the quantized bias. per_channel (`bool`, *optional*, defaults to `False`): Whether or not to use channel-wise quantization. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. Tr Fcst||_||_tt||g|_| dt |j| dt|j|r?tt||_ | dt |j ||_ ||_ ||_||_||_ d|_tj|_dS)Nr fc_scaling_factor bias_integerF)r r in_features out_featuresrrrrrrrbiasrrrIbias_bitrr r!r")r#rdrerfrrgrIrr'r)r*rs  zQuantLinear.__init__cs*t}d|d|jd|jd}|S)N(z weight_bit=z , quant_mode=rN)r rPrr)r#sr'r)r*rPs zQuantLinear.__repr__Nc Cs |jstjj||j|jddfS|dur|jdksJd|j}|j}|j r=t j |ddd\}}t j |ddd\}}n| d}| d}t|j|||j |_||j|j|j|j|_|j|}|jdurw||j|jd||_|dd}||} tjj| |j|jd||fS)N)rrf)r zInput activation to the QuantLinear layer should be globally (non-channel-wise) quantized. Please add a QuantAct layer with `per_channel = True` before this QuantAct layerr )routFrQ)rrr+linearrrfshaper-r.rIrr/r1r0r2rrbr"rr rgrcrV) r#r3prev_act_scaling_factorr6r7r8_r9bias_scaling_factorx_intr)r)r*r;s0    zQuantLinear.forward)TrraFFNr_r)r)r'r*r`s  r`cs4eZdZdZd fdd ZddZd d d ZZS) IntGELUa} Quantized version of `torch.nn.GELU`. Adds quantization-specific arguments on top of `torch.nn.GELU`. Args: quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. force_dequant (`str`, *optional*, defaults to `"none"`): Force dequantize the layer if either "gelu" or "nonlinear" is given. Tnonecsjt||_|dvrtdd|_|jst|_d|_d|_ gd|_ |j d|j d<dS) N) nonlineargeluzForce dequantize geluFg -?)g]m{ҿgMr r) r rrloggerinforGELU activation_fnkconstcoeff)r#r force_dequantr'r)r*r7s    zIntGELU.__init__cCst|jd|}t|jd|d}t|}tt|| }|||d|}|d|jd}t|d|j}|d|j}||fSNr rwr) rfloorr~signr/abs floor_ster!r})r#rpscaling_factorb_intc_intrabs_inty_intr)r)r*int_erfGs zIntGELU.int_erfNcCs^|js ||dfS||}||||j\}}d|}|||}||d}|||fS)N?rw)rr{rr|)r#r3rrp sigmoid_intsigmoid_scaling_factor shift_intr)r)r*r;Vs   zIntGELU.forward)Trsrq)r=r>r?r@rrr;rAr)r)r'r*rr,s  rrcs:eZdZdZd fdd ZddZdd Zd d ZZS) IntSoftmaxa Quantized version of `torch.nn.Softmax`. Adds quantization-specific arguments on top of `torch.nn.Softmax`. Args: output_bit (`int`): Bitwidth for the layer output activation. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. force_dequant (`str`, *optional*, defaults to `"none"`): Force dequantize the layer if either "softmax" or "nonlinear" is given. Frscst||_d|_||_|dvrtdd|_td|jd|_d|_ d|_ gd |_ |j d |j d <|j d |j d <dS) Nra)rtsoftmaxzForce dequantize softmaxFrgvq -)gN$?g'|:?rr rrw) r r output_bitmax_bitrrxryrBactx0r}coef)r#rrrr'r)r*rrs   zIntSoftmax.__init__cCs~tt|jd|}t|jd|d}Wdn1s%wY||||}|jd|d}||fSr)rno_gradrr)r#rprrrzr)r)r*int_polynomials zIntSoftmax.int_polynomialcCstt|j|}Wdn1swYt||j|}t||}|||}|||\}}tj t|d|j|dd}|d|j}||fS)Nrwrr/) rrrrr1r}rr!rclamp)r#rprx0_intqrexp_intexp_scaling_factorr)r)r*int_exps  "zIntSoftmax.int_expc Cs|js tjj|dddfS||}|jddd\}}||}|||\}}|||\}}||}|jddd} t d|j | } t || d|j |j }dd|j }|||fS)NrQrT)rkeepdimrwr ) rrr+rr1rrrTrr!rr) r#r3rrp x_int_maxrnrrexp exp_int_sumfactorr)r)r*r;s zIntSoftmax.forward)Frs) r=r>r?r@rrrr;rAr)r)r'r*res   rcs<eZdZdZdfdd ZddZd d Zdd d ZZS) IntLayerNorma Quantized version of `torch.nn.LayerNorm`. Adds quantization-specific arguments on top of `torch.nn.LayerNorm`. Args: output_bit (`int`, *optional*, defaults to `8`): Bitwidth for the layer output activation. quant_mode (`bool`, *optional*, defaults to `False`): Whether or not the layer is quantized. force_dequant (`str`, *optional*, defaults to `"none"`): Force dequantize the layer if either "layernorm" or "nonlinear" is given. rFrscst||_||_tt||_tt||_ ||_ |dvr,t dd|_ | dtd||_d|_d|_t|j|j d|_dS)N)rt layernormzForce dequantize layernormFshiftr rar)r rnormalized_shapeepsrrrrrrfrrxryrrrdim_sqrtrB activation)r#rrrrrr'r)r*rs  zIntLayerNorm.__init__cCstA|d}tj|ddd}tt|d|j}|j}t|j||_t dt |dt |jWddS1sHwYdS)NrwTaxisrzDynamic shift adjustment: z -> ) rrrTlog2sqrtrceilr1rrxryint)r#ry_sq_intvar_intr shift_oldr)r)r* set_shifts """zIntLayerNorm.set_shiftcCs:||t|d|j}|d}tj|ddd}|S)z This fallback function is called when overflow is detected during training time, and adjusts the `self.shift` to avoid overflow in the subsequent runs. rwTr)rrr!rrrT)r#r y_int_shiftedrrr)r)r*overflow_fallbacks zIntLayerNorm.overflow_fallbackNcCs|js.|jddd}||}tj|dddd}|t|j|}||j|j}|dfS|jdurHtj|j dtj d}t| |j |_||}t |jddd}||} t| d|j} | d} tj| ddd} |jr| d|jkr|| } | d|jdksJdtt| d|j} td| }t| |d} |jd}|jj|jj}t||}| |} ||j}| |}||fS) NrwTr)dtypeg?zfError detected in overflow handling: `var_int` exceeds `self.max_bit` (the maximum possible bit width)li@)rmeanrrrrrfrtensorrlfloattodevice round_ster!rrrTrRr1rrr-r.)r#r3rryvarnrpmean_intrrrrstd_intrrfbias_intr)r)r*r;s@    zIntLayerNorm.forward)rFrsrq) r=r>r?r@rrrr;rAr)r)r'r*rs    rFc Cs|jd}t|d|d}t||d}tj||dj}|dkr(|d}n tj| |dj }|s<|}|}||fS)a Calculate the percentile max and min values in a given tensor Args: input (`torch.Tensor`): The target tensor to calculate percentile max and min. lower_percentile (`float`): If 0.1, means we return the value of the smallest 0.1% value in the tensor as percentile min. upper_percentile (`float`): If 99.9, means we return the value of the largest 0.1% value in the tensor as percentile max. output_tensor (`bool`, *optional*, defaults to `False`): If True, this function returns tensors, otherwise it returns values. Returns: `Tuple(torch.Tensor, torch.Tensor)`: Percentile min and max value of *input* rr g{Gz?)r|)rlroundrkthvaluevaluesrO) inputlower_percentileupper_percentile output_tensor input_length lower_index upper_index upper_bound lower_boundr)r)r*get_percentile_min_maxs  rcCst|jdkr|dddd}|dddd}nt|jdkr,|dd}|dd}n |d}|d}|rF|d|||Std|||S)a? Quantize single-precision input tensor to integers with the given scaling factor and zeropoint. Args: input (`torch.Tensor`): Single-precision input tensor to be quantized. scale (`torch.Tensor`): Scaling factor for quantization. zero_pint (`torch.Tensor`): Shift for quantization. inplace (`bool`, *optional*, defaults to `False`): Whether to compute inplace or not. Returns: `torch.Tensor`: Linearly quantized value of *input* according to *scale* and *zero_point*. rQr rwr)lenrlrVmul_add_round_rr)rscale zero_pointinplacer)r)r*linear_quantize5s   rcCstKd|dd}|r-tjtj||gdddd\}}tj|dd|}nt||}tj|dd|}Wd|SWd|S1sRwY|S)a/ Compute the scaling factor with the given quantization range for symmetric quantization. Args: saturation_min (`torch.Tensor`): Lower bound for quantization range. saturation_max (`torch.Tensor`): Upper bound for quantization range. per_channel (`bool`, *optional*, defaults to `False`): Whether to or not use channel-wise quantization. Returns: `torch.Tensor`: Scaling factor that linearly quantizes the given range between *saturation_min* and *saturation_max*. rwr rg:0yE>rN)rrr1stackrr)num_bitssaturation_minsaturation_maxrIrrrnr)r)r*r2Xs (     r2c@(eZdZdZeddZeddZdS)r zw Class to quantize the given floating-point values using symmetric quantization with given range and bitwidth. cCsNtjd|jd}d|dd}t|||dd}t|| |d}||_|S)a6 Args: x (`torch.Tensor`): Floating point tensor to be quantized. k (`int`): Quantization bitwidth. percentile_mode (`bool`): Whether or not to use percentile calibration. scale (`torch.Tensor`): Pre-calculated scaling factor for *x*. Note that the current implementation of SymmetricQuantFunction requires pre-calculated scaling factor. Returns: `torch.Tensor`: Symmetric-quantized value of *input*. g)rrwr F)r)rrrrrr)ctxr3r|rrrr new_quant_xr)r)r*r;}s zSymmetricQuantFunction.forwardcCsb|j}t|jdkr|dddd}nt|jdkr!|dd}n|d}||ddddfS)NrrQr rw)rrrlrVclone)r grad_outputrr)r)r*backwards zSymmetricQuantFunction.backwardNr=r>r?r@ staticmethodr;rr)r)r)r*r xs  r c@r)rz; Straight-through Estimator(STE) for torch.floor() cC t|Srq)rrrr3r)r)r*r; zfloor_ste.forwardcC|Srqrrrr)r)r*rzfloor_ste.backwardNrr)r)r)r*r  rc@r)rz; Straight-through Estimator(STE) for torch.round() cCrrq)rrrr)r)r*r;rzround_ste.forwardcCrrqrrr)r)r*rrzround_ste.backwardNrr)r)r)r*rrrcCs|}|d}t|\}}g}|D]}tt|d|j tdtj d}| |qt |}t ||}t||j|t||j|fS)z Decompose the scaling factor into mantissa and twos exponent. Args: scaling_factor (`torch.Tensor`): Target scaling factor to decompose. Returns: ``Tuple(torch.Tensor, torch.Tensor)`: mantisa and exponent rQrw1)rounding)sizerVnpfrexpcpunumpyrdecimalDecimalquantize ROUND_HALF_UPappendarrayrr from_numpyrr)inputsrshape_of_inputoutput_moutput_etmp_mm int_m_shiftedr)r)r* batch_frexps "   r c@s.eZdZdZe  dddZeddZdS)rUaQ Function to perform fixed-point arithmetic that can match integer arithmetic on hardware. Args: pre_act (`torch.Tensor`): Input tensor. pre_act_scaling_factor (`torch.Tensor`): Scaling factor of the input tensor *pre_act*. bit_num (`int`): Quantization bitwidth. z_scaling_factor (`torch.Tensor`): Scaling factor of the output tensor. identity (`torch.Tensor`, *optional*): Identity tensor, if exists. identity_scaling_factor (`torch.Tensor`, *optional*): Scaling factor of the identity tensor *identity*, if exists. Returns: `torch.Tensor`: Output tensor(*pre_act* if *identity* is not given, otherwise the addition of *pre_act* and *identity*), whose scale is rescaled to *z_scaling_factor*. NcCst|jdkr dd}ndd}||_d|dd}t||}|dur,||}||_t||} |tj} |tj tj} | | } || } t | \} }| tj| tj}t|d|}|durt||}|tj} |tj tj} | | } || } t | \}}|tj|tj}t|d|}||}t |tj | d|WdS1swYdS)NrcSs|Srqr)r3r)r)r*sz'FixedPointMul.forward..cSs|dddS)Nr rQ)rVr r)r)r*r srwr r) rrlrXrrz_scaling_factorrtypedoublerr r)rpre_actrWbit_numr rXrYreshaperz_int_A_B new_scalereoutputwx_intm1e1output1r)r)r*r;s<      $zFixedPointMul.forwardcCs8d}|jdur||j}||jdddd|dfSrq)rXrr )rr identity_gradr)r)r*r/s zFixedPointMul.backwardr<rr)r)r)r*rUs 4rU)F)r)rrrrrtorch.autogradrutilsr get_loggerr=rxModulerrBr`rrrrrrr2r rrr rUr)r)r)r*s*    SjP9G e $ # - "