o b¦µi^ƒã@s ddlZddlZddlmZddlmZddlmZddlm Z ddlm Z ddl m Z ddl m Z dd lmZdd lmZ ed d gƒGd d„dƒƒZedƒdde ¡dfdd„ƒZedƒGdd„deƒƒZdd„ZGdd„deƒZedƒ   d>dd„ƒZ ed ƒd?d!d"„ƒZed#ƒd$d%„ƒZed&ƒd'd(„ƒZed)ƒd@d*d+„ƒZed,ƒd@d-d.„ƒZGd/d0„d0eƒZddd1dd2d3œd4d5„Zd6d7„Z d8d9„Z!d:d;„Z"dhs û r>cCsÆt |jd¡}t ||¡}t ||¡}d|>d}|sdnd}t ||¡}t |||¡}t |||¡} |t ||¡} t | ||¡} t | ¡} t  t || ¡|¡} t  t || ¡|¡}| ||| fS)z>Adjusts and nudges the quantization range for better accuracy.Úfloat32r$r) rÚ result_typer(rr5Úsubtractr,r1r2r0)Ú min_rangeÚ max_rangeÚnum_bitsÚ narrow_rangeÚ compute_dtypeÚ quant_maxÚ quant_minÚ diff_ranger;Ú inv_scaleÚzero_point_from_minÚ zero_pointÚnudged_zero_pointÚ nudged_minÚ nudged_maxrrrÚadjust_and_nudge‡s       rUcs.eZdZd ‡fdd„ Zdd„Zdd „Z‡ZS) ÚFakeQuantWithMinMaxVarséFNcs tƒ ¡||_||_||_dSr)ÚsuperrrIrJr%)rrIrJr%©Ú __class__rrr©s  z FakeQuantWithMinMaxVars.__init__cCst||||j|j|jdS)N)rIrJr%)Úfake_quant_with_min_max_varsrIrJr%©rr6Úmin_valsÚmax_valsrrrÚcall¯súzFakeQuantWithMinMaxVars.callcCst|j|jdS)Nr')rÚshaper(r\rrrÚcompute_output_spec¹sz+FakeQuantWithMinMaxVars.compute_output_spec©rWFN)rrr rr_raÚ __classcell__rrrYrrV¨s rVz-keras.quantizers.fake_quant_with_min_max_varsrWc sLt|fƒr tƒ |||¡St |¡}t |¡}t |¡}tˆƒ‰ˆdur*tˆ|jƒ‰t ¡dkr•ddl }t  |j ¡}ˆdurd|j j t |d¡t t |d¡d¡t t |d¡d¡ˆˆd}tj||dS|jd} t |ˆ| ¡}|j jt |d¡t |d¡t |d¡ˆˆd}tj||d}t || ˆ¡Stj‡‡‡fd d „ƒ} | |||ƒS) auPerform per-tensor or per-channel fake quantization. `[min_vals, max_vals]` define the clamping range for the `inputs`. The `inputs` are quantized into the quantization range: - `[0, 2^num_bits - 1]` when `narrow_range=False` - `[1, 2^num_bits - 1]` when `narrow_range=True` After quantization, the values are dequantized and output as floats within the `[min_vals, max_vals]` interval. This operation supports gradient computation, allowing `min_vals` and `max_vals` to be trained. Args: inputs: Input Keras tensor of float dtype. min_vals: A global minimum scalar or a per-channel minimum tensor. max_vals: A global maximum scalar or a per-channel maximum tensor. num_bits: Quantization bit width (e.g., `8` for int8). Defaults to `8`. narrow_range: Whether to use narrow quantization range. Defaults to `False`. axis: Axis along which to perform per-channel quantization. If `None`, per-tensor quantization is performed. Defaults to `None`. Returns: Tensor: A Keras tensor with fake quantization applied. NÚ tensorflowrrDr)rIrJr'r$c sât ˆj¡}t||ˆˆƒ\‰‰}}t t t ˆ |¡d¡¡}t ˆt  ˆˆj¡t  ˆˆj¡¡}t  |ˆ¡}t t t t  t ||¡|¡d¡¡|¡} tj | |d} t  t  ˆˆ¡t  ˆˆ¡¡‰ddœ‡‡‡‡‡fdd„ } | | fS)Ngà?r')Úupstreamcsº|dur|\}t ˆ|d¡}‡fdd„tt|jƒƒDƒ}t ˆˆ¡}t ||d¡}ˆdur5tj||d}nt |¡}t ˆˆ¡}t ||d¡}ˆdurStj||d}nt |¡}|||fS)Nçcóg|]}|ˆkr|‘qSrr©Ú.0Úi©r%rrÚ 7ózqfake_quant_with_min_max_vars.._fake_quant_with_min_max_vars_per_channel..grad..rk)rÚwhereÚrangeÚlenr`Ú less_equalÚsumÚ greater_equal)reÚargsÚdxÚaxesÚmin_maskÚgrad_minÚmax_maskÚgrad_max)r%ÚmasksrTrSrrrÚgrad1s     z]fake_quant_with_min_max_vars.._fake_quant_with_min_max_vars_per_channel..grad)rr)r(rUrÚfloorr-r0r1r5rFÚ logical_andrsrq) rÚmin_valÚmax_valr(r;rOÚ quant_zeroÚ x_clampedÚx_clamped_shiftedÚresultr|©r%rJrI)r{rTrSrrÚ)_fake_quant_with_min_max_vars_per_channels8  ÿÿÿ  ÿüÿ÷ ÿzOfake_quant_with_min_max_vars.._fake_quant_with_min_max_vars_per_channel)rrVÚ symbolic_callrr4r@rÚndimrrdr)r(Ú quantizationr[r5ÚreshapeÚswapaxesÚ(fake_quant_with_min_max_vars_per_channelÚcustom_gradient) r6r]r^rIrJr%Útfr(r<Ú last_axisr†rr…rr[½sH %ÿ       û    û ?r[z%keras.quantizers.compute_float8_scalecCsRt |¡}t t ||¡d|¡}t |dk||¡}t t |¡||¡}t |¡S)Nérf)rÚ reciprocalr,rnÚisfinite)Úamaxr;Ú dtype_maxÚmarginÚsfrrrÚcompute_float8_scaleUs  r—z,keras.quantizers.compute_float8_amax_historycCsDt t t |¡¡|j¡}t tj|dddggt |dg¡¡}|S)Néÿÿÿÿ)Úshiftrr$)rr5r.r/r(Úscatter_updateÚrollrŠ)rÚ amax_historyÚ amax_updateÚnew_amax_historyrrrÚcompute_float8_amax_historybs  ýrŸz(keras.quantizers.quantize_and_dequantizecCsht tt |¡jƒ|¡}t |t ||¡¡}t || |¡}t ||¡}t t ||¡t ||¡¡}|Sr) rr5ÚfloatÚ ml_dtypesÚfinfor.r,r1r0)r6r;Úquantized_dtyperKÚquantized_dtype_maxrrrrÚquantize_and_dequantizemsÿ r¥zkeras.quantizers.pack_int4csž|dvr td|›dƒ‚t |j¡|kr#td|›dt |j¡›dƒ‚t|jddƒp.t|jƒ}ˆd kr7ˆ|7‰ˆg‡fd d „t|ƒDƒ‰‡fd d „t|ƒDƒ}t   |ˆ¡}t  |¡d }t   t   |d ¡d¡}|dd…dfd }t j ||gd d} |t  |d¡} | d| …df} | ddd …df} | ddd …df} t jd|d}t  | |¡}t  | |¡}t  |t  |d¡¡}t  ||¡}t   ||¡}|}|t  |¡|fS)ae Pack an int4 tensor into an int8 tensor with packed nibbles. The input values must already be int8 in the signed range `[-8, 7]` and represent the desired int4 values. Packing is performed along the specified axis (default is 0). For every two consecutive rows, the **low nibble** of the output byte stores the value from the first row, and the **high nibble** stores the value from the second row. Args: arr: An `int8` or `uint8` tensor containing int4 values in the range `[-8, 7]`. axis: The axis along which to pack the tensor. Defaults to 0. dtype: The data type of the input and packed tensor. Can be `"int8"` or `"uint8"`. Defaults to `"int8"`. Returns: tuple: A tuple `(packed, packed_shape, orig_rows)` where `packed` is the packed int8 tensor with int4 values stored in nibbles, `packed_shape` is the shape of the packed tensor, and `orig_rows` is the original (unpacked) row count prior to any padding that may have been inserted when an odd number of rows is supplied. Example: ```python >>> import numpy as np >>> from keras.quantizers import pack_int4, unpack_int4 # Example with axis=0 # Original array has shape (3, 2) >>> original_array = np.array([[-3, 7], [2, -8], [1, 0]], dtype=np.int8) # Pack the array along axis 0. Since the length of axis 0 (3) is # odd, it will be padded to a length of 4. The packed array will # have a shape of (ceil(3/2), 2) = (2, 2). >>> packed, packed_shape, orig_len = pack_int4(original_array, axis=0) >>> print("Packed array: ", packed) Packed array: [[ 45 -121] [ 1 0]] # Now, unpack the array back to its original form >>> unpacked = unpack_int4(packed, orig_len, axis=0) >>> print("Unpacked array: ", unpacked) Unpacked array: [[-3 7] [ 2 -8] [ 1 0]] >>> np.allclose(original_array, unpacked) True # Example with axis=1 # Original array has shape (2, 3) >>> original_array = np.array([[-3, 7, 2], [-8, 1, 0]], dtype=np.int8) # Pack along axis 1. Length of axis 1 (3) is padded to 4. # The new shape is (2, ceil(3/2)) = (2, 2). >>> packed, packed_shape, orig_len = pack_int4(original_array, axis=1) >>> print("Packed array: ", packed) Packed array: [[ 125 2] [ 24 0]] # Unpack the array >>> unpacked = unpack_int4(packed, orig_len, axis=1) >>> print("Unpacked array: ", unpacked) Unpacked array: [[-3 7 2] [-8 1 0]] >>> np.allclose(original_array, unpacked) True ``` ©r Úuint8ú1Expected dtype to be 'int8' or 'uint8', but got 'ú'.z Expected z tensor for packing, got Ú.ÚrankNrcrgrrrhrkrrrlÙrmzpack_int4..cóg|]}ˆ |¡‘qSr©Úindexrh©ÚpermrrrlÚórr$.rkÚint32ér'é)Ú ValueErrorrr)r(Ú TypeErrorÚgetattrr`rprorÚ transposeÚequalÚmodÚ concatenater5ÚarrayÚ bitwise_andÚ bitwise_orÚ left_shift)Úarrr%r(r«Úinv_permÚ transposedÚrowsÚ needs_padÚzero_rowÚ padded_fullÚ rows_packedÚpaddedÚlowÚhighÚmaskÚlow_uÚhigh_uÚpackedÚorig_lenr©r%r°rÚ pack_int4|s@M ÿ ÿÿ     rÑzkeras.quantizers.unpack_int4cs|dvr td|›dƒ‚t |j¡dvrtd|j›ƒ‚dd„}t|jddƒp+t|jƒ}ˆd kr4ˆ|7‰ˆd krŽ|d krŽtj d |jd }t  ||¡}t  t  |d ¡|¡}|dkr`||ƒ}||ƒ}t  ||¡} t  ||¡} tj | | gdd} t | dtt |¡dd…ƒ¡} | d|…dfSˆg‡fdd„t|ƒDƒ‰‡fdd„t|ƒDƒ} t |ˆ¡}tj d |jd }t  ||¡}t  t  |d ¡|¡}|dkrÑ||ƒ}||ƒ}t  ||¡}t  ||¡}tj ||gdd} t | dtt |¡dd…ƒ¡} | d|…df} t | | ¡} | S)aƒ Unpack a packed int4 back to an int8 tensor in the range [-8, 7]. This function reverses the packing performed by `pack_int4`, restoring the original int8 tensor (values in the range [-8, 7]) from a packed int8 tensor where each element contains two int4 values (one in the lower nibble, one in the upper nibble). The function restores the original axis order and removes any padding that was added during packing. Args: packed: An int8 tensor containing packed int4 values along the specified axis. Each int8 value encodes two int4 values. orig_len: The original (unpadded) length of the axis that was packed. This is used to remove any padding that may have been added during packing to ensure an even number of rows. axis: The axis along which the tensor was packed. Defaults to 0. dtype: The data type of the input and unpacked tensor. Can be `"int8"` or `"uint8"`. Defaults to `"int8"`. Returns: unpacked: An int8 tensor with the same shape as the original (unpacked) tensor, with values in the range [-8, 7]. Example: ```python >>> import numpy as np >>> from keras.quantizers import pack_int4, unpack_int4 # Example with axis=0 # Original array has shape (3, 2) >>> original_array = np.array([[-3, 7], [2, -8], [1, 0]], dtype=np.int8) # Pack the array along axis 0. Since the length of axis 0 (3) is # odd, it will be padded to a length of 4. The packed array will # have a shape of (ceil(3/2), 2) = (2, 2). >>> packed, packed_shape, orig_len = pack_int4(original_array, axis=0) >>> print("Packed array: ", packed) Packed array: [[ 45 -121] [ 1 0]] # Now, unpack the array back to its original form >>> unpacked = unpack_int4(packed, orig_len, axis=0) >>> print("Unpacked array: ", unpacked) Unpacked array: [[-3 7] [ 2 -8] [ 1 0]] >>> np.allclose(original_array, unpacked) True # Example with axis=1 # Original array has shape (2, 3) >>> original_array = np.array([[-3, 7, 2], [-8, 1, 0]], dtype=np.int8) # Pack along axis 1. Length of axis 1 (3) is padded to 4. # The new shape is (2, ceil(3/2)) = (2, 2). >>> packed, packed_shape, orig_len = pack_int4(original_array, axis=1) >>> print("Packed array: ", packed) Packed array: [[ 125 2] [ 24 0]] # Unpack the array >>> unpacked = unpack_int4(packed, orig_len, axis=1) >>> print("Unpacked array: ", unpacked) Unpacked array: [[-3 7 2] [-8 1 0]] >>> np.allclose(original_array, unpacked) True ``` r¦r¨r©z1Expected int8 or uint8 tensor for unpacking, got cSs:t |j¡}t d|¡}t d|¡}t ||k|||¡S)z9Converts unpacked nibbles [0, 15] to signed int4 [-8, 7].rWé)rr)r(rr5rn)rÚdtype_xÚeightÚsixteenrrrÚ to_signedVs   zunpack_int4..to_signedr«Nrrr³r'r´r r$rk)r˜.crgrrrhrkrrrlwrmzunpack_int4..cr¬rr­rhr¯rrrlxr±)rµrr)r(r¶r·r`rprr¼r½Ú right_shiftr5ÚstackrŠrAror¸)rÎrÏr%r(rÖr«rËÚ low_unpackedÚ high_unpackedÚ low_finalÚ high_finalÚstackedÚunpackedrÁrÂrÉrÊrrÐrÚ unpack_int4ÿsPM ÿ ÿ   "    " rßcsLeZdZdZeddddfdd„Zddd „Z‡fd d „Zed d „ƒZ ‡Z S)Ú GPTQQuantizeraA class that handles the quantization of weights using GPTQ method. This class provides methods to find quantization parameters (scale and zero) for a given tensor and can be used to quantize weights in a GPTQ context. Args: weight_bits: (int) The number of bits to quantize to (e.g., 4). per_channel: (bool) A flag indicating whether quantization is applied per-channel (`True`) or per-tensor (`False`). Defaults to `False`. symmetric: (bool) A flag indicating whether symmetric (`True`) or asymmetric (`False`) quantization is used. Defaults to `False`. group_size: (int) The size of weight groups for quantization. A value of -1 indicates that grouping is not used. Defaults to -1. N)Ú tokenizerÚdatasetrDcCsFt |¡|j|_|j|_|j|_|j|_||_d|_d|_d|_ dSr) r rÚ weight_bitsÚ per_channelÚ symmetricÚ group_sizerKr;ÚzeroÚmaxq)rrrKrrrr¥s  zGPTQQuantizer.__init__Tc Cs<t||j|j|j|j||jd\|_|_|_|j|j|jfS)zBFinds quantization parameters (scale and zero) for a given tensor.)ÚbitsråräræÚweightrK) Úcompute_quantization_parametersrãrårärærKr;rçrè)rÚ input_tensorrêrrrÚ find_params¶sù zGPTQQuantizer.find_paramscs*tƒ ¡}| |j|j|j|jdœ¡|S)N)rãräråræ)rXrÚupdaterãräråræ)rrrYrrrÃs üÿzGPTQQuantizer.get_configcCs,tdd|d|d|d|dd}||ƒS)Nrãräråræ)rárârãrärårær )rrÚgptqrrrrÏsúzGPTQQuantizer.from_config)T) rrr Ú__doc__r rrírr!rrcrrrYrrà“s  ý   ràr˜rD)rårärærêrKc Cs^|dur td|›dƒ‚|r#t|jƒdkr#td|›dt|jƒ›dƒ‚t |¡dkr.td ƒ‚|j}|rM|rL|d krBt |d |g¡}nt ||dd g¡}nt |d d g¡}tj|d d } tj|d d } |r|t t  | ¡| ¡} t  t  | d¡t  | ¡| ¡} t  | | ¡} t  | t | d ¡| ¡} t  | t | d ¡| ¡} t t t d|¡d ¡|¡} t t | | ¡| ¡} |rÁt | t t | d ¡d¡¡}n t t t  | ¡| ¡¡}t  t | d¡d | ¡} |r|rò|d kròt | d d g¡} t |d d g¡}n|s|d}t t | d¡|d f¡} t t |d¡|d f¡}|r$t | d d g¡} t |d d g¡}t |d¡}| || fS)a… Computes the scale and zero-point for quantization. This function calculates the scale and zero-point required for quantizing a given tensor `x` based on the specified parameters. It supports grouped, per-channel, per-tensor, symmetric, and asymmetric quantization - along with any combinations of these. Args: x: KerasTensor. The input tensor to quantize. bits: int. The number of bits to quantize to (e.g., 4). symmetric: bool. Whether to use symmetric quantization. per_channel: bool. Whether to quantize per channel. group_size: int. The group size for quantization. weight: bool. Whether the input tensor is a weight tensor. Returns: scale: KerasTensor. The scale tensor for quantization. zero: KerasTensor. The zero tensor for quantization. maxq: scalar. The maximum quantization value. Nz Input tensor z cannot be None.rzInput weight tensor z. must have a rank of at least 2, but got rank rªrz!Input tensor 'x' cannot be empty.r˜r$rkç:Œ0âŽyE>)r$r$r§)rµrpr`rÚsizerŠÚminr.Úmaximumr/rnÚlessÚnegativer¹rFr-r5Úpowerr,Ú full_liker2rqÚtile)rrérårärærêrKÚoriginal_shapeÚinput_reshapedÚ min_valuesÚ max_valuesÚ zero_rangerèr;rçÚnum_rowsrrrrëÜs^ÿÿ€ÿ    rëc Cs\tjd|jd}t t |d¡||¡}t t t ||¡t ||j¡¡¡}t |d|¡}|S)a Quantize a float tensor into discrete levels [0, maxq] using per-tensor/per-channel/grouped scaling. Returns `q` (same dtype as inputs/scales; float is fine) where values are in [0, maxq]. Args: input_tensor: KerasTensor. The input tensor to quantize. scale: KerasTensor. The scale tensor for quantization. zero: KerasTensor. The zero tensor for quantization. maxq: KerasTensor. The maximum quantization value. Returns: KerasTensor. The quantized tensor. rñr'r) rr5r(rnr¹r2r-r,r1)rìr;rçrèr8Ú safe_scaleÚquantized_tensorrrrÚquantize_with_zero_pointAsÿÿrc Cst |t |t ||j¡¡¡S)a` Dequantizes a quantized tensor using the provided scale and zero tensors. Args: input_tensor: KerasTensor. The quantized tensor to dequantize. scale: KerasTensor. The scale tensor for dequantization. zero: KerasTensor. The zero tensor for dequantization. Returns: KerasTensor. The dequantized tensor. )rr0rFr5r()rìr;rçrrrÚdequantize_with_zero_point^s ÿrcCs:t |d¡}tj||dd}tj||dd}t||||ƒS)ašQuantize the weight matrix from group params. This function uses the provided scale and zero tensors to quantize the input weights_matrix according to the group indices. It maps each column of the weights_matrix to its corresponding group parameters and performs the quantization operation. Args: weights_matrix: 2D tensor of shape [out_features, in_features]. scale: Per-group scale tensor of shape [out_features, n_groups]. zero: Per-group zero-point tensor of shape [out_features, n_groups]. g_idx: Integer tensor of shape [in_features,] mapping each column to its group index. maxq: Scalar (float) representing the maximum integer quantization level (e.g., 2^bits - 1). Returns: A tensor with the same shape as `weights_matrix` containing the quantized weights produced using the provided group parameters. r²r$rk)rr5Útaker)Úweights_matrixr;rçÚg_idxrèÚgroupsÚ scale_colsÚ zero_colsrrrÚquantize_with_sz_mapos r cCsRt |d¡}tj||dd}tj||dd}t ||j¡}t t ||¡|¡}|S)a©Rebuild a dequantized weight matrix from group params. This function uses the provided scale and zero tensors to dequantize the input weights_matrix according to the group indices. It maps each column of the weights_matrix to its corresponding group parameters and performs the dequantization operation. Args: weights_matrix: 2D tensor of shape [out_features, in_features]. scale: Per-group scale tensor of shape [out_features, n_groups]. zero: Per-group zero-point tensor of shape [out_features, n_groups]. g_idx: Integer tensor of shape [in_features,] mapping each column to its group index. maxq: Scalar (float) representing the maximum integer quantization level (e.g., 2^bits - 1). Returns: A tensor with the same shape as `weights_matrix` containing the dequantized weights produced using the provided group parameters. r²r$rk)rr5rr(r0rF)rr;rçrrÚ scales_mappedÚ zeros_mappedÚ quantizedrrrÚdequantize_with_sz_mapŒs  ÿrrb)r)rr )$r¡Únumpyr+Ú keras.srcrrÚkeras.src.api_exportrÚkeras.src.backendrrÚ&keras.src.backend.common.backend_utilsrrÚkeras.src.ops.operationr Ú keras.src.quantizers.gptq_configr r r8r=r>rUrVr[r—rŸr¥rÑrßràrërrr rrrrrÚsl          1 ú$! ú     M øe