o biN@sddlZddlmZddlmZddlmZddlmZddlmZddl m Z ddl m Z dd l mZdd l mZdd l mZdd l mZd dZddddedddZGdddZdS)N)partial)ops) quantizers)Dense) EinsumDense)linalg) GPTQConfig) GPTQQuantizer)compute_quantization_parameters)dequantize_with_zero_point)quantize_with_zero_pointcCs^t|d}tjd|dd}tt|dt|d}t|t|d}tt|S)zReturn a stable permutation that sorts `metric` in descending order. Uses an index-based jitter to break ties deterministically.rint32dtypefloat32-q=) rshapearangedividecastaddmultiplyargsortnegative)metricnidxjittermetric_jitteredrM/home/ubuntu/.local/lib/python3.10/site-packages/keras/src/quantizers/gptq.py_stable_permutations r!F) blocksize group_sizeactivation_order order_metriccompute_scale_zeroc. Cst|d}|rM|durttt|d}nt|d}tt||t|}t |}t |} tj ||dd}tj tj ||dd|dd}nd}} |} tj|dd} g} g} |d krd|n|}t d||D]D}t |||}||}| dd||f}t|}|||||f}d}d}d}d }t |D]}||}|dd|f}||ks|||}||krt |||}| dd||f}||\}}}| || ||}|||}} }!n|dur|| \}}}| || |d}|||}} }!tt|d|| |!}"t| d|ft|"d} t|"|| dddf}#|||f}$tt||#|$}%t|d|ft|%d}||dkrtt|%dt|||ddfd}&|dd|ddf}'t|d|dft|'|&}q||krt|||||df}(tj| ddd|ft| dd|df|(gdd} ql|})tjd|dd}*t|*|)}*t|*d}*|rtj |*| dd}*tj | | dd} t| dkr||\}+},}-|+}|,} ntj| dd}tj| dd} | || |*fS) ab Implements the GPTQ error correction updates. For a single column update (column j): e = invH[j, j] * (w_j - q_j) W[:, j+1:] -= e * invH[j, j+1:] where: - w_j is the original column, - q_j is the quantized column, - invH is the inverse Hessian, - e is the propagated error term. Across entire blocks: W[:, future] -= E_block * invH[block, future] where: - E_block is the quantization error accumulated for the current block, - invH[block, future] denotes the cross-block slice of the inverse Hessian, - W[:, future] are the columns yet to be quantized. Args: weights_transpose: Transposed weight matrix [out_features, in_features] to quantize. inv_hessian: Inverse Hessian matrix [in_features, in_features] for error propagation. blocksize: Size of the blocks to process (default: 128). group_size: Size of the groups for parameter reuse (default: -1, no grouping). activation_order: Whether to apply activation-order permutation (default: False). order_metric: Metric for ordering features (default: None, uses 1 / diag(invH)). compute_scale_zero: Function to compute scale and zero for quantization. Returns: quantized_weights: Quantized weight matrix [out_features, in_features]. scale: float32. Scale parameters for quantization [out_features, num_groups]. zero: Zero-point parameters for quantization [out_features, num_groups]. g_idx: int32. Group indices for each feature [in_features]. Nrr)axisrr rr#)rr reciprocalrdiagonalrwhereisfinite zeros_liker!rtakerangeminappendr expand_dims slice_updater rsubtractmatmul concatenaterlen).weights_transpose inv_hessianr$r%r&r'r( in_featuresperminv_permweights_bufferquantized_weights_buffer scale_chunks zero_chunkseffective_group block_start block_end block_size block_weights block_errorblock_inv_hessian cached_scale cached_zero cached_maxqcached_group_start block_idx global_idx weight_column group_start group_end group_slicescalezeromaxqquantized_columndequantized_colcurrent_block_influenceerrupdatetail total_update base_groupg_idxsz_rrr gptq_quantize_matrixs3                      rcc@s<eZdZedddfddZddZ d dd Zd d ZdS) GPTQN) tokenizerdatasetc CsF||_d|_||_t||jd|_t|ts!t|tr6|j j dkr6|j j |_ |j d|_ |j d|_||_n_t|tr|j j dkr|j j |_ t|j }|t|}|dkrg|\}}}|t|||_ |_n|dvr{|\}}}t||||_ |_tjt|j |j |jfd|_n tdt|tj|j |j fd d |_dS) Nr) compute_dtyper))r)rh)kernelz!Unsupported layer type for GPTQ: rr)original_layer num_samplesconfigr variable_dtype quantizer isinstancerrrjndimr kernel_shaperowscolumnslayerlistindexmaxrrtypesSimpleNamespacereshape TypeErrortypezeroshessian) selfrurmrd_model_dim_indexr<headshead_dim out_featuresrrr __init__sB              z GPTQ.__init__c Csp|durtdt|jdkrtdt|jdt|dkr%tdt|jdkr7t|d|jdf}t|d }t|d}|j}t||}t|j dt|dkrrtd t|j dd t|ddt t ||}t t|t |d }|jdkrt |j t |||_ t|j t t d |||_ |jt|dpd|_dS) a Updates the running average of the Hessian matrix with a new batch. This method computes the Hessian matrix for a given batch of input activations and updates the accumulated Hessian (`self.hessian`) using a numerically stable running average. This allows the Hessian to be computed over a large dataset without loading all samples into memory at once. The input tensor is first reshaped into a 2D matrix [num_samples, num_features] before the Hessian is calculated. Args: input_batch: A 2D or higher-dimensional tensor of input activations from a calibration batch. Raises: ValueError: If the feature dimension of the input tensor `input_batch` does not match the dimensions of the pre-initialized Hessian matrix `self.hessian`. NzInput tensor cannot be None.rhz+Input tensor must have rank >= 2 (got rank z).rzInput tensor cannot be empty.r#rzHessian dimensions (z) do not match input features (g@) ValueErrorr9rrsizer{rrlrrr7 transposerr)r input_batchxnum_new_samplesnum_prev_samples total_samples gram_matrixrrr update_hessian_with_batchDsF    zGPTQ.update_hessian_with_batchr"c Cs`t|jj}t|j}t|d}t|d|}t|jt t|dt |}t |j j t|}t||}tt|t t|t |}t|}t||||j j|j jt|t|jjddd\}} } } t||jjj}|j jdkrtj|ddd \}} } |j`|jj ||jj! | |jj" | |jj# | d|j_$d S) a8 Performs GPTQ quantization and correction on the layer's weights. This method implements the core logic of the "Optimal Brain Quant" (OBQ) method, as applied by GPTQ, to quantize the weights of a single layer. It iteratively quantizes blocks of weights and corrects for the quantization error by updating the remaining weights. The algorithm follows these main steps: 1. Initialization: It optionally reorders the weight columns based on activation magnitudes (`activation_order=True`) to protect more salient weights. 2. Hessian Modification: The Hessian matrix, pre-computed from calibration data, is dampened to ensure its invertibility and stability. 3. Iterative Quantization: The function iterates through the weight columns in blocks (`blocksize`). In each iteration, it: a. Quantizes one column. b. Calculates the quantization error. c. Updates the remaining weights in the *current* block by distributing the error, using the inverse Hessian. 4. Block-wise Correction: After a block is quantized, the total error from that block is propagated to the *next* block of weights to be processed. 5. Finalization: The quantized weights are reordered back if `activation_order` was used, and the layer's weights are updated. This implementation is based on the official GPTQ paper and repository. For more details, see: - Paper: https://arxiv.org/abs/2210.17323 - Original Code: https://github.com/IST-DASLab/gptq Args: blocksize: (int, optional) The size of the weight block to process at a time. Defaults to 128. gg?T)weight)r;r$r%r&r'r(ruint8)r*rN)%rrrurjr,requalr-rdiagr/rrmhessian_dampingmeanr6rinvrcr%r&rro find_paramsrrkquantized_kernelr weight_bitsr pack_int4_kernelassign kernel_scale kernel_zeror_is_gptq_calibrated) rr$weights_matrixhessian_diagonal dead_diagonalhessian_matrixdamping_factorinverse_hessian quantizedrTrUr_rbrrr quantize_and_correct_layersV)        zGPTQ.quantize_and_correct_layercCs |`|`dS)N)rru)rrrr freesz GPTQ.free)r")__name__ __module__ __qualname__rrrrrrrrr rds 2G _rd)ry functoolsr keras.srcrrkeras.src.layersrr keras.src.opsr keras.src.quantizers.gptq_configrkeras.src.quantizers.quantizersr r r r r!rcrdrrrr s*            v