o *ih?@sUddlZddlZddlmZmZmZddlZddlmZm Z m Z m Z m Z m Z mZddlmZddlmZmZmZddlmZddlmZddlmZmZmZdd lmZgd Zd gZe e!Z"ej#e$d < dEd edede deedeeeff ddZ% dEdede dej&jdeefddZ'de de(defddZ)dede*fddZ+dede*fdd Z,dede(fd!d"Z-ed#d$dedeee(efddffd%d&Z.ed#d$ ' ( (dFded)e*d*e*d+e*deee(efddff d,d-Z/dejde0fd.d/Z1dejd0d1de*fd2d3Z2ed4ede*fd5d6Z3e eej4fd7ejd8ejd9ee d:ee d;eej5f de0d?ee d@e*de0f dAdBZ7d;ej5de8fdCdDZ9dS)HN) GeneratorOptionalTuple) FP4_E2M1_DATA FP8_E4M3_DATA FloatArgsQuantizationArgsQuantizationStrategyQuantizationTyperound_to_quantized_type_dtype)QuantizationScheme)generate_mxfp4_scalesmaybe_convert_from_mxfp4_expshould_generatre_mxfp4_scales) deprecated)logger) FloatTensor IntTensorTensor)Module)is_module_quantizedis_model_quantized module_typeget_torch_bit_depth can_quantizeKV_CACHE_TARGETSis_kv_cache_quant_schemeiter_named_leaf_modulesiter_named_quantizable_modulescompute_dynamic_scales_and_zpcalculate_rangecalculate_qparamsgenerate_gparam strategy_cdivzre:.*self_attn$_LOGGERmin_valsmax_valsquantization_args global_scalereturnc Cst|t|}t|t|}|j}t||\}}||}|jrMtt|t|}t|dr:t |d} n|t |d} tj | j ||j d} n$|jdkr\|jtjkr\td||t |} ||| } t| ||} |dury|| } |jdurt| |jd} t|| } t|jdur|jn| j d} t| d ktj| | j |d | } t| |jd d } | jd kr| d } | d } | | fS)a :param min_vals: tensor of min value(s) to calculate scale(s) and zero point(s) from :param max_vals: tensor of max value(s) to calculate scale(s) and zero point(s) from :param quantization_args: settings to quantization :param global_scale: additional global scale to scale the locally generated scale currently only applied/supported for Fp4 :return: tuple of the calculated scale(s) and zero point(s). For FP4, the calculated scale is of dtype FP8 )args)x)devicedtypez0Asymmetric Quantization is not supported for FP4Nr.r)r.r-F)r.cast_to_original_dtype)torchmin zeros_likemaxr-r symmetricabsrr floatzerosshaper.num_bitstyper FLOATNotImplementedErrorclamp scale_dtyper r_get_dtype_epswheretensorzp_dtypendimreshape) r%r&r'r(r-bit_minbit_max bit_range max_val_posscales zero_pointsepsrOj/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/compressed_tensors/quantization/utils/helpers.pyr!AsV           r!valuer*modulec sd}|jtjkrddhtfddt|jD}n;|jtjkr$d}n2|jtjtjfvrFd}d}t |j d|j |j f}| d|}ntjtjtjtjf}td ||s`t|\}} ntj|||d }tj|||d } t|| ||d S) a Returns the computed scales and zero points for dynamic activation quantization. :param value: tensor to calculate quantization parameters for :param args: quantization args :param reduce_dims: optional tuple of dimensions to reduce along, returned scale and zero point will be shaped (1,) along the reduced dimensions :return: tuple of scale and zero point derived from the observed tensor Trr2c3s|] }|vr|VqdSNrO).0idxdimrOrP sz0compute_dynamic_scales_and_zp..NFz+Dynamic quantization is only supported for )rWkeepdims)r()strategyr TOKENtuplerangerFTENSOR TENSOR_GROUPGROUPmathceilr; group_size unflatten ValueErrorr3aminmaxaminamaxr!) rQr*rRr( keep_dims reduce_dims reshaped_dimssupported_strategiesmin_valmax_valrOrVrPrs<  rr-cCs|jtjkr$d|j}tj|dd|d}tj| d|d}||fS|jtjkr`|jdkrCtjtj|d}tjtj |d}||fS|jdkr\tjt j|d}tjt j |d}||fSt dt d|j)a Calculated the effective quantization range for the given Quantization Args :param quantization_args: quantization args to get range of :param device: device to store the range to :return: tuple endpoints for the given quantization range r,r2)r-r/z1Range calculation only supported for 4 and 8 bitszInvalid quantization type ) r=r INTr<r3rDr>rr6r4rr?rf)r'r-rJq_maxq_minrOrOrPr s$    r cCsBt|dsdS|jjdurdS|jjdurdS|jjdurdSdS)z Check if a module is quantized, based on the existence of a non-empty quantization scheme :param module: pytorch module to check :return: True if module is quantized, False otherwise quantization_schemeFNT)hasattrrtweightsinput_activationsoutput_activationsrRrOrOrPrs    rmodelcCstdd|DS)z Check if any modules in a model are quantized, based on the existence of a non-empty quantization scheme in at least one module :param model: pytorch model :return: True if model is quantized, False otherwise css|]}t|VqdSrS)r)rT submodulerOrOrPrXsz%is_model_quantized..)anymodules)rzrOrOrPrsrcCs t|jS)z Gets a string representation of a module type :module: pytorch module to get type of :return: module type as a string )r=__name__ryrOrOrPrs rzThis function will be removed in a future release. Please use `model.named_modules()` and filter by compressed_tensors.InternalModule if neceessary)messageccs|D]F\}}t|}t|dkrd|vr||fVqt|dkr/tt|\}}d}tt|D] }||}d|vrCd}q7|sK||fVqdS)z Yields modules that do not have any submodules except observers. The observers themselves are not yielded :param model: model to get leaf modules of :returns: generator tuple of (name, leaf_submodule) robserverFTN) named_moduleslistchildrenlenzipnamed_childrenr^)rznamer{rrhas_non_observer_childreni child_namerOrOrPrs"    rTFinclude_children include_attn include_mlpc cs|D]`\}}|rMt|}t|dkr!d|vr!||fVn,t|dkr1tt|\}}d}tt|D] } || } d| vrEd}q9|sM||fV|rY|drY||fV|re|dre||fVqdS)aU Yield name and submodule of - leaf modules, set by include_children - attention modyles, set by include_attn :param model: model to get leaf modules of :param include_children: flag to get the leaf modules :param inlcude_attn: flag to get the attention modules :returns: generator tuple of (name, submodule) rrFT self_attnmlpN)rrrrrrr^endswith) rzrrrrr{rrrrrrOrOrPr:s0        rcCs8z t|jj}W|Styt|jj}Y|Sw)z Determine the number of bits used to represent the dtype of a tensor :param value: tensor to check bit depth of :return: bit depth of each element in the value tensor )r3finfor.bits TypeErroriinfo)rQ bit_depthrOrOrPrhs r quant_argsrcCs:t|}|j}||jkrtd|d|d||jkS)aI Checks if value can be quantized by quant_args. :param value: tensor to check for quantization :param quant_args: QuantizationArgs to use for quantization :return: False if value is already quantized to quant_args or value is incompatible with quant_args, True if value can be quantized with quant_args z%Can't quantize tensor with bit depth z to zH.The QuantizationArgs provided are not compatible with the input tensor.)rr<r$warn)rQrrrequested_depthrOrOrPrws   rschemecCs|jD] }|tvr dSqdS)a  Check whether the QuantizationScheme targets the kv cache. It does if all the following criteria are met: - the scheme targets either exactly match the KV_CACHE_TARGETS or the match KV_CACHE_TARGETS regex pattern - the scheme quantizes output_activations (we want to quantize the outputs from the KV_CACHE_TARGETS, as their correspond to the keys and values that are to be saved in the cache) :param scheme: The QuantizationScheme to investigate :return: boolean flag TF)targetsr)rtargetrOrOrPrs rupdated_min_valupdated_max_val scale_data quant_datar.c Cs^t|t|}t|t|}tt|t|}|j|j|}||dgS)ah Generate a global scale for an entire tensor (input_tensor). Goal of the scale is to ensure that the quantization (local) scale falls into the approproiate dtype range. E.g. for NVFP4, group (local) scales are in dtype FP8. The global_scale attempts to use the entire FP8 dtype range while mapping a per-group max to the FP4 max. r2)r3r4r5r6r8torG) rrrrr.r%r&rKr(rOrOrPr"s r"divisorr[strictcCsVt||}|||kr)|d|d|d|}|r t|tjdd||S)NzJ quantization strategy requires strict division of weight/activation size z and group/block size z[. consider reducing the group/block size or ignoring modules with weights not divisible by T)log_once)rbrcrfrbindwarning)rQrr[rdividendrrOrOrPr#s r#cCs@|tjkrdS|tjkrdSttjg|drt|jSdS)Ng?g?r0r2)rr.rr3is_floating_pointrDrrNr0rOrOrPrBs   rBrS)TFF)F):loggingrbtypingrrrr3*compressed_tensors.quantization.quant_argsrrrrr r r ,compressed_tensors.quantization.quant_schemer 1compressed_tensors.quantization.utils.mxfp4_utilsr rrcompressed_tensors.utilsrlogururrrrtorch.nnr__all__r getLoggerr~r$Logger__annotations__r!nnrstrr boolrrrrrintrrrfloat32r.r"r#r9rBrOrOrOrPs $      Z ;  & )