o i*@sddlZddlmZddlmZmZddlZddlmZddl m Z m Z m Z m Z mZddlmZmZmZgdZGdd d eZ d ed eed efd dZd ed efddZeZee deje jddde je jddZee deje jdddde je jd e deje jde jdde je jd dZ ee deje j!dddej"ej"ddZ#ee deje j!dddej"ej"de deje j!dddej"ej"ddZ$ee dej%e j&ddde dej%e j'dddddZ(ee dej%e j&ddddZ)ee dej%e j!dddd dZ*ee dej%e j!dddd dZ+ee dej%de j!ddd!e dej%e j'dddddZ,ee deje j-ddde deje j-ddddZ.ee deje j&ddde deje j'dddddZ/ee deje j0ddddgd"e deje j!ddddd#dZ1ee)e*e+e(e(e,e.e/e1ee e#e$d$Z2dS)%N)deepcopy)ListOptional)CompressionFormat) FP8_E4M3_DATA DynamicTypeQuantizationArgsQuantizationStrategyQuantizationType) BaseModel ConfigDictmodel_validator)QuantizationSchemepreset_name_to_schemeis_preset_schemec@s|eZdZUdZeeed<dZee ed<dZ ee ed<dZ ee ed<dZ eeed<e dd dd d ZeddZdS)ra Set of QuantizationArgs defining how the weights, inputs and outputs of target list of modules should be quantized :param targets: list of modules to apply the QuantizationArgs to, can be layer names, layer types or a regular expression, typically ["Linear"] :param weights: quantization config for layer weights :param input_activations: quantization config for layer inputs :param output_activations: quantization config for layer outputs :param format: CompressionFormat for the layer targetsNweightsinput_activationsoutput_activationsformatafter)modemodelreturncCs|j}|j}|j}|j}|dur@|jtjtjtjtj tj fvr7|jtjkr.|j dur.t dt d|jd|j dur@td|durM|j durMtd|tjjkrWtd|r~|r~|jtjkr~|jtjkr~|j|jkr~tjd|jd |jd td d |S) NTzDStatic and local group-wise activation quantization is not supportedzUsing z6 strategy is not supported for activation quantizationz*Cannot apply actorder to input activationsz+Cannot apply actorder to output activationszBmixed-precision cannot be set as a format for a QuantizationSchemezXUsing GROUP strategy for both weights and input_activations with different group sizes (z vs zu) may complicate fused kernel implementations. Consider using TENSOR_GROUP strategy for both or matching group sizes.) stacklevel)rrrrstrategyr TOKENTENSORGROUP TENSOR_GROUP ATTN_HEADdynamicNotImplementedErroractorder ValueErrorrmixed_precisionvalue group_sizewarningswarn UserWarning)rinputsoutputsrrr.`/home/ubuntu/.local/lib/python3.10/site-packages/compressed_tensors/quantization/quant_scheme.pyvalidate_model_after8s\          z'QuantizationScheme.validate_model_afterforbid)extra)rrrr)__name__ __module__ __qualname____doc__rstr__annotations__rrrrrrr r0r model_configr.r.r.r/r%s   ;rnamerrcCsJ|}|tvrtd|dtttt|}tdd|i|S)a :param name: preset quantization settings name. must exist in upper case in PRESET_SCHEMES :param targets: list of quantization targets to be passed to the Scheme :return: new QuantizationScheme for a given name with the given targets zUnknown preset scheme name z, available names: rNr.)upperPRESET_SCHEMESKeyErrorlistkeysrr)r:r scheme_argsr.r.r/r|s  rcCs |tvS)zn :param name: preset quantization settings name :return: True if the name is a preset scheme name )r;r<)r:r.r.r/rs rTF)num_bitstyper symmetricr"r( scale_dtypezp_dtype)r static_minmax) rCrDrrEr"r(observerrFrG)rr )rCrDrr"rEr(rFrG)rCrDrrEr")rCrDrrEr"rI)rCrDrr(rEr")rCrDr(rrEr")rCrDrrEr"block_structure)rCrDrrEr"rIr() UNQUANTIZEDW8A16W4A16 W4A16_ASYMW8A8INT8W4A8FP8 FP8_DYNAMIC FP8_BLOCKNVFP4A16NVFP4MXFP4A16MXFP4)3r)copyrtypingrrtorchcompressed_tensors.configr*compressed_tensors.quantization.quant_argsrrrr r pydanticr r r __all__rr7rboolrdictrNFLOATr dtyperXLOCALrYruint8rZr[INTCHANNELr INT8_W8A8rOrPrQ INT8_W4A8rrUrVBLOCKrWr<r.r.r.r/s  R