o *iw/ @sDddlZddlmZmZmZddlZddlmZmZm Z m Z ddl m Z m Z mZmZmZmZmZddlmZddlmZddlmZmZmZmZmZmZddlmZm Z gd Z!e"e#Z$  dd ed eed e%fddZ&d efddZ' dd ede(dedeee)dfdej*d e%f ddZ+d ed ed e%fddZ,d efddZ-dS)N)OptionalTupleUnion) IMPL_ATTR KV_CACHE_ATTRQuantizedAttentionImplQuantizedKVCache)ActivationOrdering DynamicTypeQuantizationArgsQuantizationMetadataQuantizationSchemeQuantizationStatusQuantizationStrategy)wrap_module_forward_quantized) strategy_cdiv)disable_hf_hookget_execution_device get_head_dimget_num_attn_headsget_num_kv_headsregister_offload_parameter)Module Parameter)"initialize_module_for_quantizationis_attention_moduleinitialize_qparamsinitialize_attn_qparamsTmoduleschemeforce_zero_pointcCsb|pt|dd}|durdSt|t|rt|||nt|tjjs/t dt |t |dr@|j }t|tjs?Jnt dt |dt |dS|jdurgt|d|j|jdd|j|d |jduryt|d|j|j|j|d |jdurt|d |j|jdd|j|d t| t||Wdn1swY||_tj|_dS) aa Attaches appropriate scales, zero points, and observers to a layer given its target quantization scheme. Previously initialized scales and zero points will be removed from module if they no longer apply to the scheme :param module: module to set for calibration :param scheme: scheme to use for quantization. if None is provided, will attempt to use scheme stored in the module under `quantization_scheme`, if not provided, the layer will be skipped :param force_zero_point: whether to force initialization of a zero point for symmetric quantization quantization_schemeNz&Attempting to quantize module of type weightz module type zR targeted for quantization but has no attribute weight, skipping quantization for inputobserved_shapeobserved_dtyper output)getattrr clear_all_qparamsrr isinstancetorchnnLinear_LOGGERwarningtypehasattrr"Tensorinput_activationsrshapedtypeweightsoutput_activationsrrr!r INITIALIZEDquantization_status)rrr r"r;q/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/compressed_tensors/quantization/lifecycle/initialize.pyr=sb          rcCs.d|jjvot|dpt|dpt|dS)N attentionk_projv_projqkv_proj) __class____name__lowerr2)rr;r;r<rs  r base_namequantization_argsr&r'cCs|j}|j}|j}t|} |durdS|tjkr.ttjdtj | ddd} t ||d| |t j kr5dS|tj kr=d} n|tjkrFtd |tjkr\t|d krUtd |d df} n|tjtjfvr|jduskJt|dkrutd |j} t|d| |} g|dd| R} |tjkrttj|dfd| tjddd}t ||d|nP|tjkr|jdusJt|d krtd|j}t|d |d |}t|d|d|}||f} n|tjkrt|dkrtd|dddf} nJd||}|tjtjtj tjfvr tj}ttj| || ddd}t ||d||s)|jsAttj| | |j ddd}t ||d|dSdS)a Initialize quantization parameters for a given basename according to the passed quantization args. The shape and dtype of the observed weight/activation must also be provided. Scales will always be initialized. Global scales are initialized depending on args. Zero points will be initialized if not symmetric or if `force_zero_point` is True. :param module: module to register qparams to :param base_name: base name of qparams, for example "input", "weight", "k", "v" :param quantization_args: arguments for quantization :param observed_shape: last (right-most) known dimensions of the observed weight/act :param observed_dtype: dtype of the observed weight/actt :param force_zero_point: force the zero_point parameter to be initialized TN)r6deviceF) requires_grad _global_scale)rFz(Cannot perform static token quantizationz5Channel quant requires at least 2 observed dimensionsz2Group quant requires at least 1 observed dimensionr$)rGr6_g_idxz3Block quant requires at least 2 observed dimensionsz7Attention quant requires at least 3 observed dimensionszUnknown strategy _scale _zero_point)!strategydynamicactorderrr TENSOR_GROUPrr,emptyfloat32rr LOCALTENSORTOKEN ValueErrorCHANNELlenGROUP group_sizerr fullintBLOCKblock_structure ATTN_HEADfloat16bfloat16float64 symmetriczeroszp_dtype)rrDrEr&r'r rQrRrSrGinit_global_scaleexpected_shaper^ num_groups init_g_idxrbnum_rowsnum_cols scale_dtype init_scaleinit_zero_pointr;r;r<rs              rc Cst|td}t|td}|dur|durtdtdtdt||j}t|}t|}t|}|d|f} |d|f} t | j } |durRt |d|j | | |d|durnt |d|j | | |dt |d|j | | |ddSdS) z(Initlaize k_scale, v_scale for self_attnNz0Attention module has quantization scheme but no z or zc attributes. Please ensure that these attributes are initialized using `apply_quantization_config`.qr%kv)r)rrrZ_validate_attention_schemeconfigrrrnext parametersr6rr4) rrr implkv_cacherwnum_attn_heads num_kv_headshead_dimq_observed_shapekv_observed_shaper'r;r;r<rsV      rcCs:|jdur td|jdurtd|jdurtddS)NzmCannot apply weight quantization to attention. Instead, target the (q|k|v)_proj submodule layers of attentionzHCannot apply attention quantization without specifying input activationsz-Cannot apply output quantization to attention)r7rZr4r8)rr;r;r<rvNs   rv)NT)T).loggingtypingrrrr,compressed_tensors.modelingrrrrcompressed_tensors.quantizationr r r r r rr1compressed_tensors.quantization.lifecycle.forwardr%compressed_tensors.quantization.utilsrcompressed_tensors.utilsrrrrrrtorch.nnrr__all__ getLoggerrBr/boolrrstrr`r6rrrvr;r;r;r<sV$    U { 9