o پip @sJddlZddlZddlmZmZddlmZeeZ GdddeZ dS)N)QuantizationConfigQuantizeMethodBase) is_fp8_fnuzc@sReZdZdZdefddZdejjfddZ dejjdej fd d Z dd d Z d S)BaseKVCacheMethoda Quant method that adds `k_scale` and `v_scale` attributes to the Attention layer to support loading those scaling factors from checkpoints. The k/v_scale will be used to: - quantize k/v_cache entries before saving them to the cache - dequantize k/v_cache entries before fetching them from the cache :param quant_config: the appropriate QuantizationConfig quant_configcCs ||_dS)N)r)selfrr[/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/layers/quantization/kv_cache.py__init__s zBaseKVCacheMethod.__init__layercCs@tjjtjdtjddd|_tjjtjdtjddd|_dS)zS Create "weight" (aka k_scale and v_scale) for an attention layer. g)dtypeF) requires_gradN)torchnn Parametertensorfloat32k_scalev_scalerr rrr create_weightss  z BaseKVCacheMethod.create_weightsreturncCst|jjd)Nz.apply should not be called.) RuntimeError __class____name__rrrr apply,szBaseKVCacheMethod.applyNcCs|jdkr&|jdkr&|jd}|jd}tr%|d9}|d9}n6|jdkr5|jdkr5d}d}n'|jdkss