o *iH @sLddlZddlmZmZmZmZmZmZddlm Z m Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZdd lmZmZmZgd Zd ZGd ddeZdedeedeeefdeeeeeefffddZdedefddZ dedeeegeefdefddZ!dedeeegeefdefddZ"dS)N)AnyCallableDictListOptionalTuple) ReferenceTyperef)forward_quantize) getattr_chain)InternalModule)Tensor)Module)RemovableHandle)CachePretrainedConfigPreTrainedModel)QuantizedKVCacheinitialize_hooked_kv_cacheregister_key_hookregister_value_hook KV_CACHE_ATTRkv_cachecspeZdZdZdedeffdd ZdeeeffddZ d ed edeeeffd d Z d e e fddZ ZS)rai QuantizedKVCache module which wraps the functionality of any existing kvcache args. Unlike transform Cache instances, this cache is a `torch.nn.Module` which can be hooked to trigger transforms and calibration hooks. This module works by being registered as a submodule to attention modules via `initialize_hooked_kv_cache`, then adding a hook which replaces `past_key_values` kwargs with this module. This module adopts the functionality of the replaced cache, preserving caching functionality such as sliding window attention, ect. :param attn_module: parent attention module config attn_modulecs$t||_t||_d|_dSN)super__init__rr rpast_key_values)selfrr __class__`/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/compressed_tensors/modeling/kvcache.pyr6s   zQuantizedKVCache.__init__returncOs||i|Srr")rargskwargsr"r"r#update<szQuantizedKVCache.update key_states value_statesc Os|}d}t||d}t|dd}|dur&|r&t||d|}t||d|}|jdur;|j||g|Ri|} n||f} d|_| S)Nz%quantization_scheme.input_activationsquantization_enabledTkv)rr getattrr rr') rr(r)r%r&modulequant_args_attr quant_args quant_enabledretr"r"r#forward?s$    zQuantizedKVCache.forwardrcCs |dur t||_dSd|_dSr)r r)rrr"r"r#add_past_key_valuesZs z$QuantizedKVCache.add_past_key_values)__name__ __module__ __qualname____doc__rrrrr r'r3rrr4 __classcell__r"r"r r#r(s   rr.r%r&r$cCsJdt|jjvr dnd}||d}t|t}|||||<||fS)aZ Hook which should be called before each quantized attention forward pass. This hook dynamically replaces the `past_key_values` kwarg to the attention forward function. The original kvcache object is assigned to QuantizedKVCache().past_key_values as a weakref to maintain original cache functionality and compute savings rpast_key_valueN)inspect signaturer3 parametersgetr-rr4)r.r%r& _past_kv_namercacher"r"r#_kv_cache_attention_hookds    rAmodelcCs4t|ts|tt|j||jtdddSdS)z Initialize a `QuantizedKVCache` instance attached to attention :param model: parent model of attention module :param module: attention module to initialize with T with_kwargsN)hasattrrregister_modulerrregister_forward_pre_hookrA)rBr.r"r"r#r}s rhookc,tt}dtffdd }|j|ddS)z Register a hook which takes post-rope key states as an argument and returns the modified key states or `None` :param module: attention module to add hook to :param hook: key hook function r@cFt|jj|i|}|jd}|dur||jd<|j|jfS)Nr(r;r<r3bind argumentsr%r&r@r%r&boundvaluerHr.r"r#_hook   z register_key_hook.._hookTrCr-rrrGr.rHrrRr"rQr#r rcrI)z Register a hook which takes value states as an argument and returns the modified value states or `None` :param module: attention module to add hook to :param hook: value hook function r@crJ)Nr)rKrNrQr"r#rRrSz"register_value_hook.._hookTrCrTrUr"rQr#rrVr)#r;typingrrrrrrweakrefrr 1compressed_tensors.quantization.lifecycle.forwardr compressed_tensors.utilsr !compressed_tensors.utils.internalr torchr torch.nnrtorch.utils.hooksr transformersrrr__all__rrstrrArrrr"r"r"r#sJ        <