o ˜à·i1 ã@sœddlmZmZddlmZddlmZddlZddlm Z ddl m Z ddl m Z mZddlmZer@dd lmZdd lmZe eƒZGd d „d eƒZdS) é)ÚABCÚabstractmethod)ÚIterator)Ú TYPE_CHECKINGN)Ú init_logger)ÚAttentionBackend)Ú LoadStoreSpecÚOffloadingManager)ÚOffloadingHandler)Ú VllmConfig)Ú KVCacheConfigc @szeZdZdZ    ddd„Zedefd d „ƒZed ee e j fd ee e e fdeee ee eeffd d„ƒZdS)ÚOffloadingSpecz Spec for an offloading connectorÚ vllm_configr Úkv_cache_configúKVCacheConfig | NonecCsht d¡||_||_|j}|dusJ‚|j|_|jj|_ t |j  d|j ¡ƒ|_ |j |j dks2J‚dS)NzsInitializing OffloadingSpec. This API is experimental and subject to change in the future as we iterate the design.Ú block_sizer) ÚloggerÚwarningrrÚkv_transfer_configÚkv_connector_extra_configÚ extra_configÚ cache_configrÚgpu_block_sizeÚintÚgetÚoffloaded_block_size)Úselfrrr©rúM/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/v1/kv_offload/spec.pyÚ__init__sÿ  ÿzOffloadingSpec.__init__ÚreturncCódS)z§ Get an OffloadingManager that will be used by the scheduler-side offloading connector to track offloaded blocks and manage evictions. Nr)rrrrÚ get_manager-szOffloadingSpec.get_managerÚ kv_cachesÚ attn_backendscCr!)aM Get offloading handlers along with their respective src and dst types. Args: kv_caches: A dictionary of layer_name -> gpu_kv_cache tensor. attn_backends: A dictionary of layer_name -> AttentionBackend. Yields: Tuples of (src_type, dst_type, offloading_handler). Nr)rr#r$rrrÚ get_handlers6szOffloadingSpec.get_handlersN)rr rr)Ú__name__Ú __module__Ú __qualname__Ú__doc__rrr r"ÚdictÚstrÚtorchÚTensorÚtyperrÚtuplerr r%rrrrr s"ÿ ÿ þýür )ÚabcrrÚcollections.abcrÚtypingrr,Ú vllm.loggerrÚvllm.v1.attention.backendrÚvllm.v1.kv_offload.abstractrr Ú vllm.v1.kv_offload.worker.workerr Ú vllm.configr Úvllm.v1.kv_cache_interfacer r&rr rrrrÚs