o .i1 @sddlmZmZddlmZddlmZddlZddlm Z ddl m Z ddl m Z mZddlmZer@dd lmZdd lmZe eZGd d d eZdS) )ABCabstractmethod)Iterator) TYPE_CHECKINGN) init_logger)AttentionBackend) LoadStoreSpecOffloadingManager)OffloadingHandler) VllmConfig) KVCacheConfigc @szeZdZdZ    dddZedefd d Zed ee e j fd ee e e fdeee ee eeffd dZdS)OffloadingSpecz Spec for an offloading connector vllm_configr kv_cache_configKVCacheConfig | NonecCshtd||_||_|j}|dusJ|j|_|jj|_ t |j d|j |_ |j |j dks2JdS)NzsInitializing OffloadingSpec. This API is experimental and subject to change in the future as we iterate the design. block_sizer) loggerwarningrrkv_transfer_configkv_connector_extra_config extra_config cache_configrgpu_block_sizeintgetoffloaded_block_size)selfrrrrT/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/v1/kv_offload/spec.py__init__s  zOffloadingSpec.__init__returncCdS)z Get an OffloadingManager that will be used by the scheduler-side offloading connector to track offloaded blocks and manage evictions. Nr)rrrr get_manager-szOffloadingSpec.get_manager kv_caches attn_backendscCr!)aM Get offloading handlers along with their respective src and dst types. Args: kv_caches: A dictionary of layer_name -> gpu_kv_cache tensor. attn_backends: A dictionary of layer_name -> AttentionBackend. Yields: Tuples of (src_type, dst_type, offloading_handler). Nr)rr#r$rrr get_handlers6szOffloadingSpec.get_handlersN)rr rr)__name__ __module__ __qualname____doc__rrr r"dictstrtorchTensortyperrtuplerr r%rrrrr s"  r )abcrrcollections.abcrtypingrr, vllm.loggerrvllm.v1.attention.backendrvllm.v1.kv_offload.abstractrr vllm.v1.kv_offload.worker.workerr vllm.configr vllm.v1.kv_cache_interfacer r&rr rrrrs