o Tio!@sddlZddlmZddlmZmZmZmZddlZddl m Z ddl m Z ddlmZddlmZddlmZd d lmZd d lmZmZmZd ejd eejejffddZGdddZdS)N)reduce)AnyIterableOptionalTuple)ReduceOp)get_accelerator) elem_size)inference_logger)BlockedAllocator)AllocationMode KVCacheConfig MemoryConfigkv_cachereturncCs`|jdkrtd|jd|dddddddddf|dddddddddffS)a Split a KV cache instance into its key and value components. Parameters: kv_cache (torch.Tensor): The KV-cache to split. This should be a 5D tensor with the following shape: [num_blocks, block_size, 2, num_heads, head_size] Returns: Tuple[torch.Tensor, torch.Tensor]: The key and value components of the KV-cache. Both tensors will have the shape [num_blocks, block_size, num_heads, head_size]. z%KV-cache must have 5 dimensions, got .Nrr )ndim ValueError)rrZ/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/ragged/kv_cache.pysplit_kvs Drc @s(eZdZUeejdfed< eedfed< eedfed<  d!deedfde d e e d e d df d d Z d"deded ejfddZd"deeded dfddZd"deeded ejfddZd"deeded ejfddZd"deded ejfddZed ejfddZed efdd ZdS)#BlockedKVCache._caches _allocators_configsNFconfigs memory_configmp_groupoffloadrc Cs||_||_||_|jrtdt|jjtjurd}|jD]}ttj |j |j }|d9}||t |j 7}qtj|ddkrTtjdtjtd}tj|tj|dtt|jj} t} td| d | d || |krtd |d | | |} tj|ddkrtj| tjtd} tj| tj|d| } ~ tn|jj} g} g}t |jD]B\}}|j d}|j d}|j d}|| |j d||f}t!d |d|d| d| "tj#||j td|"t$| qt%| |_&t%||_'dS)a Create a container that will maintain the storage and allocations for a set of blocked KV-caches. Parameters: config (KVCacheConfig): The configuration of the KV-cache. slack (int): The amount of slack space to reserve in GPU memory for the cache. enable_offload (bool): Whether to enable offloading of the cache to the host. blocks (int): The number of blocks to pre-allocate for the cache. If this is set, slack will be ignored. z-Offloading of KV-caches is not yet supported.rr )groupr )dtypedevice)opr"z6Memory usage before KV-cache allocation: total_memory=z, available_kv_memory=z, total_per_block_footprint=z5Insufficient memory to allocate KV-caches. Required: z , Available: zAllocating KV-cache z with shape: z consisting of z blocks.N)(r_memory_config_enable_offloadNotImplementedErrorrmodeRESERVEroperatormul cache_shape block_sizer cache_dtypedistget_world_sizetorchtensorint32rcurrent_device all_reducerMIN empty_cacheavailable_memorysize total_memoryr debugritem enumerateinfoappendemptyr tuplerr)selfrrr r!total_per_block_footprintconfigper_block_footprint dummy_tensoravailable_kv_memoryr; num_blocks reduce_tensorcaches allocatorscache_group_id num_caches num_heads head_size alloc_shaperrr__init__<sb        zBlockedKVCache.__init__rrI cache_groupcCs|j||S)a3 Reserve a number of blocks from the cache. This will return a 1D tensor of block_ids that have been marked as reserved. Parameters: num_blocks (int): The number of blocks to reserve. cache_group (int): The cache group to reserve from. Default is 0. )rallocate)rCrIrSrrrreserves zBlockedKVCache.reserveblockscCs|j||dS)a  Free a set of blocks from the cache. This will mark the blocks as free in the allocator. Parameters: blocks (Iterable[int]): The blocks to free. cache_group (int): The cache group to free from. Default is 0. N)rfreerCrVrSrrrrWs zBlockedKVCache.freecCtd)z Offload KV-cache blocks from accelerator memory to the host. Parameters: blocks (Iterable[int]): The blocks to offload. cache_group (int): The cache group to offload from. Default is 0. Offloading is not yet supported.r(rXrrrr!zBlockedKVCache.offloadcCrY)z Restore KV-cache blocks from the host to accelerator memory. Parameters: blocks (Iterable[int]): The blocks to restore. cache_group (int): The cache group to restore to. Default is 0. rZr[rXrrrrestorer\zBlockedKVCache.restorecache_idcCs|j||S)z Get the tensor associated with the given cache ID. Parameters: cache_id (int): The ID of the cache tensor to get. cache_group (int): The cache group to get from. Default is 0. )r)rCr^rSrrr get_cacheszBlockedKVCache.get_cachecCsdd|jDS)z@ Return the number of free blocks in each cache cSsg|]}|jqSr) free_blocks).0 allocatorrrr sz.BlockedKVCache.free_blocks..)rrCrrrr`szBlockedKVCache.free_blockscCs t|jS)z- Return the number of caches )lenrrdrrrrNs zBlockedKVCache.num_caches)NF)r)__name__ __module__ __qualname__rr2Tensor__annotations__r rrrrboolrRintrUrrWr!r]r_propertyr`rNrrrrr(s:    T     r)r+ functoolsrtypingrrrrr2deepspeed.commcommr0deepspeed.comm.reduce_oprdeepspeed.acceleratorrinference_utilsr loggingr blocked_allocatorr manager_configsrrrrirrrrrrs