o Ti@sddlZddlmZmZmZmZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZmZdd lmZGd d d ZdS) N)AnyDictOptionalTuple)get_accelerator)RaggedUtilsBuilder)logger)BlockedAllocator)BlockedKVCache)DSStateManagerConfig KVCacheConfig)DSSequenceDescriptorc @seZdZUdZeed< eeed< eed< e e e fed< e ed<ee jdfed<ee jdfed < d,d ed eedfd eedd fddZd-de de de jfddZde dd fddZde dee fddZde de fddZde de fddZede e e ffdd Zede fd!d"Zede fd#d$Zede fd%d&Zede jfd'd(Zd-d)e de de jfd*d+Zd S).DSStateManagerzu Base abstract class for managing blocked KV caches. Will probably have a single implementation for now. _config _kv_configs _kv_cache_seqs_tracking_allocator._all_block_ids_all_block_ids_shadowNconfig kv_configs base_mp_groupreturncCs||_||_t|_t|jj|_g}g}|jD]%}|jj|j|j f}| t j |t j td| |j|dqt||_t||_i|_t|j|jj||jjd|_dS)z| The key Parameters: block_size (int): The number of tokens to allocate in each block. )dtypedevice)mp_groupoffloadN)rrrload _ragged_utilsr max_tracked_sequencesrnum_allocation_groupsmax_blocks_per_allocation_groupappendtorchzerosint32rcurrent_deviceallocate_fast_host_buffertuplerrrr memory_configrr)selfrrr all_block_idsall_block_ids_shadow cache_config ids_shaper2`/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/ragged/ragged_manager.py__init__7s*      zDSStateManager.__init__rcache_id cache_groupcC|jj||dS)z Return the Tensor associated with the given cache id in the specified cache group. Arguments: cache_group (str): The KV cache group. cache_id (int): The cache id within that group. r6)r get_cache)r-r5r6r2r2r3r9dszDSStateManager.get_cacheuidcCsl||jvrtd|ddS|j|}t|jD]}|jj|j|d|dq|j|j |j|=dS)zK Free all resources associated with the given sequence id. zAttempting to flush sequence z which does not exist.Nr8) rrwarningrangen_kv_cache_groupsrfreer.r tracking_id)r-r:seqir2r2r3flush_sequencens   zDSStateManager.flush_sequencecCs|j|dS)z Get the sequence descriptor for the given sequence id. If the sequence does not exist, then None is returned. N)rget)r-r:r2r2r3 get_sequence}szDSStateManager.get_sequencecCs ||}|dur |S||S)aA Get the existing sequence descriptor for a given uid or initialize one if it does not exist. NOTE: This will always return a valid sequence descriptor if one may be allocated and should not be used from APIs that are attempting to test the schedulability of a hypothetical batch. N)rD_create_sequence)r-r:r@r2r2r3get_or_create_sequences  z%DSStateManager.get_or_create_sequencecs||jvr td|dz |jdWnty&td|dwtfdd|jD}tfdd|jD}t |||j j d |j|<t d |d d |j|S) zM Create a new sequence descriptor for the given sequence id. z Sequence z already exists.r z,Unable to create tracking slot for sequence z% since the metadata buffers are full.c3|]}|VqdSNr2).0r. tracking_slotr2r3 sz2DSStateManager._create_sequence..c3rGrHr2)rIr/rJr2r3rLs) max_contextzCreated sequence z with tracking slot .)r ValueErrorrallocateitem RuntimeErrorr+rrrrrMrdebug)r-r: seq_block_idsseq_block_ids_shadowr2rJr3rEs(      zDSStateManager._create_sequencecCs|jS)z/ Return the tracked sequences. )rr-r2r2r3tracked_sequencessz DSStateManager.tracked_sequencescCs t|jS)zC Return the number of sequences currently tracked. )lenrrVr2r2r3n_tracked_sequencess z"DSStateManager.n_tracked_sequencescC|jjS)z8 Return the block size of the KV cache. ) _kv_config block_sizerVr2r2r3 kv_block_sizezDSStateManager.kv_block_sizecCrZ)z1 Return the number of KV caches. )r num_cachesrVr2r2r3r=r^z DSStateManager.n_kv_cache_groupscCrZ)zC Return the number of free blocks in the KV cache. )r free_blocksrVr2r2r3r`r^zDSStateManager.free_blocksn_blockscCr7)Nr8)rreserve)r-rar6r2r2r3allocate_blocksszDSStateManager.allocate_blocksrH)r)__name__ __module__ __qualname____doc__r __annotations__rr r rintrr r&Tensorrrr4r9rBrDrFrEpropertyrWrYr]r=r`rcr2r2r2r3rsN    -  r)r&typingrrrrdeepspeed.acceleratorrdeepspeed.ops.op_builderrdeepspeed.utils.loggingrblocked_allocatorr kv_cacher manager_configsr r sequence_descriptorrrr2r2r2r3s