o Ti*@sNddlmZmZmZddlZGdddZGdddeZGdddeZdS) )ListTupleUnionNc@sNeZdZedefddZed dedefddZed dedefdd Zd S) BaseSequenceDescriptorreturncCt)z\ The number of tokens for this sequence that have completed a forward pass. NotImplementedErrorselfr e/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/ragged/sequence_descriptor.py seen_tokens z"BaseSequenceDescriptor.seen_tokensr cache_groupcCr)zP The number of KV blocks currently allocated for this sequence. rr rr r r cur_allocated_blocksrz+BaseSequenceDescriptor.cur_allocated_blockscCr)zA The pointer to the KV blocks for this sequence. rrr r r kv_blocks_ptrrz$BaseSequenceDescriptor.kv_blocks_ptrNr)__name__ __module__ __qualname__propertyintrrrr r r r r src@s^eZdZdZddddZedefddZedd edefd d Zedd edefd d Z dS)PlaceholderSequenceDescriptorz The DummySequenceDescriptor is an empty object that allows us to perform schedulability checks before formally tracking a sequence. rrNcCs||_||_||_dSN) _seen_tokens_cur_allocated_blocks_kv_blocks_ptr)r rrrr r r __init__)s z&PlaceholderSequenceDescriptor.__init__cC|jSrrr r r r r.z)PlaceholderSequenceDescriptor.seen_tokensrcCr r)rrr r r r2r"z2PlaceholderSequenceDescriptor.cur_allocated_blockscCr r)rrr r r r6r"z+PlaceholderSequenceDescriptor.kv_blocks_ptr)rrrrNr) rrr__doc__rrrrrrr r r r r#s rc @seZdZUeed< eed< eed< eedfed< eejdfed< eejdfed<eejdfed< eed < d2d ed eejdfd eejdfdeddf ddZ e defddZ e defddZ e defddZ e defddZe d3dedefddZd4ded edejfd!d"Ze d3dedefd#d$Zd3dedejfd%d&Zd'eddfd(d)Zd5d*d+Zd3d,eeejejfdeddfd-d.Zd3d/eeejejfdeddfd0d1ZdS)6DSSequenceDescriptorr_in_flight_tokens _max_context._num_allocation_groups_blocks_per_allocation_group _kv_cache_ids_kv_cache_ids_shadow _tracking_id tracking_id kv_cache_idskv_cache_ids_shadow max_contextrNcCs||_||_||_||_t||_d|_d|_tdd|D|_ tdd|j D|_ t |D]\}}|j ||j dks@Jt|j dksIJq0dS)a Create the metadata to track a single sequence in the system. Arguments: tracking_id (int): The slot in the tracking buffers used to track this sequence. kv_cache_ids (Tuple[torch.Tensor, ...]): The KV-cache IDs for the sequence. The shape of the tensor should be [num_allocation_groups, max_blocks_per_allocation_group]. There should be one tensor per cache group. kv_cache_ids_shadow (Tuple[torch.Tensor, ...]): The shadow tensor for the KV-cache IDs. This tensor should be allocated on the host and should have the same shape as the tensor provided in ``kv_cache_ids``. There should be one tensor per cache group. max_context (int): The maximum number of tokens this sequence may eventually include. Currently unused but may be used in future implementations for speculative caching. rcss|]}|jdVqdS)rN)shape).0r0r r r ~sz0DSSequenceDescriptor.__init__..css"|] }tj|tjddVqdS)cpu)dtypedeviceN)torchzerosint32)r3 num_groupsr r r r4s N) r,r*r+r'len_n_cache_groupsrr&tupler(r) enumerater2)r r.r/r0r1rr r r rbs"   zDSSequenceDescriptor.__init__cCr )zV Number of tokens in the sequence that have completed a forward pass. r!r r r r rrz DSSequenceDescriptor.seen_tokenscCr )z[ Number of tokens that have begun a forward pass but not yet completed it. r&r r r r in_flight_tokensrz%DSSequenceDescriptor.in_flight_tokenscCr )zO Maximum number of tokens for this sequence. Currently unused. )r'r r r r r1rz DSSequenceDescriptor.max_contextcCr )zV Return the slot in the tracking buffers used to track this sequence. )r,r r r r r.rz DSSequenceDescriptor.tracking_idrrcCs.t|jdkr|jdS|j|S)z Returns the number of blocks currently allocated for this sequence in the specified cache group. Arguments: cache_group (int): The cache group to query. r)r=r)itemsumrr r r rs z)DSSequenceDescriptor.cur_allocated_blocksF on_devicecCs|r|j|S|j|S)a= Returns the Tensor containing the block IDs for this sequence on the appropriate device for the specified cache group. Arguments: cache_group (int): The cache group to query. on_device (bool): Whether or not to return the Tensor on the device or on the host. )r*r+)r rrFr r r r/s  z!DSSequenceDescriptor.kv_cache_idscCs|j|S)z Get the device pointer to the base of the KV-cache ids for the specified cache group and sequence. Arguments: cache_group (int): The cache group to query. )r*data_ptrrr r r rs z"DSSequenceDescriptor.kv_blocks_ptrcCs@g}t|j||j|D] \}}||d|q t|S)z Return the Tensor containing all block IDs for this sequence in the specified cache group. Arguments: cache_group (int): The cache group to query. N)zipr*r)appendr8cat)r r block_idsallocation_group num_blocksr r r all_block_idss    z"DSSequenceDescriptor.all_block_ids num_tokenscCs ||_dS)z Update the state of the sequence before a forward pass. Arguments: num_tokens (int): The number of tokens in the sequence that will be executed during the next forward pass of the model. NrA)r rOr r r pre_forwards z DSSequenceDescriptor.pre_forwardcCs|j|j7_d|_dS)a Update the state of the sequence after a forward pass. This should be called after the forward pass completes. NOTE: due to the asynchronous nature of the accelerator, this may be called before the forward pass completes on the device itself. rN)rr&r r r r post_forwards z!DSSequenceDescriptor.post_forwardnew_idsc Cst|tjr |g}t||j|kr!tdt|d|j|t|D]K\}}|}|dkr2q%|j||}|j ||}|j ||}|||| |||||j ||||dd|j |||7<q%dS)a Extend the KV-cache for the sequence. Arguments: new_ids (Union[List[torch.IntTensor], torch.IntTensor]): For each allocation group, the IDs to add to the KV-cache. If there is only one allocation group, a single tensor can be provided. Otherwise, a list of tensors should be provided. The tensors do not need to have the same shape. zOnly z& allocation groups provided, expected rT) non_blockingN) isinstancer8Tensorr=r( ValueErrorr@numelr+r*r)copy_) r rRrgroup_id new_group_ids new_blocksshadow_alloc_group alloc_group cur_blocksr r r extend_kv_caches* z$DSSequenceDescriptor.extend_kv_cachefree_idscCstd)a Free blocks from the KV-cache for the sequence. Arguments: free_ids (Union[List[torch.IntTensor], torch.IntTensor]): The ids of blocks to free from the KV-cache. If there is only one allocation group, a single tensor can be provided. Otherwise, a list of tensors should be provided. The tensors do not need to have the same shape. z.Partial KV-cache freeing is not yet supported.r)r r`rr r r free_kv_caches z"DSSequenceDescriptor.free_kv_cache)r-r)rFr#)rrrr__annotations__rr8 IntTensorrUrrrrBr1r.rboolr/rrNrPrQrrr_rar r r r r%;sZ    %   ( ,#r%)typingrrrr8rrr%r r r r s