o TiM@s*ddlmZmZddlZGdddZdS))IterableUnionNc@seZdZUdZeed<ejed<eed<eed<deddfd d Zdedejfd d Z d e e eefddfddZ e defddZdS)BlockedAllocatora Allocator class for managing which blocks are free/used in the blocked KV-cache. This is a simple allocator that uses a linked list to keep track of which blocks are free/used. The cost of allocation/deallocation is O(blocks), where blocks is the number of blocks to allocate/deallocate. TODO(cmikeh2): Evaluate performance of this allocator and migrate to C++ if necessary. _num_blocks_blocks_head _free_blocks num_blocksreturnNcCsH|dkr td|||_tjd|dtjddd|_d|_||_dS)z Initialize an allocator with `num_blocks` blocks. This requires at least `num_blocks` * 4 bytes of host memory. Parameters: num_blocks (int): The number of blocks to allocate. z6Blocked KV-cache must have at least 1 block, provided cpuT)dtypedevice pin_memoryrN) ValueErrorrtorcharangeint32rrr)selfr rc/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/ragged/blocked_allocator.py__init__!s   zBlockedAllocator.__init__cCst||jkr td|dtj|tjd}t|D]}|j||<|j|j|_d|j||<|jd8_q|S)a Allocate a list of blocks from the associated KV-caches. This will return `num_blocks` blocks from the KV-cache if they are available, or raise an exception if there are not enough free blocks. Parameters: num_blocks (int): The number of blocks to allocate. Returns: List[int]: The list of blocks allocated. z3Not enough free blocks in the KV-cache to allocate z blocks)r r ) rrrzerosrrangerritem)rr allocated_blocksirrrallocate2s   zBlockedAllocator.allocateblockscCst|tr|g}|D]"}|dks||jkrtd|d|j|dkr,td|dq |D]}|j|j|<||_|jd7_q/dS) a Return a list of blocks to the free pool. If a single invalid block is provided (i.e., one that is out of range of the allocator or is already free), then an exception is raised and no blocks are freed. Parameters: blocks (Union[Iterable[int], int]): The list of blocks to free. If only one block is to be freed, this can be alone as an integer. rzInvalid block z provided to freerzBlock z is already freer N) isinstanceintrrrrr)rrblockrrrfreeJs  zBlockedAllocator.freecCs|jS)zC Return the number of free blocks in the KV-cache. )r)rrrr free_blocksdszBlockedAllocator.free_blocks)__name__ __module__ __qualname____doc__r!__annotations__rTensorrrrrr#propertyr$rrrrr s  r)typingrrrrrrrrs