o Ti1@shddlmZddlZddlmZddlmZddlmZddl m Z de d e fd d Z Gd d d Z dS))OptionalN)get_accelerator)RaggedUtilsBuilder)DSSequenceDescriptor)DSStateManagerConfig original_sizereturncCs0dtdtdtfdd}|dkrdnd}|||S) z0 Pad to a backend friendly granularity. val pow_2_valr cSs||d|d@S)Nr)r r r r `/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/ragged/ragged_wrapper.py_pad_to_mul_of_pow2sz&to_padded.._pad_to_mul_of_pow2i@)int)rr granularityr r r to_paddeds rc@seZdZUdZejed<ejed< ejed<ejed< ejed<ejed< ejed<ejed < ejed <ejed < d ed dfddZd/ddZ d0de dejd dfddZ e d ejfddZ d1deed dfddZd0ded ejfdd Zd0ded ejfd!d"Zd0ded ejfd#d$Zd0ded ejfd%d&Zd0ded ejfd'd(Zd0ded eejfd)d*Ze d efd+d,Ze d efd-d.ZdS)2RaggedBatchWrapperaY Container for all the auxiliary Tensors used in the management of a ragged batch. For each Tensor, we maintain a shadow Tensor on the host. This Tensor is what is directly populated when constructing the ragged batch. The shadow Tensors, when possible, should be allocated so as to support fast host-to-accelerator copies. _input_ids_shadow _input_ids_batch_metadata_storage_batch_metadata_storage_shadow_token_to_seq_storage_token_to_seq_storage_shadow_inflight_seq_descriptors _inflight_seq_descriptors_shadow_kv_ptrs_kv_ptrs_shadowconfigr NcCs||_tj|jjtjtd|_tjdtjtd|_ tj|jjtjtd|_ tj|jj dftjtd|_ tj|jj tjtd|_ t|_|jj}||j|_||j |_||j |_||j |_||j |_d|_d|_d|_g|_g|_g|_g|_dS)a' Convenience wrapper around the data structures used to represent a ragged batch for inference. Only a single `RaggedBatchWrapper` should be used per ragged inference engine. The underlying data structures are implemented in `ragged_batch_descriptor.h`. )dtypedeviceFrN)_configtorchzerosmax_ragged_batch_sizeint64rcurrent_devicerint32rrmax_ragged_sequence_countrrrload _utils_moduleallocate_fast_host_bufferrrrrr _is_padded_current_tokens_current_sequences _batch_tokens$_inflight_seq_descriptors_shadow_buf_kv_blocks_ptr_buf _token_to_seq_storage_shadow_buf)selfr host_allocr r r __init__Os@          zRaggedBatchWrapper.__init__cCs(d|_d|_g|_g|_g|_g|_dS)zb Clear the ragged batch. This will reset the number of tokens and sequences to 0. rN)r0r1r2r3r4r5r6r r r clear{s  zRaggedBatchWrapper.clearTseq_descriptortokenscCs|jtdkrtd|jd|r#|j|jjkr#td|jj|}|r<|j||jjkr._noblock_copy)rCrflattenlenr3rTr%tensorrcatr2rr5rr4rrArfill_r/Tensorrrrrr)r6rNrL padded_toksrArUr r r finalizes4 " " "zRaggedBatchWrapper.finalize on_devicecC|r |jd|jS|jS)z The input ids tensor for the ragged batch. If the device Tensor is requested, the Tensor is truncated to the number of tokens in the batch. N)rrMrr6r^r r r input_idszRaggedBatchWrapper.input_idscCs|r|jS|jS)z Buffer associated with the batch metadata tensor that can be populated in preparation for passing a new input to the device. )rrr`r r r batch_metadata_buffersz(RaggedBatchWrapper.batch_metadata_buffercCr_)z Mapping of token to which sequence it belongs to in the ragged batch. If the device Tensor is requested, the Tensor is truncated to the number of tokens in the batch. N)rrMrr`r r r tokens_to_seqrbz RaggedBatchWrapper.tokens_to_seqcCr_)z Buffer associated with the metadata of each sequence in the ragged batch. If the device Tensor is requested, the Tensor is truncated to the number of sequences in the batch. N)rrArr`r r r inflight_seq_descriptorsrbz+RaggedBatchWrapper.inflight_seq_descriptorscCr_)z Pointer to where the list of KV ids associated with a sequence are. If the device Tensor is requested, the Tensor is truncated to the number of sequences in the batch. N)rrArr`r r r kv_ptrsrbzRaggedBatchWrapper.kv_ptrscCsdS)z Placeholder for supporting complex masks. Currently not supported. Models that will need this will be BERT-like, not generative. Nr r`r r r masksszRaggedBatchWrapper.maskscC|jSrK)r0r9r r r rCz!RaggedBatchWrapper.current_tokenscCrh)z The number of sequences in the in-flight ragged batch. This will not trigger synchronization with the device. )r1r9r r r rAriz$RaggedBatchWrapper.current_sequences)r N)T)F)__name__ __module__ __qualname____doc__r%r[__annotations__rr8r:rrJpropertyrMrboolr]rarcrdrerfrgrrCrAr r r r rsB           , & $     r)typingrr%deepspeed.acceleratorrdeepspeed.ops.op_builderrsequence_descriptorrmanager_configsrrrrr r r r s