o iq @sddlZddlmZmZmZmZmZmZmZmZm Z ddl Z Gdddej Z Gdddej Z Gdddej ZedeejZGd d d ej Zd efd efd edeefge_edeeddZGdddZddZdedededede jf ddZdS)N) CFUNCTYPEPOINTERc_intc_int64c_size_tc_uint8c_uint16c_void_ppointerc@s"eZdZdefdefdefgZdS) DLDataTypecodebitslanesN)__name__ __module__ __qualname__rr_fields_rrR/home/ubuntu/vllm_env/lib/python3.10/site-packages/flashinfer/comm/dlpack_utils.pyr s r c@seZdZdefdefgZdS)DLDevice device_type device_idN)rrrrrrrrrr(src@sBeZdZdefdefdefdefdeefdeefde fgZ dS) DLTensordatadevicendimdtypeshapestrides byte_offsetN) rrrr rrr rrrrrrrrr/s rc@s eZdZdS)DLManagedTensorN)rrrrrrrr Esr dl_tensor manager_ctxdeletercCsdS)Nr)dmt_ptrrrr no_op_deleterQsr%c@seZdZdZddZdS)CapsuleWrapperaE A wrapper class that holds references to the PyCapsule and its associated data. This class prevents Python's garbage collector from collecting the shape_array and managed_tensor objects while the capsule is still in use. It serves as a container to maintain the lifecycle of all DLPack-related objects. cCs||_||_||_dS)aF Initialize the CapsuleWrapper with the necessary objects. Parameters: capsule: The PyCapsule object that follows the DLPack protocol shape_array: The array containing tensor shape information managed_tensor: The DLManagedTensor instance that the capsule points to N)capsule _shape_array_managed_tensor)selfr' shape_arraymanaged_tensorrrr__init__as  zCapsuleWrapper.__init__N)rrr__doc__r-rrrrr&Xs r&cCsd}d}|tjtjtjtjtjtjfvrt|j}d}n.|tj tj tj tj fvr2t |j}d}n|tjtjtjtjfvrGt |j}d}nt||d}td} | |||} | ||d} td|d} t||dd} t}t||_| |_d|_| |_t| tt|_t| tt|_ d|_!t"}||_#d|_$t%|_&tj'j(}t|_)ttj*tg|_+t,|}||dd}t|tj-j.}t/|| |}|S) a Parameters: ptr: GPU memory address obtained from cudaMalloc (Python int) segment_size: Memory size of each segments in bytes segment_stride: Memory stride size between segments in bytes num_segments: Number of segments torch_dtype: torch dtype dev_id: device id. Returns: A PyCapsule object compliant with DLPack specification, which can be directly converted to a tensor using torch.utils.dlpack.from_dlpack r)rr)r r rNsdltensor)0torch float8_e5m2 float8_e4m3fnbfloat16float16float32float64finfor int8int16int32int64iinfouint8uint16uint32uint64NotImplementedErrorrrr rr rrrrctypescastrrrrr r!r"r%r# pythonapi PyCapsule_Newrestypec_char_pargtypesr py_objectvaluer&)ptr segment_sizesegment_stride num_segments torch_dtypedev_idbits_per_elementsdldata_type_codebytes_per_elementShapeArrayTyper+ stride_arrayrrdltensorr,rGmanaged_tensor_ptr capsule_ptrr'capsule_wrapperrrrcreate_dlpack_capsulessZ       r\rMrNrOrPrcCs,t||||||}tjj|j}||_|S)ax Pack GPU memory into a PyTorch tensor with specified stride. Parameters: ptr: GPU memory address obtained from cudaMalloc segment_size: Memory size of each segment in bytes segment_stride: Memory stride size between segments in bytes num_segments: Number of segments dtype: PyTorch data type for the resulting tensor dev_id: CUDA device ID Returns: PyTorch tensor that references the provided memory Note: This function creates a new DLPack capsule each time it's called, even with the same pointer. Each capsule is consumed only once. )r\r2utilsdlpack from_dlpackr'_capsule_wrapper)rMrNrOrPrrRr[ torch_tensorrrrpack_strided_memorys  rb)rDrrrrrrrr r r2 Structurer rrDLManagedTensorDeleterr rr%r&r\intrrbrrrrs:,    L