o -i@sddlZddlZddlmZddlmZddlmZmZddl m Z m Z ddd Z d d Z Gd d d ejjZ     dde eejfdedededede ejde efddZdS)N) _topk_forward)_topk_backward)Tensor Bitmatrix)OptionalUnionTcCst|ts&|dur|jdn||jdg}|jd|jdg}t|||d}dd} d} d} d} t|jdks9J|jd d ksBJ|dksHJ|sLJ|j\}} |j\}}|j}tj||f|j|d }|durkd }n tj||ftj |d }d }| | | | }|d}tj|| |ddftj |d }t |ddd|}| | | }|| }tj|ftj |d }t | || |}t|f||d|||d|||d|d|| || || | |||d||dg}|dg}t||||d}|||fS)Nrr)shape shape_maxcSs||d|S)Nr)abr r a/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/third_party/triton_kernels/topk.pysztopk_forward.. i)dtypedeviceTF)BLOCK_MBLOCK_N APPLY_SOFTMAX N_EXPTS_PAD N_EXPTS_ACT)r r scratchpad) isinstancerr lenr rtorchemptyrint16uint32 transposeint32maxrstrider)xk apply_softmaxdimreturn_bitmatrixy_indxn_rowsx_shape x_shape_maxcdivrrBLOCK_Sn_cols n_rows_max_devy_valsuse_provided_indx n_cols_pad n_cols_words bitmatrixs_blockss_colsrpidsbitmatrix_shapebitmatrix_shape_maxr r r topk_forward sR          r?cCs|jd|ks Jt|jd}t|}t|jdf||d||d||d||d|jd||jd|||d|S)Nrr)rrr)r tritonnext_power_of_2r empty_likerr%)r&r+dy_valsr'r,r( n_expts_paddxr r r topk_backward9s rFc@s$eZdZeddZeddZdS)TopKc CsBt|||||||\}}} |||||_||_||_||| fSN)r?save_for_backwardr(r'r,) ctxr&r'r(r)r*r+r,r5r9r r rforwardGs   z TopK.forwardcCs4|j\}}t||||j|j|j}|ddddddfSrH) saved_tensorsrFr'r,r()rJrC_0_1r&r+rEr r rbackwardPs z TopK.backwardN)__name__ __module__ __qualname__ staticmethodrKrOr r r rrGEs  rGr&r'r(r)r*r+r,c Cst|||||||}|S)a Computes the top-k values and indices along a specified dimension of a tensor. Note that the input can be either a `Tensor` or a `torch.Tensor`, but the output will always be a `torch.Tensor`. Parameters ---------- x : Union[triton_kernels.Tensor, torch.Tensor] Input tensor of shape (n_tokens, n_expts). k : int Number of top elements to retrieve. apply_softmax : bool, default True Whether to apply softmax to the input tensor before computing top-k. dim : int, default 1 Dimension along which to compute top-k. return_bitmatrix : bool, default True A bitmatrix of shape (n_tokens, cdiv(n_expts, 32)). Each bit on [t, b] indicates whether the b-th expert was selected for the t-th token. y_indx : torch.Tensor, optional Pre-allocated tensor for storing indices of top-k elements with shape (n_tokens, k). If provided, we skip the computation of top-k indices and use this tensor instead. n_rows : int, optional Number of rows to apply top-k on. If None, we consider all rows in `x`. Returns ------- (expt_scal, expt_indx, bitmatrix) : Tuple[torch.Tensor, torch.Tensor, Bitmatrix] )rGapply)r&r'r(r)r*r+r,retr r rtopkWs$rV)TrTNN)rr@)triton_kernels.topk_details._topk_forwardr*triton_kernels.topk_details._topk_backwardrtriton_kernels.tensorrrtypingrrr?rFautogradFunctionrGintboolrVr r r rs:   0