o پi$<@sddlZddlmZddlZddlmmZddlm Z m Z Gdddej j Z e jZGdddej j ZejZddd Zd d ZdddZ     dddZ     dddZ    d ddZ               d!deejfddZdS)"N)Optional) rearrangerepeatc@$eZdZeddZeddZdS)IndexFirstAxisc Csh|||jdks J|jd|jdd|_}|}tt|ddt|d|dj dg|RS)Nrb ... -> b (...)z -> z dd) save_for_backwardndimshapefirst_axis_dimnumeltorchgatherrrreshape)ctxinputindices other_shape second_dimrR/home/ubuntu/.local/lib/python3.10/site-packages/flash_attn_origin/cute/testing.pyforward s  zIndexFirstAxis.forwardcCs|j\}|jdks J|jdd}t|d}tj|j|jdg|j|jd}| dt |d|jdd||j |jg|RdfS)Nrrr devicedtyperr r ) saved_tensorsrrrrzerosrrr scatter_rr)r grad_outputrr grad_inputrrrbackwards zIndexFirstAxis.backwardN__name__ __module__ __qualname__ staticmethodrr&rrrrr s   rc@r)IndexPutFirstAxiscCsZ|||jdks J|jdksJtj|g|jddR|j|jd}|||<|S)Nrrr)rrrr"rrr )rvaluesrroutputrrrr)s  zIndexPutFirstAxis.forwardcCs|j\}||}|ddfSN)r!)rr$r grad_valuesrrrr&4s zIndexPutFirstAxis.backwardNr'rrrrr,(s   r,c Cs|dur||n|}|jdtjd}|jdtjd}tj|dd}|}ttj |dtjdd}t t |d|||||fS)Nr )dimr F)as_tupler)rrb s ... -> (b s) ...) sumrint32nonzeroflattenmaxitemFpadcumsumindex_first_axisr) hidden_statesattention_mask unused_mask all_masksseqlens_in_batchused_seqlens_in_batchrmax_seqlen_in_batch cu_seqlensrrr unpad_input>s rFcCst||||}t|d|dS)Nz(b s) ... -> b s ...b)index_put_first_axisr)r>rbatchseqlenr.rrr pad_inputNsrLrandomFcCs|dvsJ|dkrtj|df||tjd}n1|dkr2tjt|r"dnd|d|d|df|d}ntjt|r9dnd|d |d|df|d}|r_t|D] }|d dkrZd||<qNd|d <ttj||dd |d |k}|S)N)fullrMthirdrNrrrMr)rr zs -> b srG)rrNr5randintr8rangerarange) max_seqlen batch_sizermode zero_lengthslengthsi padding_maskrrrgenerate_random_padding_maskSs2   r]c "sN|r|rJ|j\} } |jd} |j\} }} |j|| fks$J|j|| fks/J|dus7| dur?|r;J|r?J|durdt|||\}}}}fdd}|durat|dnd}n*t|d}tjddtj|jd}d}}fd d}|durt|dnd}|durt||| \}}}}t||| ^}} nt|d}t|d}tjddtj|jd}d}}|r ||ksJ| |ksJtj|||gdd }tj|||gd d }|durfd d}nfd d}| ||| ||fS|rTtj||gdd }tj||gd d }|}|dur1fdd} nfdd} | | ||||| | ||| f S|}|durdfdd}!nfdd}!| | | |dur| nd||||||| | | |dur| nd|||!fS)Nr ct|Sr/rL output_unpadrW indices_qseqlen_qrrszgenerate_qkv..r3zb s h d -> (b s) h drr)stepr rct|ddSNz(b s) h d -> b s h drGrr`rWrrrer1rcr^r/r_ dqkv_unpadrbrrrecrgNz(b s) t h d -> b s t h drGrirmrjrrrerkcr^r/r_ dkv_unpadrW indices_kseqlen_krrrerocrgrprirqrjrrrerkcr^r/r_dk_unpadrsrrrerocrgrhrirvrjrrrero) rrFrrrUr5rallstackdetachrequires_grad_)"qkvquery_padding_maskkey_padding_maskqvkvpacked qkvpackedquery_unused_maskkey_unused_masknheadsr d_v_nheads_kq_unpad cu_seqlens_q max_seqlen_q seqused_q output_pad_fnqv_unpadk_unpad cu_seqlens_k max_seqlen_k seqused_kv_unpad qkv_unpadqkv dqkv_pad_fnkv_unpadkv dq_pad_fn dkv_pad_fn dk_pad_fnr)rWrtrcrurdr generate_qkvqs                        rNNc Cs<ttj||tjdd}tj||tjd} |dur3t|d}t| d|jdd} t| |k| |d} |dur9|nt|dd} |durG|nt|dd} |ddura| || | |d kS|durkt| |n| } |d durx| | k} n| t || | |d | k} t | t | || | |dk| |kS) Nrs -> s 1 b -> b 1 1 1 s -> b 1 1 srrGr r) rrrUlongrrwherer4 full_likeminimum logical_or logical_and) rdru window_sizesink_token_lengthrr key_leftpadrrow_idxcol_idxsksqlocal_mask_leftrrrconstruct_local_masks4      rc Csttj||tjdd}tj||tjd}|dur3t|d}t|d|jdd}t||k||d}|dur9|nt|dd} |durG|nt|dd} |durYt||n| } || | || | |} t || k|| |kS) NrrrrrrGrr ) rrrUrrrrr4rr) rdruattention_chunkrrrrrrrrcol_limit_left_chunkrrrconstruct_chunk_masks&  rTlearnable_sinkc( Cs| r|ddf}|j}|r%|||}}}| dur#| nd} | durRt| d|jd|jdd} || |j}| durP| | | jnd} | durd|t| dj|jd}| durv|t| dj|jd}|jd|jd}}t|d|jd|jdd}t|d|jd|jdd}|jd }|jd }d t| dur|n||}|st d |||}n t d |||}| dur|t d | ||}|dkrt |||}|dur| t|d td d}|ddus |ddurt ||||||||j d}|dkr4t|||||||j d}|dur2t||n|}|durA| |td |durJ||}|dur[tj|d d|j}n6|tj} tj| d dd}!t|d}t||!}"t| |"}#|#jd ddt||"}$|#|$|j}|dur|t|dd}|dur|t|d d}|dur|tj|d ddd}d d|}%|dur||d}&n|}&|dur|&||&j}&t d|&||%}'|dur|' t|dd|'j|d|j|dfS)Nrzb h -> b 1 (h g) 1r)gzb h -> b 1 h 1)r rzb s h d -> b s (h g) dr g?zbthd,bshd->bhtszb s -> b 1 1 sz-inf)rrrlT)r1keepdimz h -> h 1 1zb s -> b 1 s 1rzbhts,bshd->bthdzb s -> b s 1 1)r floatrrtormathsqrtreinsumtanh masked_fill_rrrrsoftmaxfloat32amaxmaximumexpr4 masked_fillrx)(r|r}r~rrr attn_bias dropout_p dropout_maskcausalr q_descale k_descale v_descalerrrrsoftcapupcast reorder_opsintermediate_dtypedtype_ogrdrur dv softmax_scalescores local_mask chunk_mask attention scores_fp32 logits_maxlogits_or_sinks_maxunnormalized_scores normalizerdropout_scalingattention_dropr.rrr attention_ref8s                   rr/)rMF)NNNFFNN)rrNNNN)NNNN)NNNNrNFNNNNrrrNrTFN)rtypingrrtorch.nn.functionalnn functionalr:einopsrrautogradFunctionrapplyr=r,rIrFrLr]rrrTensorrrrrrsn   "  . $