o iD/%@sddlmZmZddlZddlmZddlmZdddd d dd dddd d d dejdejdejdejdejdede de de de de de de deejdeejdeejde d e f$d!d"Z dS)#)OptionalTupleN)+BlockDiagonalCausalWithOffsetPaddedKeysMask)_is_triton_availableg@g?Fg@g0@g@@T) theta linear_scaleuse_dynamic_scalingdynamic_old_context_lendynamic_scale_factordynamic_low_freq_factordynamic_high_freq_factorout_q first_seqposseqpos adjacentsinternal_dtypexqxkxvcache_kcache_v attn_biasrr r r r r rrrrrrc 4 Cstr|js|js|js|js|js| durtdts Jddl}ddlm}|jj d}|j j d}|j }|dvrAtd| }| }| }| }| }|j }|j }|d }|}|d dkrid}|} |dkry|d dkry|d } |d kr|j \}!}"}#}$|"|ksJ|d|||$fkrtd |d d|||$f|j d|||$fkrtd |j d d|||$f|d|| |$fkrtd|j d|| |$fkrtdd}%d|#|$|$df}&n|j \}!}"}%}#}$|"|ksJ|d||%||$fkrtd |d d||%||$f|j d||%||$fkrtd |j d d||%||$f|d||%| |$fkr7td|d d||%| |$f|j d||%| |$fkrRtd|j d d||%| |$fd|#|$|%|#|$|$df}&|!dkrhtd|ddkrstd|ddkr~td|ddkrtd|ddkrtd|ddkrtd|#d|}'|'|}(|#})| dur||j } n| j |j krtd| }&|&ddkrtd| dusJt|jj d}*|dur|durtdd}+|dur|j |*fkr t|j },td|,d|*d| d}+n!|dur3|j |fkr.t|j },td|,d|d| d}+d |}-t|-||$}.t|.d!}.t|.d"}.tt|.d#dd$}/|j}0|jj}1|j j}2|j j}3|1j|0ksv|2j|0ksv|3j|0krztd%|d&vsJtj|j||jj|*|'|%fg|||| |||1|2|3||||r| nd|r| nd|r| nd|r| nd|||)|(|%|$|d|d'kr|dnd|d |d|d'kr|dnd|d |d|d'kr|dnd|d |d|d'kr|dnd|d |d|d'kr-|dnd|d |1 d|2 d|3 d|&d|d'krO|&dnd|&d |+|Rd(dd|.||/d)Wd| S1spwY| S)*u Performs RoPE (rotary embeddings) and kv-cache emplacement for a heterogeneous batch for inference in the style given by BlockDiagonalCausalWithOffsetPaddedKeysMask. The batch is concatenated along the sequence dimension, so the actual dim-0 length of all tensors is 1. xq, xk and xv should be (1, slen, n_heads, dim), where xq's n_heads can differ from xk and xv. This function places the roped xk in the right place in cache_k, and xv (unmodified) in the right place in cache_v, and returns out_q (the roped xq) such that things are ready to call xformers.ops.memory_efficient_attention( out_q, cache_k, cache_v, attn_bias=attn_bias ) This functionality is experimental. Its API might be changed without warnings. Use it at your own risk. Arguments: xq: tensor of queries to apply rope to xk: tensor of keys to apply rope to xv: tensor of values to copy into cache_v cache_k: cache of keys, MODIFIED IN PLACE cache_v: cache of values, MODIFIED IN PLACE attn_bias: details the layout of caches. Used to determine frequencies for the RoPE calculation as well as the locations in cache_k and cache_v to write to. Must be on the device. first_seqpos: Optionally a tensor containing the sequence position of the beginning of the cache for each batch element. Providing a tensor of zeros is the same as providing None. This affects the numerical calculation but not which memory locations are read or written. seqpos: Optionally a 1D tensor containing the sequence position of each query. This should have length equal to xq.shape[1] . This affects the numerical calculation but not which memory locations are read or written. adjacents: If True, the inputs are in adjacent pairs along the final dim axis. This is like the released LLaMA model. If False, the dim axis is split in two equal pieces. I.e. the features are ordered with all the real parts before all the imaginary parts. This matches HuggingFace, e.g. https://github.com/huggingface/transformers/blob/ f143037789288ba532dada934a118e648e715738/ src/transformers/models/llama/modeling_llama.py#L126-L130 linear_scale: A scaling factor to apply to the sequence ids when computing the RoPE frequencies. When set to K, all sequence indices are divided by K. use_dynamic_scaling: If true, dynamic scaling in use, using a scaling like “YaRN: Efficient Context Window Extension of Large Language Models” dynamic_old_context_len: used with use_dynamic_scaling dynamic_scale_factor: used with use_dynamic_scaling dynamic_low_freq_factor: used with use_dynamic_scaling dynamic_high_freq_factor: used with use_dynamic_scaling internal_dtype: set to "f32" or "f64" to enforce dtype in the calculation NzGradients not supported.r)_rope_padded_kernel)zUnexpected xq dimensionrzunexpected k shape z : expected zunexpected v shape zunexpected cache_k shapezunexpected cache_v shapezunexpected cache_k shape zunexpected cache_v shape zHExpected batch size dimension to be 1 as batches should be concatenated.zEach q head must be contiguouszEach k head must be contiguouszEach v head must be contiguousz$Each cache_k head must be contiguousz$Each cache_v head must be contiguousrzUnexpected shape of out_qz"Each out_q head must be contiguousz0seqpos and first_seqpos may not both be providedzfirst_seqpos.shape z but (z ,) expected.z seqpos.shape iiz:`attn_bias` must be on the same device as the other inputs)rf32f64rF)const_batch_stridescache_padding_length seqlenk_shift BLOCK_SIZEr num_warps)torchis_grad_enabled requires_grad ValueErrorrtriton_triton.rope_padded_kernelsr q_seqinfo seqstart_py k_seqinfondimstrideshape new_emptylentuple element_sizeminnext_power_of_2maxdeviceseqstartseqlencuda max_seqlen)4rrrrrrrr r r r r rrrrrrr.rn_total_queries cache_lengthr3 xq_stride xk_stride xv_stridecache_k_stridecache_v_stride cache_k_shapexk_shape n_kv_headsexpected_kv_headsexpected_cache_headsbszq_len n_q_headsdimn_groups out_q_stride n_total_headsv_startk_start logical_bsz stride_seqposr5MAX_FUSED_SIZEr(r)r= seqstartq seqstartkseqlenkr]L/home/ubuntu/.local/lib/python3.10/site-packages/xformers/ops/rope_padded.py rope_paddedsP                                    !"#$%&'()*+,-./0 88r_) typingrrr*xformers.ops.fmha.attn_biasrrrTensorfloatboolstrr_r]r]r]r^sj