o پiA@srUdZddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZddlmZeeZdd dd d ejd ejd ejdeededeejdeejejffddZdejdejfddZdejdejfddZ dSdejdejdejdededejf ddZedGd d!d!eZGd"d#d#eZGd$d%d%ejjZGd&d'd'ejjZ dTdee!ed)fBd*ede!ed)ffd+d,Z"d(dej#d-d.ee!ed)fBd/ee!ed)fBd*ed0ej$e%BdBd1ej&dejf d2d3Z'd4d5d5ej#dfd*ed6ej(eBd7e)d8e)d9e)d1ej&d0ej$e%BdBde!ejejffd:d;Z*d4d5d5ddded?ed@ed1ej&dAed0ej$e%BdBde!ejejffdBdCZ,d5d5dej#ddfd>ed1ej&dAed0ej$e%BdBde!ejejff dDdEZ-iZ.e/e!efe0dF<ea1dGe0dH<eZ2dIe0dJ< K   5dUdedLedMedNee)BdedOe/e%efdBd1ej&dBdPe)defdQdRZ3dS)VzRotary Positional Embeddings.N) OrderedDict)AnyOptionalTuple)apply_rotary_embedding) get_sp_group)CustomOp) init_loggerF) head_sizeis_neox positionsqk cos_sin_cacher r r returnc Cs$|dks |dkrtdt|jdt|j|j|jkr.td|jd|jt|tjr:|dks>td|j\}}}} |durK| }|| krYtd| d |zd d lm} Wnt yd dl } | j d dd |jdd} |dur|d|d| f |j } |d|| df |j }| d |dd||d} |d |dd||d}n!| |jd}||d| f |j } ||| df |j }||||| }||||| }t|| || d}t|| || d}||||| ||||| fYSw|dur0tj||jtjd}|dkr*|n||}n0t|tjrE|j tjkrE|dksItd|||kr`td||d|||||| }||||| }| |||| ||d||||| ||||| fS)Nz>Expected q/k to be 4D [bsz, seqlen, nheads, head_size], got q:z k:z&q and k must have the same shape, got z vs z'cos_sin_cache must be a 2D torch.Tensorzhead_size mismatch: inferred z, but head_size=r)%apply_rope_with_cos_sin_cache_inplacez8FlashInfer not available, using Triton fallback for RoPE) stacklevel) interleaveddevicedtypez(positions must be a 1D torch.long Tensorz$positions length must be bsz*seqlen=z, got )r querykeyr rr )dim ValueErrortupleshape isinstancetorchTensorflashinfer.roper ImportErrorwarningswarntor unsqueezeexpandreshaperviewrarangelongrepeatnumel contiguous)r rrr r r bszseqlennheadsdrr& half_sizecossinq_flatk_flatq_rotk_rotpos_1dr>i/home/ubuntu/.local/lib/python3.10/site-packages/sglang/multimodal_gen/runtime/layers/rotary_embedding.py apply_flashinfer_rope_qk_inplace+s   "$   r@xcCsH|dd|jddf}|d|jdddf}tj| |fddS)N.rrr)r r"catrAx1x2r>r>r? _rotate_neox}srGcCsB|ddddf}|ddddf}tj| |fdd}|dS)N.rrrrB)r"stackflattenrDr>r>r? _rotate_gptjs rKr7r8 is_neox_stylerc Cs|rR|d}|d}|rtj|ddd\}}n|ddddf}|ddddf}|||||}|||||}tj||fddSt||||S)a Args: x: [num_tokens, num_heads, head_size] or [num_tokens, head_size] cos: [num_tokens, head_size // 2] sin: [num_tokens, head_size // 2] is_neox_style: Whether to use the Neox-style or GPT-J-style rotary positional embeddings. rHrrrB.Nr)r)r"chunkfloattype_asrCr) rAr7r8rLrrErFo1o2r>r>r?_apply_rotary_embs  rRrotary_embeddingcseZdZdZdedededeeBdedejdd ffd d Z deeBdej fd d Z dej fddZ de fddZ ddej dej dej dej d Bdeej ej ff ddZdefddZZS)RotaryEmbeddingz%Original rotary positional embedding.r rotary_dimmax_position_embeddingsbaserLrrNcsXt||_||_||_||_||_||_|}| |}||j d|dddS)NrF) persistent) super__init__r rUrVrWrLr_compute_cos_sin_cacher(register_buffer)selfr rUrVrWrLrcache __class__r>r?rZs  zRotaryEmbedding.__init__cCs(d|tjd|jdtjd|j}|S)zCompute the inverse frequency.?rrr)r"r-rUrN)r]rWinv_freqr>r>r?_compute_inv_freqs z!RotaryEmbedding._compute_inv_freqcCsR||j}tj|jtjd}td||}|}|}tj ||fdd}|S)zCompute the cos and sin cache.rb i,j -> ijrrB) rdrWr"r-rVrNeinsumr7r8rCr]rctfreqsr7r8r^r>r>r?r[s z&RotaryEmbedding._compute_cos_sin_cachecOs|j|i|SN)forward_native)r]argskwargsr>r>r? forward_cudaszRotaryEmbedding.forward_cudar rroffsetscCs|dur||}|}|jd}|jd|}|jddd\}}|j} ||d|j}|dd|jf} |d|jdf} t| |||j } t j | | fdd | }|j} ||d|j}|dd|jf} |d|jdf}t| |||j } t j | |fdd | }||fS)z-A PyTorch-native implementation of forward().NrrrrB.) rJr r index_selectrMr,r rUrRrLr"rCr+)r]r rrro num_tokenscos_sinr7r8 query_shape query_rot query_pass key_shapekey_rotkey_passr>r>r?rks& zRotaryEmbedding.forward_nativecCs@d|jd|j}|d|j7}|d|jd|j7}|S)Nz head_size=z , rotary_dim=z, max_position_embeddings=z, base=z, is_neox_style=)r rUrVrWrL)r]sr>r>r? extra_reprszRotaryEmbedding.extra_reprrj)__name__ __module__ __qualname____doc__intrNboolr"rrZr#rdr[rrnrrkstrrz __classcell__r>r>r_r?rTsB  rTcsTeZdZdedededeeBdedejdedd ffd d Zdej fd d Z Z S)LinearScalingRotaryEmbeddingr rUrVrWrLrscaling_factorrNcs&t||_tj||||||ddS)N)r rUrVrWrLr)rNrrYrZ)r]r rUrVrWrLrrr_r>r?rZs  z%LinearScalingRotaryEmbedding.__init__cCs\||j}tj|jtjd}||j}td||}|}| }tj ||fdd}|S)NrbrerrB) rdrWr"r-rVrNrrfr7r8rCrgr>r>r?r[s  z3LinearScalingRotaryEmbedding._compute_cos_sin_cache) r{r|r}rrNrr"rrZr#r[rr>r>r_r?rs& rcseZdZdZdddejddfdedededed ejd e d e ffd d Z ddZ dej fddZ ejdddedededeej ej ffddZdej deej ej ffddZejdddededeej ej ffddZZS) OneDRotaryEmbeddingz,1D rotary positional embedding with caching.@raFrthetatheta_rescale_factorinterpolation_factorruse_realrepeat_interleave_realcsHt|ddks J||_||_||_||_||_||_||_dS)Nrr) rYrZrrrrrrr)r]rrrrrrrr_r>r?rZ(s  zOneDRotaryEmbedding.__init__c CsBd|jtjd|jd|j|dd|jd|jj|d}|S)Nrarrrrr)rr"r-rrr()r]rrir>r>r? build_freqs=s  zOneDRotaryEmbedding.build_freqsposcCs|j}|jdkr||j|j|jd9}||}t||j|}|}|}|j r?|j r?|j ddd}|j ddd}| | fS)NrarrrB) rrrrr"outerrr7r8rrrepeat_interleaverN)r]rrrri freqs_cos freqs_sinr>r>r?build_freqs_outerIs   z%OneDRotaryEmbedding.build_freqs_outermaxsizeseq_len start_pos device_strrcCs:t|}tj||||j|d}|||\}}||fS)Nr)r"rr-rr)r]rrrrrrrr>r>r?forward_from_grid\s z%OneDRotaryEmbedding.forward_from_gridcCs"t|}t|j}|||S)z Calculates 1D rotary embeddings for the given positions. This method converts the input tensor to a hashable representation and calls a cached helper method to perform the computation. )rtolistrr_forward_cached)r]r pos_tuplerr>r>r?forwardhs   zOneDRotaryEmbedding.forwardrcCs4t|}tj||j|d}|||\}}||fS)z} The core implementation that computes 1D rotary embeddings. This method is wrapped by an LRU cache. r)r"r as_tensorrr)r]rrrrrrr>r>r?rss z#OneDRotaryEmbedding._forward_cached)r{r|r}r~r"float32rrNrrrZrr#r functools lru_cacherrrrrrr>r>r_r?r%sX  rcsbeZdZdZddddejfdeededeeeBdeeeBde d e d ej ffd d Z d ej de ej ej ffddZejddde e edfdfdede ej ej ffddZdej de ej ej ffddZ   d%de edfdeded ejeBdBde ej ej ff d!d"Zejddde edfdededede ej ej ff d#d$ZZS)&NDRotaryEmbeddingz*N-dimensional rotary positional embedding.raF rope_dim_list rope_thetarrrrrc st||_t||_||_||_t|tt fr"|g|j|_ nt|t r7t|dkr7|dg|j|_ n||_ t|j |jksFJdt|tt frU|g|j|_ nt|t rjt|dkrj|dg|j|_ n||_ t|j |jksyJdt j|_i}g|_t|jD]@} |j| } |j | } |j | } | | | ||f} | |vrt| |j| | |j||d}t|j|| <|j||| }|j|qdS)Nrrr?rZsX       zNDRotaryEmbedding.__init__r rcCs(ttt|}t|j}|||S)a, Calculates n-d rotary embeddings for given absolute positions. Args: positions (torch.Tensor): A tensor of shape `[num_tokens, ndim]` containing the integer coordinates for each token. Returns: A tuple of (cos, sin) tensors. )rmaprrrr)r]r rrr>r>r?rs  zNDRotaryEmbedding.forwardrrr.rcCs(t|}tj|tj|d}|j|dS) The core implementation that computes embeddings from a position tensor. This method is wrapped by an LRU cache. r)r)r"rtensorr.forward_uncached)r]rrrr r>r>r?rs  z!NDRotaryEmbedding._forward_cachedrcCs |j}|jd}|jd}|jr|jrt|j}nt|jd}tj||f||j d}tj||f||j d}d}t |j D]?} |dd| f |j } |j | } |j| } | | \} }| jd}| |dd|||f<||dd|||f<||7}q=||fS)rrrrNr)rr rrrsumrr"emptyrrrr(rrN)r]rrrqfirst_generatorhead_dimr7r8 col_offsetrpos_irrcos_1dsin_1d slice_widthr>r>r?rs&         z"NDRotaryEmbedding.forward_uncachedrN grid_size shard_dim start_framercCs$|durt|nd}|||||S)z' Handles sp internally Ncpu)r_forward_cached_from_grid)r]rrrrrr>r>r?rs z#NDRotaryEmbedding.forward_from_gridc"CsZt|}t}|j}|j}t||jd} d|j} t| } dg|j} |dkrO| ||dks?Jd|d| |d|| ||} || | |<| | |<d}| D]}|t|9}qSt |j d}tj ||f||j d }tj ||f||j d }d}t |jD]}|j |}|d}t| |}| |}|dkr|dkr||7}|dkr||kr|| |7}|j|}|j|}||||\}}d}t |d|jD] }|t| |9}qd}t d|D] }|t| |9}q|j|dd} |j|dd}!|dkr| |d} |!|d}!| |d d |||f<|!|d d |||f<||7}q||fS) z Computes embeddings for a structured grid, using a highly efficient implementation that avoids materializing the full position tensor. This method is wrapped by an LRU cache. rBrrr Dimension with size 2 is not divisible by sequence parallel world size rrN)r"rr rank_in_group world_size _to_tuplerrrrrrrrrrrrr/rN)"r]rrrrrsp_groupsp_rank sp_world_sizesizesstarts shard_sizes shard_offsets shard_sizerqry head_dim_halfr7r8rrdim_i dim_i_halfsize_i base_offsetrrrrrepeats_per_entryj tile_count cos_expanded sin_expandedr>r>r?rsj              z+NDRotaryEmbedding._forward_cached_from_grid)rrN)r{r|r}r~r"rrrrNrrrZr#rrrrrrrrrrrr>r>r_r?rst   @  (    rr.rcCs8t|tr |f|St||kr|Std|d|)NzExpected length z or int, but got )r!rrr)rArr>r>r?rhs   r)rrrstartrlrrc s2t|dkrt|d}d||nIt|dkr6t|dt|d|dtfddt|D}n&t|dkrSt|dt|d|dt|d|d}n tdt|g}t|D]%}||||}} } tj|| | d||d d | } || qbtj|d d i} tj | dd} | S) a> Get n-D meshgrid with start, stop and num. Args: start (int or tuple): If len(args) == 0, start is num; If len(args) == 1, start is start, args[0] is stop, step is 1; If len(args) == 2, start is start, args[0] is stop, args[1] is num. For n-dim, start/stop/num should be int or n-tuple. If n-tuple is provided, the meshgrid will be stacked following the dim order in n-tuples. *args: See above. dim (int): Dimension of the meshgrid. Defaults to 2. Returns: grid (np.ndarray): [dim, ...] rrBrrc3 |] }||VqdSrjr>.0rrstopr>r? z"get_meshgrid_nd..r'len(args) should be 0, 1 or 2, but got rNindexingij) rrrrrr"linspacermeshgridrI) rrrrrlnum axis_gridrabnggridr>rr?get_meshgrid_ndqs*         rrrarrrrc Cst|trtj|||d}nt|tjr%|dur%|jt|kr%||}|dkr3||||d9}d|tjd|d|dd|d||j|d}t|||}|}| } || fS)a Precompute the frequency tensor for complex exponential (cis) with given dimensions. (Note: `cis` means `cos + i * sin`, where i is the imaginary unit.) This function calculates a frequency tensor with complex exponential using the given dimension 'dim' and the end index 'end'. The 'theta' parameter scales the frequencies. Args: dim (int): Dimension of the frequency tensor. pos (int or torch.FloatTensor): Position indices for the frequency tensor. [S] or scalar theta (float, optional): Scaling factor for frequency computation. Defaults to 10000.0. theta_rescale_factor (float, optional): Rescale factor for theta. Defaults to 1.0. interpolation_factor (float, optional): Factor to scale positions. Defaults to 1.0. Returns: freqs_cos, freqs_sin: Precomputed frequency tensor with real and imaginary parts separately. [S, D] rNrarrr) r!rr"r-r#rr(rr7r8) rrrrrrrrirrr>r>r?get_1d_rotary_pos_embeds(   (rr) rrrrrrrrrrrrrc % st|} t| dkrt|| d} d| nIt| dkr8t|| dt| d| dtfddt| D} n&t| dkrUt|| dt| d| d}t| d| d} n tdt| || kslJd |d | t| }dg| }|dkr| ||dksJd |d | |d || ||}||||<|||<t|ttBr|g| }nt|trt|dkr|dg| }t|| ksJdt|ttBr|g| }nt|trt|dkr|dg| }t|| ksJdd}|D]}|t|9}qt |d}t j ||f| |d}t j ||f| |d}d}t| D]}t||}|d}t||}|}|dkrC| dkrC|| 7}|dkrS||krS|||7}t j || |d|}t ||||||||| d\}}d} t|d| D] }!| t||!9} qwd}"td|D] }!|"t||!9}"q|j| dd}#|j| dd}$|"dkr|#|"d}#|$|"d}$|#|dd|||f<|$|dd|||f<||7}q||fS)a This is a n-d version of precompute_freqs_cis, which is a RoPE for tokens with n-d structure. Supports sequence parallelism by allowing sharding of a specific dimension. Args: rope_dim_list (list of int): Dimension of each rope. len(rope_dim_list) should equal to n. sum(rope_dim_list) should equal to head_dim of attention layer. start (int | tuple of int | list of int): If len(args) == 0, start is num; If len(args) == 1, start is start, args[0] is stop, step is 1; If len(args) == 2, start is start, args[0] is stop, args[1] is num. *args: See above. theta (float): Scaling factor for frequency computation. Defaults to 10000.0. theta_rescale_factor (float): Rescale factor for theta. Defaults to 1.0. interpolation_factor (float): Factor to scale positions. Defaults to 1.0. shard_dim (int): Which dimension to shard for sequence parallelism. Defaults to 0. sp_rank (int): Rank in the sequence parallel group. Defaults to 0. sp_world_size (int): World size of the sequence parallel group. Defaults to 1. Returns: Tuple[torch.Tensor, torch.Tensor]: (cos, sin) tensors of shape [HW, D/2] rrBrrc3rrjr>rrstopsr>r?rrz*get_nd_rotary_pos_embed..rrz shard_dim z( must be less than number of dimensions rrrrrr)rrrrrN)rrrrrrr!rrNrr"rr-rrr/)%rrrrrrrrrrrrlrr_rrrrqryrr7r8rrrrrrrrrrrrrrr>rr?get_nd_rotary_pos_embeds#                        rc sd|||durfddtD}t|ks Jdt|t|t|tr/t|nt|t|tr;t|nt||f} | tv} | rQt| } | t| <nt|||||d} | t| <t tdkrmtt t t| j t |dd|| | d \}}||fS) a| Generate rotary positional embeddings for the given sizes. Args: rope_sizes: Tuple of dimensions (t, h, w) hidden_size: Hidden dimension size heads_num: Number of attention heads rope_dim_list: List of dimensions for each axis, or None rope_theta: Base for frequency calculations theta_rescale_factor: Rescale factor for theta. Defaults to 1.0 interpolation_factor: Factor to scale positions. Defaults to 1.0 shard_dim: Which dimension to shard for sequence parallelism. Defaults to 0. Returns: Tuple of (cos, sin) tensors for rotary embeddings Ncsg|]}qSr>r>)rrr target_ndimr>r? sz(get_rotary_pos_embed..z>sum(rope_dim_list) should equal to head_dim of attention layer)rrrrrrrB)rrrr)rrrrNr!r_ND_ROPE_CACHEpoprrnextiterrr) rope_sizes hidden_size heads_numrrrrrrrrr cache_hitrope_embrrr>rr?get_rotary_pos_embedesL        r  _ROPE_DICTz%OrderedDict[tuple, NDRotaryEmbedding]rz5OrderedDict[tuple, tuple[torch.Tensor, torch.Tensor]]_ROPE_3D_CACHETrU max_positionrW rope_scalingpartial_rotary_factorc CsL|durt}|durdd|D}t|} nd} |dkr(t||}|} d} |durb|d|dd} | dvrAd}n!| dkrbt|dd} |d d} | durbt| tt| | } ||| ||| |f}|tvrst|S|durt ||| |||}n| dkrt|dd} t ||| |||| d }nt d ||t|<|S) NcSs(i|]\}}|t|trt|n|qSr>)r!rr)rrvr>r>r? szget_rope..ra rope_typetype)Ndefaultlinearfactor original_max_position_embeddings)r rUrVrWrLrrzUnknown RoPE scaling ) r"get_default_dtypeitemsrrgetrNmaxr rTrr)r rUrrWrLrrrrope_scaling_tuplerope_scaling_argsrVrr original_maxr rotary_embr>r>r?get_ropesf      r!)F)r)TNNra)4r~r collectionsrtypingrrrr")sglang.jit_kernel.diffusion.triton.rotaryr8sglang.multimodal_gen.runtime.distributed.parallel_stater.sglang.multimodal_gen.runtime.layers.custom_opr1sglang.multimodal_gen.runtime.utils.logging_utilsr r{loggerr#rrr@rGrKrRregisterrTrrModulerrrrrrrrr FloatTensorrNrrrr r dict__annotations__rr r!r>r>r>r?sB      R   Z \,h   6  9           Y