o پi$@s\ddlmZddlmZmZddlmZmZddlm Z ddl m Z m Z m Z mZddlZddlZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lm Z m!Z!ddl"m#Z#e rlddl$m%Z%e!Z&ee'e e'fZ( eGdddZ)d5ddZ*d6ddZ+d7ddZ,d8d!d"Z-Gd#d$d$Z.  %d9d:d2d3Z/e#ed&d'gd4ZdS);) annotations)IterableMapping) dataclassfield) lru_cache) TYPE_CHECKINGAnyOptionalTupleN)can_use_fused_inplace_qknormfused_inplace_qknorm)envs)RadixAttention) SWAKVPool)get_is_capture_mode) ForwardBatch)get_current_device_stream_fastis_cuda)register_custom_op)RMSNormc@speZdZUdZeedZded<eedZded<eedZ ded<dd d Z dddZ dddZ dddZ dS) WeightsMapperzBMaps the name of each weight if they match the following patterns.)default_factoryWeightsMappingorig_to_new_substrorig_to_new_prefixorig_to_new_suffixkeystrreturn Optional[str]cCst|jddddD]\}}||vr$|durdS|||d}nq t|jddddD]\}}||rJ|durAdS|||d}nq1t|jddddD]\}}||rs|durgdS|| |d}|SqW|S)NcS t|dSNrlenir'K/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/models/utils.py4 z)WeightsMapper._map_name..T)rreversecSr!r"r#r%r'r'r(r)>r*cSr!r"r#r%r'r'r(r)Hr*) sortedritemsreplacer startswithrendswithjoinrsplit)selfrsubstrnew_keyprefixsuffixr'r'r( _map_name2s:  zWeightsMapper._map_nameweights"Iterable[tuple[str, torch.Tensor]]cfdd|DS)Nc3s.|]\}}|dur|fVqdSNr9).0namedataout_namer4r'r( Vsz&WeightsMapper.apply..r')r4r:r'rBr(applySs zWeightsMapper.applyvalues list[str]cr<)Ncs"g|] }|durqSr=r>)r?r@rBr'r( ]s z,WeightsMapper.apply_list..r'r4rFr'rBr( apply_list\s zWeightsMapper.apply_listdict[str, Any]csfdd|DS)Ncs(i|]\}}|dur|qSr=r>)r?r@valuerBr'r( ds z,WeightsMapper.apply_dict..)r.rIr'rBr( apply_dictcs zWeightsMapper.apply_dictN)rrrr )r:r;rr;)rFrGrrG)rFrKrrK)__name__ __module__ __qualname____doc__rdictr__annotations__rrr9rErJrNr'r'r'r(r*s   ! r forward_batchrcCs,tot|jdo|jjtjkot|jt S)z?Enable fused set_kv_buffer only on CUDA with bfloat16 KV cache.dtype)_is_cudahasattrtoken_to_kv_poolrVtorchbfloat16 isinstancer)rUr'r'r(enable_fused_set_kv_bufferks   r]rL torch.TensorlayerrcCsbddlm}|j}|j}||}||}||||jdd||jdd|j|j |j dS)Nr)FusedSetKVBufferArg)rLk_bufferv_bufferk_scalev_scale cache_loc) sgl_kernelr`layer_idrYget_key_bufferget_value_bufferviewshaperdre out_cache_loc)rLr_rUr`rhrYrbrcr'r'r(create_fused_set_kv_buffer_argus   rnpermrcCs*t|}tj||j|jd||<|S)N)devicerV)rZ empty_likearangenumelrprV)roinv_permr'r'r( permute_invs rugrid_thwcCs|jjdks Jd|}t|dddf|dddf|dddfjdtjd}ttjdtjd|g}t |}|S) z Compute cu_seqlens from grid_thw using NumPy. grid_thw: [T, 3] int tensor on CPU. columns: [repeat_count, H, W] Returns: cu_seqlens: 1D int32 tensor on CPU, shape [N + 1] cpuz7compute_cu_seqlens_from_grid_numpy expects a CPU tensorNr,r)axisrV)rV) rptypenumpynprepeatcumsumint32 concatenatezerosrZ from_numpy)rvarr cu_seqlensr'r'r("compute_cu_seqlens_from_grid_numpys 4 rc@s$eZdZeeddd d d Zd S) RotaryPosMixini)maxsizehintwspatial_merge_sizerr^cCst|tjr t|}t|tjrt|}t|tjr$t|}tt||d||f}||}||}|||||}| dddd}| }tt|d|||f}|||||}| dddd}| }t tj ||gddS)Nr,rrxra)ry) r\rZTensorritemr| broadcast_torrreshape transposeflattenrstack)rrrhpos_idsh_divw_divwpos_idsr'r'r( rot_pos_idss6      zRotaryPosMixin.rot_pos_idsN)rrrrrrrr^)rOrPrQ staticmethodrrr'r'r'r(rsrTqkq_normrk_normhead_dimr alt_streamOptional[torch.cuda.Stream] allow_inplacebool!Tuple[torch.Tensor, torch.Tensor]c Cs6|d}|j}|j} tr8|r8|| kr8tjs8t||jr8t| |d|| |d||j |j ||d||fS|durwt rwt } | | |d|} || } tj||d|} || } Wdn1slwY| |n|d|} || } |d|} || } | |j}| |j}||fS)a Apply QK normalization for query and key tensors. If eligible, we will use JIT fused inplace QK normalization for better performance. Args: q: Query tensor of shape [batch_size, ...] k: Key tensor of shape [batch_size, ...] q_norm: RMSNorm layer for query normalization k_norm: RMSNorm layer for key normalization head_dim: Dimension of each attention head alt_stream: Optional alternative CUDA stream for overlapping computation allow_inplace: Whether to allow inplace normalization. (True for better performance) Returns: Tuple of normalized query and key tensors rra)rrq_weightk_weightrepsN)sizevariance_epsilonrWr%SGLANG_ENABLE_DETERMINISTIC_INFERENCEgetr rVr rkweightrr wait_streamrrZcudastreamrl) rrrrrrr batch_sizeq_epsk_epscurrent_stream q_by_head k_by_headr'r'r( apply_qk_normsL             r) mutates_args)rUr)rLr^r_rrUr)ror^rr^)rvr^rr^)NT)rr^rr^rrrrrrrrrrrr)0 __future__rcollections.abcrr dataclassesrr functoolsrtypingrr r r r{r|rZsglang.jit_kernel.normr r sglang.srt.environr!sglang.srt.layers.radix_attentionr$sglang.srt.mem_cache.swa_memory_poolr+sglang.srt.model_executor.cuda_graph_runnerr,sglang.srt.model_executor.forward_batch_inforsglang.srt.utilsrrsglang.srt.utils.custom_oprsglang.srt.layers.layernormrrWrrrr]rnrurrrr'r'r'r(s>          @  * B