o i<@sddlZddlmZddlmZddlmZmZmZddlm Z m Z m Z ddl m Z mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZd dlmZmZmZmZd dl m!Z!m"Z"m#Z#d dl$m%Z%d dl&m'Z'm(Z(d dl)m*Z*m+Z+d dl,m-Z-ee.Z/dej0dej0dej0dej0dej0dej0de1ddfddZ2dej0dej0dej0dej0dej0dej0de1ddfddZ3ede2dge3d Gd!d"d"ej4e%Z5dS)#N) rearrange)nn) CacheConfig ModelConfigget_current_vllm_config)divideget_tensor_model_parallel_rank$get_tensor_model_parallel_world_size)ForwardContextget_forward_context) init_logger)sharded_weight_loader)set_weight_attrs)direct_register_custom_op)AttentionMetadata)GDNAttentionMetadata)FusedRMSNormGated chunk_kdafused_kda_gatefused_recurrent_kda)ColumnParallelLinearReplicatedLinearRowParallelLinear) MambaBase)MambaStateDtypeCalculatorMambaStateShapeCalculator)causal_conv1d_fncausal_conv1d_update)QuantizationConfig q_proj_states k_proj_states v_proj_statesg1beta core_attn_out layer_namereturnc Cs*t}|j|}|j||||||ddS)N)r r!r"r#r$r%)r no_compile_layers_forward) r r!r"r#r$r%r&forward_contextselfr,T/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/model_executor/layers/kda.py kda_attention)s  r.cCsdS)Nr,)r r!r"r#r$r%r&r,r,r-kda_attention_fake>s r/)op_nameop_func mutates_args fake_implcseZdZedefddZdeejejejejffddZ deee dfee dfee dfee dfffddZ d"d e d e de d Bde d Bded Bdededd ffdd Zdejdejdejdd fddZdejdejdejdejdejdejdd fd d!ZZS)#KimiDeltaAttentionr'cCsdS)N gdn_attentionr,r+r,r,r- mamba_typeSszKimiDeltaAttention.mamba_typecCs0|jdus |jdurtdt|jj|jjS)Nz)model_config and cache_config must be set) model_config cache_config ValueErrorrkda_state_dtypedtypemamba_cache_dtyper6r,r,r-get_state_dtypeWs  z"KimiDeltaAttention.get_state_dtype.cCstj|j|j|j|jdS)N)conv_kernel_size)rkda_state_shapetp_size num_headshead_dim conv_sizer6r,r,r-get_state_shape`sz"KimiDeltaAttention.get_state_shapeNh㈵> layer_idx hidden_size quant_configr9r8 rms_norm_epsprefixc stt|_t|_||_||_||_|durt d|j } | d|_ | d|_ ||_ ||_|j |jdks;Jt|j |j|_|j |j } | d|_t|j| d||dd|_t|j| d||d d|_t|j| d||d d|_t|j|j d||d d|_t|j | d||d d|_ttjt| |jtjd |_t|jdt dit|j|j d||dd|_!t|j| dtj|dd|_"t|j| dtj|dd|_#t|j| dtj|dd|_$|j"j%j&'d|j"j%_&|j#j%j&'d|j#j%_&|j$j%j&'d|j$j%_&ttjdd|jdtjd |_(t|j(dt dit|j|j d||dd|_)t|j | d||dd|_*t+|j |dd|_,t-| |jd||dd|_.t/j0} || j1vrkt d||| j1|<dS)Nzmodel_config must be providedrCrBrshort_conv_kernel_sizeFz.q_proj)biasrJrLz.k_projz.v_projz .f_a_projz .f_b_proj)r< weight_loaderz.b_projz .q_conv1d) input_size output_sizerN params_dtyperLz .k_conv1dz .v_conv1drz .g_a_projz .g_b_projsigmoid)eps activationz.o_projzDuplicate layer name: )2super__init__r rArtp_rankrIr8r9r:linear_attn_configrCrBrHrLrlocal_num_headsrDrq_projk_projv_projrf_a_projf_b_projr Parametertorchemptyfloat32dt_biasrr b_projq_conv1dk_conv1dv_conv1dweightdata unsqueezeA_logg_a_projg_b_projro_normro_projrcompilation_configstatic_forward_context) r+rHrIrJr9r8rKrLkwargs kda_configprojection_sizerr __class__r,r-rXgs       zKimiDeltaAttention.__init__ hidden_states positionsoutputc Cs |d}||d}||d}||d}||d}|||dd} t | |j |j |j d} | d}| d} |||dd} t| d|j d} tjd||j|j f|j|jd} tjj|||| || |j|| | } t| d} || d|dd<dS)Nr)g_biasz... (h d) -> ... h ddr)r<devicez1 n h d -> n (h d))sizer\r]r^rffloatrTr`r_rrmrCrerlrornrrbzerosr[r<ropsvllmr.rLrprq) r+ryrzr{ num_tokensqkvr$r# g_proj_statesg2r%r,r,r-forwards8    zKimiDeltaAttention.forwardr r!r"r#r$r%c st}|j}|dur dSt|tsJ|j}t|tsJ|j} |j} |j} |j } j |j } |d| }|d| }|d| }|d| }|d| }| \}}}}| dd}| dd}| dd}j jj jdj jd}jjjjdjjd}jjjjdjjd}|jdkr| dd}| dd}| dd}t||j jd|| | | |d dd}t||jjd|| | | |d dd}t||jjd|| | | |d dd}n.| d|j }t|||j jd|dd }t|||jjd|dd }t|||jjd|dd }tfd d |||f\}}}|jdkrT| | }d||<|| }t||||||dd| d \}}||| <nt||||||d| d|jd| d \}}|dd| f|dd| f<dS)NrrSrsilu)rV conv_stateshas_initial_state cache_indicesquery_start_locmetadataT)rVconv_state_indices validate_datacst|djdS)Nzn (h d) -> 1 n h dr})rrC)xr6r,r-sz-KimiDeltaAttention._forward..) rrrgr$ initial_stateoutput_final_stateuse_qk_l2norm_in_kernel cu_seqlens) rrrrr$rrrssm_state_indices)r attn_metadata isinstancedictrLrrnon_spec_query_start_locnon_spec_state_indices_tensornum_actual_tokenskv_cachevirtual_engine transposergrjviewrrhri num_prefillsrrNrmap contiguousrr num_decodes)r+r r!r"r#r$r%r*rrrrrconstant_caches conv_state_q conv_state_k conv_state_vrecurrent_stateq_conv_weightsk_conv_weightsv_conv_weightsrrrdecode_conv_indiceszero_idxrcore_attn_out_non_speclast_recurrent_stater,r6r-r)s                              zKimiDeltaAttention._forward)NNNrFrG)__name__ __module__ __qualname__propertystrr7tuplerbr<r>intrErrrrrXTensorrr) __classcell__r,r,rwr-r4Rsp  .    &r4)6rbeinopsrr vllm.configrrrvllm.distributedrrr vllm.forward_contextr r vllm.loggerr -vllm.model_executor.model_loader.weight_utilsr vllm.model_executor.utilsrvllm.utils.torch_utilsrvllm.v1.attention.backendr#vllm.v1.attention.backends.gdn_attnr fla.ops.kdarrrrlinearrrrmamba.abstractrmamba.mamba_utilsrrmamba.ops.causal_conv1drrquantization.base_configrrloggerrrr.r/Moduler4r,r,r,r-sx