o پiQ@s~dZddlmZmZddlmZddlmZddlm Z m Z GdddeZ z e de Wd Se y>e ejd<Yd Sw) a%LFM2-MoE (Liquid Foundation Model 2 - Mixture of Experts) configuration Note: HF transformers has Lfm2MoeConfig in v5.0.0rc2 (unreleased). Once released, we could inherit from it like Lfm2Config does with HFLfm2Config. For now, we define a standalone config to support the model immediately. )ListOptional)CONFIG_MAPPING)PretrainedConfig)Mamba2CacheParamsMamba2StateShapec5s$eZdZdZdZdgZ                 d:dedededededededededed ed!ed"ed#ed$ed%e e d&ed'ed(ed)ed*ed+ed,ed-ed.e e e f2fd/d0 Z ed1e efd2d3Zed1e efd4d5Zed1efd6d7Zed1e efd8d9ZZS); Lfm2MoeConfiga Configuration for LFM2-MoE models (e.g., LiquidAI/LFM2-8B-A1B). LFM2-MoE is a hybrid architecture with: - Attention layers and ShortConv layers (like dense LFM2) - MoE (Mixture of Experts) FFN layers with sigmoid routing Key MoE specifics: - First `num_dense_layers` use dense MLP, rest use MoE - Sigmoid routing (not softmax) with expert_bias for load balancing - expert_bias is fp32 for numerical stability lfm2_moepast_key_values {Gz?h㈵>TrNF? vocab_size hidden_sizeintermediate_sizemoe_intermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_headsmax_position_embeddingsinitializer_rangenorm_eps use_cache pad_token_id bos_token_id eos_token_idtie_word_embeddingsrope_parameters conv_bias conv_L_cachenum_dense_layers num_expertsnum_experts_per_tokuse_expert_biasrouted_scaling_factornorm_topk_prob layer_typesc s||_||_||_||_||_||_||_||_| |_| |_ | |_ ||_ ||_ ||_ ||_||_||_||_||_||_||_|durVt||krVtdt|d|d|d|}tjd| | ||d|dS)Nzlayer_types length (z ) must match num_hidden_layers () tie_embedding)r$r%r&r')rrrrrrrr r!r"r#r)r*r+r,r-r.r/r0r1r(len ValueErrorpopsuper__init__)selfrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1kwargs __class__r4O/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/configs/lfm2_moe.pyr9-sH   zLfm2MoeConfig.__init__returncC"|jdurgSddt|jDS)z0Return indices of attention layers for KV cache.NcSsg|] \}}|dkr|qS)full_attentionr4.0iltr4r4r> sz:Lfm2MoeConfig.full_attention_layer_ids..r1 enumerater:r4r4r>full_attention_layer_ids|s z&Lfm2MoeConfig.full_attention_layer_idscCr@)z3Return indices of conv layers for conv state cache.NcSsg|] \}}|dvr|qS))conv short_convr4rBr4r4r>rFsz2Lfm2MoeConfig.linear_layer_ids..rGrIr4r4r>linear_layer_idss zLfm2MoeConfig.linear_layer_idscCsdS)z@Return chunk size for Mamba2 backend. LFM2 doesn't use chunking.rr4rIr4r4r>mamba_chunk_sizeszLfm2MoeConfig.mamba_chunk_sizec Cstddlm}|j}|s dS|j}t|j}z|}Wn ttfy'd}Ynwtj ||d||d|d}t ||dS)z Get cache params for HybridReqToTokenPool initialization. LFM2-MoE uses ShortConv layers with a small fixed-size cache. r)get_attention_tp_sizeNr) tp_world_sizern_groups num_headshead_dim state_size conv_kernel)shapelayers) sglang.srt.layers.dp_attentionrOrMrintr*AssertionError RuntimeErrorrcreater)r:rOconv_layer_idsrrUtp_sizerVr4r4r>mamba2_cache_paramss0    z!Lfm2MoeConfig.mamba2_cache_params)r r r rrrrrrrTrrrTNFrrrrTrTN)__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencerYfloatboolrdictrstrr9propertyrJrMrNrr_ __classcell__r4r4r<r>rs       Orr N)rctypingrr transformersr transformers.configuration_utilsrsglang.srt.configs.mamba_utilsrrrregister Exception_extra_contentr4r4r4r>s   !