o پi:@sNdZddlmZddlmZddlmZmZmZe e Z GdddeZ dS)zFalcon-H1 model configuration)PretrainedConfig)logging)Mamba2CacheParamsMamba2StateShapemamba2_state_dtypecseZdZdZdZdgZ                                  d%fdd ZeddZedd Z ed!d"Z ed#d$Z Z S)&FalconH1Configa0 This is the configuration class to store the configuration of a [`FalconH1Model`]. It is used to instantiate a FalconH1Model model according to the specified arguments, defining the model architecture. Instantiating a configuration with defaults taken from [ibm-fms/FalconH1-9.8b-2.2T-hf](https://huggingface.co/ibm-fms/FalconH1-9.8b-2.2T-hf). The FalconH1Model is a hybrid [mamba2](https://github.com/state-spaces/mamba) architecture with SwiGLU. The checkpoints are jointly trained by IBM, Princeton, and UIUC. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 128000): Vocabulary size of the FalconH1 model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`FalconH1Model`] tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether the model's input and output word embeddings should be tied. Note that this is only relevant if the model has a output word embedding layer. hidden_size (`int`, *optional*, defaults to 4096): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 14336): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the Transformer encoder. num_key_value_heads (`int`, *optional*, defaults to 8): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `8`. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. num_logits_to_keep (`int` or `None`, *optional*, defaults to 1): Number of prompt logits to calculate during generation. If `None`, all logits will be calculated. If an integer value, only last `num_logits_to_keep` logits will be calculated. Default is 1 because only the logits of the last prompt token are needed for generation. For long sequences, the logits for the entire sequence may use a lot of memory so, setting `num_logits_to_keep=1` will reduce memory footprint significantly. pad_token_id (`int`, *optional*, defaults to 0): The id of the padding token. bos_token_id (`int`, *optional*, defaults to 1): The id of the "beginning-of-sequence" token. eos_token_id (`int`, *optional*, defaults to 2): The id of the "end-of-sequence" token. max_position_embeddings (`int`, *optional*, defaults to 8192): Max cached sequence length for the model attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. mamba_d_ssm (`int`, *optional*, defaults to 1024): The dimension of the SSM state space latents. mamba_n_heads (`int`, *optional*, defaults to 128): The number of mamba heads used in the v2 implementation. mamba_d_head (`int`, *optional*, defaults to `"auto"`): Head embedding dimension size mamba_n_groups (`int`, *optional*, defaults to 1): The number of the mamba groups used in the v2 implementation. mamba_d_state (`int`, *optional*, defaults to 256): The dimension the mamba state space latents mamba_d_conv (`int`, *optional*, defaults to 4): The size of the mamba convolution kernel mamba_expand (`int`, *optional*, defaults to 2): Expanding factor (relative to hidden_size) used to determine the mamba intermediate size mamba_chunk_size (`int`, *optional*, defaults to 256): The chunks in which to break the sequence when doing prefill/training mamba_conv_bias (`bool`, *optional*, defaults to `True`): Flag indicating whether or not to use bias in the convolution layer of the mamba mixer block. mamba_proj_bias (`bool`, *optional*, defaults to `False`): Flag indicating whether or not to use bias in the input and output projections (["in_proj", "out_proj"]) of the mamba mixer block mamba_norm_before_gate (`bool`, *optional*, defaults to `True`): Whether to use RMSNorm before the gate in the Mamba block mamba_rms_norm (`bool`, *optional*, defaults to `False`): Whether to use RMSNorm instead of LayerNorm in the Mamba block projectors_bias (`bool`, *optional*, defaults to `False`): Flag indicating whether or not to use bias in the input and output projections (["in_proj", "out_proj"]) of the attention block rope_theta (`float`, *optional*, defaults to 100000.0): The theta value used for the RoPE embeddings. rope_scaling (`float`, *optional*): The scaling value used for the RoPE embeddings. If `None`, no scaling is applied. lm_head_multiplier (`float`, *optional*, defaults to 1.0): The multiplier for the LM head. This is used to scale the output of the LM head. embedding_multiplier (`float`, *optional*, defaults to 1.0): The multiplier for the embedding layer. This is used to scale the output of the embedding layer. mlp_multipliers (`list[float]`, *optional*): The multipliers for the MLP layers. This is used to scale the output of the MLP layers. The first value is the multiplier of gate layer, the second value is the multiplier of the down_proj layer. key_multiplier (`float`, *optional*): The multiplier for the key layer. This is used to scale the output of the key layer. attention_out_multiplier (`float`, *optional*): The multiplier for the attention output layer. This is used to scale the output of the attention output attention_in_multiplier (`float`, *optional*): The multiplier for the attention input layer. This is used to scale the output of the attention input layer. ssm_multipliers (`list[float]`, *optional*): The multipliers for the SSM layers. This is used to scale the output of the SSM layers. ssm_in_multiplier (`float`, *optional*): The multiplier for the SSM input layer. This is used to scale the output of the SSM input layer. ssm_out_multiplier (`float`, *optional*): The multiplier for the SSM output layer. This is used to scale the output of the SSM output layer. falcon_h1past_key_valuesF8 silu{Gz?h㈵>Tr autoj@N?c*, s||_||_||_||_||_||_||_d|_d|_|dur!|}||_ ||_ | |_ | |_ | |_ | |_||_d|_| |_||_|durG||n||_}+|+|dkrVtd|dkr^|+|}|||+krhtd||_||_||_||_||_||_||_||_||_||_||_||_|!|_ |"|_!|#dur|#|_"nddg|_"|%dur|%|_#nd|_#|&dur|&|_$nd|_$|$dur|$|_%nd|_%|'dur|'|_&ngd|_&|(dur|(|_'nd|_'|)dur|)|_(nd|_(t)j*d | |||d|*dS) NFrz4mamba_n_heads must divide mamba_expand * hidden_sizerzPThe dimensions for the Mamba head state do not match the model intermediate_sizer)rrrrr) pad_token_id bos_token_id eos_token_idtie_word_embeddings)+ vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsmax_position_embeddingsattention_dropoutattention_biasmlp_biasnum_key_value_heads hidden_actinitializer_range rms_norm_eps use_cachenum_logits_to_keep rope_theta rope_scalingprojectors_biasmamba_intermediate ValueError mamba_d_ssm mamba_n_heads mamba_d_headmamba_n_groups mamba_d_state mamba_d_conv mamba_expandmamba_chunk_sizemamba_conv_biasmamba_proj_biasmamba_norm_before_gatemamba_rms_normlm_head_multiplierembedding_multipliermlp_multipliersattention_out_multiplierattention_in_multiplierkey_multiplierssm_multipliersssm_in_multiplierssm_out_multipliersuper__init__),selfr"r r#r$r%r&r+r,r-r.r/r0rrrr'r(r6r7r8r9r:r;r<r=r>r?r@rAr3r1r2rBrCrDrGrErFrHrIrJkwargsr4 __class__r!P/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/configs/falcon_h1.pyrLs-     zFalconH1Config.__init__cCsddt|jDS)NcSsg|]}dqS)rr!).0ir!r!rQ sz4FalconH1Config.layers_block_type..ranger%rMr!r!rQlayers_block_typesz FalconH1Config.layers_block_typecC t|jSNrUrWr!r!rQfull_attention_layer_ids" z'FalconH1Config.full_attention_layer_idscCrYrZrUrWr!r!rQlinear_layer_ids'r\zFalconH1Config.linear_layer_idsc CsFddlm}tj||j|j|j|j|j|j d}t ||j t |dS)Nr)get_attention_tp_size) tp_world_sizer$n_groups num_headshead_dim state_size conv_kernel)shapelayersdtype) sglang.srt.layers.dp_attentionr^rcreater4r9r7r8r:r;rr]r)rMr^rer!r!rQmamba2_cache_params,s  z"FalconH1Config.mamba2_cache_params))r Fr r r r rrrrTrrrrrrrrrrrrrrTFTFFrNrrNNNNNNN) __name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencerLpropertyrXr[r]rj __classcell__r!r!rOrQrsnj   rN) rn transformers.configuration_utilsrtransformers.utilsrsglang.srt.configs.mamba_utilsrrr get_loggerrkloggerrr!r!r!rQs