o پi*@sndZddlmZmZddlmZddlmZee Z iZ GdddeZ GdddeZ Gd d d eZd S) zDbrx configuration.)AnyOptional)PretrainedConfig)loggingc s^eZdZdZ    ddedeeded ed ef fd d Ze d e d eddfddZ Z S)DbrxAttentionConfigaBConfiguration class for Dbrx Attention. [`DbrxAttention`] class. It is used to instantiate attention layers according to the specified arguments, defining the layers architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: attn_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the attention layers. clip_qkv (`float`, *optional*, defaults to None): If not `None`, clip the queries, keys, and values in the attention layer to this value. kv_n_heads (Optional[int]): For grouped_query_attention only, allow user to specify number of kv heads. rope_theta (float): The base frequency for rope. rN@ attn_pdropclip_qkv kv_n_heads rope_thetakwargsc sdtjdi|||_||_||_||_dD] }||vr"||qt|dkr0td|dS)N model_typerFound unknown kwargs=) super__init__r r r r poplen ValueError)selfr r r r r k __class__rK/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/configs/dbrx.pyr"s  zDbrxAttentionConfig.__init__pretrained_model_name_or_pathreturnrcK||||j|fi|\}}|ddkr|d}d|vr5t|dr5|d|jkr5td|d|j|j|fi|S)Nrdbrx attn_configYou are using a model of type %s to instantiate a model of type %s. This is not supported for all configurations of models and can yield errors._set_token_in_kwargsget_config_dictgethasattrrloggerwarning from_dictclsrr config_dictrrrfrom_pretrained6$  z#DbrxAttentionConfig.from_pretrained)rNrr) __name__ __module__ __qualname____doc__floatrintrr classmethodstrr- __classcell__rrrrrs2rcs~eZdZdZ        ddeed ed ed ed eed edeedede ffdd Z e de de ddfddZ ZS) DbrxFFNConfigaIConfiguration class for Dbrx FFN. [`DbrxFFN`] class. It is used to instantiate feedforward layers according to the specified arguments, defining the layers architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: ffn_act_fn (dict, optional): A dict specifying activation function for the FFN. The dict should have a key 'name' with the value being the name of the activation function along with any additional keyword arguments. ffn_hidden_size (int, optional): The hidden size of the feedforward network. moe_num_experts (int, optional): The number of experts in the mixture of experts layer. moe_top_k (int, optional): The number of experts to use in the mixture of experts layer. moe_jitter_eps (float, optional): The jitter epsilon for the mixture of experts layer. moe_loss_weight (float, optional): The loss weight for the mixture of experts layer. moe_normalize_expert_weights (float, optional): The normalization factor for the expert weights. uniform_expert_assignment (bool, optional): Whether to use uniform expert assignment. This should only be used for benchmarking purposes. Nr{Gz?F ffn_act_fnffn_hidden_sizemoe_num_experts moe_top_kmoe_jitter_epsmoe_loss_weightmoe_normalize_expert_weightsuniform_expert_assignmentr c  st|dur ddi}||_||_||_||_||_||_||_||_ dD] } | | vr2| | q't | dkr@t d| dS)Nnamesilurrr) rrr<r=r>r?r@rArBrCrrr) rr<r=r>r?r@rArBrCr rrrrrjs$   zDbrxFFNConfig.__init__rrrcKr)Nrr ffn_configr!r"r*rrrr-r.zDbrxFFNConfig.from_pretrained)Nr9r:rNr;rF)r/r0r1r2rdictr4r3boolrrr5r6r-r7rrrrr8SsJ  r8cseZdZdZdZdddddZ      ddedededededededee dee de dede dede ffdd Z ZS) DbrxConfiga Configuration class for Dbrx. [`DbrxModel`]. It is used to instantiate a Dbrx model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: d_model (`int`, *optional*, defaults to 6144): Dimensionality of the embeddings and hidden states. n_heads (`int`, *optional*, defaults to 48): Number of attention heads for each attention layer in the Transformer encoder. n_layers (`int`, *optional*, defaults to 40): Number of hidden layers in the Transformer encoder. max_seq_len (`int`, *optional*, defaults to 32768): The maximum sequence length of the model. vocab_size (`int`, *optional*, defaults to 100352): Vocabulary size of the Dbrx model. Defines the maximum number of different tokens that can be represented by the `inputs_ids` passed when calling [`DbrxModel`]. resid_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability applied to the attention output before combining with residual. emb_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the embedding layer. attn_config (`dict`, *optional*): A dictionary used to configure the model's attention module. ffn_config (`dict`, *optional*): A dictionary used to configure the model's FFN module. use_cache (`bool`, *optional*, defaults to `False`): Whether or not the model should return the last key/values attentions (not used by all models). initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. output_router_logits (`bool`, *optional*, defaults to `False`): Whether or not the router logits should be returned by the model. Enabling this will also allow the model to output the auxiliary loss. See [here]() for more details router_aux_loss_coef (`float`, *optional*, defaults to 0.001): The aux loss factor for the total loss. Example: ```python >>> from transformers import DbrxConfig, DbrxModel >>> # Initializing a Dbrx configuration >>> configuration = DbrxConfig() >>> # Initializing a model (with random weights) from the configuration >>> model = DbrxModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` rn_headsd_modeln_layers max_seq_len)num_attention_heads hidden_sizenum_hidden_layersmax_position_embeddings}NT{Gz?F皙? vocab_size resid_pdrop emb_pdropr rF use_cacheinitializer_rangeoutput_router_logitsrouter_aux_loss_coefr c s|dur t|_nt|trtdi||_n||_| dur#t|_nt| tr1tdi| |_n| |_||_||_||_||_ ||_ ||_ ||_ | |_ | |_| |_| |_|dd}|ratdtjdd|i|dS)Ntie_word_embeddingsFz5tie_word_embeddings is not supported for Dbrx models.r)rr isinstancerGr8rFrKrJrLrMrYrZr[r\r]r^r_rrrr)rrKrJrLrMrYrZr[r rFr\r]r^r_r r`rrrrs:       zDbrxConfig.__init__) rRrSrTrRrUrVrVNNTrWFrX)r/r0r1r2r attribute_mapr4r3rrr8rHrrr7rrrrrIsd7      rIN)r2typingrr transformers.configuration_utilsrtransformers.utilsr get_loggerr/r'"DBRX_PRETRAINED_CONFIG_ARCHIVE_MAPrr8rIrrrrs   CR