o ¾e¦iŒ%ã@s*ddlmZmZGdd„deƒZdgZdS)é)ÚPreTrainedConfigÚlayer_type_validationc,seZdZdZdZdgZddddddddœZdgdgfd d gd gfd gd gfd œZd Zd ddddddddddddgd¢ddddddddfd e d!e d"e d#e d$e d%e d&e d'e d(e d)e d*e d+e d,e dBd-e d.e d/e d0e d1e d2e dBd3e d4ee dBf*‡fd5d6„ Z‡ZS)7Ú CwmConfigaÙ Configuration for Code World Model (CWM). This is an inherited Llama3-compatible configuration with layer-interleaved sliding-window attention. Configures a `CwmModel`. Designed to yield a configuration mirroring the model in the [facebook/cwm](https://huggingface.co/facebook/cwm) architecture by default. Other models include: - [facebook/cwm-sft](https://huggingface.co/facebook/cwm-sft) - [facebook/cwm-pretrain](https://huggingface.co/facebook/cwm-pretrain) Args: vocab_size (`int`, *optional*, defaults to 128256): Vocabulary size of the CWM model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`CwmModel`] hidden_size (`int`, *optional*, defaults to 6144): Dimension of the hidden representations intermediate_size (`int`, *optional*, defaults to 21504): Dimension of the MLP representations num_hidden_layers (`int`, *optional*, defaults to 64): Number of hidden layers in the Transformer decoder num_attention_heads (`int`, *optional*, defaults to 48): Number of attention heads for each attention layer in the Transformer decoder num_key_value_heads (`int`, *optional*, defaults to 8): This is the number of key_value heads that should be used to implement Grouped Query Attention (GQA). If it is not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 128): The attention head dimension. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 131072): The maximum sequence length that this model might ever be used with. CWM's attention allows sequence lengths up to 131072 tokens. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*): Padding token id. eos_token_id (`int` or `list[int]`, *optional*, defaults to `[128001, 128008, 128009]`): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. bos_token_id (`int`, *optional*, defaults to 128000): The id of the *beginning-of-sequence* token. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. pretraining_tp (`int`, *optional*, defaults to 1): Tensor parallelism degree used during pretraining. See [this document](https://huggingface.co/docs/transformers/parallelism) and [this issue](https://github.com/pytorch/pytorch/issues/76232). mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. sliding_window (`int`, *optional*, defaults to 8192): Sliding window attention window size. layer_types (`List[str]`, *optional*): List of layer types for each layer. Each element should be either "full_attention" or "sliding_attention". If not specified, will default to alternating pattern based on the provided window pattern. ÚcwmÚpast_key_valuesÚcolwiseÚrowwise)zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.o_projzlayers.*.mlp.gate_projzlayers.*.mlp.up_projzlayers.*.mlp.down_projÚ input_idsÚ inputs_embedsÚ hidden_statesÚattention_mask)Ú embed_tokensÚlayersÚnorm瀄.AiõiiTé@é0éé€Úsiluig{®Gáz”?gñh㈵øä>TN)iôiôi ôiôFgéé Ú vocab_sizeÚ hidden_sizeÚintermediate_sizeÚnum_hidden_layersÚnum_attention_headsÚnum_key_value_headsÚhead_dimÚ hidden_actÚmax_position_embeddingsÚinitializer_rangeÚ rms_norm_epsÚ use_cacheÚ pad_token_idÚ bos_token_idÚtie_word_embeddingsÚattention_dropoutÚpretraining_tpÚmlp_biasÚrope_parametersÚsliding_windowÚ layer_typesc s|dur dddddddœ}|durd‰‡fd d „t|ƒDƒ}nt||ƒ|r*t|ƒnd|_t|ƒ|_||_| |_||_||_ ||_ ||_ |durJ|}||_ ||_ | |_| |_||_| |_||_||_|durh|n|j|j |_||_||_| |_||_||_tƒjd i|¤ŽdS) Nrg0@g@gð?rÚllama3)Ú rope_thetaÚfactorÚhigh_freq_factorÚlow_freq_factorÚ original_max_position_embeddingsÚ rope_typeécs g|] }|ˆdkr dnd‘qS)éÚfull_attentionÚsliding_attention©)Ú.0Úi©Úwindow_patternr8úg/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/cwm/configuration_cwm.pyÚ •sÿÿz&CwmConfig.__init__..r8)ÚrangerÚintr+Úlistr,rr rrrrrrr!r"r(r#r'r)rr*r&r$r%Ú eos_token_idÚsuperÚ__init__)Úselfrrrrrrrrr r!r"r#r$rBr%r&r'r(r)r*r+r,Úkwargs©Ú __class__r;r=rDmsNú  þ  zCwmConfig.__init__)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú model_typeÚkeys_to_ignore_at_inferenceÚbase_model_tp_planÚbase_model_pp_planÚ default_thetar@ÚstrÚfloatÚboolÚdictrArDÚ __classcell__r8r8rGr=rs¤@ù  ýèþýüûúùø ÷ ö õ ô óòðïîíìëé èrN)Úconfiguration_utilsrrrÚ__all__r8r8r8r=Ús #