o ¾e¦i”8ã@s\dZddlmZddlmZddlmZe e¡Z Gdd„deƒZ Gdd„deƒZ dgZ d S) zEvolla model configurationé)ÚPreTrainedConfig)ÚRopeParameters)ÚloggingcsDeZdZdZ           d‡fdd„ Z‡ZS)Ú SaProtConfigaÊ This is the configuration class to store the configuration of a [`EvollaSaProtProteinEncoder`]. It is used to instantiate a SaProt model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 446): Vocabulary size of the protein sequence model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`EvollaModel`]. mask_token_id (`int`, *optional*, defaults to 4): The id of the *mask* token in the protein sequence model. pad_token_id (`int`, *optional*, defaults to 1): The id of the *padding* token in the protein sequence model. hidden_size (`int`, *optional*, defaults to 1280): Dimensionality of the protein sequence model layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 33): Number of hidden layers in the protein sequence model. num_attention_heads (`int`, *optional*, defaults to 20): Number of attention heads for each attention layer in the protein sequence model. intermediate_size (`int`, *optional*, defaults to 5120): Dimensionality of the intermediate layers in the protein sequence model. hidden_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the hidden layers in the protein sequence model. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention probabilities in the protein sequence model. max_position_embeddings (`int`, *optional*, defaults to 1026): The maximum sequence length that the protein sequence model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). layer_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon value for the layer normalization layer in the protein sequence model. position_embedding_type (`str`, *optional*, defaults to `"rotary"`): The type of position embedding to use in the protein sequence model. Currently only `"rotary"` is supported. emb_layer_norm_before (`bool`, *optional*, defaults to `False`): Whether to apply layer normalization before the position embedding in the protein sequence model. token_dropout (`bool`, *optional*, defaults to `True`): Whether to apply dropout to the tokens in the protein sequence model.é¾éééé!ééçš™™™™™¹?éç{®Gáz”?çñh㈵øä>ÚrotaryFTc s|tƒjdi|¤Ž||_||_||_||_||_||_||_||_ ||_ ||_ | |_ | |_ | |_| |_| |_||_||_dS)N©)ÚsuperÚ__init__Ú pad_token_idÚ mask_token_idÚ is_decoderÚadd_cross_attentionÚ vocab_sizeÚ hidden_sizeÚnum_hidden_layersÚnum_attention_headsÚintermediate_sizeÚhidden_dropout_probÚattention_probs_dropout_probÚmax_position_embeddingsÚinitializer_rangeÚlayer_norm_epsÚposition_embedding_typeÚemb_layer_norm_beforeÚ token_dropout)Úselfrrrrrrrrrr r!r"r#r$r%rrÚkwargs©Ú __class__rúm/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/evolla/configuration_evolla.pyr?s$ zSaProtConfig.__init__)rrrr r r r r r rrrrFTFF)Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚ __classcell__rrr(r*rs((îrc@sveZdZdZdZdeiZdZ                       d8dedBde dBde dBde dBde dBde dBde dBde dBde dBd e dBd!e ee e fBdBd"e dBd#e dBd$e dBd%e dBd&e dBd'e dBd(e dBd)e dBd*e dBd+e dBd,e dBd-e dBd.e dBd/e dBd0e dBd1e dBd2e dBd3e dBd4e dBd5e dBf>‡fd6d7„ Z‡ZS)9Ú EvollaConfiga© This is the configuration class to store the configuration of a [`EvollaModel`]. It is used to instantiate an Evolla model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Evolla-10B. e.g. [westlake-repl/Evolla-10B-hf](https://huggingface.co/westlake-repl/Evolla-10B-hf) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: protein_encoder_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`SaProtConfig`]. vocab_size (`int`, *optional*, defaults to 128256): Vocabulary size of the Evolla llama model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`EvollaModel`]. hidden_size (`int`, *optional*, defaults to 4096): Dimensionality of the llama layers and the pooler layer. intermediate_size (`int`, *optional*, defaults to 14336): Dimensionality of the intermediate layers in the llama model. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the llama model. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the llama model. num_key_value_heads (`int`, *optional*, defaults to 8): Number of key-value pairs for each attention layer in the llama model. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the llama model. If string, `"gelu"`, `"relu"`, `"selu"` and `"silu"` are supported. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon value for the RMS-norm layer in the llama model. rope_parameters (`float`, *optional*): The scaling factor for the RoPE layer in the llama model. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the attention layer. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention layer. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the MLP layer. aligner_ffn_mult (`int`, *optional*, defaults to 4): The FFN multiplier for the aligner layer. aligner_enable_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the aligner layer. aligner_attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention probabilities in the aligner layer. aligner_num_add_layers (`int`, *optional*, defaults to 8): The number of additional layers for the aligner layer. resampler_depth (`int`, *optional*, defaults to 6): The depth of the resampler layer in the llama model. resampler_dim_head (`int`, *optional*, defaults to 64): The dimension of the heads in the resampler layer in the llama model. resampler_heads (`int`, *optional*, defaults to 8): The number of heads in the resampler layer in the llama model. resampler_num_latents (`int`, *optional*, defaults to 64): The number of latents in the resampler layer in the llama model. resampler_ff_mult (`int`, *optional*, defaults to 4): The FFN multiplier for the resampler layer. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. pad_token_id (`int`, *optional*): The id of the *padding* token. bos_token_id (`int`, *optional*, defaults to 128000): The id of the *beginning-of-sequence* token. eos_token_id (`int`, *optional*, defaults to 128009): The id of the *end-of-sequence* token. use_cache (`bool`, *optional*, defaults to `False`): Whether or not the model should return the last key/values attentions (not used by all models). tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether or not to tie the input and output word embeddings. is_decoder (`bool`, *optional*, defaults to `False`): Whether to only use the decoder in an encoder-decoder architecture, otherwise it has no effect on decoder-only or encoder-only architectures. add_cross_attention (`bool`, *optional*, defaults to `False`): Whether cross-attention layers should be added to the model. Example: ```python >>> from transformers import EvollaModel, EvollaConfig >>> # Initializing a Evolla evolla-10b style configuration >>> configuration = EvollaConfig() >>> # Initializing a model from the evolla-10b style configuration >>> model = EvollaModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```Ú EvollaModelÚprotein_encoder_configg€„ANéõéé8é éÚsilué rFçrTr éé@réôé ôrrrrrÚnum_key_value_headsÚ hidden_actr Ú rms_norm_epsÚrope_parametersÚattention_biasÚattention_dropoutÚmlp_biasÚaligner_ffn_multÚaligner_enable_biasÚ$aligner_attention_probs_dropout_probÚaligner_num_add_layersÚresampler_depthÚresampler_dim_headÚresampler_headsÚresampler_num_latentsÚresampler_ff_multr!rÚ bos_token_idÚ eos_token_idÚ use_cacheÚtie_word_embeddingsrrc ! sö||_||_||_||_||_||_||_||_||_| |_ | |_ ||_ | |_ | |_ ||_||_||_||_||_||_||_||_||_||_||_||_| |_|dur\i}t d¡tdi|¤Ž|_||_ ||_||_ ||_!t"ƒj#di| ¤ŽdS)NzX`protein_encoder_config` is `None`. Initializing the `SaProtConfig` with default values.r)$rrrrrrrr?r@r rArRrCrDrErFrGrHrIrQr!rJrKrLrMrNrBÚloggerÚinforr2rrOrPrr)!r&r2rrrrrr?r@r rArBrCrDrErFrGrHrIrJrKrLrMrNr!rrOrPrQrRrrr'r(rr*rËsH# zEvollaConfig.__init__)Nr3r4r5r6r6r7r8r9rNFr:FrTr r7r;r<r7r<rrNr=r>FFFF)r+r,r-r.Ú model_typerÚ sub_configsÚ default_thetaÚdictÚintÚstrrÚboolÚfloatrr/rrr(r*r0isÆ]àþýüûúùø ÷ ö õ ô óòñðïîíìëêéèçæåäãâá àr0N) r.Úconfiguration_utilsrÚmodeling_rope_utilsrÚutilsrÚ get_loggerr+rSrr0Ú__all__rrrr*Ús    Q 0