o ߥi.@sTddlZddlZddlmZddlmZeZGdddeZGdddeZ dS)N)PretrainedConfig)loggercseZdZdZ                  d#fdd ZeddZeddZddZddZ dd Z d!d"Z Z S)$ PlugNLUConfigplugNLUTR gelu皙?hn|?linear{Gz??镲 ?TFh㈵>c stjdd|i|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ | |_||_||_||_||_||_||_||_||_||_||_||_dSNlayer_norm_eps)super__init__ vocab_sizeoriginal_vocab_size hidden_sizenum_hidden_layersnum_attention_heads hidden_actintermediate_sizehidden_dropout_probattention_probs_dropout_probmax_position_embeddingstype_vocab_sizeinitializer_rangelr_decay_style weight_decay clip_gradwarmuppre_lnfp16fp32_layernormfp32_embeddinglayernorm_epsilonfp32_tokentypesdec_hidden_layers attn_separate)selfrrrrr r"r!r#r$r%r&r'r(r)r*r+r,r-r.r/r1r0r2r3kwargs __class__r\/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/plug/configuration.pyrs2 zPlugNLUConfig.__init__cCs&t}|D] \}}||j|<q|S)zAConstructs a `BertConfig` from a Python dictionary of parameters.)ritems__dict__)cls json_objectconfigkeyvaluerrr8 from_dictSs zPlugNLUConfig.from_dictcCsFt|ddd }|}Wdn1swY|t|S)z9Constructs a `BertConfig` from a json file of parameters.rzutf-8)encodingN)openreadr@jsonloads)r; json_filereadertextrrr8from_json_file[s zPlugNLUConfig.from_json_filecCs6|j}|jD]\}}||vrq ||j|<q |S)z;merge values a `BertConfig` from a json file of parameters.)r:keysr9)r4args local_keysr>r?rrr8 merge_argsbs  zPlugNLUConfig.merge_argscCs t|S)N)strto_json_stringr4rrr8__repr__ks zPlugNLUConfig.__repr__cCst|j}|S)z0Serializes this instance to a Python dictionary.)copydeepcopyr:)r4outputrrr8to_dictns zPlugNLUConfig.to_dictcCstj|ddddS)z*Serializes this instance to a JSON string.T)indent sort_keys )rEdumpsrVrQrrr8rPsszPlugNLUConfig.to_json_string)rrrr r r r r r rrrrrrrTTTFFrrF) __name__ __module__ __qualname__ model_typer classmethodr@rJrNrRrVrP __classcell__rrr6r8rsF5   rcsVeZdZdZdZ                  dfdd ZZS) PlugNLGConfiga This is the configuration class to store the configuration of a [`PlugModel`]. It is used to instantiate a PLUG understanding model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the PLUG [PLUG](https://modelscope.cn/models/damo/nlp_plug_text-generation_27B/summary) architecture. Configuration objects inherit from [`PlugNLUConfig`] and can be used to control the model outputs. Read the documentation from [`PlugNLUConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 21504): Padded vocabulary size of the PLUG model for vocab tensor parallel. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`PlugModel`]. original_vocab_size (`int`, *optional*, defaults to 21128): True vocabulary size of the PLUG model. Defines the number of different tokens that can be represented. hidden_size (`int`, *optional*, defaults to 8192): Dimensionality of the encoder layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 24): Number of hidden layers in the Transformer encoder. dec_hidden_layers (`int`, *optional*, defaults to 6): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 128): Number of attention heads for each attention layer in the Transformer encoder. intermediate_size (`int`, *optional*, defaults to 32768): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act (`str`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported. hidden_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the Transformer Attention. max_position_embeddings (`int`, *optional*, defaults to 2048): The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). type_vocab_size (`int`, *optional*, defaults to 3): The vocabulary size of the `token_type_ids` passed when calling [`PlugModel`]. initializer_range (`float`, *optional*, defaults to 0.00707): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. lr_decay_style (`str`, *optional*, defaults to 'linear'): The decay style of learning rate during fine-tunining. If string, `"linear"`, `"cosine"`, `"exponential"`, `"constant"`, `"None"` are supported. weight_decay (`float`, *optional*, defaults to 1e-2): Decoupled weight decay to apply. clip_grad (`float`, *optional*, defaults to 1.0): Maximum gradient norm for gradient clipping. warmup (`float`, *optional*, defaults to 0.01): Ratio of total training steps used for a linear warmup from 0 to `learning_rate`. pre_ln (`boolean`, *optional*, defaults to `True`): Whether or not to apply LayerNorm to the input instead of the output in the blocks. fp16 (`boolean`, *optional*, defaults to `True`): Whether to use fp16 16-bit (mixed) precision training instead of 32-bit training. fp32_layernorm (`boolean`, *optional*, defaults to `True`): Whether to use fp32 32-bit precision LayerNorm training while the argument `fp16` set to `True`. fp32_embedding (`boolean`, *optional*, defaults to `False`): Whether to use fp32 32-bit precision Embedding training while the argument `fp16` set to `True`. fp32_tokentypes (`boolean`, *optional*, defaults to `False`): Whether to use fp32 32-bit precision token types training while the argument `fp16` set to `True`. layernorm_epsilon (`float`, *optional*, defaults to 1e-5): The epsilon to use in the layer normalization layers. attn_separate (`boolean`, *optional*, defaults to `False`): Whether or not to separate query-key-value to query, key, value in the Attention. Example: >>> # The PLUG model has 27B parameters and usually need to run on multiple GPUs. The example given >>> # here only initializes a slice of the model on a single GPU. >>> # Check out the [`~DistributedPipeline.__init__`] method to initialize entire PLUG model. >>> from modelscope.models.nlp.plug import PlugNLGConfig, PlugModel >>> # Initializing a Plug configuration >>> configuration = PlugNLGConfig() >>> # Initializing a model from the configuration >>> model = PlugModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config plugNLGrrrr rr r r r rrrrrrTF-q=c stjdd|i|||_||_||_||_||_||_| |_| |_ | |_ | |_ | |_ ||_ ||_||_||_||_||_||_||_||_||_||_||_dSr)rrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3)r4rrrrr2r r"r!r#r$r%r&r'r(r)r*r+r,r-r.r/r1r0r3r5r6rr8rs0 zPlugNLGConfig.__init__)rrrr rr r r r r rrrrrrrTTTFFrdF)r\r]r^__doc__r_rrarrr6r8rbxs8Prb) rSrE transformersrmodelscope.utilsrlogging get_loggerrrbrrrr8s  ]