o Ti@sBddlZddlZddlmZmZGdddZGdddeZdS)N)ActivationFuncTypeNormTypec@seZdZddZdS)TransformerConfigcCs"d|_||_||_||_||_dS)N)layer_id hidden_sizeintermediate_sizeheadsnum_hidden_layers)selfrrr r r ^/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/ops/transformer/inference/config.py__init__ s  zTransformerConfig.__init__N)__name__ __module__ __qualname__rr r r r r s rc$seZdZdZdddddddejdejddddddddddej dddddddddddddd df$fd d Z e d d Z e ddZ ZS)DeepSpeedInferenceConfigaWInitialize the DeepSpeed Transformer Config. Arguments: hidden_size: The hidden size of the transformer layer intermediate_size: The intermediate size of the feed-forward part of transformer layer heads: The number of heads in the self-attention of the transformer layer num_hidden_layers: The number of transformer layers layer_norm_eps: The epsilon value for the layer norm local_rank: Optional: The rank of GPU running the transformer kernel, it is not required to use if the model already set the current device, otherwise need to set it so that the transformer kernel can work on the right device mp_size (optional): This argument is mainly used to create the parameters on the kernel side using model-parallel architecture. If the client model already takes care of this, there is no need to pass this argument. pre_layer_norm: Select between Pre-LN or Post-LN transformer architecture stochastic_mode: Enable for high performance, please note that this flag has some level of non-determinism and can produce different results on different runs. However, we have seen that by enabling it, the pretraining tasks such as BERT are not affected and can obtain a high accuracy level. On the other hand, for the downstream tasks, such as fine-tuning, we recommend to turn it off in order to be able to reproduce the same result through the regular kernel execution. scale_attention: If true, both q and k are scaled by 1/sqrt(attention_heads) before attention computation. return_tuple: if True, returns the transformer output as a tuple, otherwise returns as a tensor bigscience_bloom: This flag is added temporarily for supporting the BLOOM-176B model architecture. use_triton: This flag is to enable triton kernels in inference or not. invert_mask: If True, the attention mask is inverted when passed to attention block. rg-q=TFii'c%%stt|||dkr |nd|||||_| |_| |_||_| |_||_||_ | |_ | |_ ||_ ||_ ||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |_|!|_|"|_ |#|_!|$|_"dS)Nr)#superrrdtypepre_layer_norm norm_type local_rankstochastic_modeepsilonmp_sizescale_attentiontriangular_maskinglocal_attention window_size rotary_dim rotate_halfrotate_every_two return_tuplemlp_after_attnmlp_act_func_typetraining_mp_sizebigscience_bloommax_out_tokensmin_out_tokensscale_attn_by_inverse_layer_idxenable_qkv_quantizationuse_mupreturn_single_tupleset_empty_paramstransposed_mode use_tritontriton_autotunenum_kv rope_theta invert_mask)%r rrr r layer_norm_epsrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r-r.r,r/r0r1r2r3r4r5r6 __class__r r r1sL% z!DeepSpeedInferenceConfig.__init__cCs&t}|D] \}}||j|<q|S)N)ritems__dict__)cls json_objectconfigkeyvaluer r r from_dictzs z"DeepSpeedInferenceConfig.from_dictcCsFt|ddd }|}Wdn1swY|t|S)Nrzutf-8)encoding)openreadrAjsonloads)r< json_filereadertextr r r from_json_files z'DeepSpeedInferenceConfig.from_json_file)rrr__doc__torchfloat16r LayerNormrGELUr classmethodrArK __classcell__r r r8r rsVI r)rFrMdeepspeed.utils.typesrrrrr r r r s