o wib@sddlZddlmZddlmZddlmZmZmZm Z m Z ddl Z ddl m Z mZddlmZddlmZddlmZdd lmZdd lmZmZmZmZmZdd lmZmZdd l m!Z!dd l"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3m4Z4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;erddlm?Z?ddl@mAZAe;dd\ZBZCe;dd\ZDZCe;dd\ZEZCdd d!efd"d#ZFeGd$d%d%e.ZGeGd&d'd'eGZHeGd(d)d)eGZIeGd*d+d+eGZJGd,d-d-e/ZKe4LeKd.Gd/d0d0e4jMd1eKfZNe4OeKd.Gd2d3d3e4jMeKd1fZPGd4d5d5eZQGd6d7d7eBZRGd8d9d9eZSGd:d;d;e j*jTZUdd?ZWd@dAZXgdBZYdS)CN) dataclass)Path) TYPE_CHECKING AnnotatedCallableOptionalUnion)parallel_statetensor_parallel)get_bias_dropout_add)FusedScaleMaskSoftmax)PackedSeqParams)ColumnParallelLinear)MegatronModule ModuleSpecTransformerConfigTransformerLayerTransformerLayerSubmodules) SelfAttentionSelfAttentionSubmodules) AttnMaskType)MLP MLPSubmodules)attention_mask_func)divide)Tensornn) openai_gelu) GPTConfigGPTModel)Config)OptimizerModuleioteardown) TransformFns) dtype_from_hf)safe_import_from)GemmaForCausalLM AutoTokenizer) TokenizerSpecz+megatron.core.extensions.transformer_engineTERowParallelLinearTENormTELayerNormColumnParallelLinearconfigrreturnc CsBtttttdtjitttt ddt tt t tt ddt ddS) attn_mask_type) linear_qkvcore_attention linear_proj)moduleparams submodules) linear_fc1 linear_fc2)r5r7)self_attention self_attn_bdamlpmlp_bda) rrrrrcausalrr-Gemma2DotProductAttentionTERowParallelLinearLayerNormr rrr.rBb/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/gemma2.pygemma2_layer_spec?s. rDc@seZdZUdZdZeed<eZe ed<dZ e ed<dZ eed<d Z e ed <d Zeed <d Zeed<dZeed<dZeed<dZe ed<dZe ed<dZeed<dZeed<dZeed<dZeed<d Ze ed<eZeee dgeffed<dZ eed <d!Z!eed"<d#Z"eed$<d%S)& Gemma2ConfigzGemma2 basic configRMSNorm normalizationactivation_funcTgated_linear_unitropeposition_embedding_typeFadd_bias_lineari seq_length kv_channelsattention_dropouthidden_dropout#share_embeddings_and_output_weightslayernorm_zero_centered_gammagư>layernorm_epsiloni' rotary_base)ir window_sizei vocab_sizegradient_accumulation_fusionrtransformer_layer_specquery_pre_attn_scalargI@attn_logit_softcappingg>@final_logit_softcappingN)#__name__ __module__ __qualname____doc__rGstr__annotations__rrHrrIboolrKrLrMintrOrQfloatrRrSrTrUrVrWtuplerXrYrDrZrrr\r]r^rBrBrBrCrE^s,                   rEc@ZeZdZUdZdZeed<dZeed<dZeed<dZ eed <d Z eed <d Z eed <dS)Gemma2Config2BzGemma2 2B config num_layersi hidden_sizenum_attention_headsnum_query_groupsi$ffn_hidden_sizerNr\N r_r`rarbrlrfrdrmrorqrrr\rBrBrBrCrj{      rjc@ri)Gemma2Config9BzGemma2 9B config*rlirmrornrqi8rrrNr\NrsrBrBrBrCrurtruc@ri)Gemma2Config27BzGemma2 27B config.rlirm rorwrqirrr\NrsrBrBrBrCrxrtrxc sreZdZdZ    d deeeeefdeededdee e j ge j fffdd Z fd d Z ZS) Gemma2Modelr0Nr.optim tokenizerr*model_transformcstj|pt|||ddS)N)r}r~r)super__init__rE)selfr.r}r~r __class__rBrCrszGemma2Model.__init__csRddlm}ttjddr||jjttj ddr'||jj t dSdS)Nr)extend_instanceF)ignore_virtual) #nemo.collections.common.parts.utilsrrconfigure_modelr is_pipeline_first_stager5 embeddingEmbeddingScalingMixinis_pipeline_last_stage output_layerGemma2OutputLayer)rrrrBrCrs    zGemma2Model.configure_model)NNNN)r_r`rarbrrrEr r!rrModulerr __classcell__rBrBrrCr|s  r|hfc@sXeZdZdZdefddZdedefddZdd Ze dd d Z e de fd dZ dS)HFGemmaImporterr0r/cCst|j|jdS)N)r~)r|r.r~rrBrBrCinitszHFGemmaImporter.init output_pathcCshddlm}|jt|dd}|}||}||||||td|t ||~~|S)Nr)Gemma2ForCausalLMauto) torch_dtypez/Converted Gemma2 model to Nemo, model saved to ) transformersrfrom_pretrainedrcr nemo_setup convert_state nemo_saveprintr#)rrrsourcetargettrainerrBrBrCapplys     zHFGemmaImporter.applyc CLddddddddd }tjd d tjd tjd dtjd g}tj||||dS)r0 embedding.word_embeddings.weight2decoder.layers.*.self_attention.linear_proj.weight&decoder.layers.*.mlp.linear_fc2.weight.make_vocab_size_divisible_byrlrmrrroinit_method_stdrUrqrVr\r]r^rWrITrrXrSfp16bf16 params_dtypegeneration_configNrB)rrrrrcrEnum_hidden_layersrmintermediate_sizeroinitializer_range rms_norm_epsnum_key_value_heads rope_thetar\r]r^sliding_windowrXr%torchfloat16bfloat16)r HFGemmaConfigrrrroutputrBrBrCr.s^        zHFGemmaImporter.configN)r/r)) r_r`rarbr|rrrrpropertyr~rEr.rBrBrBrCrs# rr'c@sNeZdZdZdddZdedefddZd d Zed d Z edddZ dS)HFGemmaExporterr0r/r'cCsNddlm}ddlm}|||jWdS1s wYdS)Nr)AutoModelForCausalLM)no_init_weights)rrtransformers.modeling_utilsr from_configr.)rrrrBrBrCr&s   $zHFGemmaExporter.initrcCsH|}|t|\}}|||}|}|||j||Sr0)r nemo_loadrcrcpusave_pretrainedr~)rrrr_rBrBrCr-s   zHFGemmaExporter.applyc Cr)r0rrrrrrrr)rrrrrrrrrrrrrr)r"rr$ split_qkv split_fc1rrrBrBrCr:s* zHFGemmaExporter.convert_statecCstt|jjjSr)r" load_contextrcmodelr~rrBrBrCr~^szHFGemmaExporter.tokenizerrEcCsztjt|dd}ddlm}|dg|j|j|j|j|j dur"|j n|j|j|j |j |j |j |jj|j|j|j|jdS)r0z model.config)subpathr)rErN) architecturesrrmrrohead_dimmax_position_embeddingsrrrrXrr\r]r^)r"rrcrrErlrmrrrorOrMrrUrqr~rXrVr\r]r^)rrrrBrBrCr.ds*   zHFGemmaExporter.configN)r/r')r/rE) r_r`rarbrrrrrr~r.rBrBrBrCr"s  $ rc sbeZdZdZ ddededededef fdd Z  dd e d e d e d e dede f ddZ Z S)r?a Region where selective activation recomputation is applied. This region is memory intensive but less compute intensive which makes activation checkpointing more efficient for LLMs (20B+). See Reducing Activation Recomputation in Large Transformer Models: https://arxiv.org/abs/2205.05198 for more details. We use the following notation: h: hidden size n: number of attention heads p: number of tensor model parallel partitions b: batch size s: sequence length Nr. layer_numberr1attention_typerQc s(tj|d||_|jjdksJdtd||_d|_|jddkr(|j|_||_||_|jj |jj }t }t |||_t ||j |_t |jj ||_t |jj||_d} t|j|_|jjrn|j} |j| 9_t|jj|jj|j|jjt|jj| d|_tj !|dur|jj"n||_"dS)NrAz?Context parallelism is only supported by TEDotProductAttention!rr) input_in_fp16 input_in_bf16r1scaled_masked_softmax_fusion mask_funcsoftmax_in_fp32scale)#rrr.context_parallel_sizemaxrrWr1rrOror $get_tensor_model_parallel_world_sizerhidden_size_per_partitionhidden_size_per_attention_head!num_attention_heads_per_partitionrqnum_query_groups_per_partitionmathsqrtr\ norm_factorapply_query_key_layer_scalingr rrmasked_softmax_fusionrattention_softmax_in_fp32scale_mask_softmaxrrDropoutrQ) rr.rr1rrQkwargsprojection_size world_sizecoeffrrBrCrsD     z"Gemma2DotProductAttention.__init__querykeyvalueattention_maskpacked_seq_paramscKs`|dusJd|j|jdkr&|j|j|jdd}|j|j|jdd}|d|d|d|df}||d|d|dd}||d|d|dd}t|d|d|d|df|j d } t j | | dd| dd ddd d |j d } t| |jj} | j|} |dur|jdurt|d|d|j}|| |} |jjst || } Wdn1swYn|| } |d|d|d|df}||d|d|dd}| |d|d|dd} t | | dd} | j|} | dddd} | dd |jf}| j|} | S)zForward. Modified from mcore.transformer.dot_product_attention to support Gemma2-specific final_logit_softcapping. NzaPacked sequence is not supported by DotProductAttention.Please use TEDotProductAttention instead.rr)dimrmpurPg?)betaalpha)rrrepeat_interleavesizereshapeviewr get_global_memory_buffer get_tensordtyperbaddbmm transposerlogit_softcappingr.r]rWget_swarsequence_parallelr get_cuda_rng_trackerforkrQbmmpermute contiguousr)rr r rrr1rr output_sizematmul_input_buffer matmul_resultattention_scoresattention_probscontextnew_context_shaperBrBrCforwardsj             z!Gemma2DotProductAttention.forward)N)NN)r_r`rarbrrfrrcrgrrr r0rrBrBrrCr?s:Ar?cs:eZdZdZdededeffdd ZfddZZS) r@z=Modified From TERowParallelLinear with an additional Post-LN. input_sizer)r.c s*tj||fd|i|t|||_dS)Nr.)rrr,post_layernorm)rr1r)r.rrrBrCrRsz%TERowParallelLinearLayerNorm.__init__cst|\}}|||fS)z)Forward with additional Post LN on output)rr0r2)rxrbiasrrBrCr0bsz$TERowParallelLinearLayerNorm.forward) r_r`rarbrfrrr0rrBrBrrCr@Osr@c eZdZdZfddZZS)rz:Extends from ColumnParallelLinear with logit soft capping.cs,tj|i|\}}t||jj}||fS)z Forward with logit soft capping.)rr0r!r.r^)rargsrrr4rrBrCr0kszGemma2OutputLayer.forwardr_r`rarbr0rrBrBrrCrhsrcr5)rz A mixin class for scaling embeddings in Megatron GPT. The scaling is applied only if the configuration (accessible via `self.config`) includes `apply_embedding_scaling` set to True. c s.tjdi|}|tj|jjd|jdS)z Forward pass that scales the output embeddings from the `forward` method of the superclass by the square root of the hidden size specified in the configuration. g?)rNrB)rr0rtensorr.rmr)rr embeddingsrrBrCr0yszEmbeddingScalingMixin.forwardr7rBrBrrCrrsrlogitsrcCs|s|S|t||S)zIPrevents logits from growing excessively by scaling them to a fixed range)rtanh)r:rrBrBrCr!sr!cCsRtj||tjdd}tj||||dd}tj||||dd}|}|S)zECreate the equivalent attention mask fro SWA in [seq_q, seq_kv] shapecuda)rdevicer)diagonalr)ronesretriutril)seq_qseq_kvwmmumlrBrBrCr"s r")rErjrurxr|)Zr dataclassesrpathlibrtypingrrrrrr megatron.corer r (megatron.core.fusions.fused_bias_dropoutr #megatron.core.fusions.fused_softmaxr megatron.core.packed_seq_paramsr megatron.core.tensor_parallelrmegatron.core.transformerrrrrr#megatron.core.transformer.attentionrrmegatron.core.transformer.enumsrmegatron.core.transformer.mlprrmegatron.core.transformer.utilsrmegatron.core.utilsrrr"nemo.collections.llm.fn.activationr#nemo.collections.llm.gpt.model.baserrnemo.collections.llm.utilsr nemo.lightningr!r"r#nemo.lightning.io.stater$nemo.lightning.pytorch.utilsr%nemo.utils.import_utilsr&rr'rr)1nemo.collections.common.tokenizers.tokenizer_specr*r+rr,r-rDrErjrurxr|model_importerModelConnectorrmodel_exporterrr?r@rrrrgr!r"__all__rBrBrBrCsn                     j`M