o wi\@sdZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZmZdd lm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5m6Z6m7Z7ddl8m9Z9ddl:m;Z;ddlm/Z?ddl@mAZAddlBmCZCddlDmEZEe=dd\ZFZGe=dd\ZHZGe=dd \ZIZGe=dd!\ZJZG d"e.d#efd$d%ZKeGd&d'd'e.ZLeGd(d)d)eLZMeGd*d+d+eLZNeGd,d-d-eLZOeGd.d/d/eLZPGd0d1d1e/ZQGd2d3d3eZRGd4d5d5eZSd6eTd7eeTeTfd#eUfd8d9ZVGd:d;d;e ZWGdGd?d@d@e6jZdAeQfZ[e6\eQd>GdBdCdCe6jZeQdAfZ]gdDZ^dS)EzGemma3 language modelN) dataclass) lru_cache)Path) TYPE_CHECKING AnnotatedCallableOptionalTupleUnion)get_bias_dropout_add)BaseInferenceContext)LanguageModelEmbedding)RotaryEmbedding)PackedSeqParams) ModuleSpecTransformerConfigTransformerLayerTransformerLayerSubmodules) SelfAttentionSelfAttentionSubmodules) AttnBackend AttnMaskType)MLP MLPSubmodules)Tensornn) openai_gelu) GPTConfigGPTModel)TERowParallelLinearLayerNorm)Config)OptimizerModuleioteardown) TransformFns) dtype_from_hf)safe_import_from)rGemma3ForCausalLM AutoTokenizer) TokenizerSpecz+megatron.core.extensions.transformer_engineTERowParallelLinearTENormTELayerNormColumnParallelLinearTEDotProductAttentionconfigreturnc CsFtttttdtjitttt t t ddt tt t tt ddt ddS)zGemma3 custom layer spec.attn_mask_type) linear_qkvcore_attention q_layernorm k_layernorm linear_proj)moduleparams submodules) linear_fc1 linear_fc2)r8r:)self_attention self_attn_bdamlpmlp_bda)rrrGemma3SelfAttentionrcausalrr.Gemma3TEDotProductAttentionr-rr rr)r0rDb/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/gemma3.pygemma3_layer_spec^s2 rFcsHeZdZUdZdZeed<dZeed<dZe ed<dZ e ed <d Z e ed <d Ze ed <d Ze ed<dZeed<dZe ed<dZe ed<dZeed<dZeed<ejZeed<d Ze ed<dZe ed<eZeed<dZe ed<dZe ed<dZ e ed<e!Z"e#e$ed ge$ffed!<d Z%e ed"< # # #d)d$e&ed%d&ffd'd( Z'Z(S)* Gemma3ConfigzGemma3 basic configi seq_lengthi@ vocab_sizeropeposition_embedding_type)'@B rotary_baseT#share_embeddings_and_output_weightsRMSNorm normalizationlayernorm_zero_centered_gammagư>layernorm_epsilon window_size)interleaved_attn_patterngattention_dropouthidden_dropoutattention_backendgated_linear_unitFadd_bias_linearactivation_funcis_vision_language flash_decodegradient_accumulation_fusionrtransformer_layer_spec#scatter_embedding_sequence_parallelNvp_stager1 MCoreGPTModelc s|jdkr tdt|dddur|dusJd|j\}}||_tj||||d}||f|_t|drCt||j|j |j |j d|_ t |jd |j|j|d |j|j|d |_t|ds`t|d rd||S) z7Configure and instantiate a megatron-core Gemma3 model.rWz3Context Parallel is not supported for Gemma3 model.$virtual_pipeline_model_parallel_sizeNzKVirtual pipeline model parallel size is not yet supported for Gemma3 model.)rd embedding)r0rImax_sequence_lengthrKscatter_to_sequence_parallel?F) kv_channelsrotary_percentrotary_interleavedseq_len_interpolation_factorrN rope_scalingrope_scaling_factoruse_cpu_initializationrotary_base_local output_layer)context_parallel_size ValueErrorgetattrrNsuperconfigure_modelhasattrGemma3LanguageModelEmbeddingrIrHrKrcrgGemma3RotaryEmbeddingrkrmrnrprqrotary_pos_emb!setup_embeddings_and_output_layer)self tokenizer pre_process post_processrdrrrotary_base_globalmodel __class__rDrErxs>     zGemma3Config.configure_model)NNN))__name__ __module__ __qualname____doc__rHint__annotations__rIrKstrrNtuplerOboolrQrRrSfloatrUrXrYrZrflashr[r\r]rr^rr_r`rarFrbr rrcrrx __classcell__rDrDrrErG{s>                    rGc@seZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z eed <d Z eed <dZ eed<dZeed<dZeed<dZeed<dZeed<dZeed<dS)Gemma3Config1BzGemma3 1B configFr_ num_layersi hidden_sizenum_attention_headsrWnum_query_groupsrkiffn_hidden_sizerTrUrjrpirHTbf16irIN)rrrrr_rrrrrrrrkrrUrprrHrrIrDrDrDrErs            rc@~eZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z eed <d Z eed <dZ eed<dZeed<dZeed<dS)Gemma3Config4BzGemma3 4B configTr_"ri rrrrrrki(rrU @rpNrrrrr_rrrrrrrrkrrUrprrDrDrDrEr         rc@r)Gemma3Config12BzGemma3 12B configTr_0rirrrrrrki<rrrUrrpNrrDrDrDrErrrc@seZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z eed <d Z eed <de dZeed<dZeed<dZeed<dZeed<dS)Gemma3Config27BzGemma3 27B configTr_>rir rrrrkrj softmax_scaleiTrrrUrrpN)rrrrr_rrrrrrrrkmathsqrtrrrUrprrDrDrDrErs         rc speZdZdZddddgfdeeeeefdeededdee e j ge j fdee f fd d Z ZS) Gemma3ModelzGemma3 base modelNr0optimrr+model_transformmodel_context_managerscs tj|pt||||ddS)N)rrrr)rw__init__rG)r~r0rrrrrrDrErs zGemma3Model.__init__)rrrrrrrGr r!rrModulelistrrrDrDrrErs$rc s4eZdZdZd dedededeffdd ZZS) rzzQGemma3 language token embedding. Adds a normalization to the embedding. N input_ids position_ids tokentype_idsr1cs$t|||}||jjd}|S)z!Calculate embedding and normalizeg?)rwforwardr0r)r~rrr embeddingsrrDrEr(sz$Gemma3LanguageModelEmbedding.forwardN)rrrrrrrrrDrDrrErz"s(rzc sfeZdZdZ    ddededed effd d Zed d ddededede ffdd Z Z S)r{zpGemma3 position rope embedding. Calculates rope embeddings for both local and global attention layers. FrrMrLrorprNrrc sL|dusJtjd||d||j|_td||d||_dS)NF)rorNrD)rwrinv_freqr rope_local)r~rorprNrrkwargsrrDrEr5s  zGemma3RotaryEmbedding.__init__r)maxsizer max_seq_lenoffset packed_seqr1cs(t|||}|j|||}||fS)z#Get global and local rope embedding)rwrr)r~rrr rope_globalrrrDrErPszGemma3RotaryEmbedding.forward)FrrMrL)rF) rrrrrrrrrrrrrDrDrrEr{/s"*r{ layer_number layer_patterncCst|}||dkS)Nr)sum)rr pattern_sizerDrDrE_is_local_attn_layerXs rcseZdZdZ        ddddededeedeedeeeeeeffd eed eed eed ee d ee deedeeefffddZ Z S)rAzyGemma3 self attention. Uses local rope embedding for local layers, global rope embedding for global layers. N)inference_params hidden_statesattention_maskkey_value_statesinference_contextr|rotary_pos_cosrotary_pos_sinattention_biaspacked_seq_paramssequence_len_offsetrr1c  sft|tsJ|dur|dusJt|j|jjr|d} n|d} tj||||| |||| | | d S)z>Switch to either local or global rope embedding before forwardNrrW) rrrrr|rrrrrr) isinstancerrrr0rXrwr) r~rrrrr|rrrrrrfinal_rotary_pos_embrrDrErgs$ zGemma3SelfAttention.forward)NNNNNNNN) rrrrrrr r r rrrrrDrDrrErA`sJ       rAc s>eZdZdZ d dededededee f fdd Z Z S) rCzGemma3 core attention. Switches between global and local sliding window attention based on the layer_number and pre-defined layer pattern. Nr0rr2attention_typerYc sXt|}t||jr|jdf|_nd|_|jrtj}tj d|||||d|dS)Nr)r0rr2rrYrD) copydeepcopyrrXrUr_r arbitraryrwr)r~r0rr2rrYrrrDrErs  z$Gemma3TEDotProductAttention.__init__r) rrrrrrrrrrrrrDrDrrErCs rChfc@sXeZdZdZdefddZdedefddZdd Ze dd d Z e de fd dZ dS)HFGemma3ImporterzGemma3 Huggingface importerr1cCst|j|jdS)N)r)rr0rr~rDrDrEinitszHFGemma3Importer.init output_pathcCshddlm}|jt|dd}|}||}||||||td|t ||~~|S)Nrr'auto) torch_dtypez,Converted HF Gemma3 model to Nemo, saved to ) transformersr(from_pretrainedrr nemo_setup convert_state nemo_saveprintr#)r~rr(sourcetargettrainerrDrDrEapplys     zHFGemma3Importer.applyc CPddddddddd d d }tjd d tjdtjddtjdg}tj||||dS)N embedding.word_embeddings.weights                    R )  - ' d b