o }oi) @sddlZddlZddlmZmZddlmZddlmZddl m Z m Z m Z m Z mZmZmZmZmZddlZddlmmZddlZddlmZddlmZmZmZddlmZdd lm Z dd l!m"Z"dd l#m$Z$m%Z%m&Z&dd l'm(Z(dd l)m*Z*ddl+m,Z,m-Z-m.Z.ddl/m0Z0ddl1m2Z2z ddl3m4Z4dZ5Wn e6ydZ5Ynwe rddl7mZ8ddl9m:Z:m;Z;ddlddld?d?eTZUeGd@dAdAeTZVGdBdCdCeZWGdDdEdEeWZXe%YeWdFGdGdHdHe%jZdIeWfZ[e%\eWdFGdJdKdKe%jZeWdIfZ]e%\eWdLGdMdNdNe]Z^ O P Q Rd^dSe_dTe_dUe_dVe`fdWdXZaebdYe,dZejcfd[d\Zdgd]ZedS)_N) dataclassfield)partial)Path) TYPE_CHECKING AnnotatedAnyCallableDictListOptionalTupleUnion)nn) GPTConfigGPTModeltorch_dtype_from_mcore_configget_llama4_layer_spec)Config)load_distributed_model_weights)OptimizerModuleioteardown)ADAPTER_META_FILENAME)ckpt_to_weights_subdir) TransformCTX TransformFns _ModelState) dtype_from_hf)logging) ModuleSpecTF)r)AutoPeftModelForCausalLM PeftConfig LlamaConfig)LlamaForCausalLM AutoTokenizer) TokenizerSpecc@seZdZUdZdZeed<ejZ e ed<dZ e ed<dZ eed<d Ze ed <d Zeed <d Zeed<d Zeed<d Ze ed<dZe ed<dZe ed<dZe ed<dZe ed<dZe ed<dZee ed<dS)r%zConfiguration class for Llama models. Extends GPTConfig with specific settings optimized for Llama architectures. Includes configurations for normalization, activation functions, and various architecture-specific options. RMSNorm normalizationactivation_funcTgated_linear_unitropeposition_embedding_typeFadd_bias_linear seq_lengthattention_dropouthidden_dropout#share_embeddings_and_output_weightsbias_activation_fusionmasked_softmax_fusionpersist_layer_normbias_dropout_fusionapply_rope_fusionNuse_transformer_engine_op_fuser)__name__ __module__ __qualname____doc__r+str__annotations__Fsilur,r r-boolr/r0r2intr4floatr5r6r7r8r9r:r;r<r rHrHX/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/llama.pyr%9s"              r%c@NeZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed <d S) Llama2Config7BzConfiguration for a 7B parameter Llama 2 model. Specific configuration for the 7B Llama 2 model with 32 layers, 4096 hidden size, and 32 attention heads. num_layersr1 hidden_sizenum_attention_headsnum_query_groupsi+ffn_hidden_sizeN r=r>r?r@rMrFrBrNrOrPrQrHrHrHrIrKU     rKc@rJ) Llama2Config13BzConfiguration for a 13B parameter Llama 2 model. Specific configuration for the 13B Llama 2 model with 40 layers, 5120 hidden size, and 40 attention heads. (rMrNrOrPi6rQNrRrHrHrHrIrTdrSrTc@sNeZdZUdZdZeed<dZeed<dZeed<dZ eed <d Z eed <d S) Llama2Config70BzConfiguration for a 70B parameter Llama 2 model. Specific configuration for the 70B Llama 2 model with 80 layers, 8192 hidden size, and 64 attention heads with 8 query groups. PrM rN@rOrPprQNrRrHrHrHrIrWsrSrWc@seZdZUdZdZeed<dZeed<dZ eed<dZ e ed<d Z eed <d Z eed <d Zeed<ejZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<d Zeed<dZe ed<dZeed<dS) Llama3ConfigzConfiguration for Llama 3 models. Base configuration for Llama 3 architecture with common settings across different model sizes, including group query attention (GQA) and architecture-specific settings. r[rPr3r5r4r*r+g{Gz?init_method_stdgh㈵>layernorm_epsilonFr0r,Tr-r7r8r9r:r;r6r.r/?rotary_percentN)r=r>r?r@rPrFrBr5rGr4r+rAr^r_r0rErCrDr,r r-r7r8r9r:r;r6r/rarHrHrHrIr]s&                r]csbeZdZUdZdZeed<dZeed<dZeed<dZ e ed <d Z eed <ddfdd Z Z S) Llama31ConfigzConfiguration for Llama 3.1 models. Extends Llama3Config with specific settings for Llama 3.1 models, including RoPE scaling parameters. @ scale_factorr`low_freq_factor@high_freq_factorrYold_context_leng{Gz?r^Nreturn MCoreGPTModelcs8t||||}t|jj|j|j|j|jd|j_|S)aConfigure and instantiate a Megatron Core Llama 3.1 model. Extends the base configuration with Llama 3.1 specific RoPE scaling. Args: tokenizer: Tokenizer used with the model pre_process: Whether to include pre-processing in the model post_process: Whether to include post-processing in the model Returns: MCoreGPTModel: Configured Megatron Core GPT model instance )factorrergrh) superconfigure_modelapply_rope_scalingrotary_pos_embinv_freqrdrergrh)self tokenizer pre_process post_processvp_stagemodel __class__rHrIrms  zLlama31Config.configure_model)NNN)rirj)r=r>r?r@rdrGrBrergrhrFr^rm __classcell__rHrHrwrIrbs      rbc@ZeZdZUdZdZeed<dZeed<dZeed<dZ eed <d Z eed <dZ eed <d S)Llama3Config8BzConfiguration for an 8B parameter Llama 3 model. Specific configuration for the 8B Llama 3 model with 32 layers, 4096 hidden size, and 32 attention heads.  rotary_baserYr2rLrMr1rN8rQrON r=r>r?r@r}rFrBr2rMrNrQrOrHrHrHrIr{s      r{c@sreZdZUdZdZeed<dZeed<dZeed<dZ eed<d Z eed <d Z eed <d Z e ed<dZeed<dS)Llama3Config70BzConfiguration for a 70B parameter Llama 3 model. Specific configuration for the 70B Llama 3 model with 80 layers, 8192 hidden size, and 64 attention heads. r|r}rYr2rXrMrNr\rQrZrOgܠ[;Q?r^make_vocab_size_divisible_byN)r=r>r?r@r}rFrBr2rMrNrQrOr^rGrrHrHrHrIrs        rc@rz)Llama31Config8BzConfiguration for an 8B parameter Llama 3.1 model. Specific configuration for the 8B Llama 3.1 model with 32 layers, 4096 hidden size, and 32 attention heads, supporting a longer context length of 131K tokens. r|r}r2rLrMr1rNr~rQrONrrHrHrHrIrs      rc@feZdZUdZdZeed<dZeed<dZeed<dZ eed <d Z eed <d Z eed <dZ eed<dS)Llama31Config70BzConfiguration for a 70B parameter Llama 3.1 model. Specific configuration for the 70B Llama 3.1 model with 80 layers, 8192 hidden size, and 64 attention heads, supporting a longer context length of 131K tokens. r|r}rr2rXrMrYrNr\rQrZrOrrN r=r>r?r@r}rFrBr2rMrNrQrOrrHrHrHrIr       rc@sfeZdZUdZdZeed<dZeed<dZeed<dZ eed <d Z eed <d Z eed <d Z eed<dS)Llama31Config405BzConfiguration for a 405B parameter Llama 3.1 model. Specific configuration for the 405B Llama 3.1 model with 126 layers, 16384 hidden size, and 128 attention heads, supporting a longer context length of 131K tokens. r|r}rr2~rM@rNirQrrOrNrrHrHrHrIr rrc@seZdZUdZdZeed<dZeed<dZ e ed<dZ e ed <d Z e ed <d Z e ed <dZe ed<dZe ed<dZe ed<dZe ed<dS)Llama32Config1BzConfiguration for a 1B parameter Llama 3.2 model. Specific configuration for the 1B Llama 3.2 model with 16 layers, 2048 hidden size, and 32 attention heads (8 query groups). g@@rdTr6r|r}rr2rMirNrYrQrLrOr[rPrrN)r=r>r?r@rdrGrBr6rEr}rFr2rMrNrQrOrPrrHrHrHrIr          rc@seZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z eed <d Z eed <dZ eed<dZeed<dZeed<dZeed<dS)Llama32Config3BzConfiguration for a 3B parameter Llama 3.2 model. Specific configuration for the 3B Llama 3.2 model with 28 layers, 3072 hidden size, and 24 attention heads (8 query groups). rLrdTr6r|r}rr2rMi rNrYrQrOr[rPrrN)r=r>r?r@rdrFrBr6rEr}r2rMrNrQrOrPrrHrHrHrIr1rrc@*eZdZUdZdZeed<dZeed<dS)CodeLlamaConfig7BzConfiguration for a 7B parameter CodeLlama model. Extends Llama2Config7B with modified settings specifically for code generation, including longer context length and different rotary base. @Br}rr2Nr=r>r?r@r}rFrBr2rHrHrHrIrE  rc@r)CodeLlamaConfig13BzConfiguration for a 13B parameter CodeLlama model. Extends Llama2Config13B with modified settings specifically for code generation, including longer context length and different rotary base. rr}rr2NrrHrHrHrIrQrrc@r)CodeLlamaConfig34BzConfiguration for a 34B parameter CodeLlama model. Specific configuration for the 34B CodeLlama model with 48 layers, 8192 hidden size, and 64 attention heads (8 query groups). 0rMrYrNrZrOr[rPiVrQrr}rr2N) r=r>r?r@rMrFrBrNrOrPrQr}r2rHrHrHrIr]s       rc@seZdZdZdS)CodeLlamaConfig70BzConfiguration for a 70B parameter CodeLlama model. Extends Llama2Config70B with settings specifically for code generation. N)r=r>r?r@rHrHrHrIrnsrc@seZdZUdZdZeed<dZeed<dZeed<dZ eed <d Z eed <d Z eed <dZ eed<dZ eed<dZeed<dZeed<dZeed<dZeed<edddZeeedgeffed<dZeed<dZeed<dZeed<d Zeed!<dZeed"<d#Zeed$<d%Zeed&<d'Z e!eed(<dZ"eed)<dZ#eed*<dZ$eed+<dZ%eed,<dZ&eed-<d.Z'e(ed/<dZ)eed0<d'S)1 Llama4Configz2 Configuration for Llama4 language model. r|r}rYr2rrMrVrNrrQrUrOi@ vocab_sizeFr0Tr-rotary_interleavedr;nope_layer_intervalcCstSNrrHrHrHrIszLlama4Config.default_factoryr%transformer_layer_specmoe_grouped_gemm#moe_shared_expert_intermediate_sizemoe_ffn_hidden_sizemoe_router_topkmoe_router_pre_softmaxsigmoidmoe_router_score_functionalltoallmoe_token_dispatcher_typeNmoe_router_dtypemoe_apply_probs_on_inputmoe_shared_expert_overlapmoe_permute_fusion qk_l2_norm rope_scalingrcrope_scaling_factorattention_chunk_size)*r=r>r?r@r}rFrBr2rMrNrQrOrr0rEr-rr;rrrrr!r rrrrrrrArrr rrrrrrrGrrHrHrHrIrxs@                          rc@sBeZdZUdZdZeed<dZeed<dZ e ed<dZ eed<d S) Llama4Experts16Configz4 Configuration for llama4 16-experts model. rnum_moe_expertsTrrcrrN) r=r>r?r@rrFrBrrErrGrrHrHrHrIrs    rc@sXeZdZUdZdZeed<dZeed<e dddZ e ee efed <dZ eed <d S) Llama4Experts128Configz5 Configuration for llama4 128-experts model. rrFrcCs ddgdS)NrrrrHrHrHrHrIrs zLlama4Experts128Config.rmoe_layer_freqrN)r=r>r?r@rrFrBrrErrrr rrHrHrHrIrs   "rc speZdZdZddddgfdeeeeefdeededdee e j ge j fdee f fd d Z ZS) LlamaModelzLlama model implementation based on the GPT model architecture. This class provides a high-level interface for Llama models, implementing the specific architecture and settings needed for Llama models. Nconfigoptimrrr)model_transformmodel_context_managerscs tj|pt||||ddSN)rrrrr)rl__init__r%rqrrrrrrrwrHrIrs zLlamaModel.__init__)r=r>r?r@rr r%rrr rModuler rryrHrHrwrIrs$rc sfeZdZdZ    d deeeeefdeededdee e j ge j fffdd Z Z S) MLPerfLoRALlamaModela|Memory-optimized Llama model implementation for MLPerf LoRA fine-tuning. This class wraps LlamaModel and adds context managers around configure_model to reduce memory consumption during initialization. It applies techniques like avoiding unnecessary gradients and using FP8 parameter initialization. Changes made here are experimental, proceed with caution. Nrrrrr)rcs*tg}tj|p t||||ddSr)torchno_gradrlrr%rrwrHrIrs  zMLPerfLoRALlamaModel.__init__)NNNN)r=r>r?r@rr r%rrr rrrryrHrHrwrIrs rhfc@sleZdZdZdefddZdedefddZdd Ze dd d Z d e j de fddZe defddZdS)HFLlamaImporterzImporter for converting Hugging Face Llama models to NeMo format. This class handles the conversion of Hugging Face's LlamaForCausalLM models to NeMo's LlamaModel format, including weight mapping and configuration translation. ricCst|j|jdS)zInitialize a NeMo LlamaModel instance. Returns: LlamaModel: Initialized NeMo Llama model with the appropriate configuration and tokenizer. )rr)rrrrrqrHrHrIinitszHFLlamaImporter.init output_pathc Csddlm}m}|t|}t|ddkr)ddlm}|jt|dd}|j}n |jt|dd}|}| |}| ||| ||t d|d |j d t||~~|S) zApply the conversion from HF to NeMo format. Args: output_path: Path where the converted model will be saved Returns: Path: Path to the saved NeMo model r) AutoConfigAutoModelForCausalLM model_typellama4)Llama4ForConditionalGenerationauto torch_dtypez.Converted Llama model to Nemo, model saved to z in .) transformersrrfrom_pretrainedrAgetattrrlanguage_modelr nemo_setup convert_state nemo_saveprintdtyper) rqrrr hf_configrsourcetargettrainerrHrHrIapplys      zHFLlamaImporter.applycCsdddddd}t|jddr|d =tjd d tjd g}d t|jdvrN||}dddddddd}|||tjddtj d tjddtj d gn|ddd| tjddtj d tj ||||dS)aJConvert state dict from HF format to NeMo format. Maps the weights from the HF model to the NeMo model according to the appropriate mapping scheme. Args: source: Source HF model target: Target NeMo model Returns: The result of applying the transforms embedding.word_embeddings.weight2decoder.layers.*.self_attention.linear_proj.weight.make_vocab_size_divisible_byr rope_typellama3rkrc)rdrr)rrrrrT)rrFinterleave_moe_layer_steprrrMrNrQintermediate_size_mlprOr^r_rPr2r}r-rr6rfp16bf16 params_dtypegeneration_configr kv_channelshead_dimrH)"rrr$rrA Exceptionrrgetrrbr%r text_confignum_experts_per_tokr use_qk_normintermediate_sizernum_hidden_layersr)rNr*rOinitializer_range rms_norm_epsnum_key_value_headsmax_position_embeddings rope_thetarrrfloat16bfloat16) rqrr$rr.rclsargspattern num_patternsoutputrHrHrIrs                      zHFLlamaImporter.configN)rir()r=r>r?r@rrrrrpropertyrrrrrrr%rrHrHrHrIrs !f  $rr&c@seZdZdZejfdddZdedefddZdd d Z e dd dZ e dddZ ddZ ddZdedeeeffddZddZd S)HFLlamaExporterzExporter for converting NeMo Llama models to Hugging Face format. This class handles the conversion of NeMo's LlamaModel to Hugging Face's LlamaForCausalLM format, including weight mapping and configuration translation. rir&cCsRddlm}ddlm}||j|j|dWdS1s"wYdS)zInitialize a HF LlamaForCausalLM instance. Args: dtype: Data type for model parameters Returns: LlamaForCausalLM: Initialized HF Llama model r)r)no_init_weightsrN)rrtransformers.modeling_utilsrF from_configr)rqrrrFrHrHrIrs $zHFLlamaExporter.initrcCs|r ||\}}n |t|\}}|j}|t|}||||}|}|jj r?| }| d|j ||dn| |z |j j |W|Sty]tdY|Sw)zApply the conversion from NeMo to HF format. Args: output_path: Path where the converted model will be saved Returns: Path: Path to the saved HF model r)rzFailed to save tokenizer) is_llama4 ckpt_load nemo_loadrArrrrcpurrrsave_pretrainedrrr1r warning)rqrr source_config_rrrHrHrIr!s&    zHFLlamaExporter.applyNcCs|}|r|dus J|||}dddddd}tjdd tjd tjd d tjd tjd dtjd g}|jj sD| tjddtjd |rk| d| dddddd| tjddtjd tjd dtjd gtj||||dS)aConvert state dict from NeMo format to HF format. Maps the weights from the NeMo model to the HF model according to the appropriate mapping scheme. Args: source: Source NeMo model target: Target HF model source_config: Source NeMo config (optional, used for Llama4) Returns: The target model with weights transferred from source Nrrrrr)rrrrrrrrrrrrrrrrrz-model.layers.*.feed_forward.experts.down_projz0model.layers.*.feed_forward.experts.gate_up_proj)rrrz.decoder.layers.*.mlp.experts.linear_fc2.weightz.decoder.layers.*.mlp.experts.linear_fc1.weightrrrr)rIrrrr split_qkv split_fc1 prune_paddingrrr rrr r )rqrrrOrIrrrHrHrIrCs~      zHFLlamaExporter.convert_stater)cCstjt|ddjS)zzGet the tokenizer from the NeMo model. Returns: TokenizerSpec: Tokenizer from the NeMo model rvsubpath)r load_contextrArrrrHrHrIrrszHFLlamaExporter.tokenizer HFLlamaConfigcCsVtjt|dd}t|tr||Sddlm}d}t|tr,|j |j |j |j dd}|didd gd |j d |jd |jd |jd|jdurL|jn2|j|jd|jd|jd|jd|jd|jd|jjd|jd|d|jjd|jjSd|jd|jd|jd|jd|jd|jjd|jd|d|jjd|jjS)zCreate a HF LlamaConfig from the NeMo model config. Translates the NeMo configuration parameters to the equivalent HF configuration. Returns: HFLlamaConfig: HF configuration for Llama models model.configrTrr$Nr(rkrerg original_max_position_embeddingsr' architecturesr&r7rNr6rOr0r;r8r9r:r<rrr bos_token_id eos_token_idrH)rrVrArrcreate_llama4_configrr%rbrdrergrhrMrNrQrOr/r2r^r_rPr}rrrr6bos_ideos_id)rqrrWrrHrHrIrs            zHFLlamaExporter.configcCstjt|dd}t|tS)z(Check if the model config is for Llama4.rXrT)rrVrArr)rqrrHrHrIrIs zHFLlamaExporter.is_llama4cCs@ddlm}t|ts J|jr|jdddddnd}t|d d}d}|durCt|tr/|}nt|tr<|j t |}nt d ||d id |j d |j d |j d|jd|jd|jd|jd|jd|jd|jd|jd|jd|jd|jd|jd|d|d|jdgdd|jjd|jj}|S)!z8Create a HF Llama4TextConfig from the NeMo Llama4Config.r)Llama4TextConfigr`rfrYr(rYNrzUnexpected moe_layer_freq r0r7rNr6r*rOr4rr;r8r9r:r5r<rrr) pad_token_idz<|finetune_right_pad|>r\r]rH)rrarrrrrrFrrMsum ValueErrorr/rNrrQrOrrr2r^r_rPrr}rrr tokens_to_idsr_r`)rqrHFLlama4TextConfigrrr)rrHrHrIr^s|          z$HFLlamaExporter.create_llama4_configpathc s|dd}|stdt|d }t|}Wdn1s#wY|d}i}fdd|d }|j}t|d } | D]7\} } d | vrOqF| d d } d| vry| d|kryt | dD]} | | || ddt | <qg| || <qF||fS)a This function loads the state dict directly from a distributed checkpoint, and modify the state dict so that it is consistent with the key names you would get from loading the checkpoint into a model. This is a more memory-efficient method to obtain a state dict without initializing the nemo model. Args: path (Path): The path from which the model will be loaded. Returns ------- Tuple[Dict, Any]: The loaded state dict and the yaml config object. contextz model.yamlz@model.yaml is not found in the context folder of the checkpoint.rNr!cs,t|trtddfdd|DS|S)NrrHcsi|] \}}||qSrHrH).0kkvv dict_to_objrHrI 4sz?HFLlamaExporter.ckpt_load....)rdicttypeitems)drmrHrIr3s,z+HFLlamaExporter.ckpt_load..rT _extra_statezmodule.layersrzlayers.) existsFileNotFoundErroropenyaml safe_loadrMrrrreplacesizerrA)rqrg model_yamlstreamrdist_ckpt_folderr config_objlangauge_layersdistributed_model_weightskvnew_kirHrmrIrJs*         zHFLlamaExporter.ckpt_loadcCst|jD]g}d}t|jtrt|j|jksJ|j|}|rQ|d|d}|ddd|d|d<|d|d}|ddd|d|d <qd|d |vs[J|d|d }||d|d <qt ||}|S) z For MoE layer, we transpose the gate_up_proj and down_proj to match HF implementation. For dense layer, we change the name for the post attention layer norm to avoid the many-to-one mapping in the conversion confi. Tdecoder.layers.z&.mlp.experts.experts.linear_fc1.weightrr%rz.mlp.experts.linear_fc1.weightz&.mlp.experts.experts.linear_fc2.weightz.mlp.experts.linear_fc2.weightz!.mlp.linear_fc1.layer_norm_weightz.pre_mlp_layernorm.weight) rrMrrrrrpermute contiguousr)rqrrOrrr rrHrHrIrEs.   z+HFLlamaExporter._modify_llama4_source_state)rir&r)rir))rirW)r=r>r?r@rr>rrrrrDrrrrIr^r r rrJrrHrHrHrIrE s "c  26 (rEzhf-peftcsPeZdZdZejfdfdd ZdedefddZd d Z e dd d Z Z S)HFLlamaPEFTExporterzExporter for converting NeMo Llama models with PEFT adapters to Hugging Face format. This class extends HFLlamaExporter to handle Parameter-Efficient Fine-Tuning (PEFT) adapters, specifically LoRA and DoRA adapters. rir"csddlm}tj|d}tt|ddt}t|d}t |d}Wdn1s.wYd | d d d|_ |||j dd S) zInitialize a HF PEFT model. Args: dtype: Data type for model parameters Returns: AutoPeftModelForCausalLM: Initialized HF PEFT model r)get_peft_model)rF) is_savingrimodel_ckpt_pathN/)autocast_adapter_dtype)peftrrlrrrArryjsonloadjoinsplit name_or_path peft_config)rqrrrvadapter_meta_pathfrrwrHrIrns  zHFLlamaPEFTExporter.initrcCspddlm}m}m}tjt|dd|_|t|\}}| t |j }| ||}| }|j|dd|S)zApply the conversion from NeMo PEFT model to HF format. Args: output_path: Path where the converted model will be saved Returns: Path: Path to the saved HF PEFT model r) CanonicalLoRADoRALoRAzmodel.model_transformrTF)save_embedding_layers)nemo.collections.llm.peftrrrrrVrApeft_objrKrrrrrLrM)rqrrrrrrPrrHrHrIrs  zHFLlamaPEFTExporter.applyc Cs<ddlm}d}d}d}d}d}d} |d |d |d |d |d |d |d |d |d|d |d |di} g} t|j|r| |d |d|d|d |d|d|d |d|d|d |d|d|d |d|d|d |d|d|d | d|d|d | d|d|d | d |d!|d | d"|d#i nY| tj|d$|d|d|dftj d%tj|d&|d|d|dftj d%tj|d'|d!|dftj d%tj|d(|d#|dftj d%gtj ||| | d)S)*aConvert state dict from NeMo PEFT model to HF PEFT format. Maps the weights from the NeMo model to the HF model according to the appropriate mapping scheme for PEFT adapters. Args: source: Source NeMo model with PEFT adapters target: Target HF model Returns: The target model with weights transferred from source r)rrzbase_model.model.model.layers.z"self_attention.linear_proj.adapterzmlp.linear_fc2.adapterz!self_attention.linear_qkv.adapterzmlp.linear_fc1.adapterz*.z.linear_in.weightz(*.self_attn.o_proj.lora_A.default.weightz.linear_out.weightz(*.self_attn.o_proj.lora_B.default.weightz%*.mlp.down_proj.lora_A.default.weightz%*.mlp.down_proj.lora_B.default.weightz.adapter_q.linear_in.weightz(*.self_attn.q_proj.lora_A.default.weightz.adapter_q.linear_out.weightz(*.self_attn.q_proj.lora_B.default.weightz.adapter_k.linear_in.weightz(*.self_attn.k_proj.lora_A.default.weightz.adapter_k.linear_out.weightz(*.self_attn.k_proj.lora_B.default.weightz.adapter_v.linear_in.weightz(*.self_attn.v_proj.lora_A.default.weightz.adapter_v.linear_out.weightz(*.self_attn.v_proj.lora_B.default.weightz.adapter_up.linear_in.weightz#*.mlp.up_proj.lora_A.default.weightz.adapter_up.linear_out.weightz#*.mlp.up_proj.lora_B.default.weightz.adapter_gate.linear_in.weightz%*.mlp.gate_proj.lora_A.default.weightz.adapter_gate.linear_out.weightz%*.mlp.gate_proj.lora_B.default.weightz4*.self_attention.linear_qkv.adapter.linear_in.weightrz5*.self_attention.linear_qkv.adapter.linear_out.weightz)*.mlp.linear_fc1.adapter.linear_in.weightz**.mlp.linear_fc1.adapter.linear_out.weightr)rrrrrr rrr duplicate3rQ duplicate2rRr ) rqrrrpnphp_projp_fc2p_qkvp_fc1rrrHrHrIrs    )z!HFLlamaPEFTExporter.convert_stater#c Csddlm}ddlm}|jjr|jjdksJddgdgdggd d gd gd gd d gd gd }g}|jjD] }|||q7||jj ||jj |jjt |j|dS)zCreate a PEFT config for the HF model. Translates the NeMo PEFT configuration to the equivalent HF PEFT configuration. Returns: PeftConfig: HF PEFT configuration r) LoraConfig)rprez5LoRA dropout_position must be 'pre' to convert to HF.q_projk_projv_proj)rrro_projup_proj gate_proj down_proj) linear_qlinear_klinear_v linear_qkv linear_proj linear_fc1_uplinear_fc1_gate linear_fc1 linear_fc2)ritarget_modules lora_alpha lora_dropoutuse_dora) rrrrrdropoutdropout_positionrr ralphar)rqrrNEMO2HFhf_target_modulestmrHrHrIrs6   zHFLlamaPEFTExporter.peft_config)rir")rir#) r=r>r?r@rr>rrrrrDrryrHrHrwrIrfsgrrcr`rfrYrkrergrhc Cstd|d|d|d|d ||}||}dtj|}t||k|||}|||||} d| ||| |} ||k||k} t| | |}|S)aZApply RoPE scaling for extending context length in Llama models. This implements the NTK-aware RoPE scaling method used in Llama 3.1 models to extend context length beyond the original training length. Args: inv_freq: Original inverse frequency tensor factor: Scaling factor for context length extension low_freq_factor: Factor for low frequency components high_freq_factor: Factor for high frequency components old_context_len: Original context length Returns: torch.Tensor: Modified inverse frequency tensor for extended context zApply rope scaling with factor=z, low_freq_factor=z, high_freq_factor=z, old_context_len=rr%r)r infomathpirwhere) rprkrergrhlow_freq_wavelenhigh_freq_wavelenwaveleninv_freq_llama smooth_factorsmoothed_inv_freqis_medium_freqrHrHrIrn+s rnctxtensorcCs"|jj}|j}tj||dd}|S)a! Split interleave-concatenated MoE expert weights. Args: ctx: Transformation context containing model configuration. tensor: The tensor containing concatenated expert weights. Returns: A list of tensors, each corresponding to an expert's weights. rr)rrrrr)rrmegatron_configrexpert_tensorsrHrHrI split_moeVs r)r%rKrTrWr{rrrrrrrrrrrrrrr)rcr`rfrY)frr dataclassesrr functoolsrpathlibrtypingrrrr r r r r rrtorch.nn.functionalr functionalrCrz#nemo.collections.llm.gpt.model.baserrr+nemo.collections.llm.gpt.model.llama4_utilsrnemo.collections.llm.utilsr.nemo.export.trt_llm.nemo_ckpt_loader.nemo_filernemo.lightningrrrnemo.lightning.ckpt_utilsrnemo.lightning.io.plrnemo.lightning.io.staterrrnemo.lightning.pytorch.utilsr nemo.utilsr $megatron.core.transformer.spec_utilsr!HAVE_TE ImportError"megatron.core.models.gpt.gpt_modelrjrr"r#rr%rWr&rr(1nemo.collections.common.tokenizers.tokenizer_specr)rKrTrWr]rbr{rrrrrrrrrrrrrrrmodel_importerModelConnectorrmodel_exporterrErrGrFrn staticmethodTensorr__all__rHrHrHrIs  ,               &   '     ]G +