o }oi|@s^ddlZddlmZddlmZddlmZddlmZm Z m Z m Z m Z ddl Z ddl mZddlmZmZmZddlmZmZmZmZmZdd lmZmZdd lmZdd lmZm Z m!Z!dd l"m#Z#dd l$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-e-d\Z.Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5erddl6m7Z7m8Z8ddl9mZ:ddl9m;Z;ddlm?Z?eGdddeZ@GdddeZAdd d!e5fd"d#ZBeGd$d%d%ee3ZCeGd&d'd'ee3ZDGd(d)d)eZEe FeEd*Gd+d,d,e jGd-eEfZHe IeEd*Gd.d/d/e jGeEd-fZJe IeEd0Gd1d2d2eJZKgd3ZLdS)4N) dataclass)partial)Path) TYPE_CHECKING AnnotatedCallableOptionalUnion)nn) GPTConfigGPTModeltorch_dtype_from_mcore_config) Llama31ConfigLlama31Config8BLlama31Config70BLlama31Config405B LlamaConfig)1LLAMA_31_NEMOTRON_ULTRA_253B_HETEROGENEOUS_CONFIG0LLAMA_33_NEMOTRON_SUPER_49B_HETEROGENEOUS_CONFIG)Config)OptimizerModuleioteardown)ADAPTER_META_FILENAME)ckpt_to_weights_subdir) TransformFns) dtype_from_hf)logging) safe_importtransformer_engine) get_gpt_heterogeneous_layer_spec)HeterogeneousTransformerConfig) ModuleSpec)AutoPeftModelForCausalLM PeftConfigr)LlamaForCausalLM AutoTokenizer) TokenizerSpecc@eZdZUdZdZeed<dS)Llama31NemotronNano8BConfigz1Configuration for an Llama31-Nemotron-Nano model. kv_channelsN__name__ __module__ __qualname____doc__r-int__annotations__r5r5a/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/llama_nemotron.pyr+;s r+c@r*)Llama31Nemotron70BConfigz0Configuration for an Llama31-Nemotron-70B model.r,r-Nr.r5r5r5r6r7Bs r7configr returncCs t|tdS)a%Determine the most appropriate layer specification based on availability. Uses Transformer Engine specs if available, otherwise falls back to local implementation. Args: config: GPT configuration object Returns: ModuleSpec: The selected module specification )use_te)r HAVE_TE)r8r5r5r6heterogeneous_layer_specIs r<c@leZdZUdZdZeed<dZeed<dZeed<dZ e ed <e Z e ed <e Zeeed geffed <dS) Llama33NemotronSuper49BConfigz2Configuration for an Llama31-Nemotron-Super model.i hidden_size@num_attention_headsP num_layersN heterogeneous_layers_config_path(heterogeneous_layers_config_encoded_jsonr transformer_layer_spec)r/r0r1r2r?r3r4rArCrDstrrrEr<rFr r"rr5r5r5r6r>W      "r>c@r=) Llama31NemotronUltra253BConfigz2Configuration for an Llama31-Nemotron-Ultra model.i@r?r,rArCNrDrEr rF)r/r0r1r2r?r3r4rArCrDrGrrEr<rFr r"rr5r5r5r6rIcrHrIc sfeZdZdZ    d deeeeefdeededdee e j ge j fffdd Z Z S) LlamaNemotronModelzLlama-Nemotron model implementation based on the GPT model architecture. This class provides a high-level interface for Llama-Nemotron models, implementing the specific architecture and settings needed for Llama-Nemotron models. Nr8optim tokenizerr)model_transformcstj|pt|||ddS)N)rLrMrN)super__init__r+)selfr8rLrMrN __class__r5r6rPvs zLlamaNemotronModel.__init__)NNNN)r/r0r1r2rrrrrrr ModulerP __classcell__r5r5rRr6rKosrKhfc@sXeZdZdZdefddZdedefddZdd Ze dd d Z e de fd dZ dS)HFLlamaNemotronImporteraImporter for converting Hugging Face Llama-Nemotron models to NeMo format. This class handles the conversion of Hugging Face's LlamaForCausalLM models to NeMo's LlamaNemotronModel format, including weight mapping and configuration translation. r9cCst|j|jdS)zInitialize a NeMo LlamaModel instance. Returns: LlamaModel: Initialized NeMo Llama model with the appropriate configuration and tokenizer. )rM)rKr8rMrQr5r5r6initszHFLlamaNemotronImporter.init output_pathcCsddlm}m}tdt|dt|vr"|jt|dd}n |jt|ddd}td |}||}| ||| ||t d |d |j d t ||~~|S) zApply the conversion from HF to NeMo format. Args: output_path: Path where the converted model will be saved Returns: Path: Path to the saved NeMo model r)AutoModelForCausalLMr&zLoad HF model Nanoauto torch_dtypeTtrust_remote_coder_z$Initialize NeMo Nemotron-Llama modelz7Converted Llama-Nemotron model to Nemo, model saved to z in .) transformersr[r&rinforGfrom_pretrainedrY nemo_setup convert_state nemo_saveprintdtyper)rQrZr[r&sourcetargettrainerr5r5r6applys       zHFLlamaNemotronImporter.applycCs^dddddddd}t|jd d r|d =tjd d tjdtjddtjdg}tj||||dS)aJConvert state dict from HF format to NeMo format. Maps the weights from the HF model to the NeMo model according to the appropriate mapping scheme. Args: source: Source HF model target: Target NeMo model Returns: The result of applying the transforms embedding.word_embeddings.weight2decoder.layers.*.self_attention.linear_proj.weight&decoder.layers.*.mlp.linear_fc2.weight.make_vocab_size_divisible_by rope_scalingz-Llama-Nemotron model should have rope scaling block_configsrBfactorg @)rErD scale_factornum_query_groups )rrCr?ffn_hidden_sizerAr-head_dimrinit_method_stdlayernorm_epsilon seq_length rotary_basegated_linear_unitr#share_embeddings_and_output_weightsr}Ffp16bf16 params_dtypegeneration_configr5) rcrrrerG Exceptionrnum_hidden_layersr>rIrto_json_stringrgetrAr attentionn_heads_in_groupr+r7num_key_value_headsr?intermediate_sizeinitializer_range rms_norm_epsmax_position_embeddings rope_thetarrtorchfloat16bfloat16) rQrrrkrr target_classclsoutputr5r5r6r8sr          zHFLlamaNemotronImporter.configN)r9r() r/r0r1r2rKrYrrnrgpropertyrMrr8r5r5r5r6rWs +  rWr&c@s\eZdZdZejddfdddZddedefd d Zd d Z e dddZ e dddZ dS)HFLlamaNemotronExporteraExporter for converting NeMo Llama-Nemotron models to Hugging Face format. This class handles the conversion of NeMo's LlamaNemotronModel to Hugging Face's LlamaForCausalLM format, including weight mapping and configuration translation. It supports both homogeneous (Nano/70B) and heterogeneous (Super/Ultra) model architectures. The exporter performs the following key operations: 1. Initializes a Hugging Face model with appropriate configuration 2. Maps weights from NeMo format to Hugging Face format 3. Handles special cases for heterogeneous architectures 4. Saves the converted model and tokenizer to the specified output path Attributes: tokenizer: The tokenizer associated with the NeMo model config: The configuration for the Hugging Face model Methods: init: Initialize a Hugging Face model instance apply: Convert and save the model to Hugging Face format convert_state: Convert model weights from NeMo to Hugging Face format FNr9r&c Csddlm}m}ddlm}|7|r#|j|j|dWdS|dus)J|j|dd}|j|d|d}t| d |WdS1sKwYdS) aInitialize a Hugging Face LlamaForCausalLM model instance. This method creates a new Hugging Face model instance with the appropriate configuration and data type. It handles both homogeneous and heterogeneous model architectures. Args: dtype (torch.dtype, optional): Data type for model parameters. Defaults to torch.bfloat16. from_config (bool, optional): Whether to initialize from config or load from pretrained. Set to True for homogeneous models (Nano/70B), False for heterogeneous models (Super/Ultra). Defaults to False. model_name (str, optional): Name of the pretrained model to load for heterogeneous architectures. Required when from_config is False. Defaults to None. Returns: LlamaForCausalLM: Initialized Hugging Face Llama model instance Raises: AssertionError: If model_name is not provided for heterogeneous models r)rr[)no_init_weightsr^NTrr`r[) rcrr[transformers.modeling_utilsr from_configr8retyperegister_for_auto_class) rQrjr model_namerr[rr8hf_modelr5r5r6rYCs   $zHFLlamaNemotronExporter.initrZcCstd|t|\}}t|jt}|dur0|r0|jj}|dkr%d}n |dkr,d}ntd|j t |j| |d}| ||}| }| ||jj ||S) aConvert and save a NeMo Llama-Nemotron model to Hugging Face format. This method performs the complete conversion process: 1. Loads the NeMo model checkpoint 2. Determines the appropriate target model configuration 3. Initializes the Hugging Face model 4. Converts and transfers the weights 5. Saves the converted model and tokenizer Args: output_path (Path): Directory path where the converted model will be saved target_model_name (str, optional): Name of the target Hugging Face model. Required for heterogeneous models (Super/Ultra). For homogeneous models, this is determined automatically. Defaults to None. Returns: Path: Path to the saved Hugging Face model directory Raises: ValueError: If the target model is not supported or if target_model_name is missing for heterogeneous models z(Loading Llama-Nemotron NeMo checkpoint..NrB&nvidia/Llama-3_3-Nemotron-Super-49B-v1rJ'nvidia/Llama-3_1-Nemotron-Ultra-253B-v1^Unknown target model. Currently only support exporting Llama-Nemotron Nano/Super/Ultra models.rr)rrd nemo_loadrG isinstancer8r!rC ValueErrorrYr rgcpusave_pretrainedrM)rQrZtarget_model_namerk_is_heterogeneousrCrlr5r5r6rnns.    zHFLlamaNemotronExporter.applycCsfdddddd}tjddtjd tjd d tjd tjd d tjd tjddtjd g}tj||||dS)aZConvert state dict from NeMo format to HF format. Maps the weights from the NeMo model to the HF model according to the appropriate mapping scheme. Args: source: Source NeMo model target: Target HF model Returns: The target model with weights transferred from source rwrxryrzr{)rprqrrrsrtrr~rrrrorvrur|r)rrr split_qkv split_fc1 prune_paddingrrr5r5r6rgsB  z%HFLlamaNemotronExporter.convert_stater)cCstjt|ddjS)zzGet the tokenizer from the NeMo model. Returns: TokenizerSpec: Tokenizer from the NeMo model modelsubpath)r load_contextrGrMrXr5r5r6rMsz!HFLlamaNemotronExporter.tokenizer HFLlamaConfigcCstjt|dd}t|trJddlm}d}t|tr)|j|j |j |j dd}||j |j |j|j|j|j|j|j|j|j|jj|j||jj|jjdS) aJCreate a HF LlamaConfig from the NeMo model config. This function should only be invoked for Non-heterogeneous transformers (i.e. Nano). Translates the NeMo configuration parameters to the equivalent HF configuration. Returns: HFLlamaConfig: HF configuration for Llama models z model.configrrr%Nllama3)rlow_freq_factorhigh_freq_factor original_max_position_embeddings rope_type)rr?rrArrrrrrrr}r bos_token_id eos_token_id)rrrGrr!rcrrrrrold_context_lenrCr?rrAr-rrrrrrMrrbos_ideos_id)rQrkrrr5r5r6r8s8   zHFLlamaNemotronExporter.config)r9r&N)r9r))r9r) r/r0r1r2rrrYrrnrgrrMr8r5r5r5r6r+s+57 rzhf-peftcsVeZdZdZejddfdfdd Zddedefd d Zd d Z e dddZ Z S)HFLlamaNemotronPEFTExporterzExporter for converting NeMotron Llama models with PEFT adapters to Hugging Face format. This class extends HFLlamaNemotronExporter to handle Parameter-Efficient Fine-Tuning (PEFT) adapters, specifically LoRA and DoRA adapters. FNr9r#c sddlm}tj|||d}tt|ddt}t|d}t |d}Wdn1s0wYd | d d d|_ |||j dd S) zInitialize a HF PEFT model. Args: dtype: Data type for model parameters Returns: AutoPeftModelForCausalLM: Initialized HF PEFT model r)get_peft_model)rjrrF) is_savingrmodel_ckpt_pathN/)autocast_adapter_dtype)peftrrOrYrrGropenjsonloadjoinsplit name_or_path peft_config) rQrjrrrradapter_meta_pathfrrRr5r6rYs  z HFLlamaNemotronPEFTExporter.initrZc Csddlm}m}m}tjt|dd|_|t|\}}t |j t }|dur?|r?|j j } | dkr4d}n | dkr;d }nt d |jt|j | |d } ||| } | } | j|d d |S)zApply the conversion from NeMo PEFT model to HF format. Args: output_path: Path where the converted model will be saved Returns: Path: Path to the saved HF PEFT model r) CanonicalLoRADoRALoRAzmodel.model_transformrNrBrrJrrrF)save_embedding_layers)nemo.collections.llm.peftrrrrrrGpeft_objrrr8r!rCrrYr rgrr) rQrZrrrrrkrrrCrlr5r5r6rn.s.   z!HFLlamaNemotronPEFTExporter.applyc Cs<ddlm}d}d}d}d}d}d} |d |d |d |d |d |d |d |d |d|d |d |di} g} t|j|r| |d |d|d|d |d|d|d |d|d|d |d|d|d |d|d|d |d|d|d | d|d|d | d|d|d | d |d!|d | d"|d#i nY| tj|d$|d|d|dftj d%tj|d&|d|d|dftj d%tj|d'|d!|dftj d%tj|d(|d#|dftj d%gtj ||| | d)S)*aConvert state dict from NeMo PEFT model to HF PEFT format. Maps the weights from the NeMo model to the HF model according to the appropriate mapping scheme for PEFT adapters. Args: source: Source NeMo model with PEFT adapters target: Target HF model Returns: The target model with weights transferred from source r)rzdecoder.layers.zbase_model.model.model.layers.z"self_attention.linear_proj.adapterzmlp.linear_fc2.adapterz!self_attention.linear_qkv.adapterzmlp.linear_fc1.adapterz*.z.linear_in.weightz(*.self_attn.o_proj.lora_A.default.weightz.linear_out.weightz(*.self_attn.o_proj.lora_B.default.weightz%*.mlp.down_proj.lora_A.default.weightz%*.mlp.down_proj.lora_B.default.weightz.adapter_q.linear_in.weightz(*.self_attn.q_proj.lora_A.default.weightz.adapter_q.linear_out.weightz(*.self_attn.q_proj.lora_B.default.weightz.adapter_k.linear_in.weightz(*.self_attn.k_proj.lora_A.default.weightz.adapter_k.linear_out.weightz(*.self_attn.k_proj.lora_B.default.weightz.adapter_v.linear_in.weightz(*.self_attn.v_proj.lora_A.default.weightz.adapter_v.linear_out.weightz(*.self_attn.v_proj.lora_B.default.weightz.adapter_up.linear_in.weightz#*.mlp.up_proj.lora_A.default.weightz.adapter_up.linear_out.weightz#*.mlp.up_proj.lora_B.default.weightz.adapter_gate.linear_in.weightz%*.mlp.gate_proj.lora_A.default.weightz.adapter_gate.linear_out.weightz%*.mlp.gate_proj.lora_B.default.weightz4*.self_attention.linear_qkv.adapter.linear_in.weightrz5*.self_attention.linear_qkv.adapter.linear_out.weightz)*.mlp.linear_fc1.adapter.linear_in.weightz**.mlp.linear_fc1.adapter.linear_out.weightr)rrrrupdateextendrrr duplicate3r duplicate2rr) rQrkrlrpnphp_projp_fc2p_qkvp_fc1rrr5r5r6rgVs    )z)HFLlamaNemotronPEFTExporter.convert_stater$c Csddlm}ddlm}|jjr|jjdksJddgdgdggd d gd gd gd d gd gd }g}|jjD] }|||q7||jj ||jj |jjt |j|dS)zCreate a PEFT config for the HF model. Translates the NeMo PEFT configuration to the equivalent HF PEFT configuration. Returns: PeftConfig: HF PEFT configuration r) LoraConfig)rprez5LoRA dropout_position must be 'pre' to convert to HF.q_projk_projv_proj)rr r o_projup_proj gate_proj down_proj) linear_qlinear_klinear_v linear_qkv linear_proj linear_fc1_uplinear_fc1_gate linear_fc1 linear_fc2)rtarget_modules lora_alpha lora_dropoutuse_dora) rrrrrdropoutdropout_positionrrdimalphar)rQrrNEMO2HFhf_target_modulestmr5r5r6rs6   z'HFLlamaNemotronPEFTExporter.peft_config)r9r#r)r9r$) r/r0r1r2rrrYrrnrgrrrUr5r5rRr6rs(gr)rKr+r>rIr7)Mr dataclassesr functoolsrpathlibrtypingrrrrr rr #nemo.collections.llm.gpt.model.baser r r $nemo.collections.llm.gpt.model.llamarrrrr4nemo.collections.llm.gpt.model.llama_nemotron_configrrnemo.collections.llm.utilsrnemo.lightningrrrnemo.lightning.ckpt_utilsrnemo.lightning.io.plrnemo.lightning.io.staternemo.lightning.pytorch.utilsr nemo.utilsrnemo.utils.import_utilsrrr;@megatron.core.models.gpt.heterogeneous.heterogeneous_layer_specsr rIrKmodel_importerModelConnectorrWmodel_exporterrr__all__r5r5r5r6s^                      ) f X