o }oi5@sfddlmZddlmZddlmZmZmZmZddl Z ddl m m Z ddl m Z ddlmZmZmZddlmZddlmZmZmZdd lmZdd lmZerkdd lmZdd lmZdd l m!Z!ddl"m#Z#eGdddeZ$eGddde$Z%GdddeZ&e'e&dGdddej(de&fZ)e*e&dGdddej(e&dfZ+gdZ,dS)) dataclass)Path) TYPE_CHECKING AnnotatedCallableOptionalN)nn) GPTConfigGPTModeltorch_dtype_from_mcore_config)Config)OptimizerModuleioteardown) TransformFns) dtype_from_hfGPTBigCodeConfigGPTBigCodeForCausalLM AutoTokenizer) TokenizerSpecc@seZdZUdZdZeed<ejZ e ed<dZ e ed<dZ eed<d Zeed <d Zeed <d Zeed <dZeed<dZeed<dZe ed<dZeed<dZeed<dZe ed<dZe ed<dZe ed<dS)StarcoderConfigzR Configuration class for the Starcoder Config, inheriting from GPTConfig. LayerNorm normalizationactivation_funcTadd_bias_lineari seq_lengthlearned_absoluteposition_embedding_typeg?hidden_dropoutattention_dropoutg{Gz?init_method_stdgh㈵>layernorm_epsilonF#share_embeddings_and_output_weightsN kv_channelsnum_query_groupsattention_softmax_in_fp32bias_activation_fusionbias_dropout_fusion)__name__ __module__ __qualname____doc__rstr__annotations__Fgelurrrboolrintr r!floatr"r#r$r%r&r(r)r*r+r7r7\/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/starcoder.pyr%s"              rc@sNeZdZUdZdZeed<dZeed<dZeed<dZ eed <d Z e ed <d S) StarcoderConfig15Bz\ Configuration class for the Starcoder 15B Config, inheriting from StarcoderConfig. ( num_layersi hidden_sizei`ffn_hidden_size0num_attention_headsg{Gz?r#N) r,r-r.r/r;r5r1r<r=r?r#r6r7r7r7r8r9=s     r9c sfeZdZdZ    d deeeeefdeededdee e j ge j fffdd Z Z S) StarcoderModelz Starcoder model implementation based on the GPT model architecture. This class provides a high-level interface for Starcoder models, implementing the specific architecture and settings needed for Starcoder models. Nconfigoptim tokenizerrmodel_transformcstj|pt|||ddS)N)rBrCrD)super__init__r)selfrArBrCrD __class__r7r8rFRs zStarcoderModel.__init__)NNNN)r,r-r.r/rrrr r rrModulerF __classcell__r7r7rHr8r@Js r@hfc@sXeZdZdZdefddZdedefddZdd Ze dd d Z e de fd dZ dS)HFStarcoderImporterz Importer for converting Hugging Face Starcoder models to NeMo format. This class handles the conversion of Hugging Face's GPTBigCodeForCausalLM models to NeMo's Starcoder format, including weight mapping and configuration translation. returncCst|j|jdS)z Initialize a NeMo StarcoderModel instance. Returns: StarcoderModel: Initialized NeMo Starcoder model with the appropriate configuration and tokenizer. )rC)r@rArCrGr7r7r8initgszHFStarcoderImporter.init output_pathcCshddlm}|jt|dd}|}||}||||||td|t ||~~|S)z Apply the conversion from HF to NeMo format. Args: output_path: Path where the converted model will be saved Returns: Path: Path to the saved NeMo model rrauto torch_dtypez2Converted Starcoder model to Nemo, model saved to ) transformersrfrom_pretrainedr0rP nemo_setup convert_state nemo_saveprintr)rGrQrsourcetargettrainerr7r7r8applyqs     zHFStarcoderImporter.applycCsziddddddddd d d d d ddddddddddddddddddd d!d"}tj|||d#S)$aS Convert state dict from HF format to NeMo format. Maps the weights from the HF model to the NeMo model according to the appropriate mapping scheme. Args: source: Source HF model target: Target NeMo model Returns: The result of applying the transforms transformer.wte.weight embedding.word_embeddings.weighttransformer.wpe.weight$embedding.position_embeddings.weight"transformer.h.*.attn.c_proj.weight2decoder.layers.*.self_attention.linear_proj.weight transformer.h.*.attn.c_proj.bias0decoder.layers.*.self_attention.linear_proj.bias"transformer.h.*.attn.c_attn.weight1decoder.layers.*.self_attention.linear_qkv.weight transformer.h.*.attn.c_attn.bias/decoder.layers.*.self_attention.linear_qkv.biastransformer.h.*.mlp.c_fc.weight&decoder.layers.*.mlp.linear_fc1.weighttransformer.h.*.mlp.c_fc.bias$decoder.layers.*.mlp.linear_fc1.bias!transformer.h.*.mlp.c_proj.weight&decoder.layers.*.mlp.linear_fc2.weighttransformer.h.*.mlp.c_proj.bias$decoder.layers.*.mlp.linear_fc2.biastransformer.h.*.ln_1.weight.make_vocab_size_divisible_byr'F) r;r<r=r?r#rr$r(rr%fp16bf16 params_dtype)rUrrVr0rn_layern_embdn_innern_headinitializer_range n_positionslayer_norm_epsilonrrtorchfloat16bfloat16)rGHFStarcoderConfigr[routputr7r7r8rAs&   zHFStarcoderImporter.configN)rNr) r,r-r.r/r@rPrr^rXpropertyrCrrAr7r7r7r8rM^s $  rMrc@sTeZdZdZejfdddZdedefddZd d Z e d d Z e dddZ dS)HFStarcoderExporterz Exporter for converting NeMo StarcoderModel to Hugging Face format. This class handles the conversion of NeMo's StarcoderModel to Hugging Face's GPTBigCodeForCausalLM format, including weight mapping and configuration translation. rNrcCsRddlm}ddlm}||j|j|dWdS1s"wYdS)z Initialize a HF GPTBigCodeForCausalLM instance. Args: dtype: Data type for model parameters Returns: GPTBigCodeForCausalLM: Initialized HF Starcoder model rr)no_init_weightsrSN)rUrtransformers.modeling_utilsr _from_configrA)rGdtyperrr7r7r8rPs $zHFStarcoderExporter.initrQcCsP|t|\}}|t|j}|||}|}|||j||S)N) nemo_loadr0rPr rArXcpusave_pretrainedrC)rGrQr[_r\r7r7r8r^s   zHFStarcoderExporter.applycCsZddddddddd d d d d ddd}tjddtjdtjddtjdg}tj||||dS)ac Convert state dict from NeMo format to HF format. Maps the weights from the NeMo model to the HF model according to the appropriate mapping scheme. Args: source: Source NeMo model target: Target HF model Returns: The target model with weights transferred from source rarcrergrirkrmrorqrsrurwryr{r})rbrdrfrhrjrlrnrprrrtrvrxrzr|r~r`r_) source_key target_keyfnrr)r transforms)rstate_transformr prune_paddingr)rGr[r\rrr7r7r8rXs8 z!HFStarcoderExporter.convert_statecCstt|jjjS)z Get the tokenizer from the NeMo model. Returns: TokenizerSpec: Tokenizer from the NeMo model )r load_contextr0modelrCrOr7r7r8rC4szHFStarcoderExporter.tokenizerrcCsnddlm}tjt|dd}|dg|j|j|j|j|j dur"|j n|j|j|j |j |j |j |j|jjd S)zCreate a HF GPTBigCodeConfig from the NeMo model config. Translates the NeMo configuration parameters to the equivalent HF configuration. Returns: HFStarcoderConfig: HF configuration for Starcoder models rrz model.config)subpathrN) architecturesnum_hidden_layersr<intermediate_sizer?head_dimtie_word_embeddingsmax_position_embeddingsrnorm_epsnum_key_value_headsr)rUrrrr0r;r<r=r?r&r%rr#r$r(rCr)rGrr[r7r7r8rA>s$   zHFStarcoderExporter.configN)rNr)rNr) r,r-r.r/rrrPrr^rXrrCrAr7r7r7r8rs .  r)rr9r@)- dataclassesrpathlibrtypingrrrrrtorch.nn.functionalr functionalr2#nemo.collections.llm.gpt.model.baser r r nemo.collections.llm.utilsr nemo.lightningr rrnemo.lightning.io.staternemo.lightning.pytorch.utilsrrUrrrrr1nemo.collections.common.tokenizers.tokenizer_specrrr9r@model_importerModelConnectorrMmodel_exporterr__all__r7r7r7r8s6             }