o wi?@sddlmZddlmZddlmZmZmZmZm Z ddl Z ddl m m Zddl m Z ddlmZmZddlmZddlmZmZmZdd lmZdd lmZerkdd lmZdd lmZdd l m!Z!ddl"m#Z#eGdddeZeGdddeZ$eGdddeZ%eGdddeZ&GdddeZ'e(e'dGdddej)de'fZ*e+e'dGdddej)e'dfZ,gdZ-dS) ) dataclass)Path) TYPE_CHECKING AnnotatedCallableListOptionalN)nn) GPTConfigGPTModel)Config)OptimizerModuleioteardown) TransformFns) dtype_from_hfStarcoder2ConfigStarcoder2ForCausalLM AutoTokenizer) TokenizerSpecc@seZdZUdZdZeed<ejZ e ed<dZ e ed<dZ eed<d Zeed <d Zeed <d Zeed<d Zeed<dZeed<dZe ed<dZeed<dZeed<dZeeeed<dZe ed<dZe ed<dZe ed<dZeed<dS)rzS Configuration class for the Starcoder2 Config, inheriting from GPTConfig. LayerNorm normalizationactivation_funcTadd_bias_lineari@ seq_lengthropeposition_embedding_typeg?rotary_percentghidden_dropoutattention_dropoutg{Gz?init_method_stdF#share_embeddings_and_output_weightsN kv_channelsnum_query_groups window_sizeattention_softmax_in_fp32bias_activation_fusionbias_dropout_fusiongh㈵>layernorm_epsilon)__name__ __module__ __qualname____doc__rstr__annotations__Fgelurrrboolrintrr floatr!r"r#r$r%r&r'rrr(r)r*r+r7r7f/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/starcoder2.pyr%s&               rc@feZdZUdZdZeed<dZeed<dZeed<dZ eed <d Z eed <d Z e ed <dZ e ed<dS)Starcoder2Config3Bz] Configuration class for the Starcoder2 3B Config, inheriting from Starcoder2Config.  num_layersi hidden_sizei0ffn_hidden_sizer&num_attention_headsVy?r#gR~.A rotary_baseNr,r-r.r/r<r5r1r=r>r&rAr#r6rCr7r7r7r8r:?       r:c@r9)Starcoder2Config7Bz] Configuration class for the Starcoder2 7B Config, inheriting from Starcoder2Config. r<ir=iHr>r&$rArBr#i@BrCNrDr7r7r7r8rFNrErFc@r9)Starcoder2Config15Bz^ Configuration class for the Starcoder2 15B Config, inheriting from Starcoder2Config. (r<ir=i`r>rHr&0rAg&1?r#irCNrDr7r7r7r8rJ]rErJc sfeZdZdZ    d deeeeefdeededdee e j ge j fffdd Z Z S) Starcoder2Modelz Starcoder2 model implementation based on the GPT model architecture. This class provides a high-level interface for Starcoder2 models, implementing the specific architecture and settings needed for Starcoder2 models. Nconfigoptim tokenizerrmodel_transformcstj|pt|||ddS)N)rOrPrQ)super__init__r)selfrNrOrPrQ __class__r7r8rSts zStarcoder2Model.__init__)NNNN)r,r-r.r/rrrr r rr ModulerS __classcell__r7r7rUr8rMls rMhfc@sXeZdZdZdefddZdedefddZdd Ze dd d Z e de fd dZ dS)HFStarcoder2Importerz Importer for converting Hugging Face Starcoder2 models to NeMo format. This class handles the conversion of Hugging Face's Starcoder2ForCausalLM models to NeMo's Starcoder2 format, including weight mapping and configuration translation. returncCst|j|jdS)z Initialize a NeMo Starcoder2Model instance. Returns: Starcoder2Model: Initialized NeMo Starcoder2 model with the appropriate configuration and tokenizer. )rP)rMrNrPrTr7r7r8initszHFStarcoder2Importer.init output_pathcCshddlm}|jt|dd}|}||}||||||td|t ||~~|S)z Apply the conversion from HF to NeMo format. Args: output_path: Path where the converted model will be saved Returns: Path: Path to the saved NeMo model rrauto) torch_dtypez3Converted Starcoder2 model to Nemo, model saved to ) transformersrfrom_pretrainedr0r] nemo_setup convert_state nemo_saveprintr)rTr^rsourcetargettrainerr7r7r8applys     zHFStarcoder2Importer.applycCsXddddddddd d d d d dd}tjddtjdtjddtjdg}tj||||dS)aS Convert state dict from HF format to NeMo format. Maps the weights from the HF model to the NeMo model according to the appropriate mapping scheme. Args: source: Source HF model target: Target NeMo model Returns: The result of applying the transforms embedding.word_embeddings.weight2decoder.layers.*.self_attention.linear_proj.weight0decoder.layers.*.self_attention.linear_proj.bias&decoder.layers.*.mlp.linear_fc1.weight$decoder.layers.*.mlp.linear_fc1.bias&decoder.layers.*.mlp.linear_fc2.weight$decoder.layers.*.mlp.linear_fc2.bias.make_vocab_size_divisible_byF)r<r=r>rAr#rr+r&rCrr$fp16bf16 params_dtype)rarrbr0num_hidden_layersr=intermediate_sizerAinitializer_rangemax_position_embeddings norm_epsilonnum_key_value_heads rope_thetarrtorchfloat16bfloat16)rTHFStarcoder2Configrgroutputr7r7r8rNs(   zHFStarcoder2Importer.configN)r[r) r,r-r.r/rMr]rrjrdpropertyrPrrNr7r7r7r8rZs 5  rZrc@sNeZdZdZdddZdedefddZd d Zed d Z edddZ dS)HFStarcoder2Exporterz Exporter for converting NeMo Starcoder2Model to Hugging Face format. This class handles the conversion of NeMo's Starcoder2Model to Hugging Face's Starcoder2ForCausalLM format, including weight mapping and configuration translation. r[rcCsNddlm}ddlm}|||jWdS1s wYdS)z Initialize a HF Starcoder2ForCausalLM instance. Args: dtype: Data type for model parameters Returns: Starcoder2ForCausalLM: Initialized HF Starcoder2 model rr)no_init_weightsN)rartransformers.modeling_utilsr _from_configrN)rTrrr7r7r8r]s  $zHFStarcoder2Exporter.initr^cCsH|}|t|\}}|||}|}|||j||S)N)r] nemo_loadr0rdcpusave_pretrainedrP)rTr^rhrg_r7r7r8rj/s   zHFStarcoder2Exporter.applyc Cstddddddddd d d d d }tjddtjdtjddtjdtjddtjdtjddtjdg}tj||||dS)ac Convert state dict from NeMo format to HF format. Maps the weights from the NeMo model to the HF model according to the appropriate mapping scheme. Args: source: Source NeMo model target: Target HF model Returns: The target model with weights transferred from source rzr{r|r}r~rrrrrrr) rlrmrnrorprqrrrsrtrurvrwrrrrrrkryrxrr)rrr split_qkvsplit_qkv_bias prune_paddingrrr7r7r8rd:sP  z"HFStarcoder2Exporter.convert_statecCstt|jjjS)z Get the tokenizer from the NeMo model. Returns: TokenizerSpec: Tokenizer from the NeMo model )r load_contextr0modelrPr\r7r7r8rP|szHFStarcoder2Exporter.tokenizerrcCsvddlm}tjt|dd}|dg|j|j|j|j|j dur"|j n|j|j|j |j |j |j |j|j|j|jjdS)zCreate a HF HFStarcoder2Config from the NeMo model config. Translates the NeMo configuration parameters to the equivalent HF configuration. Returns: HFStarcoder2Config: HF configuration for Starcoder2 models rrz model.config)subpathrN) architecturesrr=rrAhead_dimtie_word_embeddingsrrnorm_epsrrpartial_rotary_factorr)rarrrr0r<r=r>rAr%r$rr#r+r&rCr rPr)rTrrgr7r7r8rNs(   zHFStarcoder2Exporter.configN)r[r)r[r) r,r-r.r/r]rrjrdrrPrNr7r7r7r8rs  B  r)rr:rFrJrM). dataclassesrpathlibrtypingrrrrrrtorch.nn.functionalr functionalr2#nemo.collections.llm.gpt.model.baser r nemo.collections.llm.utilsr nemo.lightningr rrnemo.lightning.io.staternemo.lightning.pytorch.utilsrrarrrrr1nemo.collections.common.tokenizers.tokenizer_specrr:rFrJrMmodel_importerModelConnectorrZmodel_exporterr__all__r7r7r7r8s@