o }oic@s\ddlmZddlmZddlmZmZmZmZm Z m Z m Z m Z ddl Z ddlZddlZddlmZddlmZddlmZmZmZddlmZdd lmZdd lmZdd lmZm Z dd l!m"Z"m#Z#dd l$m%Z%ddl$m&Z'ddl(m)Z)ddl*m+Z+m,Z,ddl-m.Z.ddl/m0Z0m1Z1ddl2m3Z3ddl4m5Z5erddl6m Z7ddl8m9Z9ddl:m;Z;ddlZ>dee?ej@ffddZAdejBdee?ej@fdej@fdd ZCeGd!d"d"ZDeGd#d$d$e#eDZEGd%d&d&eEZFGd'd(d(e ZGe,HeGd)Gd*d+d+e,jId,eGfZJe,KeGd)Gd-d.d.e,jIeGd,fZLGd/d0d0e1ZMd1d2ZNdS)3) dataclass)Path) TYPE_CHECKING AnnotatedCallableDictLiteralOptionalTupleUnionNparallel_state)ColumnParallelLinear) Float16Modulefloat16_to_fp32fp32_to_float16) ModuleSpec)get_batch_on_this_cp_rank)nn) GPTConfigGPTModel)HFLlamaImporterLlama32Config1B)LlamaEmbeddingExporter)get_nv_embedding_layer_spec)Config)OptimizerModuleio) TransformFns)DDPMegatronLossReduction) dtype_from_hf)logging)r"AutoModelForSequenceClassification AutoTokenizer) TokenizerSpec) LlamaConfigreturncszt|}t|trt|dkr|d}n|}tdtr,ddfdd|D}t |}|S)z Setup Reranker dataloader batch.rattention_mask input_ids position_idscs*i|]\}}||vr|jddndqS)T) non_blockingN)cuda).0keyval required_keys[/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/reranker.py Cs*z&reranker_data_step..) next isinstancetuplelensetaddr is_pipeline_first_stageitemsr)dataloder_iterbatch_batchoutputr5r3r6reranker_data_step2s    rDmodelrAcCs,|d|d|dd}|jdi|}|S)a This subsets the batch keys to the ones actually used by forward pass of the model, and then calls the model's forward pass. if "cu_seqsens" are defined in the batch, then the packed sequence parameters are also passed to the model for forward pass efficiency. r,r+r-)r,r+r-Nr5)forward)rErA forward_argsscorer5r5r6reranker_forward_stepJs rIc@seZdZUdZdZeded<dZeed<dZ e ed<d Z e ed <d Z e ed <d Zeded<dZe ed<d Ze ed<dZeeded<dZe ed<dS)ReRankerBaseConfigz: Base config for Reranker Models Training configs right)leftrKtruncation_methodnum_hard_negatives2 ce_loss_scalelabel_smoothingFin_batch_negativesfirst)randomrUnegative_sample_strategyTadd_bosadd_eosavgclsrZlast weighted_avg pool_typeg? temperatureN)__name__ __module__ __qualname____doc__rMr__annotations__rOintrQfloatrSrTboolrWrXrYr_r r`r5r5r5r6rJYs       rJcsxeZdZUdZeZeeedgeffe d<e Z ee d<e Z ee d<eZeje d<eZeje d<d dfd d ZZS)Llama32Reranker1BConfigz"Config for Llama32Reranker1B modelrtransformer_layer_specforward_step_fn data_step_fn importer_cls exporter_clsNr) MCoreGPTModelcst||||}d|_|S)zConfigure the Reranker ModelF)superconfigure_model post_process)self tokenizer pre_processrrvp_stagerE __class__r5r6rqvsz'Llama32Reranker1BConfig.configure_model)NNN)r)ro)rarbrcrd"bidirectional_attention_layer_specrjr rrrerIrkrDrlrrmrModelConnectorrrnrq __classcell__r5r5rwr6rils   ric@seZdZUdZdZeed<dS)Llama32Reranker500MConfigz$Config for Llama32Reranker500M model num_layersN)rarbrcrdr~rfrer5r5r5r6r|s r|c seZdZdZ    d deeeeefdeededdee e j ge j fffdd Z e d d Zd!d eed dffdd ZddZ d!dejdejdejdeejffdd Ze ddZddZe ddZe ddZZS)" ReRankerModelzUBase model for Reranking that extends GPTModel with reranking-specific functionality.Nconfigoptimrtr'model_transformcstj|pt|||ddS)N)rrtr)rp__init__ri)rsrrrtrrwr5r6rs zReRankerModel.__init__cCs|jj|jj|jj|jjdS)z,Getter for dataset_kwargs from model config.)rOrWrXrY)rrOrWrXrYrsr5r5r6dataset_kwargss zReRankerModel.dataset_kwargsrvr)c s`|jjdvs|jjdusJd|jjdt|t|jjd|j|jjdddd|j_dS) zConfigure the underlying model if not already configured. This method ensures the model is instantiated from the configuration. r[NInvalid pool type: z4 should be in [cls, avg, last, weighted_avg] or NoneFT)r init_methodbias skip_bias_add gather_output) rr_rprqr hidden_sizermodulerH)rsrvrwr5r6rqs zReRankerModel.configure_modelc Cst|d}||dd}|jj}|dkr)|jdd|jddd}|S|dkr5|jdd}|S|dkrC|d d d f}|S|d kr}|d d d f|jd k}|rb|d d d f}|S|jddd}|jd }|tj ||j d |f}|St d|)aPool the hidden states based on the configured pooling strategy. Args: last_hidden_states: The hidden states from the transformer attention_mask: The attention mask for the input Returns: The pooled embeddings zs b h -> b s h).NrRrZrdimr^r\Nrr]devicer) einops rearrange masked_fillrhrr_sumshapetorcharanger ValueError) rslast_hidden_statesr+ last_hiddenr_emb left_paddingsequence_lengths batch_sizer5r5r6pools,     zReRankerModel.poolr,r-r+ decoder_inputc s|jdkr|dddk}n$|jdkr0|jddkr#|jddks'Jd|}|dk}ntdtj||||d}|||}|rU|j j }t ||j }d} nd } | |d } | |j j} | rjt| } | S) a2Forward pass of the reranker model. Args: input_ids: Input token IDs position_ids: Position IDs for the input attention_mask: Attention mask for the input decoder_input: Optional decoder input Returns: The pooled logits rg?rNzAttention mask shape incorrectzAttention_mask shape incorrect)r,r-r+rTFr)ndim unsqueezersqueezerrprFrhas_float16_module_wrapperrrfloat16_convertorrHrr`r) rsr,r-r+r extended_maskrCpooled_hidden_statesfloat16_moduleneed_to_convert_back pooled_logitsrwr5r6rFs0  $   zReRankerModel.forwardcCsPt|jdr |jjSt|jjdr|jjjSt|jjjds"Jd|jjjjS)z$Get the score module from the model.rHzScore module not found)hasattrrrHrr5r5r6rH s   zReRankerModel.scorecCs"t|jtrt|jjtrdSdS)z0Check if the model has a float16 module wrapper.TF)r9rrrrr5r5r6rsz(ReRankerModel.has_float16_module_wrappercC$|jstd|jj|jjd|_|jS)z'Get the training loss reduction module.Fvalidation_steprOrS)_training_loss_reduction ReRankerLossrrOrSrr5r5r6training_loss_reductionz%ReRankerModel.training_loss_reductioncCr)z)Get the validation loss reduction module.Tr)_validation_loss_reductionrrrOrSrr5r5r6validation_loss_reduction(rz'ReRankerModel.validation_loss_reduction)NNNN)N)rarbrcrdrr rrrrrModulerpropertyrrfrqrr LongTensorTensorrFrHrrrr{r5r5rwr6rsH  %:    rhfc@sfeZdZdZdefddZdedefddZede fdd Z edd d Z d ddeddfddZ dS)ReRankerImporterzHF Importer for Reranker Modelr)cCst|j|jdS)N)rt)rrrtrr5r5r6init8szReRankerImporter.init output_pathcCsNddlm}|}||}|jt|ddd}|||||||S)zApply the conversion from HF to NeMo format. Args: output_path: Path where the converted model will be saved Returns: Path: Path to the saved NeMo model rr#autoT) torch_dtypetrust_remote_code) transformersr$r nemo_setupfrom_pretrainedstr convert_state nemo_save)rsrr$targettrainersourcer5r5r6apply;s     zReRankerImporter.applycCsHddlm}|jt|dd}tt|tjkt|tjkt||j dS)z:Create a NeMo ReRankerBaseConfig from the HF model config.r) AutoConfigT)r)fp16bf16 params_dtyper~) rrrrrir!rfloat16bfloat16num_hidden_layers)rsrrr5r5r6rPs   zReRankerImporter.configr&cCsddlm}||t|S)zGet the tokenizer for the HF model. Returns: AutoTokenizer: Tokenizer instance initialized from the HF model's tokenizer rr%)=nemo.collections.common.tokenizers.huggingface.auto_tokenizerr&save_hf_tokenizer_assetsr)rsr&r5r5r6rt]s zReRankerImporter.tokenizerrr$rNc Cs|j}|||}|jjjj|jjjks&Jd|jjjd|jjjt:z |jjj |jjWnt ySt dt d|j|jjjYn wWd|SWd|S1sgwY|S)z:Convert the state of the source model to the target model.zScore weight dtype mismatch: z != zoFailed to copy score weight. This is expected if you are trying to convert model without score weights to NeMo.zinit the score weight...N)rrmrrrHweightdtyperno_gradcopy_ Exceptionr"warninginfor)rsrrtarget_connectorr5r5r6rhs.          zReRankerImporter.convert_state)r)r&) rarbrcrdrrrrrrJrrtrr5r5r5r6r4s  rr$c@sTeZdZdZejfdddZdedefddZe d d Z d d Z e dddZ dS)ReRankerExporterzExporter for converting NeMo Llama models to Hugging Face format. This class handles the conversion of NeMo's ReRankerModel to Hugging Face's AutoModelForSequenceClassification format, including weight mapping and configuration translation. r)r$cCsRddlm}ddlm}||j|j|dWdS1s"wYdS)aInitialize a HF AutoModelForSequenceClassification instance. Args: dtype: Data type for model parameters Returns: LlamaBidirectionalForSequenceClassification: Initialized HF Llama Bidirection reranker model r)no_init_weights)+LlamaBidirectionalForSequenceClassification)rN)transformers.modeling_utilsr1nemo.collections.llm.gpt.model.hf_llama_embeddingr _from_configr)rsrrrr5r5r6rs $zReRankerExporter.initrcCs|t|\}}|jjjjj}||}|||}| }| |z|j j }|j dur7|j |_ |jj|_| |W|StyMtdY|Sw)z,Apply the conversion from NeMo to HF format.NzFailed to save tokenizer) nemo_loadrr embeddingword_embeddingsrrrrcpusave_pretrainedrt pad_token eos_tokenrrM padding_siderr"r)rsrr_ source_dtyperrtr5r5r6rs"       zReRankerExporter.applyc Cstjt|dd}ddlm}m}|d|d|did|jd|jd |j d |j d |j d |j d |j d|jd|jd|jjd|jddd|jjd|jjd|jjd|jd|j|j|j|jddS)z@Create a NeMo LlamaBidirectionalConfig from the HF model config.z model.configsubpathr)LlamaBidirectionalConfigrrr$rrintermediate_sizenum_attention_headsmax_position_embeddingsinitializer_range rms_norm_epsnum_key_value_heads rope_theta vocab_sizetie_word_embeddings num_labelsr bos_token_id eos_token_id pad_token_idr` rope_scalingllama3)factorhigh_freq_factorlow_freq_factor original_max_position_embeddings rope_typeNr5)r load_contextrrrrregister_for_auto_classr~rffn_hidden_sizer seq_lengthinit_method_stdlayernorm_epsilonnum_query_groups rotary_basertr#share_embeddings_and_output_weightsbos_ideos_idr` scale_factorrrold_context_len)rsrrrr5r5r6rsX       zReRankerExporter.configcCsXddddddd}tjdd tjd tjd d tjd tjd dtjd g}tj||||dS)zConvert NeMo State dict to HF.z&model.layers.*.self_attn.o_proj.weightz#model.layers.*.mlp.down_proj.weightz%model.layers.*.input_layernorm.weightz.model.layers.*.post_attention_layernorm.weightzmodel.norm.weight score.weight)z2decoder.layers.*.self_attention.linear_proj.weightz&decoder.layers.*.mlp.linear_fc2.weightz>sz'ReRankerLoss.reduce..r cSs$g|]}|dddkr|dqS)loss_sum_and_ub_sizerrr5r.r5r5r6r0DsrrRrT)with_context_parallelgroupr)rcatmeanr!r r;vstackrtensorr/current_device distributed all_reduceget_data_parallel_group)rslosses_reduced_per_micro_batchrZlossr r1r5r5r6reduce8s(    zReRankerLoss.reduce)FTrrR)rarbrcrdrhrfrgrrrrrr rFr?r{r5r5rwr6rs2  rcCsNddlm}tdd|D}tjj||d|tjj|d}|S)z*Reduce a tensor of losses across all GPUs.rr cSsg|] }|dqS)r)clonedetachr$)r0r>r5r5r6r0[sz=average_losses_across_data_parallel_group..r3)r!r rr5r:r;r<get_world_size)lossesr averaged_lossesr5r5r6r'Ws r')O dataclassesrpathlibrtypingrrrrrr r r rpytorch_lightningLrr!r $megatron.core.tensor_parallel.layersr megatron.core.transformer.modulerrr$megatron.core.transformer.spec_utilsrmegatron.core.utilsrr#nemo.collections.llm.gpt.model.baserr$nemo.collections.llm.gpt.model.llamarr.nemo.collections.llm.gpt.model.llama_embeddingrrrynemo.collections.llm.utilsrnemo.lightningrrnemo.lightning.io.stater nemo.lightning.megatron_parallelrr nemo.lightning.pytorch.utilsr! nemo.utilsr""megatron.core.models.gpt.gpt_modelrorr$rr&1nemo.collections.common.tokenizers.tokenizer_specr'r(rrrDLightningModulerIrJrir|rmodel_importerrzrmodel_exporterrrr'r5r5r5r6sX (                $ / H Y