o wi@sddlmZmZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddl mZddl mZdd l mZdd l mZdd lmZdd lmZdd lmZddlmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)ddl*mZm+Z+mZm,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9erddl:m;Z;eGdddeZeGdddeZ?eGd d!d!eZ@eGd"d#d#eZAeGd$d%d%eZBe2Ce+d&Gd'd(d(e2jDd)e+fZEe2Fe+d&Gd*d+d+e2jDe+d)fZGd,d-ZHe2jId.d/d0d1e2jJfd2d3ZKe2jId4d5d0d1e2jJfd6d7ZLe2jId8d9d0d1e2jJfd:d;ZMe2jIdd?ZNe2jId@dAd0d1e2jJfdBdCZOe2jIdDdEd0dFdGZPe2jIdHdId0dJdKZQe2jIdLdMd0dNdOZRdPdQZ(dRe jSfdSdTZ)e2jId/dUd0d1e2jJfdVdWZTe2jId5dXd0d1e2jJfdYdZZUe2jId9d8d0d1e2jJfd[d\ZVe2jId=dDs zQwen2VLConfig2B.default_factorylanguage_transformer_configcC tdddSN ) num_layersnum_attention_headsrr.r.r.r/r0G vision_transformer_configcCtddddS)Ni input_size hidden_sizeffn_hidden_sizerr.r.r.r/r0Jvision_projection_configN __name__ __module__ __qualname____doc__ transformersr,rr3r __annotations__r;rrCr.r.r.r/r*=s  r*c@r)) Qwen2VLConfig7BzQwen2VL Config 7Brr+cCtSN)rr.r.r.r/r0TzQwen2VLConfig7B.r1r3cCr4r5rr.r.r.r/r0Vr:r;cCr<)Nr=r>rr.r.r.r/r0YrBrCNrDr.r.r.r/rKN  rKc@r)) Qwen2VLConfig72BzQwen2VL Config 72Brr+cCrLrM)rr.r.r.r/r0crNzQwen2VLConfig72B.r1r3cCr4r5rr.r.r.r/r0er:r;cCr<)Nr= r>rr.r.r.r/r0hrBrCNrDr.r.r.r/rQ]rPrQc@r)) Qwen25VLConfig3BzQwen2.5VL Config 3Brr+cCrLrM)rr.r.r.r/r0rrNzQwen25VLConfig3B.r1r3cCr4r5rr.r.r.r/r0tr:r;cCtdddddS)N mcore_mlpr=iprojector_typer?r@rArr.r.r.r/r0wrCNrDr.r.r.r/rSlrPrSc@r)) Qwen25VLConfig7BzQwen2.5VL Config 7Brr+cCrLrM)rr.r.r.r/r0rNzQwen25VLConfig7B.r1r3cCr4r5rTr.r.r.r/r0r:r;cCrU)NrVr=rOrWrr.r.r.r/r0rYrCNrDr.r.r.r/rZ}rPrZc@r)) Qwen25VLConfig32BzQwen2.5VL Config 32Brr+cCrLrM)rr.r.r.r/r0rNzQwen25VLConfig32B.r1r3cCtddddSNr6r7i )r8r9rArTr.r.r.r/r0rBr;cCstdddddS)NrVr=rWrr.r.r.r/r0rYrCNrDr.r.r.r/r[rPr[c@r)) Qwen25VLConfig72BzQwen2.5VL Config 72Brr+cCrLrM)rr.r.r.r/r0rNzQwen25VLConfig72B.r1r3cCr\r]rTr.r.r.r/r0rBr;cCrU)NrVr=rRrWrr.r.r.r/r0rYrCNrDr.r.r.r/r^rPr^hfc@sXeZdZdZdefddZdedefddZdd Ze dd d Z e de fd dZ dS)HFQwen2VLImporterzQwen2VL Model HF ImporterreturncCst|jd|jdS)Nzqwen2-vl) model_version tokenizer)rconfigrcselfr.r.r/initszHFQwen2VLImporter.init output_pathcCstjt|dd}tjt|dd}|jdk|_|}||}|t |}|t |}| ||t d|| ||t d|t ||~~|S)NTtrust_remote_code qwen2_5_vlz+Converted Qwen2VL model to Nemo, saving to !Converted Qwen2VL model saved to )r from_pretrainedstr HFAutoConfig model_typeis_v2_5rg nemo_setuptor% convert_stateprint nemo_saver#)rfrhsource hf_configtargettrainerr.r.r/applys     zHFQwen2VLImporter.applycCsddddddddd d d d d d }|jjjs|ddi|jr)|ddddn |dddddddd|jvrI|dddddnd|jvr]|ddd d!dntd"t t t t t g}|jrq|ttg7}tj||||d#S)$Nvision_model.conv1.weightIvision_model.decoder.layers.*.self_attention.linear_qkv.layer_norm_weightGvision_model.decoder.layers.*.self_attention.linear_qkv.layer_norm_bias>vision_model.decoder.layers.*.mlp.linear_fc1.layer_norm_weightHFQwen2VLImporter.config..make_vocab_size_divisible_by)r8r@rAr9init_method_stdlayernorm_epsilonnum_query_groups rotary_basegated_linear_unitrr-rfp16bf16 params_dtype)rArrrrrV)r?r@rArXrrr)rrr)r3r;rCvision_feature_layerrrr) packaging.versionrrI __version__ ValueErrorrormrnrprnum_hidden_layersr@intermediate_sizer9initializer_range rms_norm_epsnum_key_value_heads rope_thetartie_word_embeddingsr%torchfloat16bfloat16 vision_configrspatial_merge_sizer out_hidden_sizer embed_dimr ) rfrrxrqr text_configr3rr;merge_hidden_sizerCoutputr.r.r/rd#s                 zHFQwen2VLImporter.configN)rar() rErFrGrHrrgrr{rtpropertyrcr rdr.r.r.r/r`sP r`rc@speZdZdZejfdddZdedefddZd d Z e dd d Z dede e e ffddZe dddZdS)HFQwen2VLExportera  Exporter class for converting NeMo Qwen2VL model to HuggingFace format. Inherits: io.ModelConnector: Connector interface to handle setup, save, and load using the Lightning framework. Methods: init: Initializes a new HuggingFace Qwen2VL model instance. apply: Converts the NeMo model to HuggingFace format and saves it. convert_state: Maps and transforms the state dictionary from NeMo to HuggingFace format. config: Generates and returns the HuggingFace Qwen2VL config for the model. rarcCsFddlm}|tj|j|dWdS1swYdS)a0 Initializes a HuggingFace Qwen2VLForConditionalGeneration model. Args: dtype: The data type to use for the model (default: torch.bfloat16) Returns: Qwen2VLForConditionalGeneration: A HuggingFace Qwen2VL model initialized with the configuration. r)no_init_weights) torch_dtypeN)transformers.modeling_utilsrr from_configrd)rfdtyperr.r.r/rgs $zHFQwen2VLExporter.initrhcCstd||\}}tdtd|}td||||}|}||z |jj|WntyFt dYnwt d||S)a3 Converts the NeMo Qwen2VL model to HuggingFace format and saves it to the specified path. Args: output_path (Path): The path where the converted HuggingFace model will be saved. Returns: Path: The output path where the HuggingFace model was saved. z9Loading Qwen2VL NeMo checkpoint. This may take a while...zQwen2VL NeMo checkpoint loaded.zInitializing the HF model..zStart Converting the model..zFailed to save tokenizerrl) r&info ckpt_loadrgrtcpusave_pretrainedrc Exceptionwarningru)rfrhrw source_configryr.r.r/r{s      zHFQwen2VLExporter.applycCsddddddddd d d d d d }|jjr|ddin|ddi|jr0|ddddn |dddddddd|vrO|dddd d!nd"|vrb|dddd d#ntd$ttt t t g}|jrv|t t g7}tj||||d%S)&a Maps and transforms the state dictionary from NeMo to HuggingFace format. Args: source: The source NeMo model. target: The target HuggingFace model. Returns: The target HuggingFace model with the converted state. rrrrrrrrrrrrr) r|r}r~rrrrrrrrrrrrrrrr)rrrrrrrr)rrrrrrrrrrr)rrrrr)rrrrrr)r3r-rrqrrr_export_language_qkv_export_language_qkv_bias_export_vision_qkv_export_vision_qkv_bias_export_linear_fc1 _export_vision_linear_fc1_weight_export_vision_linear_fc1_biasr"r)rfrwryrrrr.r.r/rts   zHFQwen2VLExporter.convert_statercCstjt|ddjS)z~ Gets the tokenizer from the loaded model context. Returns: The tokenizer specification. modelsubpath)r" load_contextrnrcrer.r.r/rcszHFQwen2VLExporter.tokenizerpathc Cstjt|dd}|d}i}|jj}|jj}t|d}|D]>\}} d|vr)q |dd} d| vrZ| d |ksA| d |krZt | d D]} | | || dd t| <qH| || <q t |} | |fS) a This function loads the state dict directly from a distributed checkpoint, and modify the state dict so that it is consistent with the key names you would get from loading the checkpoint into a model. This is a more memory-efficient method to obtain a state dict without initializing the nemo model. Args: path (Path): The path from which the model will be loaded. Returns ------- Tuple[Dict, Dict]: The loaded state dict and the yaml config dict. model.configrweightsT _extra_statezmodule.layersrzlayers.) r"rrnr3r8r;r!itemsreplacesizeranger$) rfrrddist_ckpt_folderrlangauge_layers vision_layersdistributed_model_weightskvnew_kirwr.r.r/r!s    $  zHFQwen2VLExporter.ckpt_loadHFQwen2VLConfigcCs4ddlm}|tj|dkrtdtjdtjt|dd}|j}|j }|j }t |do3|j d k|_ |j rtdBid |jd |jd |jd |jdddt|j|jd|jddd|jd|jd|jd|jd|jdgddddddd}tdBid |d!|jd |jd"|jd#|jd$d%d&|jd|jd'|jd(|jd)|jd*|j d+d,gd-d.d/|j!ddd0d1d2d3d4d5d6d7d8d9d:d;dt|j|j|jd|j|j|j|j|jd?dd@}t#|j|j|j|j|j|j|j|j|j!|j |ddA S)Cz Generates the configuration for the HuggingFace Qwen2VL model based on the NeMo model. Returns: HFQwen2VLConfig: A configuration object for the HuggingFace Qwen2VL model. rrrrrrrfullatt_block_indexesNdepthrr@r hidden_actsilu mlp_ratio num_heads in_channels patch_sizerspatial_patch_sizetemporal_patch_sizer)tokens_per_secondrrprkrrrrrr9max_window_layersFmax_position_embeddingsrrrr rope_scalingmrope)r7r-)type mrope_sectionr bos_token_idi[P eos_token_idi]Pvision_start_token_ididPvision_end_token_idiePvision_token_idifPimage_token_idigPvideo_token_idi quick_geluqwen2_vl)rrr@rrrrr rr!r"rrpr) rr@rr9rrrrrrrrr.)$rrrIrrr"rrnr3r;rChasattrrrqHFQwen25VLVisionConfigr8rr@intrAr9 patch_dimrr!r"rto_dictHFQwen25VLConfig seq_lengthrrrrr-HFQwen2VLVisionConfigr)rfrrwlanguage_configvision_model_configrCrrxr.r.r/rdBs             zHFQwen2VLExporter.configN)rar)rar)rar)rErFrGrHrrrgrr{rtrrcrrrrdr.r.r.r/r}s _  !rcCsd|}||f|dd} ||f|dd} |j| }|j| }|j| }g} t|D]<} | || || d|ddddf| || | dddddf| || | dddddfq-t| } | jdksyJ| j| jd|d|ksJ| j| jd|ksJ| j| jd|dksJ| j| ||d||g} | S)Nrrr) r viewrappendrcatndimshapereshape)qrrhead_numrheads_per_groupr@ head_sizeold_tensor_shapenew_q_tensor_shapenew_kv_tensor_shape qkv_weights_lr qkv_weightsr.r.r/ import_qkvs$    ,$&  rS)visual.blocks.*.attn.qkv.weightz>vision_model.decoder.layers.*.self_attention.linear_qkv.weight) source_key target_keyctxc Cs|jjj}t|jdd}||jksJ|d|ddf}|||dddf}||ddddf}t||||j|j|j|j|j|j dS)NrrrrKrrLr@rM) ryrdr;r;rHr@rSr9r kv_channels)rWhf_qkv_weightsmegatron_configslicerJrrr.r.r/rs  r)visual.blocks.*.attn.qkv.biaszlanguage_model.decoder.layers.*.self_attention.linear_qkv.biasc CsH|jjj}t|d|d|d|j|j|j|jd|jddS)NrrCrX) ryrdr3rSr^r9rrYr_)rWrarbrcr[r.r.r/rs   r)'vision_model.embeddings.class_embeddingzvision_model.class_tokencCs|dddS)NrCr)rIrW cls_tokenr.r.r/_import_cls_token.srh)z#model.layers.*.mlp.gate_proj.weightz!model.layers.*.mlp.up_proj.weightz5language_model.decoder.layers.*.mlp.linear_fc1.weightcCtj||fddSNraxisrrFdowngater.r.r/r7s r)z$visual.blocks.*.mlp.gate_proj.weightz"visual.blocks.*.mlp.up_proj.weightrcCrirjrmrnr.r.r/rCr)z"visual.blocks.*.mlp.gate_proj.biasz visual.blocks.*.mlp.up_proj.biasrcCrirjrmrnr.r.r/rLrqrc s|d|}|||dg}|d}tfddt|D}t|d}td|d} ||d|} ||d|} || d|} | | | fS)Nrrc,g|]}td|d|qSrrarange.0rrLr.r/ \zexport_qkv..rC)rIr rrFrrur) linear_qkvrKrrLr@rM qkv_total_dimq_slicek_slicev_sliceq_projk_projv_projr.rxr/rUs    rqkv_biasc s|d|}|||g}tfddt|D}t|d}td|d}||d} ||d} ||d} | | | fS)z Split interleave-concatenated qkv bias to separate q, k, v bias Example: export layer linear_qkv bias to HF {q|k|v}_proj bias rcrrrsrtrvrxr.r/ryurzz#export_qkv_bias..rCr)rIrrFrrur) rrKrrLrMr|r}r~rrarbrcr.rxr/rks   rrTc CsP|jjj}|jdkr |jn|j}tjt||j |j |j |j |||j dddS)Nr8rXrrk) ryrdrrprr@rrFrr)rWqkvrxr@r.r.r/rs  rr]cCsN|jjj}|jdkr |jn|j}tjt||j |j |j |j ||j dddS)Nr8rKrrLrMrrk) ryrdrrprr@rrFrr)rWrrxr@r.r.r/rs  rcCs2|jj}t||j|j|j|j|j|j|jdSrd)ryrdrr9rr@)rWrrxr.r.r/rs   rcCs.|jj}t||j|j|j|j|j|jdS)Nr)ryrdrr9rr@)rWrrxr.r.r/rs   rrecCs|SrM)r_rfr.r.r/_export_cls_tokensrcCtj|ddd\}}||fSNrr)dimrchunk) linear_fc1 gate_projup_projr.r.r/r rcCrrr)vision_fc1_weightrrr.r.r/rrrcCrrr)vision_fc1_biasrrr.r.r/rrr)\ dataclassesrrpathlibrtypingrrrrrrI,megatron.core.transformer.transformer_configr r ror r r>r rr7transformers.models.qwen2_5_vl.configuration_qwen2_5_vlrr:3transformers.models.qwen2_vl.configuration_qwen2_vlrr@1nemo.collections.common.tokenizers.tokenizer_specrnemo.collections.llmrrrrrrrr%nemo.collections.vlm.neva.model.llavarr'nemo.collections.vlm.qwen2vl.model.baserrnemo.collections.vlm.visionr .nemo.export.trt_llm.nemo_ckpt_loader.nemo_filer!nemo.lightningr"r#nemo.lightning.io.stater$nemo.lightning.pytorch.utilsr% nemo.utilsr&rr(r*rKrQrSrZr[r^model_importerModelConnectorr`model_exporterrrSstate_transform TransformCTXrrrrrhrrrTensorrrrrrrrrr.r.r.r/s          (         M5