o }oiz@sddlmZmZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddl mZddlmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZmZmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,erddl-m.Z.eGdddeZ/eGdddeZ0eGdddeZ1e%2edGddde%j3defZ4e%5edGddde%j3edfZ6d d!Z7e%j8d"d#d$d%e%j9fd&d'Z:e%j8d(d)d$d%e%j9fd*d+Z;e%j8d,d-d$d%e%j9fd.d/Ze%j8d8d9d$d:d;Z?e%j8d#dZ@e%j8d)d?d$d%e%j9fd@dAZAe%j8d-d,d$d%e%j9fdBdCZBe%j8d1d0d$d%e%j9fdDdEZCe%j8d5dFd$d%e%j9fdGdHZDe%j8d9d8d$dIdJZEdS)K) dataclassfield)Path) TYPE_CHECKINGDictTupleUnionN)TransformerConfig Qwen2VLConfig)Qwen2VLForConditionalGenerationQwen2VLVisionConfig) TokenizerSpec) Qwen2ConfigQwen2Config1P5B Qwen2Config7BQwen2Config72B) export_qkvexport_qkv_bias)r Qwen2VLModelrMultimodalProjectorConfig)load_distributed_model_weights)ioteardown) _ModelState) dtype_from_hf)logging AutoTokenizerc@heZdZUdZddlmZedddZee d<edddZ e eefe d <ed ddZ ee d <d S) Qwen2VLConfig2BzQwen2VL Config 2BrPretrainedConfigcCs tddS)NT)#share_embeddings_and_output_weights)rr&r&^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/qwen2vl/model/qwen2vl.py2s zQwen2VLConfig2B.default_factorylanguage_transformer_configcC tdddSN ) num_layersnum_attention_headsr r&r&r&r'r(5 vision_transformer_configcCtddddS)Ni input_size hidden_sizeffn_hidden_sizerr&r&r&r'r(8vision_projection_configN __name__ __module__ __qualname____doc__ transformersr$rr+r __annotations__r3rr;r&r&r&r'r"+s  r"c@r!) Qwen2VLConfig7BzQwen2VL Config 7Brr#cCtSN)rr&r&r&r'r(BzQwen2VLConfig7B.r)r+cCr,r-r r&r&r&r'r(Dr2r3cCr4)Nr5ir6rr&r&r&r'r(Gr:r;Nr<r&r&r&r'rC<  rCc@r!) Qwen2VLConfig72BzQwen2VL Config 72Brr#cCrDrE)rr&r&r&r'r(QrFzQwen2VLConfig72B.r)r+cCr,r-r r&r&r&r'r(Sr2r3cCr4)Nr5i r6rr&r&r&r'r(Vr:r;Nr<r&r&r&r'rHKrGrHhfc@sXeZdZdZdefddZdedefddZdd Ze dd d Z e de fd dZ dS)HFQwen2VLImporterzQwen2VL Model HF ImporterreturncCst|j|jdS)N) tokenizer)rconfigrLselfr&r&r'init^szHFQwen2VLImporter.init output_pathcCstt|}|}||}|t|j}|t|j}|||t d|| ||t d|t ||~~|S)Nz+Converted Qwen2VL model to Nemo, saving to !Converted Qwen2VL model saved to ) r from_pretrainedstrrP nemo_setuptorrM convert_stateprint nemo_saver)rOrQsourcetargettrainerr&r&r'applybs    zHFQwen2VLImporter.applyc Csiddddddddd d d d d ddddddddddddddddddd d!d"d#d$d%}|jjjsF|d&d'id(|jvrZ|d(d)d*d+d,nd-|jvrn|d-d.d/d0d,ntd1tj |||t t t t tgd2S)3Nvisual.patch_embed.proj.weightvision_model.conv1.weightvisual.blocks.*.norm1.weightIvision_model.decoder.layers.*.self_attention.linear_qkv.layer_norm_weightvisual.blocks.*.norm1.biasGvision_model.decoder.layers.*.self_attention.linear_qkv.layer_norm_biasvisual.blocks.*.norm2.weight>vision_model.decoder.layers.*.mlp.linear_fc1.layer_norm_weightvisual.blocks.*.norm2.biasHFQwen2VLImporter.config..make_vocab_size_divisible_byT)r0r8r9r1init_method_stdlayernorm_epsilonnum_query_groups rotary_basegated_linear_unitrr%rfp16bf16 params_dtype)rrrr mcore_mlp)r7r8r9projector_typerrr)r+r3r;vision_feature_layerrrr)packaging.versionrrA __version__ ValueErrorr rSrTrnum_hidden_layersr8intermediate_sizer1initializer_range rms_norm_epsnum_key_value_heads rope_thetartie_word_embeddingsrtorchfloat16bfloat16 vision_configr embed_dimspatial_merge_sizer) rOrHFQwen2VLConfig hf_configr text_configr+rr3merge_hidden_sizer;outputr&r&r'rMsh             zHFQwen2VLImporter.configN)rKr ) r=r>r?r@rrPrr]rWpropertyrLr rMr&r&r&r'rJZs= rJr c@speZdZdZejfdddZdedefddZd d Z e dd d Z dede e e ffddZe dddZdS)HFQwen2VLExportera  Exporter class for converting NeMo Qwen2VL model to HuggingFace format. Inherits: io.ModelConnector: Connector interface to handle setup, save, and load using the Lightning framework. Methods: init: Initializes a new HuggingFace Qwen2VL model instance. apply: Converts the NeMo model to HuggingFace format and saves it. convert_state: Maps and transforms the state dictionary from NeMo to HuggingFace format. config: Generates and returns the HuggingFace Qwen2VL config for the model. rKr cCsFddlm}|tj|j|dWdS1swYdS)a0 Initializes a HuggingFace Qwen2VLForConditionalGeneration model. Args: dtype: The data type to use for the model (default: torch.bfloat16) Returns: Qwen2VLForConditionalGeneration: A HuggingFace Qwen2VL model initialized with the configuration. r)no_init_weights) torch_dtypeN)transformers.modeling_utilsrr _from_configrM)rOdtyperr&r&r'rPs $zHFQwen2VLExporter.initrQcCstd||\}}tdtd|}td||||}|}||z |jj|WntyFt dYnwt d||S)a3 Converts the NeMo Qwen2VL model to HuggingFace format and saves it to the specified path. Args: output_path (Path): The path where the converted HuggingFace model will be saved. Returns: Path: The output path where the HuggingFace model was saved. z9Loading Qwen2VL NeMo checkpoint. This may take a while...zQwen2VL NeMo checkpoint loaded.zInitializing the HF model..zStart Converting the model..zFailed to save tokenizerrR) rinfo ckpt_loadrPrWcpusave_pretrainedrL ExceptionwarningrX)rOrQrZ source_configr[r&r&r'r]$s      zHFQwen2VLExporter.applyc Cs iddddddddd d d d d ddddddddddddddddddd d!d"d#d$d%}|jjrF|dd&in|d'd&id(|vr`|d)d*d+d,d-nd.|vrs|d)d*d+d,d/ntd0tj|||tt t t t gd1S)2a Maps and transforms the state dictionary from NeMo to HuggingFace format. Args: source: The source NeMo model. target: The target HuggingFace model. Returns: The target HuggingFace model with the converted state. r_r^rar`rcrbrerdrgrfrirhrkrjrmrlrornrqrprsrrrurtrwrvryrxr{rzr}r|rr~rr)rrrrrrrrr)rrrrr)rrrrrr) r+r%rrrrrr_export_language_qkv_export_language_qkv_bias_export_vision_qkv_export_vision_qkv_bias_export_linear_fc1)rOrZr[rrr&r&r'rWAs       zHFQwen2VLExporter.convert_statercCstjt|ddjS)z~ Gets the tokenizer from the loaded model context. Returns: The tokenizer specification. modelsubpath)r load_contextrTrLrNr&r&r'rLszHFQwen2VLExporter.tokenizerpathc Cstjt|dd}|d}i}|jj}|jj}t|d}|D]>\}} d|vr)q |dd} d| vrZ| d |ksA| d |krZt | d D]} | | || dd t| <qH| || <q t |} | |fS) a This function loads the state dict directly from a distributed checkpoint, and modify the state dict so that it is consistent with the key names you would get from loading the checkpoint into a model. This is a more memory-efficient method to obtain a state dict without initializing the nemo model. Args: path (Path): The path from which the model will be loaded. Returns ------- Tuple[Dict, Dict]: The loaded state dict and the yaml config dict. model.configrweightsT _extra_statezmodule.layersrzlayers.) rrrTr+r0r3ritemsreplacesizeranger) rOrrMdist_ckpt_folderrlangauge_layers vision_layersdistributed_model_weightskvnew_kirZr&r&r'rs    $  zHFQwen2VLExporter.ckpt_loadrcCstjt|dd}|j}|j}|j}t|j|j|j dt |j |j |j d|j |j|j|j|jddd}t|j|j |j |j |j|j|j|j|j|j|dd S) z Generates the configuration for the HuggingFace Qwen2VL model based on the NeMo model. Returns: HFQwen2VLConfig: A configuration object for the HuggingFace Qwen2VL model. rr quick_geluqwen2_vlr)depthrr8 hidden_act mlp_ratio num_heads in_channels patch_sizerspatial_patch_sizetemporal_patch_sizer model_typer) rr8rr1rrrrrrrr)rrrTr+r3r;HFQwen2VLVisionConfigr0rr8intr9r1 patch_dimrrrrto_dictrrrrr%r)rOrZlanguage_configvision_model_configr;rr&r&r'rMsH zHFQwen2VLExporter.configN)rKr )rKr)rKr)r=r>r?r@rrrPrr]rWrrLrrrrMr&r&r&r'rs J  !rcCsd|}||f|dd} ||f|dd} |j| }|j| }|j| }g} t|D]<} | || || d|ddddf| || | dddddf| || | dddddfq-t| } | jdksyJ| j| jd|d|ksJ| j| jd|ksJ| j| jd|dksJ| j| ||d||g} | S)Nrrr) rviewrappendrcatndimshapereshape)qrrhead_numrheads_per_groupr8 head_sizeold_tensor_shapenew_q_tensor_shapenew_kv_tensor_shape qkv_weights_lr qkv_weightsr&r&r' import_qkvs$    ,$&  r)visual.blocks.*.attn.qkv.weightz>vision_model.decoder.layers.*.self_attention.linear_qkv.weight) source_key target_keyctxc Cs|jjj}t|jdd}||jksJ|d|ddf}|||dddf}||ddddf}t||||j|j|j|j|j|j dS)Nrrrr rr r8r) r[rMr3rr r8rr1r kv_channels)rhf_qkv_weightsmegatron_configslicer rrr&r&r'rs  r)visual.blocks.*.attn.qkv.biaszlanguage_model.decoder.layers.*.self_attention.linear_qkv.biasc CsH|jjj}t|d|d|d|j|j|j|jd|jddS)Nrrr) r[rMr+rrr1rrr )rr"r#r$rr&r&r'rOs   r)'vision_model.embeddings.class_embeddingzvision_model.class_tokencCs|dddS)Nrr)r r cls_tokenr&r&r'_import_cls_tokenfsr))z#model.layers.*.mlp.gate_proj.weightz!model.layers.*.mlp.up_proj.weightz5language_model.decoder.layers.*.mlp.linear_fc1.weightcCstj||fddS)Nraxis)rr)downgater&r&r'ros rrc Cs>|jjj}tjt||j|j|j|j|j|j|jdddS)Nrrr*)r[rMrrrrrrrqkvrr&r&r'r{s   rrcCs:|jjj}tjt||j|j|j|j|j|jdddS)Nr rr rrr*)r[rMrrrrrrrqkv_biasrr&r&r'rs   rcCs2|jj}t||j|j|j|j|j|j|jdSr%)r[rMrr1rr8r.r&r&r'rs   rcCs.|jj}t||j|j|j|j|j|jdS)Nr0)r[rMrr1rr8r1r&r&r'rs   rr&cCs|SrE)r r'r&r&r'_export_cls_tokensr3cCstj|ddd\}}||fS)Nrr)dim)rchunk) linear_fc1 gate_projup_projr&r&r'rs r)F dataclassesrrpathlibrtypingrrrrrrA,megatron.core.transformer.transformer_configr r rr 3transformers.models.qwen2_vl.configuration_qwen2_vlrr1nemo.collections.common.tokenizers.tokenizer_specrnemo.collections.llmrrrr%nemo.collections.vlm.neva.model.llavarr'nemo.collections.vlm.qwen2vl.model.basernemo.collections.vlm.visionr.nemo.export.trt_llm.nemo_ckpt_loader.nemo_filernemo.lightningrrnemo.lightning.io.staternemo.lightning.pytorch.utilsr nemo.utilsrrr r"rCrHmodel_importerModelConnectorrJmodel_exporterrrstate_transform TransformCTXrrrrr)rrrrrr3rr&r&r&r's              ,c