o wiB.@sddlZddlmZddlmZmZmZmZmZm Z ddl m Z ddl Z ddlmZddlmZddlmZddlmZddlmZddl mZdd lmZdd lmZmZmZm Z dd l!m"Z"dd l#m$Z$e%e&Z'ed e j(dZ)edej*dZ+ddZ,GdddZ-Gdddee"Z.dS)N)Path)AnyCallableDictOptionalTypeVarUnion)HFValidationError) CheckpointIO)get_filesystem)_PATH)nn)override)HF_ADAPTER_CONFIG_FILENAMEHF_ADAPTER_PATHHF_WEIGHTS_PATH WEIGHTS_PATH)IOMixin)ckpt_to_weights_subdirLightningModuleT)boundModuleTcCs t pt ptdkS)z=Checks whether rank=0 accounting for un-inintialized dist-envr)dist is_availableis_initializedget_rankrrQ/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/lightning/io/hf.py is_rank_0(s rc@s(eZdZdZeddZeddZdS)HFAdapterKeyRenamerzDummy class for key renamingcC"|dddddddddS)z Converts lora adapter FQNs to HF model.model.base_model.model.model. lora_a.weight lora_A.weight lora_b.weight lora_B.weightreplacexrrr nemo_to_hf0   zHFAdapterKeyRenamer.nemo_to_hfcCr )z"Converts lora adapter FQNs to NeMor'r&r#r%r$r"r!r(r*rrr hf_to_nemo9r-zHFAdapterKeyRenamer.hf_to_nemoN)__name__ __module__ __qualname____doc__ staticmethodr,r.rrrrr-s  rc seZdZdZdfdd Zeddeeefde de ed dffd d Z dd eeefde ee fde ed dfd dZede ee fd eeeffddZe   dde de ede deBd eeeffddZede d dfddZZS)HFCheckpointIOzHFCheckpointIO that utilizes :func:`torch.save` and :func:`torch.load` to save and load checkpoints respectively, common for most use cases. .. warning:: This is an :ref:`experimental ` feature. NFcst||_||_dS)a+Initializes HFCheckpointIO Args: model (nn.Module, optional): The nn.Module that's used for training. This supplies the save_pretrained function. adapter_only (bool, optional): If true, will only save LoRA adapter weights. Defaults to False. N)super__init__ adapter_onlymodel)selfr8r7 __class__rrr6Ks  zHFCheckpointIO.__init__ checkpointpathstorage_optionsreturncstsJdt|dd}|jdtksJd|t|j}t|}|j|dd|jrK|t }|j dd| | d||t ||dd Stt|jd d rt|t}|j ddd |jj|| dd t ||dd St|||td t|)a Save model/training states to a checkpoint file. Note: This function assumes it's only written by RANK=0 if executed inside a dist-env. Args: checkpoint: dict containing model and trainer state path: write-target path storage_options: not used in ``TorchCheckpointIO.save_checkpoint`` Raises ------ TypeError: If ``storage_options`` arg is passed in zExpected to run only on rank=0T) is_savingzExpected {} to end with {})exist_ok state_dict trainer.ptsave_pretrainedN)parentsrB)rCzCheckpoint was saved at: )rrpartsrformatparentr makedirsr7rmkdir_save_adapter_weights_onlypoptorchsavecallablegetattrr8rrEr5save_checkpointNotImplementedErrorstr)r9r<r=r>checkpoint_dirfs adapter_pathhf_weights_pathr:rrrRWs(   zHFCheckpointIO.save_checkpointrCc Csvddlm}t|}|D]}||}||t|<qz |||dWdSty:}ztd|d}~ww)a Saves only the adapter weights in a safetensors format. Args: state_dict (Dict[str, Any]): The state dictionary containing model weights. path (Union[str, Path]): The directory path where the adapter weights should be saved. storage_options (Optional[Any], optional): Additional storage options, if required. Raises: OSError: If saving the file fails. r) save_fileadapter_model.safetensorsz"Failed to save adapter weights: {}N) safetensors.torchrYlistkeysrMrr,OSErrorrH) r9rCr=r>rY module_namesnameparamerrrrLs   z)HFCheckpointIO._save_adapter_weights_onlyc Cst|}||std|||std|i}t|d}|s-td|t|t}|s>td|ddlm }z2||dd d }| D] }| ||t |<qQWd n 1shwYWd |iSWd |iSty}ztd |d }~ww)a Loads only the adapter weights from a safetensors checkpoint. Args: path (Union[str, Path]): The directory path where the adapter weights are stored. Returns: Dict[str, Any]: A dictionary containing the state dictionary of the adapter model. Raises: FileNotFoundError: If the checkpoint directory does not exist. ValueError: If the checkpoint path is not a directory. OSError: If loading the weights fails. zCheckpoint file not found: {}z,Checkpoints should be a directory. Found: {}rZz"Adapter weights file not found: {}z!Adapter config file not found: {}r) safe_openptcpu) frameworkdeviceNz Failed to load adapter weights: rC)r existsFileNotFoundErrorisdir ValueErrorrHrr safetensorsrcr] get_tensorrr.r^) r=rVrC adapter_file config_filercfkrbrrr_load_adapter_weights_onlys4       z)HFCheckpointIO._load_adapter_weights_only map_locationstrictStrictHandlingc Cst|}|jdtksJd|dti}|ds&td|n tj|ddddd }|j rA|t }|t |O}|St t|jd d riz |j|t|d <W|Sttfyhtd |dwtd)aLoads checkpoint using :func:`torch.load`, with additional handling for ``fsspec`` remote loading of files. Args: path: Path to checkpoint map_location: a function, :class:`torch.device`, string or a dict specifying how to remap storage locations. Returns: The loaded checkpoint. Raises ------ FileNotFoundError: If ``path`` is not found by the ``fsspec`` filesystem rAz Expected z not to end with rDzs*