o ॵig@sddlZddlZddlZddlZddlZddlmZddlmZm Z m Z ddl m Z m Z mZmZddlZddlZddlmZddlmZddlmZddlmZmZdd lmZmZdd lmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%e#Z&eZ'ddZ(     d'dejj)de*deedeedee+de,de,ddfddZ-  d(dedefddZ.  d(ddZ/de fd d!Z0  d(d"ee*ej1fd#e*d$e fd%d&Z2dS))N) OrderedDict)copytreeignore_patternsrmtree)CallableDictOptionalUnion)nn) Optimizer) _LRScheduler)File LocalStorage)ConfigJSONIteratorEncoder) ConfigFields ModelFile) copytree_py37) get_logger) is_mastercCs8t}|D] \}}|||<qt|dt|_|S)zCopy a model state_dict to cpu. Args: state_dict (OrderedDict): Model weights on GPU. Returns: OrderedDict: Model weights on GPU. _metadata)ritemscpugetattrr) state_dictstate_dict_cpukeyvalrO/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/utils/checkpoint.pyweights_to_cpus  r Tmodelfilename optimizer lr_schedulermeta with_meta with_modelreturnc Csi}|s |s td|r|duri}nt|ts!tdt|ddlm}|j|t dt|t j j j r;|j}t|drL|jdurL|j|jd||d <t|tr\||d <nt|trvi|d <|D] \} } | |d | <qi|durt|d r||d <|rt|t j j j r|j}t|} |s| }n| |d <t} t || t| |WddS1swYdS) afSave checkpoint to file. The checkpoint will have 3 fields: ``meta``, ``state_dict`` and ``optimizer``. By default, ``meta`` will contain version and time info. Args: model (Module): Module whose params are to be saved. filename (str): Checkpoint filename. optimizer (:obj:`Optimizer`, optional): Optimizer to be saved. lr_scheduler(:obj:`_LRScheduler`, optional): LRScheduler to be saved. meta (dict, optional): Metadata to be saved in checkpoint. with_meta (bool, optional): Save meta info. with_model(bool, optional): Save model states. z;Save meta by "with_meta=True" or model by "with_model=True"Nz%meta must be a dict or None, but got r) __version__) modelscopetimeCLASSES)r,r%r#rr$) ValueError isinstancedict TypeErrortyper*r)updater+asctimetorchr parallelDistributedDataParallelmodulehasattrr,r rrr ioBytesIOsaver writegetvalue) r!r"r#r$r%r&r' checkpointr)nameoptim_weightsfrrrsave_checkpoint/sL         "rCc Cstj|std|dtj|dd}|dur\d|vrTt|tr*||dn2t|t rS|d}| D]\}}||vrG|||q7t d|d|q7nt d||durtd |vrl||d nt d ||durd |vr~|n|d }||| d iS) NzCheckpoint file z does not exist!r map_locationr#zThe state dict of optimizer z% cannot be found in checkpoint file: z@The state dict of optimizer cannot be found in checkpoint file: r$zCThe state dict of lr_scheduler cannot be found in checkpoint file: rr%)ospathexistsr-r4loadr.r load_state_dictr/rloggerwarningget) r"r!r#r$r>optimizer_dictr optimizer_insrrrrload_checkpointvs>     rPc  sdddd dddfdd }  dd d  dd ddfdd }tj|tj}tj|dd}|durEt|||||ddd\} } } } t ddd rd|| dj || | | | dS)a{ Load model checkpoint file and feed the parameters into the model. Args: model_to_load: The model to be load model_local_dir: The actual checkpoint dir on local disk. default_dtype: Set the default float type by 'torch.set_default_dtype' load_state_fn: An optional load_state_fn used to load state_dict into the model. Returns: c Sst}|D](\}}||vr+d||g}|||<||tt|t|g}q|||<qg}t|dkr<|}|||fS)N.r)rrjoinremovelistsetlencopy) state_dicts head_prefix!expected_keys_without_head_prefix missing_keysnew_state_dictr?r7name_with_headmissing_head_keysrrr_add_head_prefix_to_state_dicts    zBload_task_model_checkpoint.._add_head_prefix_to_state_dictc Ssg}|rF|D]?}|} |r|d|} n|r"d|ddd} | |vrE|| j} ||j} | | krE||||j|| jf||=q|S)NrQ)rRsplitshapeappend) rXmodel_state_dict loaded_keysprefixadd_prefix_to_modelremove_prefix_from_modelignore_mismatched_sizesmismatched_keycheckpoint_key model_key model_shapecheckpoint_shaperrr_find_mismatched_keyss(    z9load_task_model_checkpoint.._find_mismatched_keysNc sg}g}D]$}d}d|vr|dd}d|vr |dd}|r,||||qt||D] \} }| |<q2tdddurO_g|dura||||ddSd dtj ffd d ||d S) Ngammaweightbetabiasr)rfhead_prefix_keyslocal_metadata error_msgsr7csndurin |ddi}||dggf}|j||jD]\}}|dur4|||dq#dS)NTrQ)rM_load_from_state_dict_modulesr)r7rfruargsr?childrvrImetadatarrrrIs   zMload_task_model_checkpoint.._load_state_dict_into_model..loadrf)rw) keysreplacerczippoprrWrr Module) r!r start_prefixrt load_state_fnold_keysnew_keysrnew_keyold_keyrr}r_load_state_dict_into_models@       z?load_task_model_checkpoint.._load_state_dict_into_modelc$ s|}t|}t|}|jtdkr0tfdd|D}tfdd|D} nd}d} | o8| } |o=| } | rSfdd|Dfdd|D}n | r`fd d|D}gtt|t|} tt|t|} |j}g}g}g}t}t}t|dkrt| dkrt |t r|g}|D]tfd d|D|<q|D]fd d|D}| |||<qtt| t|}tt| t|}  d d} d d}|dur|D] fdd| D} q|dur|D] fdd| D} q|r|| | | d}|D]}| |qt}t|dkrA|D]}||s?||||| \}}} |||<q(d}|}t}t|jdkr_t||js_|r_|jd}t|jdkrt||jr|st||j}|D] }t||||<qytfdd|Drtd|||| | |}||||}t|dkr|D]} || || d|}!| |!qt|dkrd|}"td|jjd|"t| dkrtd|jjd| d|jjd|jjd n&t|dkrtd|jjd|d|jjdn td|jjd t| dkr6td!|jjd"| d#nt|dkrMtd$|jjd%|jjd&t|dkrld'd(d|D}#td!|jjd)|#d#| | ||fS)*Nrc3|]}|VqdSN startswith.0srrr '  zGload_task_model_checkpoint.._load_checkpoint..c3rrrrrrrr)rFcsg|] }|s|qSrrrrrr 4s  zHload_task_model_checkpoint.._load_checkpoint..cs2g|]}|rd|dddn|qSrQr`NrrRrarrrrr7s$csg|] }d|gqS)rQ)rRrrrrr=sc3rrrr prefix_headrrrRrcs.g|]}|rd|dddqSrrrrrrrVs _keys_to_ignore_on_load_missing"_keys_to_ignore_on_load_unexpectedc g|] }t|dur|qSrresearchrkpatrrrmcrrrrrrrrsr)rf add_prefix remove_prefixrwrQc3s|]}|vVqdSrrrr)%expected_keys_not_base_model_prefixedrrrszjThe state dictionary of the model you are trying to load is corrupted. Are you sure it was properly saved?z z#Error(s) in loading state_dict for z: zESome weights of the model checkpoint were not used when initializing z: z, - This IS expected if you are initializing z from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForTokenClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing z from the checkpoint of a model that you expect to be exactly identical (initializing a BertForTokenClassification model from a BertForTokenClassification model).z0 - This IS Not expected if you are initializing z from the checkpoint of a model with a same task while the structure is different (e.g. initializing a BertForTokenClassification model from a BertForTokenClassification model).z9All model checkpoint weights were used when initializing z. zSome weights of zK were not initialized from the model checkpoint and are newly initialized: zo You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.zAll the weights of z were initialized from the model checkpoint If your task is similar to the task the model of the checkpoint was trained on, you can already use z* for predictions without further training. c Ss*g|]\}}}d|d|d|dqS)z- z: found shape z in the checkpoint and z in the model instantiatedr)rrshape1shape2rrrrszl were not initialized from the model checkpoint and are newly initialized because the shapes did not match: )rrTrbase_model_prefixrVanyrUrYr/r.strextendr _init_weightsr8rr-rR RuntimeError __class____name__rKrLinfo)$r!rrri _fast_initrd expected_keyskeys_from_pretrainedpretrained_has_prefix_modulemodel_expects_prefix_modulerhrgr[unexpected_keys prefix_heads&expected_head_keys_without_head_prefixr^unexpected_head_keyspretrained_has_prefix_headrtrZrruninitialized_modulesr7missing_head_keys_by_headrYr model_to_load heads_to_loadmismatched_keysrvheadlocal_error_msgs error_msgmismatched_warning)r_rorkwargsretrieve_modules_from_names)rrrfrr_load_checkpoints                               z4load_task_model_checkpoint.._load_checkpointFc Sstdd|D}|tdd|D}g}|D]4\}}|r5||r2d|dddn|}n|rFt|dkrDd||gn|}||vrO||q|S)NcSs$g|]}d|dddqS)rQNrx)rRrarrrrrs$zSload_task_model_checkpoint..retrieve_modules_from_names..cSs0g|]}|drd|dddqS)rxrQN)isdigitrRrarrrrrs  rQr`r)rUunion named_modulesrrRrarVrc) r!namesrfrr module_keysretrieved_modulesr?r7rrrrs* z?load_task_model_checkpoint..retrieve_modules_from_namescSs|r t|j|_n|j|_t|dddur1tj|jjd|jj d|jj dfdd|j_t |drAt |drC|j |_ dSdSdS)Nrsrconstant out_featuresnum_embeddings) r Parameterrqcloner functionalpadrsdatarbr8rr)output_embeddingsinput_embeddings torchscriptrrr_tie_or_clone_weightss*    z9load_task_model_checkpoint.._tie_or_clone_weightscs6|r|j}|dur|j}||dSdSdSr)rget_output_embeddingsencoderget_input_embeddings)r!tie_word_embeddingsrr)rrr tie_weightss  z/load_task_model_checkpoint..tie_weightsrrDT)rrirrr)r!r[rrrvr)NFF)F) rFrGrRrTORCH_MODEL_BIN_FILEr4rIset_default_dtyperrMr) rmodel_local_dir default_dtyperrrr ckpt_filerr[rrrvr)r_rorrrrrload_task_model_checkpoints@# 6T   rconfigcCsdt|tr |}tj|vrd|tji|tj<tj|dtd}t j |t j }t||dS)Nr1)indentcls)r.rto_dictrpipelinetaskjsondumpsrrFrGrRr CONFIGURATIONstorager<encode) target_folderrcfg_str config_filerrrsave_configuration>s  rrsave_checkpoint_name save_functionc Ks|dus t|ts td|dustj|rtd|d|dur'tdtj||}|g}t|}| t j | d| dt |drh|j durhtrhtjjd kr[t}nt}||j |t|d d z |||fi|WdSty} ztd t| jd | dd} ~ ww)asave the pretrained model, its configuration and other related files to a directory, so that it can be re-loaded Args: model (Model): Model whose params are to be saved. target_folder (Union[str, os.PathLike]): Directory to which to save. Will be created if it doesn't exist. save_checkpoint_name (str): The checkpoint name to be saved in the target_folder save_function (Callable): The function to use to save the state dictionary. Nz'A valid save function must be passed inzProvided path (z#) should be a directory, not a filez6At least pass in one checkpoint name for saving methodz *.safetensorsz.* model_dirT)ignore dirs_exist_okz)During saving checkpoints, the error of "z with msg z thrown)r.r ExceptionrFrGisfiler-rRrUaddrrr8rrsys version_infominorrrrr1r) r!rrrroutput_ckpt_pathorigin_file_to_be_ignoredignore_file_set copytree_funcerrrsave_pretrainedHsN     r )NNNTT)NN)3r9rFrrr+ collectionsrshutilrrrtypingrrrr rr4r torch.optimr torch.optim.lr_schedulerr modelscope.fileior rmodelscope.utils.configrrmodelscope.utils.constantrrmodelscope.utils.file_utilsrmodelscope.utils.loggerrmodelscope.utils.torch_utilsrrKrr rrr/boolrCrPrrPathLiker rrrrs         I + "