o ϯiY@sUddlZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZmZddlmZddlmZmZeddejd dd DZe ed fe d <ed kruddl!m"Z"ddl#m$Z$m%Z%nedkre&ej"dre&ej"drddlm"Z"ddl'm$Z$m%Z%ddgZ(eddejd dd DZe ed fe d <Gddde dde e)fde e)fde e e)e ee effgZ*GdddZ+GdddZ,dej-de e)d e e)fd!d"Z.de e)d e)fd#d$Z/de e)d e)fd%d&Z0d'ee)efd(e)d dfd)d*Z1de e)d ee)e e)ffd+d,Z2d-e e)d e)fd.d/Z3 0d3dej-d(e)d e e e)ej-ffd1d2Z4dS)4N) defaultdict) AnycastDictIOIterableList NamedTupleOptionalTuple)HTTPURLHandler PathManagercolored) DataParallelDistributedDataParallelcc|]}t|VqdSNint.0xrL/home/ubuntu/.local/lib/python3.10/site-packages/fvcore/common/checkpoint.py r.. TORCH_VERSION) ) quantization)FakeQuantizeBase ObserverBaser r#r$ CheckpointerPeriodicCheckpointerccrrrrrrrr rc@s eZdZdS)_IncompatibleKeysN)__name__ __module__ __qualname__rrrrr)#s r)IncompatibleKeys missing_keysunexpected_keysincorrect_shapesc @sLeZdZdZ d1dddejdededed d f d d Z d eded d fddZ deded d fddZ d2dede e ed eeeffddZd efddZd efddZd e efddZdddeded eeeffd d!Zd"ed d fd#d$Zd%ed eeeffd&d'Zd(ed efd)d*Zd+ed d fd,d-Zd.eeefd d fd/d0Zd S)3r'z^ A checkpointer that can save/load model as well as extra checkpointable objects. T) save_to_diskmodelsave_dirr2checkpointablesreturnNcKspt|ttfr |j}||_i|_|D] \}}|||qt t |_ ||_ ||_ t|_|jtdS)a Args: model (nn.Module): model. save_dir (str): a directory to save and find checkpoints. save_to_disk (bool): if True, save checkpoint to disk, otherwise disable saving for this checkpointer. checkpointables (object): any checkpointable objects, i.e., objects that have the ``state_dict()`` and ``load_state_dict()`` method. For example, it can be used like `Checkpointer(model, "dir", optimizer=optimizer)`. N) isinstancerrmoduler3r5itemsadd_checkpointablelogging getLoggerr*loggerr4r2r path_managerregister_handlerr )selfr3r4r2r5kvrrr__init__6s zCheckpointer.__init__keycheckpointablecCs:||jvr td|dt|dstd||j|<dS)z Add checkpointable object for this checkpointer to track. Args: key (str): the key used to save the object checkpointable: any object with ``state_dict()`` and ``load_state_dict()`` method zKey z! already used in the Checkpointer state_dictz>add_checkpointable needs an object with 'state_dict()' method.N)r5KeyErrorhasattr TypeError)r@rDrErrrr:Ws  zCheckpointer.add_checkpointablenamekwargsc Ks|jr|jsdSi}|j|d<|jD] \}}|||<q||d|}tj |j|}tj ||ks?J||j d||j|d}t|ttt|Wdn1sewY||dS)z Dump model and checkpointables to a file. Args: name (str): name of the file. kwargs (dict): extra arbitrary data to save. Nr3z{}.pthzSaving checkpoint to {}wb)r4r2r3rFr5r9updateformatospathjoinbasenamer=infor>opentorchsaverrbytestag_last_checkpoint) r@rJrKdatarDobjrR save_filefrrrrVhs   zCheckpointer.saverPcCs|s |jdiS|jd|tj|s,|j|}tj|s,Jd|||}| |}|dur?| ||durF|j n|D]}||vre|jd|||j |}| | |qH|S)a Load from the given checkpoint. Args: path (str): path or url to the checkpoint. If empty, will not load anything. checkpointables (list): List of checkpointable names to load. If not specified (None), will load all the possible checkpointables. Returns: dict: extra data loaded from the checkpoint that has not been processed. For example, those saved with :meth:`.save(**extra_data)`. z4No checkpoint found. Initializing model from scratchz"[Checkpointer] Loading from {} ...zCheckpoint {} not found!NzLoading {} from {} ...)r=rSrNrOrPisfiler>get_local_path _load_file _load_model_log_incompatible_keysr5load_state_dictpop)r@rPr5 checkpoint incompatiblerDrZrrrloads$       zCheckpointer.loadcCstj|jd}|j|S)za Returns: bool: whether a checkpoint exists in the target directory. last_checkpoint)rOrPrQr4r>exists)r@r[rrrhas_checkpoints zCheckpointer.has_checkpointcCsttj|jd}z|j|d}|}Wdn1s!wYWn ty1YdSwtj|j|S)z[ Returns: str: The latest checkpoint file in target directory. rgrNr1) rOrPrQr4r>rTreadstripIOError)r@r[r\ last_savedrrrget_checkpoint_files z Checkpointer.get_checkpoint_filecs fddjjD}|S)z} Returns: list: All available checkpoint files (.pth files) in target directory. cs>g|]}jtjj|r|drtjj|qS)z.pth)r>r]rOrPrQr4endswith)rfiler@rr sz9Checkpointer.get_all_checkpoint_files..)r>lsr4)r@all_model_checkpointsrrrrget_all_checkpoint_filess  z%Checkpointer.get_all_checkpoint_files)resumerwcCs,|r|r|}||S|j|gdS)aG If `resume` is True, this method attempts to resume from the last checkpoint, if exists. Otherwise, load checkpoint from the given path. This is useful when restarting an interrupted training job. Args: path (str): path to the checkpoint. resume (bool): if True, resume from the last checkpoint if it exists and load the model together with all the checkpointables. Otherwise only load the model without loading any checkpointables. Returns: same as :meth:`load`. )r5)rirorf)r@rPrwrrrresume_or_loads  zCheckpointer.resume_or_loadlast_filename_basenamecCsNtj|jd}|j|d}||WddS1s wYdS)z Tag the last checkpoint. Args: last_filename_basename (str): the basename of the last filename. rgwN)rOrPrQr4r>rTwrite)r@ryr[r\rrrrXs "z Checkpointer.tag_last_checkpointr\cCstj|tddS)a Load a checkpoint file. Can be overwritten by subclasses to support different formats. Args: f (str): a locally mounted file path. Returns: dict: with keys "model" and optionally others that are saved by the checkpointer dict["model"] must be a dict which maps strings to torch.Tensor or numpy arrays. cpu) map_location)rUrfdevice)r@r\rrrr_s zCheckpointer._load_filerdcCs |d}||t|d|j}g}t|D]b}||vr~||}tdkr2t|t j j r2qt |j }t ||j }||kr~tdkoOttdoOttd} | rqdtj jdtdtj jfdd } ttf} | |j|} t| | rqq||||f||q|jj|d d } t| j| j|d S) a Load weights from a checkpoint. Args: checkpoint (Any): checkpoint contains the weights. Returns: ``NamedTuple`` with ``missing_keys``, ``unexpected_keys``, and ``incorrect_shapes`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys * **incorrect_shapes** is a list of (key, shape in checkpoint, shape in model) This is just like the return value of :func:`torch.nn.Module.load_state_dict`, but with extra support for ``incorrect_shapes``. r3zmodule.r%r$r#rDr6cSs.|ddd}|}|D]}t||}q |S)Nr)splitgetattr)r3rD key_parts cur_modulekey_partrrr_get_module_for_key/s  z5Checkpointer._load_model.._get_module_for_keyF)strict)r.r/r0)rc_convert_ndarray_to_tensor_strip_prefix_if_presentr3rFlistkeysrr7nn parameterUninitializedParametertupleshaperHr"rUModulestrr$r#appendrbr)r.r/)r@rdcheckpoint_state_dictmodel_state_dictr0rA model_param shape_modelshape_checkpointhas_observer_base_classesr cls_to_skip target_modulererrrr`sV           zCheckpointer._load_modelrecCsp|jD]\}}}|jd|||q|jr(t|j|j}|r(|jt||jr6|jt |jdSdS)zZ Log information about the incompatible keys returned by ``_load_model``. zSkip loading parameter '{}' to the model due to incompatible shapes: {} in the checkpoint but {} in the model! You might want to double check if this is expected.N) r0r=warningrNr._filter_reused_missing_keysr3get_missing_parameters_messager/!get_unexpected_parameters_message)r@rerArrr.rrrraMs$z#Checkpointer._log_incompatible_keysrFcCsdt|D])}||}t|tjs"t|tjs"td|t |t|tjs/t |||<qdS)z In-place convert all numpy arrays in the state_dict to torch tensor. Args: state_dict (dict): a state-dict to be loaded to the model. Will be modified. z,Unsupported type found in checkpoint! {}: {}N) rrr7npndarrayrUTensor ValueErrorrNtype from_numpy)r@rFrArBrrrrds  z'Checkpointer._convert_ndarray_to_tensorr1r)r*r+r,__doc__rrrboolrrCr:rVr rrrfrirorvrxrXr_r)r`rarrrrrr'0sF !   *$ Oc @sneZdZdZ   ddededeedeeded df d d Zd ed e d dfddZ ded e d dfddZ dS)r(a- Save checkpoints periodically. When `.step(iteration)` is called, it will execute `checkpointer.save` on the given checkpointer, if iteration is a multiple of period or if `max_iter` is reached. Attributes: checkpointer (Checkpointer): the underlying checkpointer object Nr3 checkpointerperiodmax_iter max_to_keep file_prefixr6cCsH||_t||_||_|dur|dksJ||_g|_|j|_||_dS)a Args: checkpointer: the checkpointer object used to save checkpoints. period (int): the period to save checkpoint. max_iter (int): maximum number of iterations. When it is reached, a checkpoint named "{file_prefix}_final" will be saved. max_to_keep (int): maximum number of most current checkpoints to keep, previous checkpoints will be deleted file_prefix (str): the prefix of checkpoint's filename Nr)rrrrrrecent_checkpointsr>r)r@rrrrrrrrrCs   zPeriodicCheckpointer.__init__ iterationrKcKst|}d|i}|||d|jdkrV|jjd|j|fi||jdurV|j |j t |j|jkrV|j d}|j |rV||jdsV|j ||jdurr||jdkrt|jj|jdfi|dSdSdS)a  Perform the appropriate action at the given iteration. Args: iteration (int): the current iteration, ranged in [0, max_iter-1]. kwargs (Any): extra data to save, same as in :meth:`Checkpointer.save`. rr rz {}_{:07d}Nz _final.pth_final)rrMrrrVrNrrrrrolenrcr>rhrprmr)r@rrKadditional_statefile_to_deleterrrsteps0        zPeriodicCheckpointer.steprJcKs|jj|fi|dS)a Same argument as :meth:`Checkpointer.save`. Use this method to manually save checkpoints outside the schedule. Args: name (str): file name. kwargs (Any): extra data to save, same as in :meth:`Checkpointer.save`. N)rrV)r@rJrKrrrrVs zPeriodicCheckpointer.save)NNr3) r*r+r,rr'rr rrCrrrVrrrrr(xs(  r3rr6c st|tt}t|D]*\}}t|jddt|jddD]\}}|r*|dnd|}|||q q |D]!}tfdd|Dr\t fdd|Ds\fdd|Dq;tS) z\ Filter "missing keys" to not include keys that have been loaded with another name. F)recurserr1c3s|]}|vVqdSrrrnkeysetrrrrz._filter_reused_missing_keys..csg|] }|vr|qSr)removerrrrrssz/_filter_reused_missing_keys..) setr_named_modules_with_duprnamed_parameters named_buffersaddvaluesanyall) r3rparam_to_names module_prefixr8rJparam full_namenamesrrrrs  ,rcCs>t|}tdd|D}d}|ddd|D7}|S)z Get a logging-friendly message to report parameter names (keys) that are in the model but not found in a checkpoint. Args: keys (list[str]): List of keys that were not found in the checkpoint. Returns: str: message. css |] \}}|t|VqdSr) _group_to_strrrArBrrrrsz1get_missing_parameters_message..zBSome model parameters or buffers are not found in the checkpoint:  cSsg|]}t|dqS)bluerrrrrrssz2get_missing_parameters_message..)_group_checkpoint_keyssortedr9rQ)rgroups msg_per_groupmsgrrrrs  rcCs,t|}d}|ddd|D7}|S)z Get a logging-friendly message to report parameter names (keys) that are in the checkpoint but not found in the model. Args: keys (list[str]): List of keys that were not found in the model. Returns: str: message. zHThe checkpoint state_dict contains keys that are not used by the model: rcss*|]\}}dt|t|dVqdS)z magentaN)rrrrrrrs z4get_unexpected_parameters_message..)rrQr9)rrrrrrrs    rrFprefixcst|}tfdd|DsdS|D]}|td}||||<qz|j}Wn ty6YdSwt|D]}t|dkrFq=|td}||||<q=dS)z Strip the prefix in metadata, if any. Args: state_dict (OrderedDict): a state-dict to be loaded to the model. prefix (str): prefix. c3s&|]}t|dkp|VqdS)rN)r startswith)rrDrrrrs$z+_strip_prefix_if_present..Nr)rrrrrc _metadataAttributeErrorr)rFrrrDnewkeymetadatarrrrs"    rcCsbtt}|D](}|d}|dkr"|d|||ddg}}n|g}}|||q|S)a. Group keys based on common prefixes. A prefix is the string up to the final "." in each key. Args: keys (list[str]): list of parameter names, i.e. keys in the model checkpoint dict. Returns: dict[list]: keys with common prefixes are grouped into lists. rrNr )rrrfindextend)rrrDposheadtailrrrr s  " rgroupcCs>t|dkrdSt|dkrd|dSddt|dS)z Format a group of parameter name suffixes into a loggable string. Args: group (list[str]): list of parameter name suffixes. Returns: str: formated string. rr1r rz.{z, })rrQr)rrrrr5s   rr1ccsR||fV|jD]\}}|durq ||rdnd|}t||EdHq dS)z{ The same as `model.named_modules()`, except that it includes duplicated modules that have more than one name. Nrr1)_modulesr9r)r3rrJr8submodule_prefixrrrrFs rr)5r;rO collectionsrtypingrrrrrrr r r numpyrrUtorch.nnriopath.common.file_ior r termcolorrtorch.nn.parallelrrr __version__rrr__annotations__torch.aor"torch.ao.quantizationr#r$rHtorch.quantization__all__rr)r'r(rrrrrrrrrrrrsd  ,  2    2    J R"!