o "i@s<UddlZddlZddlZddlZddlmZmZmZddlm Z ddl m Z m Z m Z mZmZmZmZmZmZmZmZmZddlZddlmZddlmZddlmZddlm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(dd l)m*Z*m+Z+m,Z-m.Z.m/Z/m0Z0m1Z1m2Z2dd l3m4Z4m5Z5dd l6m7Z7dd l8m9Z:dd l;mdZ?dZ@dZAeeBZCee&eejDeEeFeBfZGeeGeeGeeGeeBdffZHeeBeHfZIeeIZJeeBeeIeJffZKeLZMee eNd<ejOddZPeGdddZQeGdddeQZRejSdd  dadejTdeBdeUd eUd!eCf d"d#ZVGd$d%d%ZWd&d'ZXddd(dejTd)eejYjZd*fd+eUd,eeejTd-eeQd!eRf d.d/Z[d0eeBeHfd1eKd2eRd!dfd3d4Z\d5eejTejYjZfd6eBd!e fd7d8Z]d9eeBe fd2eRd!eeBe ffd:d;Z^dejTd2eRd!eeBeHffdd?Z`d@ejYjZd!dfdAdBZad9eKd!eeBeHffdCdDZbd@ejYjZd9eeBeHfd2eRd!eKfdEdFZcdejTdGeejYjZd*fd2eRd!eKfdHdIZddejTd@ejYjZd1eKd2eRd!eKf dJdKZedejTdGeejYjZd*fd9eKd2eRd!df dLdMZfddd(dejTd,eeejTd-eeQd!eeBeHffdNdOZgddd(dejTdGeejYjZeejYjZfd,eeejTd-eeQd!eKf dPdQZhddd(dejTdGeejYjZeejYjZfd,eeejTd-eeQd!eeeBeHfeKff dRdSZidejTd9eeejTeeBeHffeeBeHffd!eeBeHffdTdUZjddVdejTd0eeBeHfd-eeQd!e7fdWdXZkddVdejTdGeejYjZeejYjZfd1eKd-eeQd!df dYdZZlddVdejTdGeejYjZeejYjZfd0eeBeHfd1eKd-eeQd!e7f d[d\ZmeddVdejTd-eeQd!dfd]d^ZneddVdejTdGeejYjZd*fd-eeQd!dfd_d`ZodS)bN)asdict dataclassfield)chain) AnyCallablecastDict GeneratorIterableList no_type_checkOptionalSetTupleUnion) ShardedTensor)_broadcast_state_dict_flatten_state_dict_gather_state_dict_offload_state_dict_to_cpu_unflatten_state_dict)DTensor)_CHECKPOINT_PREFIX)FullOptimStateDictConfigFullStateDictConfigFullyShardedDataParallelOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigStateDictConfig StateDictType)._get_module_fsdp_state_if_fully_sharded_moduleFSDP_WRAPPED_MODULE)_IncompatibleKeys)DistributedDataParallel) tree_map_only) FQNS_T PrimitiveType ValueType DictValueTypeListDictValueTypeOptimizerStateTypeStateDictOptionsget_model_state_dictget_optimizer_state_dictget_state_dictset_model_state_dictset_optimizer_state_dictset_state_dict _flat_param param_groupsparamsstater)_patched_state_dictccs@t}tzdVW|rtdSdS|rtwwN)gc isenableddisableenable) is_enabledr?e/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/torch/distributed/checkpoint/state_dict.py _gc_context[s  rAc@sfeZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed <dZ eed <d S) r-ap This dataclass specifies how get_state_dict/set_state_dict will work. - ``full_state_dict``: if this is set to True, all the tensors in the returned state_dict will be gathered. No ShardedTensor and DTensor will be in the returned state_dict. - ``cpu_offload``: offload all the tensors to cpu. To prevent CPU OOM, if ``full_state_dict`` is also true, then only the rank0 will get the state_dict and all other ranks will get empty state_dict. - ``ignore_frozen_params``: if the value is True, the returned state_dict won't contain any frozen parameters -- the ``requires_grad`` is False. The default value is False. - ``keep_submodule_prefixes`` (deprecated): when ``submodules`` is not None, this option indicates whether to keep the submodule prefixes from the state_dict keys. or example, if the submodule is ``module.pretrain`` and the full FQN of the parameter is ``pretrain.layer1.weight`` of the param. When this option is True, the parameter's key in the returned state_dict will be ``pretrain.layer1.weight``. If the options is False, the key will be ``layer1.weight``. Note that if ``keep_submodule_prefixes`` is False, there may be conflicted FQNs, hence there should be only one submodule in ``submodules``. - ``strict``: the ``strict`` option when ``set_state_dict`` calls model.load_state_dict(). - ``broadcast_from_rank0``: when the option is True, rank0 should receive a full state_dict and will broadcast the tensors in the state_dict/ optim_state_dict one by one to other ranks. Other ranks will receive the tensors and shard according to the local shards in the model and optimizer. ``full_state_dict`` must be set to True when using this option. This option currently only supports DTensor, not the legacy ShardedTensor. Ffull_state_dict cpu_offloadignore_frozen_paramsTkeep_submodule_prefixesstrictbroadcast_from_rank0flatten_optimizer_state_dictN) __name__ __module__ __qualname____doc__rBbool__annotations__rCrDrErFrGrHr?r?r?r@r-fs  $     r-c@seZdZUeedZeeee j fee e j ffe d<eedZ eeee j fee e j ffe d<eedZeee d<dZee d<dZee d<ejZee d<eedZeeje d <d S) _StateDictInfo)default_factoryfqn_param_mappingshared_params_mappingsubmodule_prefixesT handle_model handle_optim fsdp_context fsdp_modulesN)rIrJrKrdictrQr rstrtorchTensorr'rNrRsetrSrrTrMrU contextlib nullcontextrVrlistrWr nnModuler?r?r?r@rOs   rO)maxsizeTmodelnameskip_ddp_prefixskip_compiler_prefixreturnc sx|td}d|vr |hS|d}g}|}t|D]\}}t|tr4|dks)J|j}|s3||qt|trz|t |dkrf||dt krfd |t |t } rZdfdd| j DSt |t}|tkry||t ||}qt|tjjjr|dksJ|j}|s||q|||tjjjkr|t |dkrtdqt ||}qd |tdhS) a This API is used to convert the name of a parameter to the FQNs. For FSDP without `use_orig_params`, the name of FlatParameter can be mapped to multiple original parameters. As a result, the return type of this function is `Set[str]`. Args: module (nn.Module): the root model. name (str): the name skip_ddp_prefix (bool): whether to skip DDP's `module` prefix Returns: The canonical FQNs based on the model traversal. .modulecsh|]}|qSr?r?.0fqnprefixr?r@ z_get_fqns.. _orig_modz-Expect `_extra_state` to be the last obj name)replacersplit enumerate isinstanceDDPrjappendFSDPlen _FLAT_PARAMjoingetattr_fqnsr#rZ_dynamo eval_frameOptimizedModulersr`modules_EXTRA_STATE_KEY_SUFFIX RuntimeError) rcrdrerf obj_names fqn_obj_namescurr_obji curr_obj_name flat_paramr?ror@ _get_fqnssL                 rc@s eZdZdS) _EXTRA_STATEN)rIrJrKr?r?r?r@rsrc#s:tdtjdtdtffdd |dEdHdS)Nrjcurr_fqnrgc3s||r |dnd}|D]\}}|vrq||}||EdHqt|jdd|jddD]\}}||jvrBq8||}||fVq8t|jdtj j tj j krl|tj j j }|tfVdSdS)NrirhF)recurseget_extra_state)addnamed_childrenr named_buffersnamed_parameters_non_persistent_buffers_setr~ __class__r`rarrrjrr)rjrrd submodulenew_fqnobjrvisited_modulesr?r@rs*     z+_iterate_valid_model_state..recurserh)r\r`rarYr )rcr?rr@_iterate_valid_model_statesr) submodulesoptionsoptims. optim_onlyrrc Cs&|rtdt|r|std|pt}i}i}t|D]@\}}t|tr'qt||} | |d} | durIt t t || | ||||<n| ||<| D] } t|ts\||| <qQqt|D]\} } | D] } t tj| || <qjqdt} |rt|}|D]"\}}||vrqt||} t| dksJd| dd| Dq|jr|jstdt|}|r|jrt|j|jd }t|j|jp|jd }tj}nt |jd }t!|jd }tj"}t#j$d d }t%j&|||||d }nt#j'}t(dit)|||| |t t*t+j,|| t|dkdS)zW Verify the model and options passed by the user and generates _StateDictInfo. zGetting submodules only model/optim state_dict is deprecated and will be removed in 2.5. This feature can be achieved by manually filtering out the state_dict returned from get_state_dict.z;Optimizers are not passed in but optim_only is set to True.Nrkz)Submodule FQN should only have 1 instancecss|]}|dVqdS)riNr?rlr?r?r@ Asz"_verify_options..z?full_state_dict must be True when broadcast_from_rank0 is True.)offload_to_cpu rank0_only)rc ssxt-tj||||d dVWdn1swYWddSWddS1s5wYdS)Nrjstate_dict_typestate_dict_configoptim_state_dict_config)warningscatch_warningsrzrrr?r?r@$fsdp_state_dict_type_without_warning_s "z=_verify_options..fsdp_state_dict_type_without_warningrr)rQrRrSrVrWrTrUr?)-rwarn FutureWarningrr-rrwrrgetrrrYupdatecopyr_itemsrZr[r\ named_modulesr{rGrB ValueErrorrzrWrrCrr!FULL_STATE_DICTrrSHARDED_STATE_DICTr]contextmanager functoolspartialr^rOrr r`ra)rcrrrrrQrRrdparamfqnsrnparam_fqns_rSrjrWrrrrrVr?r?r@_verify_optionss              rmodel_state_dictoptim_state_dictinfocCs|jD]}t|}|dusJdq|jr3|s3|js3|js3|jr#|js3|jr3|js3t dt d|j rH|sH|jr>|jsH|jsHt d|| D]}t|vr\t |dtdqLdS)Nz)Expected a fsdp_state with a fsdp module.z}The option indicates that model state_dict is required to save or load, but model state_dict is empty.rank = dist.get_rank()=rizgThe option indicates that model state_dict is required to save, or load but optim state_dict is empty. z contains z6. This can happen if the model is not the root module.)rWr"rTrSrDrCrBrFrGrdistget_rankrUkeysr|)rrrrj fsdp_statekeyr?r?r@_verify_state_dictsZ   rrapicCs,t||}|tvrtjt|j||d}|S)N)self)r~r8rrr)rrcallr?r?r@_state_dict_fns r state_dictcCsB|jr|jr tjstnd}t||j|dS|jrt|S|S)N)r)rC ranks_only)rBrCrZ distributedis_initializedtuplerr)rrrr?r?r@_maybe_full_or_cpu_state_dictsrc Cs|jsiS|t|d}Wdn1swYt|D]:}t||}t|dks8J||ftt|}||kr_dt fdd}|||sXt d|d|| |||<q%|j ri}|D]&}|j D] }| |svqn|jr||||<qn|t|d} |||| <qnqi|}|jr|D]\}} | jrqt||}|D]}| |qqt|D]\}} t| r| jr| |qt||S)NrrkrgcSst|t|kr dS|d}|d}d}t|D]&\}}|||kr9|d7}|t|kr8|t|dkSq|dvr>qdSdS)NFrirrk)rjrsT)r{rurv)rrn fqn_split key_splitfqn_idxkey_idxkey_namer?r?r@verifys    z%_get_model_state_dict..verifyzAn unexpected key, z, exists. FQN is )rTrVrr_rrr{nextiterrMrpoprS startswithrErDr requires_gradrrZ is_tensoris_metar) rcrrrrrnrnew_state_dictrprrpr?r?r@_get_model_state_dictsP           rc Cs\|jr|s |js tiiSi}t|D]3\}}t||}t||ddd}t||D]\}} |jr6tdkrA|| krA|||| <||| <q)q|jrd} | D]\}}t |rn| dkrn| durg|j } qP| |j ksnJqP| dusuJt||| |jd| D]\}} | ||<q|ttt|d||jdWdS1swYdS)NF)rerfr)devicerFload_state_dict)rrF)rTrGr$rrziprrrrrZrdimrrrFrVrr) rcrrlocal_state_dictrvaluerfqns_with_prefixrnfqn_with_prefixr local_stater?r?r@_load_model_state_dictsL        $roptimcCs|jrdS|jD]}|tD]}|jdurtd|jr"t||_qqg}|jD]}d|vr:||dd|d<q)|j dd|jD] }d|vrQ| d|d<qD|j dddS) zH Initialize optim states by calling the step() with zero grads. Na state_dict can only be used if the optimizer states are initialized (usually after one step() with gradients) or gradients are None. For the later case, state_dict will fake the gradients as zero to initialize the optimizer states. However, the gradients are not None.lrg)closurerT) set_to_none) r7r5_PARAMSgradrrrZ zeros_likerystepr zero_grad)r param_grouprlrsr?r?r@_init_optim_state<s0       rc Csdd}i}tt|tD] \}}tt|D]\}}||||td|d|<qqtt|tD]&}|t}ttt |D]}|D]\}}||td|d|<qKqEq7|S)aI This API flattens the optimizer state_dict to support optimizer resharding for MPMD, e.g., pipeline parallelism. Without the API, the original optimizer state_dict looks like: { "state": { "layer1.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, "layer2.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, }, "param_group": [ { "lr": 0.0, "betas": (0.9, 0.95), ..., "params": ["layer1.weight", "layer2.weight"] } ] } With this API, the optimizer state_dict looks like: { "state.layer1.weight.step": 10, "state.layer2.weight.step": 10, "state.layer1.weight.exp_avg": SomeTensor, "state.layer2.weight.exp_avg": SomeTensor, "state.layer1.weight.exp_avg_sq": SomeTensor, "state.layer2.weight.exp_avg_sq": SomeTensor, "param_group.layer1.weight.lr" : 0.1, "param_group.layer2.weight.lr" : 0.1, "param_group.layer1.weight.betas" : (0.9, 0.95), "param_group.layer2.weight.betas" : (0.9, 0.95), } Note that if any of the value is a container, like the betas in the example, this API won't flattent it. cSs*t|tjttfstdt|ddS)NzUFlattening optimizer state_dict only supports tensor, int, float states now. Type is ri)rwrZr[intfloatNotImplementedErrortype)vr?r?r@_raise_if_type_not_supportedsz?_flatten_optim_state_dict.._raise_if_type_not_supportedri) rr*_STATErr+_PGrrr rY) rrretrnr7krrrr?r?r@_flatten_optim_state_dictbs* rc Cs\i}g}t|t|i}|jD]}|tgi|tD]A}|j|D]9}|dt} t| ts0J| ||js9q!i||<|j | D]} |td|d| t t ||| <qDq!qt t t|dtd} | D]=} | tkrtqm|td| d| } | |dvr| |d| <qm|d| | krtd| d| d| d|d| d qmq |S)z This API unflattens the state_dict generated by _flatten_optim_state_dict(). See the docstring of _flatten_optim_state_dict() for more detail. rirzaAll the parameters in the same parameter group should have the same saved param_group value. But z is z while other(s) is )rrr5ryrrQrwr_rr7rrr*r rYr)rrrr7pg_state return_osdrrrnr6 state_namefirst_param_fqnrrr?r?r@_unflatten_optim_state_dictsR          r optimizersc s|jsiStitgi}|D]}t|t|d}|jrm|t|||}Wdn1s2wY|s:q t |t D]}d|vrW|t ||t| dd<qB|tD]}dd|t D}||t <q\nut tdd|jD}tt|tt|} i|D](\} } t|| } t| d ksJtt| } | | vrq| | }|| <| |<qt |t D]} | } |t | |t| <q|tD]}fd d|t D|t <q|sq tt|t|ttt|t|tq |jr ttt |}t!||S) Nrrs _orig_mod.rhcSsg|]}|ddqS)rrhrtrmrr?r?r@ rrz)_get_optim_state_dict..css|]}|tVqdSr9)r)rmgr?r?r@rsz(_get_optim_state_dict..rkcsg|]}|qSr?r?)rmpidfqn_pid_mappingr?r@r s)"rUrrrrrWrVrzrr_rrrtrr from_iterabler5rXrranger{rrrrrr*rr+extendrHr,rr)rcrrrrosdrr r6param_pid_mappingrrrrnr groupr?rr@_get_optim_state_dicts\          rc Csi}g}t|t|i}i}tddtt|tDr|S|jD]}|tgi|tD]q} |j | D]i} | |j vrXd} tt |tD]} | tt t | tvrVd} nqEnd} | s]q5|dt} t| tsjJ| | | jr}tt|t| || <tt |tD]} | tt t | tvrt|td|t| <qq5q.q!tt |tD]#}|t|d}|dkrq|D]\}}|tkrq||||<qq|S)a Extract the corresponding optim state_dict from ``optim_state_dict`` for ``optim`` and return the result optim state_dict. Args: model (nn.Module): the root model. optim (torch.optim.Optimizer): the optimizer. optim_state_dict (Dict[str, ValueType]): the superset optim state_dict that contains the optim state_dict of ``optim``. info (_StateDictInfo): state dict information. Returns: The optim state_dict of ``optim``. css|]}t|tVqdSr9)rwrr r?r?r@r(s  z*_split_optim_state_dict..FTrrk)rrallrr*rr5ryrrQrRr+r rYrwr_rr{idrr)rcrrrr7rr pg_mappingrrrn in_paramsloaded_param_groupr6idxrrr?r?r@_split_optim_state_dictsb      rc s6|jsdS|D]}t||r*t|vrt||||}nt|ttttf||}ni}|j r| D]d\}}t ||}t ||dd} || krHq3t |dksPJ| | |tD]} ttttf| } fdd| tD} | | t<q\tt|t} t| D]}|vr| || |<qq3|t|||}Wdn1swYn\|jrd|_t||f|}d|_dfdd}ttj||}dusJt|\}}t|\}}t||d |D]}||vr ||vsJ||||<||||<qt ||}t!|d |d qdS) NF)rfrkcsg|]}|qSr?r )rmr)rnfqn_with_compilerr?r@r |s z*_load_optim_state_dict..Tcs4|dkrdur|j|S|jkrtd|S)NrzDevice mismatch)rrr)trr?r@_devices  z'_load_optim_state_dict.._devicer rr)"rUrrrrrr rYr)rWrrr{rrrrr*r_rrtrVrzoptim_state_dict_to_loadrGrBrr&rZr[rrrr)rcrrrrr original_fqn_rfqns_with_compilerr valr6 osd_staterrr! flatten_osd osd_mappingflatten_local_osdlocal_osd_mapping optim_keyr?)rrnrr@_load_optim_state_dictVs               r.cCsXtt|td||d}t||}t|i||WdS1s%wYdS)aH Return the model state_dict of ``model``. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. submodules (deprecated): Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``model``. :rtype: typing.Dict[str, ValueType] FrrrN)rArrrr)rcrrrrr?r?r@r.s  $r.cCstt-t|tjjr|fnt|}t||d||d}t|||}ti|||WdS1s3wYdS)a Return the combined state_dict for optimizers. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``optimizers``. :rtype: OptimizerStateType Tr/N) rArwrZr Optimizerrrrr)rcrrrrrr?r?r@r/s    $r/cCst4t|tjjr|fnt|}t||d||d}t||}t|||}t |||||fWdS1s:wYdS)a Return the model state_dict and optimizers state_dict. ``get_state_dict`` can process any module that is parallelized by PyTorch FSDP/fully_shard, DDP/replicate, tensor_parallel/parallelize_module, and any combination of these parallelisms. The main functions of ``get_state_dict`` are: 1.) returning a model and optimizer state_dict that can be resharded with a different number of trainers and/or different parallelisms. 2.) hiding the parallelism-specific state_dict APIs. Users don't have to call these APIs. 3.) sanity checking the result state_dict. The keys of the result state dictionary are the canonical FQNs (Fully Qualified Names). A canonical FQN refers to the FQN based on a parameter's position in an nn.Module hierarchy. More specifically, a canonical FQN to a parameter is the FQN returned by ``module.named_parameters()`` or ``module.named_buffers()`` when the module is not distributed by any parallelisms. Since the optimizer internally uses parameter IDs to represent a parameter, there will be a conversion from the parameter IDs to the canonical FQNs when calling this API. ``get_state_dict`` can also process a module that is not parallelized. In such a case, ``get_state_dict`` only performs one function -- converting the optimizer parameter IDs to the canonical FQNs. Example: >>> # xdoctest: +SKIP >>> import torch >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> from torch.nn.parallel import DistributedDataParallel as DDP >>> from torch.distributed.checkpoint.state_dict import get_state_dict >>> fsdp_model = FSDP(copy.deepcopy(model)) >>> fsdp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_model = DDP(copy.deepcopy(model)) >>> ddp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_state_dict, ddp_optim_state_dict = get_state_dict(ddp_model, ddp_optim) >>> fsdp_state_dict, fsdp_optim_state_dict = get_state_dict(fsdp_model, fsdp_optim) >>> # if we simply call ddp_model.state_dict() and fsdp_model.state_dict(), >>> # the asserts will fail. >>> assert ddp_state_dict == fsdp_state_dict >>> assert ddp_optim_state == fsdp_optim_state_dict Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: ``Tuple`` that contain model state_dict and optimizer state_dict. :rtype: typing.Tuple[typing.Dict[str, ValueType], OptimizerStateType] Fr/N) rArwrZrr0rrrrr)rcrrrrrrr?r?r@r0s"F    $r0c s|siSttt|tjretdtt t tjt t t ff|}i}| D]8\}}|D]/\}}||kr;q2t||}t|dksJJdtt|d|fdd| Dq2q*|St t t t f|S)NzPassing model_state_dict as a ``Dict[nn.Module, Dict[str, Any]]``is deprecated and will be removed in 2.5. If you need this feature, please preprocessing the model_state_dict to achieve the same functionality.rkz/FQNs for a submodule should only have 1 elementricsi|] \}}||qSr?r?)rmsubfqnrror?r@ usz/_unflatten_model_state_dict..)rwrrrr`rarrrrr rYr)rrrr{r) rcrcast_state_dictrrsub_state_dictrdmrr?ror@_unflatten_model_state_dict[s,  r6)rcCs^t||}tt|td|d}t|i|t|||WdS1s(wYdS)a=Load the model state_dict. The counterpart of ``get_model_state_dict`` to set the state_dict to the model. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. model_state_dict: (Dict[str, ValueType]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys :type model_state_dict: typing.Dict[str, ValueType] FrrN)r6rArrrr)rcrrrr?r?r@r1|s  $r1cCsrt,t|tjjr|fnt|}t||d|d}ti||t||||WddS1s2wYdS)aLoad the optimizers state_dict. The counterpart of ``get_optimizer_state_dict`` to set the state_dict to the optimizers. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None :type optim_state_dict: typing.OptimizerStateType Tr7N) rArwrZrr0rrrr.)rcrrrrr?r?r@r2s  "r2cCst||}t2t|tjjr|fnt|}t||| |d}t|||t ||||t |||WdS1s=wYdS)a4Load the model state_dict and optimizers state_dict. The counterpart of ``get_state_dict`` to set the state_dict to the model and optimizers. The given ``model_state_dict`` and ``optim_state_dict`` do not have to be returned by ``get_state_dict`` but must meet the following requirements: 1) all FQNs are canonical FQNs as defined in ``get_state_dict``, 2) if a tensor is sharded, it must be either a ShardedTensor or DTensor, 3) optimizer state_dict cannot contain the parameter IDs; the keys should be the canonical FQNs. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys of the model state_dict. * **unexpected_keys** is a list of str containing the unexpected keys of the model state_dict. :type model_state_dict: typing.Dict[str, ValueType] :type optim_state_dict: typing.OptimizerStateType r7N) r6rArwrZrr0rrrr.r)rcrrrrrr?r?r@r3s*    $r3csjtjt||dfdd}||_tjt||ddtttfffdd }||_t |t |dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``model``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``model`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rcrcSr9r?r?_state_dict_callr?r@state_dict_call)z0_patch_model_state_dict..state_dict_callrc|ddS)N)rr?r"_load_state_dict_callr?r@load_state_dict_call4z5_patch_model_state_dict..load_state_dict_callN) rrr.rr1r rYrrr8r)rcrr;r@r?r?r:r@_patch_model_state_dicts   rCcstjt|||dfdd}tjt|||ddtttfffdd }t|t|t |t j j r9|fnt |}|D]}||_||_q?dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Note that if there are multiple optimizers, all of the optimizers will be patched. So users only need to call one of the state_dict() to get the full result. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rcrrcr8r9r?r?r9r?r@r;er<z4_patch_optimizer_state_dict..state_dict_callrcr=)N)rr?r"r>r?r@r@orAz9_patch_optimizer_state_dict..load_state_dict_callN)rrr/r2r rYrr8rrwrZrr0rrr)rcrrr;r@rr?rBr@_patch_optimizer_state_dict?s0    rD)TT)pr]rr:r dataclassesrrr itertoolsrtypingrrrr r r r r rrrrrZtorch.distributedrrtorch.nnr`'torch.distributed._shard.sharded_tensorr#torch.distributed._state_dict_utilsrrrrrtorch.distributed._tensorr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrtorch.distributed.fsdprrrrzrrrr r!$torch.distributed.fsdp._common_utilsr"r#torch.nn.modules.moduler$torch.nn.parallelr%rxtorch.utils._pytreer&__all__r|rrrrYr'r[rrr(r)r*r+r,r\r8rNrrAr-rO lru_cacherarMrrrrr0rrrrrrrrrrrr.r.r/r0r6r1r2r3rCrDr?r?r?r@s  8     (      .  >$    $-     C  .&@  - ? H ]   )  1  Y$  %  , -  > 6