o
3wi6 @ sZ d dl Z d dlmZ d dlZd dlZd dlmZ ddlm Z m
Z
m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z eddrPd dl m Z nd dl m Z e rbd dl m! m"Z# dd l$m%Z% dd
l&m'Z' e%e(Z) d d
e*de+e, de+de+de+de-de-de de.de.fddZ/ d!ddZ0d"de-de.fd d Z1d#de-fd d Z2dS )$ N)Path)
load_model )
MODEL_NAMEOPTIMIZER_NAMERNG_STATE_NAMESAFE_MODEL_NAMESAFE_WEIGHTS_NAME SAMPLER_NAME SCALER_NAMESCHEDULER_NAME WEIGHTS_NAMEget_pretty_nameis_cuda_availableis_hpu_availableis_mlu_availableis_musa_availableis_sdaa_availableis_torch_versionis_torch_xla_availableis_xpu_availableloadsavez>=z2.4.0)
GradScaler)
get_logger) PartialStateFT
output_dir model_states
optimizers
schedulers dataloaders
process_indexstepscalersave_on_each_nodesafe_serializationc
! C sL t | } t|D ]-\}
} | stnt} |
dkr | dd|
d} | | }
t| |
|| d td|
qt|D ].\}
}| } |
dkrKt
dnt
d|
d}| |}t| ||dd td| q:t|D ].\}
}| } |
dkr~t dnt d|
d}| |}t| ||dd td | qmt|D ]e\}
}|
dkrt dnt d|
d}| |}d
d l
m}m} t|j|r| }t||rt|||dd t|d dr|
dkrd
nd|
d}| |}| } t| | td|
d| q|dur"| } | t} t| | td| i } t d| d} || d< t | d< tj | d< t | d< t rNtj | d< t rZtj | d< nt! rftj" | d< n t# rqtj$ | d< t% r|tj& | d < t' rtj( | d < t) rt* | d < | | } t| | td | | S ) aw
Saves the current states of the models, optimizers, scaler, and RNG generators to a given directory.
If `safe_serialization` is `True`, models will be saved with `safetensors` while the rest are saved using native
`pickle`.
Args:
output_dir (`str` or `os.PathLike`):
The name of the folder to save all relevant weights and states.
model_states (`List[torch.nn.Module]`):
A list of model states
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
dataloaders (`List[torch.utils.data.DataLoader]`):
A list of dataloader instances to save their sampler states
process_index (`int`):
The current process index in the Accelerator state
step (`int`):
The current step in the internal step tracker
scaler (`torch.amp.GradScaler`, *optional*):
An optional gradient scaler instance to save;
save_on_each_node (`bool`, *optional*):
Whether to save on every node, or only the main node.
safe_serialization (`bool`, *optional*, defaults to `True`):
Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
r ._)r$ r% zModel weights saved in .binFzOptimizer state saved in zScheduler state saved in r IterableDatasetShardSeedableRandomSampleruse_stateful_dataloaderdl_state_dict.bindl_state_dict_z Sampler state for dataloader z
saved in Nz Gradient scaler state saved in .pklr" random_statenumpy_random_seedtorch_manual_seedtorch_xpu_manual_seedtorch_mlu_manual_seedtorch_sdaa_manual_seedtorch_musa_manual_seedtorch_hpu_manual_seedtorch_cuda_manual_seedxm_seedzRandom states saved in )+r enumerater
r replacejoinpathr loggerinfo
state_dictr r r
data_loaderr* r+
isinstancedataset get_samplergetattrtorchr r randomgetstatenp get_state
get_rng_stater xpuget_rng_state_allr mlur sdaar musar hpur cudar xm)!r r r r r r! r" r# r$ r% istate weights_nameoutput_model_fileoptoptimizer_nameoutput_optimizer_file schedulerscheduler_nameoutput_scheduler_file
dataloader sampler_nameoutput_sampler_filer* r+ samplerdataloader_state_dict_name!output_dataloader_state_dict_filer? output_scaler_filestates states_nameoutput_states_file rg U/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/accelerate/checkpointing.pysave_accelerator_state= s~ ,
"
"
"
ri c # K s t }
|dvr td|du rd}n|dkrt j}|du r i }t| } t|D ]D\} } | dkr5d| nd}
| t |
d }| rSt | |fd
t
|i| q(| t |
d }t ||d }| j
|fi | q(td
t|D ]+\} }| dkrt d nt d| d }| |}t |fd|i|}||
| qvtd t|D ]'\} }| dkrt d nt d| d }| |}t |fi |}|
| qtd t|D ]e\} }| dkrt d nt d| d }| |}ddlm}m} t|j|r| } t| | r|t |} t|ddrA| dkr$dnd| d } | | } | rAt | fi |}|
| qtd |dur_| t} t | }!|
|! td z|t | t d| d}"d|"v rx|"d |
d< t |"d t j |"d t!"|"d t# rt!j$%|"d t& rt!j'%|"d n"t( rt!j)%|"d nt* rt!j+%|"d! nt!j,%|"d" t- rt."|"d# td$ W |
S t/y td% Y |
S w )&a
Loads states of the models, optimizers, scaler, and RNG generators from a given directory.
Args:
input_dir (`str` or `os.PathLike`):
The name of the folder to load all relevant weights and states.
models (`List[torch.nn.Module]`):
A list of model instances
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
process_index (`int`):
The current process index in the Accelerator state
scaler (`torch.amp.GradScaler`, *optional*):
An optional *GradScaler* instance to load
map_location (`str`, *optional*):
What device to load the optimizer state onto. Should be one of either "cpu" or "on_device".
load_kwargs (`dict`, *optional*):
Additional arguments that can be passed to the `load` function.
load_model_func_kwargs (`dict`, *optional*):
Additional arguments that can be passed to the model's `load_state_dict` method.
Returns:
`dict`: Contains the `Accelerator` attributes to override while loading the state.
)Ncpu on_devicezaUnsupported optimizer map location passed, please choose one of `None`, `'cpu'`, or `'on_device'`Nrj rk r r' z .safetensorsdevicer( ) map_locationz%All model weights loaded successfullyrn z(All optimizer states loaded successfullyz(All scheduler states loaded successfullyr r) r, Fr- r. z1All dataloader sampler states loaded successfullyz$GradScaler state loaded successfullyr/ r" r0 r1 r2 r3 r4 r5 r6 r8 r9 z%All random states loaded successfullyz Could not load random states)0dict TypeErrorr rm r r: r< r existsr strr r load_state_dictr= r> r r r
r@ r* r+ rA rB rC set_samplerrD r r rF setstaterH set_staterE
set_rng_stater rK set_rng_state_allr rM r rN r rO rQ r rR Exception)# input_dirmodelsr r r r! r# rn load_kwargsload_model_func_kwargsoverride_attributesrS modelendinginput_model_filer? rW rX input_optimizer_fileoptimizer_staterZ r[ input_scheduler_filescheduler_stater] r^ input_sampler_filer* r+ r` ra input_dataloader_state_dict_fileinput_scaler_file scaler_staterd rg rg rh load_accelerator_state s '
"
"
"
r indexc C sD t |d| d }tdt| d| t| ||d dS )zL
Saves the state of `obj` to `{path}/custom_checkpoint_{index}.pkl`
custom_checkpoint_r/ zSaving the state of z to )r$ N)r r= r> r r r? )objpathr r$
save_locationrg rg rh save_custom_state9 s r c C sB | d| d}t dt| d| | t|ddd dS ) z
Loads the state of `obj` at `{path}/custom_checkpoint_{index}.pkl`. Will always set `weights_only=False` when
loading the state.
z/custom_checkpoint_r/ zLoading the state of z from rj F)rn weights_onlyN)r= r> r rs r )r r r
load_locationrg rg rh load_custom_stateC s r )NFT)NNN)r F)r )3rF pathlibr numpyrH rE safetensors.torchr utilsr r r r r r
r r r
r r r r r r r r r r r torch.ampr torch.cuda.amptorch_xla.core.xla_modelcore xla_modelrR loggingr rT r __name__r= rr listro intboolri r r r rg rg rg rh sZ X
}