o
Ii
8 @ sn d dl Z d dlmZ d dlmZ d dlZd dlZd dlm Z ddl
m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m Z m Z eddrXd d l m!Z! nd d l"m!Z! e rjd dl#m$ m%Z& dd
l'm(Z( dd l)m*Z* e(e+Z,
d!de-de.e/ de.de.de.de0de0dee! de1de1fddZ2 d"ddZ3d#d e0de1fd d Z4d$d e0fd d Z5dS )% N)Path)Optional)
load_model )
MODEL_NAMEOPTIMIZER_NAMERNG_STATE_NAMESAFE_MODEL_NAMESAFE_WEIGHTS_NAME SAMPLER_NAME SCALER_NAMESCHEDULER_NAME WEIGHTS_NAMEget_pretty_nameis_cuda_availableis_hpu_availableis_mlu_availableis_musa_availableis_neuron_availableis_sdaa_availableis_torch_versionis_torch_xla_availableis_xpu_availableloadsavez>=z2.4.0)
GradScaler)
get_logger) PartialStateFT
output_dir model_states
optimizers
schedulers dataloaders
process_indexstepscalersave_on_each_nodesafe_serializationc
! C sb t | } t|D ]-\}
} | stnt} |
dkr | dd|
d} | | }
t| |
|| d td|
qt|D ].\}
}| } |
dkrKt
dnt
d|
d}| |}t| ||dd td| q:t|D ].\}
}| } |
dkr~t dnt d|
d}| |}t| ||dd td | qmt|D ]e\}
}|
dkrt dnt d|
d}| |}d
d l
m}m} t|j|r| }t||rt|||dd t|d dr|
dkrd
nd|
d}| |}| } t| | td|
d| q|dur"| } | t} t| | td| i } t d| d} || d< t | d< tj | d< t | d< t rNtj | d< t rZtj | d< nt! rftj" | d< n t# rqtj$ | d< t% r|tj& | d < t' rtj( | d < t) rtj* | d < t+ rt, | d < | | } t| | td | | S )!aw
Saves the current states of the models, optimizers, scaler, and RNG generators to a given directory.
If `safe_serialization` is `True`, models will be saved with `safetensors` while the rest are saved using native
`pickle`.
Args:
output_dir (`str` or `os.PathLike`):
The name of the folder to save all relevant weights and states.
model_states (`List[torch.nn.Module]`):
A list of model states
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
dataloaders (`List[torch.utils.data.DataLoader]`):
A list of dataloader instances to save their sampler states
process_index (`int`):
The current process index in the Accelerator state
step (`int`):
The current step in the internal step tracker
scaler (`torch.amp.GradScaler`, *optional*):
An optional gradient scaler instance to save;
save_on_each_node (`bool`, *optional*):
Whether to save on every node, or only the main node.
safe_serialization (`bool`, *optional*, defaults to `True`):
Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
r ._)r& r' zModel weights saved in .binFzOptimizer state saved in zScheduler state saved in r IterableDatasetShardSeedableRandomSampleruse_stateful_dataloaderdl_state_dict.bindl_state_dict_z Sampler state for dataloader z
saved in Nz Gradient scaler state saved in .pklr$ random_statenumpy_random_seedtorch_manual_seedtorch_xpu_manual_seedtorch_mlu_manual_seedtorch_sdaa_manual_seedtorch_musa_manual_seedtorch_hpu_manual_seedtorch_neuron_manual_seedtorch_cuda_manual_seedxm_seedzRandom states saved in )-r enumerater r
replacejoinpathr loggerinfo
state_dictr r
r data_loaderr, r-
isinstancedataset get_samplergetattrtorchr r randomgetstatenp get_state
get_rng_stater xpuget_rng_state_allr mlur sdaar musar hpur neuronr cudar xm)!r r r r! r" r# r$ r% r& r' istate weights_nameoutput_model_fileoptoptimizer_nameoutput_optimizer_file schedulerscheduler_nameoutput_scheduler_file
dataloader sampler_nameoutput_sampler_filer, r- samplerdataloader_state_dict_name!output_dataloader_state_dict_filerB output_scaler_filestates states_nameoutput_states_file rk L/home/ubuntu/.local/lib/python3.10/site-packages/accelerate/checkpointing.pysave_accelerator_state? s ,
"
"
"
rm c # K s t }
|dvr td|du rd}n|dkrt j}|du r i }t| } t|D ]D\} } | dkr5d| nd}
| t |
d }| rSt | |fd
t
|i| q(| t |
d }t ||d }| j
|fi | q(td
t|D ]+\} }| dkrt d nt d| d }| |}t |fd|i|}||
| qvtd t|D ]'\} }| dkrt d nt d| d }| |}t |fi |}|
| qtd t|D ]e\} }| dkrt d nt d| d }| |}ddlm}m} t|j|r| } t| | r|t |} t|ddrA| dkr$dnd| d } | | } | rAt | fi |}|
| qtd |dur_| t} t | }!|
|! td zt | t d| d}"d|"v rx|"d |
d< t |"d t j |"d t!"|"d t# rt!j$%|"d t& rt!j'%|"d n s\ \