o Ti @stddlmZddlZddlmZddlmZddZdded efd d Z dd efd d Z dedeej fddZ dS))SetN)get_accelerator)OffloadStateTypeEnumcCs |dS)N_offload_buffer)keyrrY/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/offload_states.py_make_offload_state_key s r F pin_memory non_blockingcsLfdd}|jD]\}}d|vr||dd|vr#||dq dS)^Move optimizer states to device. Note that this assumes the state structure of DeepSpeed Adam.csft|}||vrtj||d||<rt||||<||j||d||||_dS)N)devicer )r torch empty_likerr copy_data)stateroffload_buf_keyr r r rrmove_keysz%offload_adam_states..move_keyexp_avg exp_avg_sqNritems) optimizerr r r r_rrrroffload_adam_statess   rcsJfdd}|jD]\}}d|vr||dd|vr"||dq dS)r cs |t|jd||_dS)Nr)r tor)rrr r rr move_back_key's z)reload_adam_states..move_back_keyrrNr)rr r r rrrrrreload_adam_states$s  r!rreturncs|tjkrtfddDS|tjkr"tddDS|tjkr-jjjhS|tj krLtfddDtfddDBS|tj kr_jj dkrYiSjj jhSdS)aWRetrieve the devices of the specified state of the model. Args: model (DeepSpeedEngine): The model whose device allocations are to be checked. state (OffloadStateTypeEnum): The specific state for which the devices should be retrieved. Returns: Set[torch.device]: A set of devices of the specified state. c3s|] }j|VqdSNrget_hp_param_device.0pmodelrr =sz$get_state_devices..css|]}|jjVqdSr#) ds_tensorr r&rrrr+?sc3|] }j|dVqdS)rNr$r&r)rrr+Cc3r-)rNr$r&r)rrr+Dr.N) r hp_paramsset parameters lp_paramslp_gradsrgrad_partitions_flat_bufferr optim_statescontiguous_grad_buffer6_DeepSpeedZeroOptimizer_Stage3__ipg_bucket_flat_buffer)r*rrr)rget_state_devices1s        r8)FF)F) typingrrdeepspeed.acceleratorr%deepspeed.runtime.zero.offload_configrr boolrr!r r8rrrrs