o Ti @sVddlZddlZddlmZmZmZmZddZddZdd Z e d fd d Z dS) N)MODEL_FILE_PREFIXMODEL_FILE_SUFFIXOPTIM_FILE_SUFFIXZERO_FILE_PREFIXcCstj|t|t}|S)N)ospathjoinrr) base_folder mp_rank_str ckpt_namer N/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/checkpoint/utils.pyget_model_ckpt_name_for_rank s  rcCs6t|}dt|d}tj|||t}|S)N_02d)rrrrr r)r dp_rankmp_rank zero_prefixmp_rank_stringzero_ckpt_namer r rget_zero_ckpt_name_for_ranks  rcCs&|d|dt}tj||}|S)Nz-model_r)rrrr )r layer_idtp_rank ckpt_file ckpt_pathr r rget_layer_ckpt_name_for_ranksrcpucst|r"tturt|jkr|S|St|t r0fdd|DSt|t r@t fdd|DSt|t rTt|fdd| DS|S)a Returns a copy of ``item`` with all enclosed tensors replaced by clones on a specified device. Works on individual tensors, and tensors contained/nested in lists, tuples, and dicts. Parameters: - ``item``: tensor to clone or (possibly nested) container of tensors to clone. - ``device``: target device (defaults to 'cpu') Returns: - copy of ``item`` with cloned tensors on target device cg|]}t|qSr clone_tensors_for_torch_save.0vdevicer r =z0clone_tensors_for_torch_save..crr rr!r$r rr&?r'csi|] \}}|t|qSr r)r"kr#r$r r Asz0clone_tensors_for_torch_save..) torch is_tensortypestrr%detachcloneto isinstancelisttupledictitems)itemr%r r$rr )s       r ) rr* constantsrrrrrrrr%r r r r rs