o ߗiP @s`UddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZddlZddlmZddlmZddlmZmZddlmZdd lmZdd lmZmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,dd l-m.Z.m/Z/m0Z0m1Z1dd l2m3Z3ddl4m5Z5ddl6m7Z7e8e9Z:ej;eGddde'Z?Gddde?Z@ d6de eAe fdedeBde&fddZCd e e&de e&fd!d"ZDde eAe fd#eBde)fd$d%ZE d6d e e)d&eBdee e)effd'd(ZFde!defd)d*ZGd+ed,edeBfd-d.ZHd/ejId0edeBfd1d2ZJd3e e)dedeBfd4d5ZKdS)7N)ChainMap)reduce)AnycastDictListOptionalTupleUnion)narrow_tensor_by_index)dedup_save_plans)FLATTEN_MAPPINGflatten_state_dict)_flatten_sharded_tensors) set_element)BytesStorageMetadataChunkStorageMetadataMetadata MetadataIndexSTATE_DICT_TYPE STORAGE_TYPES StorageMetaTensorStorageMetadata)LoadPlan LoadPlannerReadItemSavePlan SavePlanner WriteItem WriteItemType)"_create_default_metadata_only_plan_create_read_items_create_write_items_init_state_dict)find_state_dict_object)DTensor)_versionlogger)DefaultSavePlannerDefaultLoadPlannercreate_default_local_load_plancreate_default_global_load_plancreate_default_local_save_plancreate_default_global_save_planc @seZdZUeed<    d"dededeeded df d d Z  d#d ed ee ded dfddZ d e fddZ de e d ee e effddZde d e fddZded eejejffddZded efddZdedefd d!ZdS)$r)mappingsTNFrflatten_sharded_tensorsdedup_replicated_tensorsdedup_save_to_lowest_rankreturncCs2||_||_i|_||_|durtddSdS)NzDefaultSavePlanner's `dedup_replicated_tensors` argument is being deprecated, and no longer has any effect. Please remove this argument from your call.)rr0r/r2r(warning)selfrr0r1r2r6j/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/torch/distributed/checkpoint/default_planner.py__init__EszDefaultSavePlanner.__init__ state_dict storage_metais_coordinatorcCs2|jr t|\}|_|jrt|}||_||_dSN)rr/r0rr9r;)r5r9r:r;r6r6r7set_up_plannerWs  z!DefaultSavePlanner.set_up_plannercCs0t|j|j}|jrtj||jd}||_|jS)N planner_data)r-r9r;r dataclassesreplacer/plan)r5rBr6r6r7create_local_plands z$DefaultSavePlanner.create_local_plan all_planscCspt||j}t|\}}|jr#dd|D}tt|}tj||d}t||s,t d||_ ||_ |j |j fS)NcSsg|]}|jqSr6r>).0pr6r6r7 xsz9DefaultSavePlanner.create_global_plan..r>zFailed to validate global plan) r r2r.rdictrr@rA_validate_global_plan ValueError global_planmetadata)r5rDrKrLplanner_data_dictmerged_mappingsr6r6r7create_global_planls     z%DefaultSavePlanner.create_global_plannew_plancCs ||_|Sr<)rBr5rPr6r6r7 finish_planszDefaultSavePlanner.finish_plan write_itemcC||j}|||Sr<) lookup_objectindextransform_object)r5rSobjectr6r6r7 resolve_data  zDefaultSavePlanner.resolve_datarVcC t|j|SzSExtension from the planner interface to make it easy to extend the default planner.r$r9r5rVr6r6r7rU z DefaultSavePlanner.lookup_objectrXcCs(|jtjkrt}t|||}|Sr\)typerBYTE_IOioBytesIOtorchsave)r5rSrXbytesr6r6r7rWs  z#DefaultSavePlanner.transform_object)TTNFNF)__name__ __module__ __qualname__r __annotations__boolrr8rrr=rrCrr rrOrRrr rdTensorrbrcrYrrrUrWr6r6r6r7r)BsL     r)c @seZdZUdZeed<eed<   d(dededed d fd d Z d)d ede e ded d fddZ d e fddZ dee d ee fddZde d e fddZdedejd d fddZdefddZded ejd d fd!d"Zd#ed ejfd$d%Zded ejfd&d'Zd S)*r*ak DefaultLoadPlanner that adds multiple features on top of LoadPlanner. In particular it adds the following: flatten_state_dict: Handle state_dict with nested dicts flatten_sharded_tensors: For FSDP in 2D parallel mode allow_partial_load: If False, will raise a runtime error if a key is present in state_dict, but not in the checkpoint. original_state_dictr/TFrr0allow_partial_loadr3NcCs"||_||_i|_i|_||_dSr<)rr0rnr/ro)r5rr0ror6r6r7r8s  zDefaultLoadPlanner.__init__r9rLr;cCsFt|||_|jrt|}|jrt|\}|_||_||_||_dSr<) r#rnr0rrr/r9rLr;)r5r9rLr;r6r6r7r=s z!DefaultLoadPlanner.set_up_plannercCs|jdusJ|jr=t|j}t|jj}||}|r=dt_t|j\}}t|}||@r:|||_|_ dt_t |j|j|j S)N2_3) rLrsetr9keysstate_dict_metadatar'_derived_versionrnr/r+ro)r5 current_keys load_keys missing_keysold_state_dict old_mappingsold_keysr6r6r7rCs" z$DefaultLoadPlanner.create_local_planrKcCst|Sr<)r,)r5rKr6r6r7rOsz%DefaultLoadPlanner.create_global_planrPcCs|Sr<r6rQr6r6r7rRzDefaultLoadPlanner.finish_plan read_itemvaluecCsH|jrt|j|j|jjtj|dddStj|dd|j|jj<dS)NF) weights_only) rrrnr/ dest_indexfqnrdloadr9)r5r|r}r6r6r7 load_bytess  zDefaultLoadPlanner.load_bytescCrTr<) lookup_tensorrtransform_tensorr5r|tensorr6r6r7resolve_tensorrZz!DefaultLoadPlanner.resolve_tensorrcCsdSr<r6rr6r6r7 commit_tensorr{z DefaultLoadPlanner.commit_tensorrVcCr[r\r]r^r6r6r7rr_z DefaultLoadPlanner.lookup_tensorcCst||j|jSr\)r dest_offsetslengthsrr6r6r7r sz#DefaultLoadPlanner.transform_tensor)TTFrg)rhrirj__doc__rrkr rlr8rrr=rrCrrOrRrrbrcrrrdrmrrrrr6r6r6r7r*sF    ' r*c s`eZdZdZdfdd Zdededefdd Z  dd e de ed eddffd d Z Z S)_EmptyStateDictLoadPlannera Extension of DefaultLoadPlanner, which rebuilds state_dict from the saved metadata. Useful for loading in state_dict without first initializing a model, such as when converting a DCP checkpoint into a Torch save file. . N.B. `state_dict` must be an empty dictionary when used with this LoadPlanner .. warning:: Because the entire state dict is initialized, It's recommended to only utilize this LoadPlanner on a single rank or process to avoid OOM. Ncs||_tj|i|dSr<)rrsuperr8)r5rrargskwargs __class__r6r7r8sz#_EmptyStateDictLoadPlanner.__init__keyrLr3cs~jdurdS|jvr g}|j|}|D]}|r*|d|dt|gq||qtfdd|Dr=dSdS)NT.c3s|]}|jvVqdSr<)rr)rEunflattened_keyr5r6r7 3szA_EmptyStateDictLoadPlanner._should_include_key..F)rrr?getappendjoinstrany)r5rrLunflattened_keysr?rr6rr7_should_include_key!s    z._EmptyStateDictLoadPlanner._should_include_keyFr9r;cs|rJ|dus J|jD]-\}}|||sqt|tr)tj|j|jj d}||j vr8t ||j ||q|||<qt |||dS)N)dtype)rsitemsr isinstancerrdemptysize propertiesrr?rrr=)r5r9rLr;kvrr6r7r=8s     z)_EmptyStateDictLoadPlanner.set_up_plannerr<rg) rhrirjrr8rrrlrrrr= __classcell__r6r6rr7rs rTr9rLstrictr3cCsg} |D]Z\}}||jvr|rtd|dq|j|}t|trDt|dddurD|j|krDtd|jd|d|t|trY|j durX|t |||7}q|t |||7}qt |S)Nz&Missing key in checkpoint state_dict: rrzSize mismatch between saved z and current: z for ) rrs RuntimeErrorrrgetattrrrJr% device_meshget_coordinater!r)r9rLrrequestsrobjmdr6r6r7r+Ps*   r+rDcCs|S)z Create global load plan used by DefaultLoadPlanner. The default load behavior involved no global coordination and this function currently doesn't change the local plans. r6)rDr6r6r7r,zs r,r;cCsTg}|D]\}}t|tr|jdur|t||7}q|t||7}qt|S)a Create the ``SavePlan`` used by DefaultSavePlanner. On non-coordinator ranks, this function ignores tensors and non-tensor objects, only producing writes for ShardedTensor objects. On the coordinator rank, produce writes for all values. N)rrr%rrr"r)r9r;rrrr6r6r7r-s  r-rewrite_index_hintsc Cs i}g}|D]}g}|jD]q}|jtjks|jj|vsJ|jtjkr0t||jj<||q |j dus7Jt t | |jjt |j j |j jgd}|}|ratj|jt|jd} tj|| d}|||j jdusvJd|jjd|j|j jq |tj||dq|t|fS)a6 Create the global plan and metadata used by DefaultSavePlanner. Metadata is produced by concatenating the metadata of all ``WriteItem`` from the supplied plans. The only global planning change is to update index hints in all ``MetadataIndex`` objects if ``rewrite_index_hints`` is True. N)rrchunks)rVzZ Cannot create MD for tensor without bounds. FQN: z )r)rr`rSHARDrVrrarr tensor_datarr setdefaultrrr@rAlenrchunkr) rDrr new_plansrB new_itemsitem tensor_mdnew_item new_indexr6r6r7r.sJ         r.cCst|}t|g\}}|S)zTReturn the ``Metadata`` if DefaultSavePlanner was used to checkpoint ``state_dict``.)r r.)r9rB_rr6r6r7_create_default_local_metadatasrbox0box1cCsdt|j}t|D]&}|j||j||j|krdS|j||j||j|kr/dSq dS)z9Check if two boxes overlap. Tuples are (offset, lengths).FT)roffsetsrangesizes)rrndimsir6r6r7_check_box_overlaps  router_box_size inner_boxcCs`tt|D]'}|j|dkrdS|j|dkrdS|j||j|||kr-dSqdS)NrFT)rrrr)rrrr6r6r7_check_box_boundssrrKc Csd}|jD]j\}}t|trqt|jdkrqd}t|jD]:\}}t|j|s5t d||j|d}|t t j |jd7}|j|ddD]}t||rYt d|||d}qHq t t j |jd} || krqt d|| |d}q|S)NTrz~ key:%s has out of bounds chunk: tensor-size:%s chunk: %s Fr&z$key:%s has overlapping chunks: %s %szq key:%s invalid fill tensor-volume: %s chunks-volume: %s )rsrrrrr enumeraterrr(r4roperatormulrr) rKrLall_goodrr} chunks_volume chunk_idxchunk0chunk1 tensor_volumer6r6r7rIsH     rI)T)Lr@rbloggingr collectionsr functoolsrtypingrrrrrr r rdtorch.distributed._shard._utilsr .torch.distributed.checkpoint._dedup_save_plansr )torch.distributed.checkpoint._nested_dictr r2torch.distributed.checkpoint._sharded_tensor_utilsr&torch.distributed.checkpoint._traverser%torch.distributed.checkpoint.metadatarrrrrrrr$torch.distributed.checkpoint.plannerrrrrrrr,torch.distributed.checkpoint.planner_helpersr r!r"r#"torch.distributed.checkpoint.utilsr$torch.distributed.tensorr%r' getLoggerrhr(Loggerrk__all__r)r*rrrlr+r,r-r.rrSizerrIr6r6r6r7s   $    ($     WvB  *    7