o "iG @sTUddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZddlZddlmZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&dd l'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.dd l/m0Z0m1Z1m2Z2m3Z3ddl4m5Z5e6e7Z8ej9e:d<gdZ;Gddde,Z d4de e?e fde!de@de(fddZAde e(de e(fdd ZBde e?e fd!e@de+fd"d#ZC d4de e+d$e@dee e+e!ffd%d&ZDde#de!fd'd(ZEd)e d*e de@fd+d,ZFd-ejGd.e de@fd/d0ZHd1e e+de!de@fd2d3ZIdS)5N)ChainMap)reduce)AnycastDictListOptionalTupleUnion)narrow_tensor_by_index)DTensor)dedup_save_plans)FLATTEN_MAPPINGflatten_state_dict)_flatten_sharded_tensors) set_element)BytesStorageMetadataChunkStorageMetadataMetadata MetadataIndexSTATE_DICT_TYPE STORAGE_TYPES StorageMetaTensorStorageMetadata)LoadPlan LoadPlannerReadItemSavePlan SavePlanner WriteItem WriteItemType)"_create_default_metadata_only_plan_create_read_items_create_write_items_init_state_dict)find_state_dict_objectlogger)DefaultSavePlannerDefaultLoadPlannercreate_default_local_load_plancreate_default_global_load_plancreate_default_local_save_plancreate_default_global_save_planc @seZdZUeed<    d"dededeeded df d d Z  d#d ed ee ded dfddZ d e fddZ de e d ee e effddZde d e fddZded eejejffddZded efddZdedefd d!ZdS)$r'mappingsTNFrflatten_sharded_tensorsdedup_replicated_tensorsdedup_save_to_lowest_rankreturncCs2||_||_i|_||_|durtddSdS)NzDefaultSavePlanner's `dedup_replicated_tensors` argument is being deprecated, and no longer has any effect. Please remove this argument from your call.)rr.r-r0r&warning)selfrr.r/r0r4j/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/torch/distributed/checkpoint/default_planner.py__init__BszDefaultSavePlanner.__init__ state_dict storage_metais_coordinatorcCs2|jr t|\}|_|jrt|}||_||_dSN)rr-r.rr7r9)r3r7r8r9r4r4r5set_up_plannerTs  z!DefaultSavePlanner.set_up_plannercCs0t|j|j}|jrtj||jd}||_|jS)N planner_data)r+r7r9r dataclassesreplacer-plan)r3r@r4r4r5create_local_planas z$DefaultSavePlanner.create_local_plan all_planscCspt||j}t|\}}|jr#dd|D}tt|}tj||d}t||s,t d||_ ||_ |j |j fS)NcSsg|]}|jqSr4r<).0pr4r4r5 usz9DefaultSavePlanner.create_global_plan..r<zFailed to validate global plan) r r0r,rdictrr>r?_validate_global_plan ValueError global_planmetadata)r3rBrIrJplanner_data_dictmerged_mappingsr4r4r5create_global_planis     z%DefaultSavePlanner.create_global_plannew_plancCs ||_|Sr:)r@r3rNr4r4r5 finish_planszDefaultSavePlanner.finish_plan write_itemcC||j}|||Sr:) lookup_objectindextransform_object)r3rQobjectr4r4r5 resolve_data  zDefaultSavePlanner.resolve_datarTcC t|j|SzSExtension from the planner interface to make it easy to extend the default planner.r%r7r3rTr4r4r5rS z DefaultSavePlanner.lookup_objectrVcCs(|jtjkrt}t|||}|SrZ)typer BYTE_IOioBytesIOtorchsave)r3rQrVbytesr4r4r5rUs  z#DefaultSavePlanner.transform_object)TTNFNF)__name__ __module__ __qualname__r__annotations__boolrr6rrr;rrArr rrMrPrr rbTensorr`rarWrrrSrUr4r4r4r5r'?sL     r'c @seZdZUdZeed<eed<   d(dededed d fd d Z d)d ede e ded d fddZ d e fddZ dee d ee fddZde d e fddZdedejd d fddZdefddZded ejd d fd!d"Zd#ed ejfd$d%Zded ejfd&d'Zd S)*r(ak DefaultLoadPlanner that adds multiple features on top of LoadPlanner. In particular it adds the following: flatten_state_dict: Handle state_dict with nested dicts flatten_sharded_tensors: For FSDP in 2D parallel mode allow_partial_load: If False, will raise a runtime error if a key is present in state_dict, but not in the checkpoint. original_state_dictr-TFrr.allow_partial_loadr1NcCs"||_||_i|_i|_||_dSr:)rr.rlr-rm)r3rr.rmr4r4r5r6s  zDefaultLoadPlanner.__init__r7rJr9cCsFt|||_|jrt|}|jrt|\}|_||_||_||_dSr:) r$rlr.rrr-r7rJr9)r3r7rJr9r4r4r5r;s z!DefaultLoadPlanner.set_up_plannercCs"|jdusJt|j|j|j Sr:)rJr)r7rmr3r4r4r5rAsz$DefaultLoadPlanner.create_local_planrIcCst|Sr:)r*)r3rIr4r4r5rMsz%DefaultLoadPlanner.create_global_planrNcCs|Sr:r4rOr4r4r5rPzDefaultLoadPlanner.finish_plan read_itemvaluecCsH|jrt|j|j|jjtj|dddStj|dd|j|jj<dS)NF) weights_only) rrrlr- dest_indexfqnrbloadr7)r3rprqr4r4r5 load_bytess  zDefaultLoadPlanner.load_bytescCrRr:) lookup_tensorrstransform_tensorr3rptensorr4r4r5resolve_tensorrXz!DefaultLoadPlanner.resolve_tensorrzcCsdSr:r4ryr4r4r5 commit_tensorroz DefaultLoadPlanner.commit_tensorrTcCrYrZr[r\r4r4r5rwr]z DefaultLoadPlanner.lookup_tensorcCst||j|jSrZ)r dest_offsetslengthsryr4r4r5rxsz#DefaultLoadPlanner.transform_tensor)TTFre)rfrgrh__doc__rrirrjr6rrr;rrArrMrPrr`rarvr{rbrkr|rrwrxr4r4r4r5r(sF     r(c s`eZdZdZdfdd Zdededefdd Z  dd e de ed eddffd d Z Z S)_EmptyStateDictLoadPlannera Extension of DefaultLoadPlanner, which rebuilds state_dict from the saved metadata. Useful for loading in state_dict without first initializing a model, such as when converting a DCP checkpoint into a Torch save file. . N.B. `state_dict` must be an empty dictionary when used with this LoadPlanner .. warning:: Because the entire state dict is initialized, It's recommended to only utilize this LoadPlanner on a single rank or process to avoid OOM. Ncs||_tj|i|dSr:)keyssuperr6)r3rargskwargs __class__r4r5r6sz#_EmptyStateDictLoadPlanner.__init__keyrJr1cs~jdurdS|jvr g}|j|}|D]}|r*|d|dt|gq||qtfdd|Dr=dSdS)NT.c3s|]}|jvVqdSr:)r)rCunflattened_keyrnr4r5 szA_EmptyStateDictLoadPlanner._should_include_key..F)rr=getappendjoinstrany)r3rrJunflattened_keysr=rr4rnr5_should_include_keys    z._EmptyStateDictLoadPlanner._should_include_keyFr7r9cs|rJ|dus J|jD]-\}}|||sqt|tr)tj|j|jj d}||j vr8t ||j ||q|||<qt |||dS)N)dtype)state_dict_metadataitemsr isinstancerrbemptysize propertiesrr=rrr;)r3r7rJr9kvrr4r5r;s     z)_EmptyStateDictLoadPlanner.set_up_plannerr:re) rfrgrhrr6rrrjrrrr; __classcell__r4r4rr5rs rTr7rJstrictr1cCsg} |D]6\}}||jvr|rtd|dq|j|}t|tr5|jdur4|t|||7}q|t|||7}qt|S)Nz&Missing key in checkpoint state_dict: r) rr RuntimeErrorrr device_meshget_coordinater"r)r7rJrrequestsrtobjmdr4r4r5r),s   r)rBcCs|S)z Create global load plan used by DefaultLoadPlanner. The default load behavior involved no global coordination and this function currently doesn't change the local plans. r4)rBr4r4r5r*Ns r*r9cCsTg}|D]\}}t|tr|jdur|t||7}q|t||7}qt|S)a Create the ``SavePlan`` used by DefaultSavePlanner. On non-coordinator ranks, this function ignores tensors and non-tensor objects, only producing writes for ShardedTensor objects. On the coordinator rank, produce writes for all values. N)rrr rrr#r)r7r9rrtrr4r4r5r+Zs  r+rewrite_index_hintsc Cs i}g}|D]}g}|jD]q}|jtjks|jj|vsJ|jtjkr0t||jj<||q |j dus7Jt t | |jjt |j j |j jgd}|}|ratj|jt|jd} tj|| d}|||j jdusvJd|jjd|j|j jq |tj||dq|t|fS)a6 Create the global plan and metadata used by DefaultSavePlanner. Metadata is produced by concatenating the metadata of all ``WriteItem`` from the supplied plans. The only global planning change is to update index hints in all ``MetadataIndex`` objects if ``rewrite_index_hints`` is True. N)rrchunks)rTzZ Cannot create MD for tensor without bounds. FQN: z )r)rr^r SHARDrTrtr_rr tensor_datarr setdefaultrrr>r?lenrchunkr) rBrr new_plansr@ new_itemsitem tensor_mdnew_item new_indexr4r4r5r,usJ         r,cCst|}t|g\}}|S)zTReturn the ``Metadata`` if DefaultSavePlanner was used to checkpoint ``state_dict``.)r!r,)r7r@_rr4r4r5_create_default_local_metadatasrbox0box1cCsdt|j}t|D]&}|j||j||j|krdS|j||j||j|kr/dSq dS)z9Check if two boxes overlap. Tuples are (offset, lengths).FT)roffsetsrangesizes)rrndimsir4r4r5_check_box_overlaps  router_box_size inner_boxcCs`tt|D]'}|j|dkrdS|j|dkrdS|j||j|||kr-dSqdS)NrFT)rrrr)rrrr4r4r5_check_box_boundssrrIc Csd}|jD]j\}}t|trqt|jdkrqd}t|jD]:\}}t|j|s5t d||j|d}|t t j |jd7}|j|ddD]}t||rYt d|||d}qHq t t j |jd} || krqt d|| |d}q|S)NTrz~ key:%s has out of bounds chunk: tensor-size:%s chunk: %s Fz$key:%s has overlapping chunks: %s %szq key:%s invalid fill tensor-volume: %s chunks-volume: %s )rrrrrr enumeraterrr&r2roperatormulrr) rIrJall_goodrrq chunks_volume chunk_idxchunk0chunk1 tensor_volumer4r4r5rGsH     rG)T)Jr>r`loggingr collectionsr functoolsrtypingrrrrrr r rbtorch.distributed._shard._utilsr torch.distributed._tensorr .torch.distributed.checkpoint._dedup_save_plansr )torch.distributed.checkpoint._nested_dictrr2torch.distributed.checkpoint._sharded_tensor_utilsr&torch.distributed.checkpoint._traverser%torch.distributed.checkpoint.metadatarrrrrrrr$torch.distributed.checkpoint.plannerrrrrrrr ,torch.distributed.checkpoint.planner_helpersr!r"r#r$"torch.distributed.checkpoint.utilsr% getLoggerrfr&Loggerri__all__r'r(rrrjr)r*r+r,rrSizerrGr4r4r4r5s~   $     ($   WUB  "    7