o 3wi;O@sddlZddlZddlZddlZddlmZddlmZddlmZm Z ddl m Z ddl Z ddlZddlmZddlmZdd lmZdd lmZdd lmZd d lmZd dlmZd dlmZm Z m!Z!m"Z"m#Z#d dl$m%Z%d dl&m'Z'd dl(m)Z)ee*Z+e"rddl,m-m.Z/dej0j1de2fddZ3dej0j1de2fddZ4dej0j1de2fddZ5dej0j1de2fddZ6dej0j1dej0j1fddZ7dej0j1fdd Z8d!ej0j1de2fd"d#Z9 %dKd&e2d'e2d(e2fd)d*Z:d+d,Z;d-ee d4re j?ne j-Z@e@jAjBe jCee jDgZEe d5reEFe jGjHdMd6d7ZId8d9ZJd:d;ZKdMdZMdeLfd?d@ZNdAdBZOdCdDZPdEeQfdFdGZRdNd!ej0j1dHe2deSej0j1fdIdJZTdS)ON)encode) OrderedDict)partialreduce) MethodType)Version) save_file)write_basic_config) get_logger) PartialState)FSDP_PYTORCH_VERSION)DistributedType)is_deepspeed_availableis_numpy_availableis_torch_distributed_availableis_torch_xla_availableis_weights_only_available)id_tensor_storage) convert_model)is_torch_versionmodulereturncCsttdsdSt|tjjjS)zD Check whether the module was compiled with torch.compile() _dynamoF)hasattrtorch isinstancer eval_frameOptimizedModulerr!S/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/accelerate/utils/other.pyis_compiled_module5s r#cCs<ttdsdS|jr|D] }t|tjjjrdSqdS)z\ Check whether the module has submodules that were compiled with `torch.compile()`. rFT)rr_modulesmodulesrrrrr submoduler!r!r"has_compiled_regions?s  r(cs$ttjjotfddDS)z Check whether the module is a repeated block, i.e. `torch.nn.ModuleList` with all children of the same class. This is useful to determine whether we should apply regional compilation to the module. c3s |] }t|djVqdS)rN)r __class__).0mr r!r" Tsz%is_repeated_blocks..)rrnn ModuleListallr r!r r"is_repeated_blocksNs$r0cCs&|jr|D] }t|rdSqdS)z Check whether the module has repeated blocks, i.e. `torch.nn.ModuleList` with all children of the same class, at any level of the module hierarchy. This is useful to determine whether we should apply regional compilation to the module. TF)r$r%r0r&r!r!r"has_repeated_blocksWs  r1c sFdtjjdtjjffdd |fi|}d|jvr!||jd<|S)a_ Performs regional compilation where we target repeated blocks of the same class and compile them sequentially to hit the compiler's cache. For example, in `GPT2LMHeadModel`, the repeated block/class is `GPT2Block`, and can be accessed as `model.transformer.h[0]`. The rest of the model (e.g. model.lm_head) is compiled separately. This allows us to speed up the compilation overhead / cold start of models like LLMs and Transformers in general. See https://pytorch.org/tutorials/recipes/regional_compilation.html for more details. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `torch.compile()`. Returns: `torch.nn.Module`: A new instance of the model with some compiled regions. Example: ```python >>> from accelerate.utils import compile_regions >>> from transformers import AutoModelForCausalLM >>> model = AutoModelForCausalLM.from_pretrained("gpt2") >>> compiled_model = compile_regions(model, mode="reduce-overhead") >>> compiled_model.transformer.h[0] OptimizedModule( (_orig_mod): GPT2Block( (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D(nf=2304, nx=768) (c_proj): Conv1D(nf=768, nx=768) (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D(nf=3072, nx=768) (c_proj): Conv1D(nf=768, nx=3072) (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) ) ``` rrcst|rtj}|D]}|tj|fi|q |St|rH|j|j}|j |j i|_ | D]\}}| ||fi|q5|Stj|fi|}|SN)r0rr-r.appendcompiler1r)__new____dict__updater$named_children add_module)rcompile_kwargs new_moduler'name_compile_regionsr!r"r>s  z)compile_regions.._compile_regions _orig_mod)rr-Moduler6)rr:r;r!r=r"compile_regionses /  rAcKsdt|r|D] }|jdi|qdSt|r(|D] }t|fi|qdS|jdi|dS)a Performs regional compilation the same way as `compile_regions`, but specifically for `DeepSpeedEngine.module`. Since the model is wrapped in a `DeepSpeedEngine` and has many added hooks, offloaded parameters, etc that `torch.compile(...)` interferes with, version of trgional compilation uses the inplace `module.compile()` method instead. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `module.compile()`. Nr!)r0r4r1childrencompile_regions_deepspeed)rr:r'childr!r!r"rCs rCmodelcs>tddr ddlmnddlmtfdd|DS)z Check if the model has DTensor parameters. Args: model (`torch.nn.Module`): The model to check. Returns: `bool`: Whether the model has DTensor parameters. >=z2.5.0rDTensorc3s|]}t|VqdSr2)r)r*prGr!r"r,sz$model_has_dtensor..)rtorch.distributed.tensorrHtorch.distributed._tensorany parameters)rEr!rGr"model_has_dtensors  rNTFkeep_fp32_wrapperkeep_torch_compile recursivec sLtjjjtjjf}t|}t|}|r|}|j}n |r"|}|jd}t r0ddl m }||f7}t dt rCtrCddlm} || f7}t||rP|j}t||sH|r\fdd|}|s|j} |jdd } | d urt| d r~| j} | | krynt| d sqt| ||_t|d d rt|d d |r|r||_|}|S|r||jd<|}|S)a Extract a model from its distributed containers. Args: model (`torch.nn.Module`): The model to extract. keep_fp32_wrapper (`bool`, *optional*): Whether to remove mixed precision hooks from the model. keep_torch_compile (`bool`, *optional*): Whether to unwrap compiled model. recursive (`bool`, *optional*, defaults to `False`): Whether to recursively extract all cases of `module.module` from `model` as well as unwrap child sublayers recursively, not just the top-level distributed containers. Returns: `torch.nn.Module`: The extracted model. r?r)DeepSpeedEnginerF)FullyShardedDataParallelcs@t|dr |j}n|}|D] \}}t|||q|S)Nr)rrr8setattr)runwrapped_moduler<rD_recursive_unwrapr!r"rWs  z6extract_model_from_parallel.._recursive_unwrap_original_forwardN __wrapped__ _converted_to_transformer_engineF)to_transformer_engine)rr-parallelDistributedDataParallel DataParallelr#r(r?r6r deepspeedrRrrr2torch.distributed.fsdp.fully_sharded_data_parallelrSrrforwardpoprrYrgetattrr) rErOrPrQoptions is_compiled has_compiledcompiled_modelrRFSDPraoriginal_forwardr!rVr"extract_model_from_parallelsT              rjcCstdS)a  Introduces a blocking point in the script, making sure all processes have reached this point before continuing. Make sure all processes will reach this instruction otherwise one of your processes will hang forever. N)r wait_for_everyoner!r!r!r"rk.s rk state_dictcstt}D]\}}t|ts|t||q dd|D}t}| D] }fdd|D}| |dd|ddD]}|=qFq,t |dkr\t d|d d dDS) z Cleans the state dictionary from a model and removes tensor aliasing if present. Args: state_dict (`dict`): The state dictionary from a model cSs"i|] \}}t|dkr||qS)r )len)r*ptrnamesr!r!r" Js"z4clean_state_dict_for_safetensors..csg|]}|vr|qSr!r!)r*r<rlr!r" Rsz4clean_state_dict_for_safetensors..r NrzRemoved shared tensor zk while saving. This should be OK, but check by verifying that you don't receive any warning while reloadingcSs*i|]\}}|t|tjr|n|qSr!)rrTensor contiguous)r*kvr!r!r"rpZs*) collections defaultdictlistitemsrstrrr3setvaluesr7rmloggerwarning)rlptrsr<tensor shared_ptrs warn_namesro found_namesr!rqr" clean_state_dict_for_safetensors;s&     rsave_on_each_nodesafe_serializationcCstjtjkr t|}|r ttddid}t|t rt |}nt j }tj r0|s0|||dStjr=|r?|||dSdSdS)a Save the data to disk. Use in place of `torch.save()`. Args: obj: The data to save f: The file (or file-like object) to use to save the data save_on_each_node (`bool`, *optional*, defaults to `False`): Whether to only save on the global main process safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save `obj` using `safetensors` or the traditional PyTorch way (that uses `pickle`). formatpt)metadataN)r distributed_typerXLAxm_maybe_convert_to_cpursafe_save_filerrrrsaveis_main_processis_local_main_process)objfrr save_funcr!r!r"r^s    rz2.0.0z1.25.0c Ksz;trtj}d|vrd|d<tjtn|ddtj|fd|i|}Wtr:tj|r:tj||StrMtj|rNtj|www)a Compatible drop-in replacement of `torch.load()` which allows for `weights_only` to be used if `torch` version is 2.4.0 or higher. Otherwise will ignore the kwarg. Will also add (and then remove) an exception for numpy arrays Args: f: The file (or file-like object) to use to load the data map_location: a function, `torch.device`, string or a dict specifying how to remap storage locations **kwargs: Additional keyword arguments to pass to `torch.load()`. weights_onlyTN map_location) rr serializationget_safe_globalsadd_safe_globalsTORCH_SAFE_GLOBALSrbloadclear_safe_globals)rrkwargsold_safe_globals loaded_objr!r!r"rs&     rcCsHt|dst|dst|d|}t|dr|jSt|dr |jSt|S)z( Gets a pretty name from `obj`. __qualname____name__r))rrcrrr{)rr!r!r"get_pretty_names   rcCs@|D]\}}t|tr||i}t||q|||<q|S)z Recursively merges two dictionaries. Args: source (`dict`): The dictionary to merge into `destination`. destination (`dict`): The dictionary to merge `source` into. )rzrdict setdefault merge_dicts)source destinationkeyvaluenoder!r!r"rs     rportcCsR|durd}ttjtj}|d|fdkWdS1s"wYdS)z Checks if a port is in use on `localhost`. Useful for checking if multiple `accelerate launch` commands have been run and need to see if the port is already in use. Ni._getattr.)rr)rrrr!r!r"recursive_getattrs r return_fqnsc Cs|s|nd|f}|g}g}|rS|}|r|\}}|D]$\}}t|tjjrA|r<|r2|d|n|} || |fq||q|rL|||fn|||s|dddS)aATraverse the model in bottom-up order and return the children modules in that order. Args: model (`torch.nn.Module`): the model to get the children of Returns: `list[torch.nn.Module]`: a list of children modules of `model` in bottom-up order. The last element is the `model` itself. rrN)rbr8rrr-r@r3) rErtopstackordered_modulescurrent_modulecurrent_module_namer<r child_namer!r!r"get_module_children_bottom_ups&   r)TTF)FFr2)F)Urwrrrcodecsrr functoolsrrtypesrnumpynprpackaging.versionrsafetensors.torchrrcommands.config.defaultr loggingr stater constantsr dataclassesrimportsrrrrrmodelingrtransformer_enginerversionsrrr~torch_xla.core.xla_modelcore xla_modelrr-r@boolr#r(r0r1rArCrNrjrkrrr_corenp_core multiarray _reconstructndarraydtyperr3dtypes UInt32DTyperrrintrrrrr{rryrr!r!r!r"sv               H U #$       (