o *i[@sdZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z mZmZddlZddlmZzddlmZmZmZmZmZddlmZmZmZmZd ZWneyod ZdZdZdZdZdZdZdZdZdZYnwgd Z d efd dZ! dej"j#dej$fddZ%dej"j#dej&de'fddZ( dee)ej$fdej$fddZ*dej"j#dej$fddZ+ dHdej"j#de'dej"j,de eej$e dffd d!Z- dHdej"j#de'd"ej&de eej$e dffd#d$Z.dej"j#de'fd%d&Z/e!e0d'ej1dej"j#fd(d)Z2e!dd' dHd*eee efd+e'd,ej&de eej$e dffd-d.Z3e!dd'd*eee efd+e'fd/d0Z4e!e0d'ej1dej"j#fd1d2Z5e!e0d'ej1 dHd3eej"j#e ej"j#fd4e ej$fd5d6Z6d7ej"j#de'dej"j#fd8d9Z7d7ej"j#de'fd:d;Z8e!dd?Z9dej"j#dej"j#fd@dAZ:ej1dBdCZ; e!d d'dej"j#dedS)Iaw Utilities associated with offloading functionality provided by `accelerate`. | ------------------------------------------------------------------------------------------------------ | # noqa: E501 | Operation | Without offloading support | With offloading support | # noqa: E501 | ---------- | -------------------------------------- | ------------------------------------------------ | # noqa: E501 | Add | module.register_parameter(name, param) | register_offload_parameter(module, name, param) | # noqa: E501 | Check | N/A | has_offloaded_params(module) | # noqa: E501 | Onload | N/A | with align_module_device(module) | # noqa: E501 | Update | module.name.data.copy_(new_data) | update_offload_parameter(module, name, new_data) | # noqa: E501 | Delete | del module.name | delete_offload_parameter(module, name) | # noqa: E501 | Add Module | module.register_module(name, child) | register_offload_module(name, child) | # noqa: E501 | Del Module | del module.name | delete_offload_module(module, name) | # noqa: E501 | ------------------------------------------------------------------------------------------------------ | # noqa: E501 N)wraps) attrgetter)AnyCallableDictIterableLiteralOptionalTupleUnion) patch_attr)AlignDevicesHookadd_hook_to_moduleattach_align_device_hooknamed_module_tensorsremove_hook_from_module)OffloadedWeightsLoaderPrefixedDatasetfind_tied_parametersset_module_tensor_to_deviceTF)get_execution_deviceget_offloaded_deviceupdate_parameter_dataregister_offload_parameterupdate_offload_parameterdelete_offload_parameterhas_offloaded_paramsdisable_hf_hookdisable_offload align_modulesalign_module_deviceregister_offload_moduledelete_offload_moduleoffloaded_dispatchdisable_offloadingremove_dispatchcast_to_devicefallbackcs dttgtfffdd }|S)Nfunccs<tsdkrt|dd}|St|fdd}|S|S)Nerrorc_std)Nz9Please install `accelerate` in order to use this function) ValueErrorargskwargsr.]/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/compressed_tensors/utils/offload.py fallback_fn`sz8check_accelerate..decorator..fallback_fncsSNr.r+r'r.r/r0hs)_has_accelerater)r(r0r2r.r/ decorator\s  z#check_accelerate..decorator)rr)r'r4r.r2r/check_accelerate[sr5modulereturncCs8t|rt|jjd}|jjj}||jSt|S)zk :param module: module to check :return: device module is offloaded to onto after forward pass r)rlist_hf_hook weights_mapkeysdatasetdevicer)r6 first_keyprefix_datasetr.r.r/rvs   rnew_param_data param_namecCst|||dS)aG Update the data of an existing parameter and its offload dict. Supports both parameters of offloaded modules and non-offloaded modules :param module: module containing the parameter to update :param new_param_data: tensor to update parameter with :param param_name: name of module parameter to update N)r)r6r@rAr.r.r/rs r device_speccCs,t|trt|dkrd|SdS|S)a Convert an integer device index or torch.device into a torch.device object. :param device_spec: Device index (int) or torch.device object. Negative integers map to CPU. :return: torch.device corresponding to the given device specification. rzcuda:cpu) isinstanceinttorchr=)rBr.r.r/r&s r&cCsf|D] }t|rt|jjSt|jddd}|dur$|jSqt d|dt dS)z Get the device which inputs should be moved to before module execution. Assume that modules execute in the same order as returned by `model.modules()` :param module: module to check, may be offloaded :return: onload device of module FrecurseNz"Unable to get execution device of z, falling back to CPUrC) modulesrr&r9execution_devicenext parametersr=warningswarnrF)r6 submoduleparamr.r.r/rs   rname parameteroffload_devicediskcCstdd|D}|||t|rI|j}|jdusJ|j|j|<t|j||j ||j|}|j dur?i|j | <|sKt ||ddSdSdS)aj Register a parameter to the given module which may be offloaded :param module: maybe offloaded module :param name: name of newly registered parameter :param parameter: parameter being registered :param offload_device: device on which weight will be offloaded to. If None is provided, then infer device from parameters on module css |] }|jtdkVqdS)metaN)r=rF).0pr.r.r/ sz-register_offload_parameter..NrU) anyrLregister_parameterrr9r:r=original_devicesoffload_to_weights_mapdatatied_params_mapdata_ptrr)r6rQrRrS has_onloadhook offloadedr.r.r/rs    rr]cCst||}|jj|jkrtd|jjd|j|jtdkr-||jur-|j|t|r>|j j }t ||||dSdS)a Update the data of an existing parameter and its offload dict. Supports both parameters of offloaded modules and non-offloaded modules :param module: module containing the parameter to update :param name: name of module parameter to update :param data: tensor to update parameter with :param offload_device: device on which weight will be offloaded to. If None is provided, then infer device from parameters on module z!Shape of parameter being updated z% does not match shape of update data rUN) getattrr]shaperMrNr=rFcopy_rr9r:r\)r6rQr]rSrPr:r.r.r/rs   rcCs,t||t|r|jj}t||dSdS)z Delete a parameter from a module which may be offloaded :param module: maybe offloaded module :param name: name of parameter being deleted N)delattrrr9r:delete_from_weights_map)r6rQr:r.r.r/rs rr2c#sBifdd}||dVD] \}}t||qdS)Ncs$t|dr|j|<t|dSdS)Nr9)hasattrr9rr6hooksr.r/ collect_hookss   z&disable_hf_hook..collect_hooks)applyitemsr)r6rlrOrar.rjr/rs   rr:keyvaluecCs*t|tr%|dkrtdt||j}|j|}t||||dSt|trN||jvr5|j |t |j dkrJ|dkrJt|j |||dSt dt|tr|dkr`tdt||dur||vrn||j}ntt|d}|durtd|j}|j|d||<dSt dt|) a Helper function which implements offloaded item assignment for PrefixedDataset, OffloadedWeightsLoader, and Dict types. :param weights_map: weight map to be updated with offload information :param key: key used to identify weight location :param value: weight being offloaded :param offload_device: device on which weight will be offloaded to. If None is provided, then infer device from parameters in weights_map rTz!Cannot offload to disk with type rz@Updating weights_map with disk offloading is not implemented yetNz2Cannot infer offload device from empty weights_mapr=>Updating offload data not implemented for weights_map of type )rDrr*typer<prefixr\rall_keysappendlenindex state_dictNotImplementedErrordictr=rKitervaluesto)r:rorprSr<tensr.r.r/r\ s@      r\cCst|tr|j}|j|}t||dSt|tr.t|jdkr*t|j|dSt dt|t r8||=dSt dt |)NrzCDelete from weights_map with disk offloading is not implemented yetrr) rDrr<rtrgrrwrxryrzr{rs)r:ror<r.r.r/rg^s"    rgccs.t|rd|j_dVd|j_dSdVdS)z Context manager to disable module onloading and offloading. Parameters will stay on their current device :param module: module to disable offloading for FNT)rr9offloadrir.r.r/r{s    rrIrJccsvt|tjjr |fn|}t }|D]}|t|||t|qdVWddS1s4wYdS)a2 Context manager for onloading modules to a device, and disabling onload and offload attempts triggered by forward calls. Used for sequential onloading of layers :param modules: `torch.nn.Module` or iterable of `torch.nn.Module`s to onload :param execution_device: device to onload to N) rDrFnnModule contextlib ExitStack enter_contextr r)rIrJstackr6r.r.r/rs "rbasec Cst|rz|j}|js J|jdusJd}d}t|j}t|}t|||dD]*\}} | |} |j dur>i|j | <t |j|d|| |j rSt|||q)|j szt|jj|jj|dd} t|j|jd| ||d|j d} t|| |||dS)z Register a submodule with offloading if the parent module is offloaded :param base: module to attach submodule to :param name: name of submodule :param module: submodule to attach NFT)include_buffersrH.)rt)rJrio_same_devicer:offload_buffersplace_submodules skip_keysr^)rr9rr:rKrLr=rrr~r^r_r\rrrr<rtr rJrregister_module) rrQr6rarrcurrent_devicerSrArPrbr:submodule_hookr.r.r/r!sD      r!cCs8t||}t|D] \}}t||q t||dS)z Delete a submodule from a model which may contain offloading :param base: parent module to delete submodule from :param name: name of submodule on parent N)rcr8named_parametersrrf)rrQr6rA_r.r.r/r"s  r"r)rCc sdkrtdt||}fdd|D}t|d}t|}i}|D]}|D]}t||} i|| <q-q)t||d||dt |dd d d |S) a Unlike `dispatch_model`, this function forces a module (and its submodules) to offload all parameters and replace them with meta tensors, utiliizing the `AlignDevicesHook` to control onloading and offloading. :param module: module containing parameters to offload :param execution_device: device that modules will onload and execute on :param offload_device: device that module parameters will offload to :return: module with offloading device hooks rTz*Disk offloading is not currently supportedcsi|] \}}||qSr.)r~)rVrovalrSr.r/ sz&offloaded_dispatch..)ryr=T)rJrr:r^ hf_device_maprCzcuda:0) fake_offload fake_exec) rzr%ryrnrrrr_rsetattr) r6rJrSryr: tied_paramsr^grouprAr_r.rr/r#s,  r#cCs.t|ddt|drt|d|d|S)z Remove any existing dispatches from module :param module: module which may be dispatched with hf hooks :return: module without dispatch TrGrrC)rrhrfr~rir.r.r/r%$s    r%c#stjtdtffdd }ttd| dVWdn1s%wYD]!\}\}}||_|jddD] \}}t|||jq=| |dq.dS)z Keep modules onloaded and disable offloading until this context exits. Affects modules which have been hooked with accelerate's `AlignDevicesHook` selfcs8||g|Ri|}|vr||jf|<d|_|S)NF)r)rr6r,r-retonloaded_modulesoriginal_pre_forwardr.r/keep_onload_pre_forward=s z3disable_offloading..keep_onload_pre_forward pre_forwardNFrG) r rr{r rnrrrr] post_forward)rr6rarrQrPr.rr/r$3sr$cCst|dot|jto|jjS)ad Checks if a module has offloaded parameters by checking if the given module has a AlignDevicesHook attached with offloading enabled Args: module (`torch.nn.Module`): The module to check for an offload hook. Returns: bool: `True` if the module has an offload hook and offloading is enabled, `False` otherwise. r9)rhrDr9r rrir.r.r/rUs  rc cst|r?|dur|jj}||j_z|j|dVW|j|d|dur-||j_dSdS|j|d|dur>||j_w|durdd|jddD}z |D]}t|||qQdVW|D] \}}t|||qbdS|D] \}}t|||qswdVdS)a~ Context manager that moves a module's parameters to the specified execution device. Args: module (`torch.nn.Module`): Module with parameters to align. execution_device (`torch.device`, *optional*): If provided, overrides the module's execution device within the context. Otherwise, use hook execution device or pass NcSsi|]\}}||jqSr.rq)rVrQrPr.r.r/rs z'align_module_device..FrG)rr9rJrrrrrn)r6rJoriginal_devicedevicesrQr=r.r.r/r js:     r r1)?__doc__rrM functoolsroperatorrtypingrrrrrr r r rFcompressed_tensors.utilsr accelerate.hooksr rrrraccelerate.utilsrrrrr3 ImportError__all__r5rrr=rTensorstrrrEr&r Parameterrrr nullcontextcontextmanagerrr\rgrrr!r"r#r%r$boolrr r.r.r.r/s   (      , !   =   9 :