o wi@sddlmZmZmZddlmZmZddlmZddl m Z ddl m Z ddl mZddlmZmZGdd d ejZGd d d e jeZeGd d d ZdS)) dataclassfield is_dataclass)AnyOptional) instantiate) OmegaConf)nn)activation_registry) access_mixinsadapter_mixin_strategiesc@s6eZdZdZdeejfddZd ddZd d Z d S) AdapterModuleUtilz_ Base class of Adapter Modules, providing common functionality to all Adapter Modules. adapter_strategycCst|dur|}t|rt|}t|dt|ts!t|r(t||_ dSt|t j r3||_ dSt d|)aY Setup adapter strategy of this class, enabling dynamic change in the way the adapter output is merged with the input. When called successfully, will assign the variable `adapter_strategy` to the module. Args: adapter_strategy: Can be a None or an implementation of AbstractAdapterStrategy. NFz)`adapter_strategy` provided is invalid : ) get_default_strategy_configrr structured set_struct isinstancedict is_configrrr AbstractAdapterStrategyAttributeError)selfrrj/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/common/parts/adapter_modules.pysetup_adapter_strategys     z(AdapterModuleUtil.setup_adapter_strategyreturnrcCtS)z< Returns a default adapter module strategy. r ResidualAddAdapterStrategyConfigrrrrr:sz-AdapterModuleUtil.get_default_strategy_configcCs|D]}|dqdS)z Sets the requires grad for all parameters in the adapter to True. This method should be overridden for any custom unfreeze behavior that is required. For example, if not all params of the adapter should be unfrozen. TN) parametersrequires_grad_)rparamrrradapter_unfreeze@s  z"AdapterModuleUtil.adapter_unfreezeN)rr) __name__ __module__ __qualname____doc__rr rrrr#rrrrr s   r c sVeZdZdZ    ddededed ed ed ejf fd d Z ddZ ddZ Z S) LinearAdaptera Simple Linear Feedforward Adapter module with LayerNorm and singe hidden layer with activation function. Note: The adapter explicitly initializes its final layer with all zeros in order to avoid affecting the original model when all adapters are disabled. Args: in_features: Input dimension of the module. Note that for adapters, input_dim == output_dim. dim: Hidden dimension of the feed forward network. activation: Str name for an activation function. norm_position: Str, can be `pre` or `post`. Defaults to `pre`. Determines whether the normalization will occur in the first layer or the last layer. Certain architectures may prefer one over the other. dropout: float value, whether to perform dropout on the output of the last layer of the adapter. adapter_strategy: By default, ResidualAddAdapterStrategyConfig. An adapter composition function object. swishpreN in_featuresdim activation norm_positiondropoutrc stt|}t|drd|_|dvsJ||_|dkr8tt|tj ||dd|tj ||dd|_ n|dkrTttj ||dd|tj ||ddt||_ |dkr_t ||_ nd|_ | ||dS) NinplaceT)r*postr*F)biasr2r+)super__init__r hasattrr1r/r Sequential LayerNormLinearmoduleDropoutr0rreset_parameters)rr,r-r.r/r0r __class__rrr5[s2     zLinearAdapter.__init__cCsb|jdkr|jdjjd9_dS|jdkr/|jdjjd9_|jdjjd9_dSdS)Nr*rr2)r/r:weightdatar3rrrrr<s  zLinearAdapter.reset_parameterscCs"||}|jdur||}|SN)r:r0)rxrrrforwards   zLinearAdapter.forward)r)r*r+N) r$r%r&r'intstrfloatr rr5r<rD __classcell__rrr=rr(Js*. r(c@steZdZUeed<eed<dZeed<dZeed<dZe ed<e d d d Z e e ed <d ejejZeed<dS)LinearAdapterConfigr,r-r)r.r*r/r+r0cCrrBrrrrrszLinearAdapterConfig.)default_factoryrz{0}.{1}_target_N)r$r%r&rE__annotations__r.rFr/r0rGrrrrformatr(rLrrrrrIs    rIN) dataclassesrrrtypingrr hydra.utilsr omegaconfrtorchr #nemo.collections.common.parts.utilsr nemo.core.classes.mixinsr r AccessMixinr Moduler(rIrrrrs    0R