o ei2@sdZddlZddlmZddlZddlmZddlmZddlm Z dZ e j Gdddej Z d d Zd ej d ed ej fddZGdddej ZGdddej ZdS)zThe SpeechBrain implementation of various pre-trained model adapters e.g. LoRA, Houlsby Authors * Titouan Parcollet 2024 * Peter Plantinga 2024 Nfnmatch)Swish) checkpointsz Torch's native multi-head attention is not adaptable since it accesses layer weights directly to pass to highly optimized fused kernels. We are excluding all native Torch MHA layers from the list of layers to adapt. cseZdZdZddggidfdejdejdedededed ed effd d Z d dZ ddZ e j ddZe jddZe jddZfddZZS) AdaptedModela Given any torch model, e.g. asr_brain.modules.Transformer, and an adapter class, e.g. HoulsbyAdapter, this class will replace the target layers with this new adapter class (while preserving the parameters). Arguments --------- model_to_adapt: nn.Module The base PyTorch model to add adapters to. adapter_class: class An (uninitialized) adapter of this SpeechBrain library. all_linear: bool Whether to add the adapter to all linear layers (default: False) all_conv: bool Whether to add the adapter to all conv layers (default: False) target_layers: list of str A list of module names in the given model that should be replaced. Supports Unix shell-style wildcards `(*, ?, [seq], [!seq])` with `fnmatch`. unfrozen_layers: list of str List of layers to be unfrozen during training. Supports Unix shell-style wildcards `(*, ?, [seq], [!seq])` with `fnmatch`. adapter_kwargs: dict Ensemble of parameters that should be given to the adapter. manual_adapter_insertion: bool The default value (`False`) leads to the adapters being inserted at the time of initialization. However, in some cases, it is preferable to wait to insert the adapters, e.g. when pretrained parameters need to be loaded. In this case, one can set this to `True` and call `insert_adapters` manually after the parameters have been loaded. Example ------- >>> from collections import OrderedDict >>> model = torch.nn.Sequential( ... OrderedDict([ ... ("layer1", torch.nn.Linear(10, 20)), ... ("layer2", torch.nn.Linear(20, 20)), ... ("layer3", torch.nn.Linear(20, 10)), ... ]) ... ) >>> lora_model = AdaptedModel( ... model_to_adapt=model, ... adapter_class=LoRA, ... target_layers=["layer[13]"], ... unfrozen_layers=["layer2"], ... adapter_kwargs={"rank": 2}, ... ) >>> lora_model AdaptedModel( (adapted_model): Sequential( (layer1): LoRA( (pretrained_module): Linear(in_features=10, out_features=20, bias=True) (adapter_down_proj): Linear(in_features=10, out_features=2, bias=False) (adapter_up_proj): Linear(in_features=2, out_features=20, bias=False) ) (layer2): Linear(in_features=20, out_features=20, bias=True) (layer3): LoRA( (pretrained_module): Linear(in_features=20, out_features=10, bias=True) (adapter_down_proj): Linear(in_features=20, out_features=2, bias=False) (adapter_up_proj): Linear(in_features=2, out_features=10, bias=False) ) ) ) Fmodel_to_adapt adapter_class all_linearall_conv target_layersunfrozen_layersadapter_kwargsmanual_adapter_insertionc st||_||_||_|D]} d| _qg|_|D]F\} t | |||rPd ddd} | | } t | tjjrIttq|jqtfdd|Dre| D]} d| _q_q|sn|dSdS)NF.c3|]}t|VqdSNr.0layernameW/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/nnet/adapters.py }z(AdaptedModel.__init__..T)super__init__ adapted_modelrr parameters requires_gradreplace_layers named_modulesis_layer_adaptablejoinsplit get_submodule isinstancetorchnnMultiheadAttentionwarningswarn MHA_WARNINGappendanyinsert_adapters) selfrrr r r r r rparammodule parent_nameparent __class__rrr[s.       zAdaptedModel.__init__cCs>|jD]}|j|}|j|fi|j}t|j||qdS)zIf this is in `__init__` it conflicts with `Pretrainer`. Ensure this function is called exactly once before training. See ``__init__.manual_adapter_insertion`` N)r!rr&rr replace_module)r1rr3 new_modulerrrr0s  zAdaptedModel.insert_adapterscOs|j|i|S)z Pass arguments to adapted model.)r)r1argskwargsrrrforwardszAdaptedModel.forwardcCs*dd|jddD}t||dS)z$Saves only the trainable parameters.cSs i|] \}}|jr||qSr)r detach)rrr2rrr s z&AdaptedModel.saver..T) keep_varsN) state_dictitemsr(save)r1pathr@rrrsaverszAdaptedModel.savercCs"~tj|dd}|j|dddS)z)Loads the base model plus trained params.cpu) map_locationF)strictN)r(loadload_state_dict)r1rC end_of_epochr@rrrloaderszAdaptedModel.loadercCs||ddS)z3Avoids warnings due to only loading trained params.TN)rK)r1rCrrrparameter_transferszAdaptedModel.parameter_transfercs,td}t||rt||St|S)z.)r'r)LinearConv1dConv2dConv3dr/)rr3r r r rrrr#s r#rQrr9cCsJz|dd\}}||}Wn ty|}|}Ynwt|||dS)aReplace layer with a new module based on a parent assignation. This is used to replace layers with an Adapter layer wrapped around the original layer. Hence, old parameters are preserved and new ones are added. Arguments --------- model: nn.Module Model containing the module to be replaced. name: str Name of the target module to replace. new_module: nn.Module New module made of the old plus the new parameters. rN)rsplitr& ValueErrorsetattr)rQrr9r4 target_name parent_modulerrrr8s r8cs6eZdZdZedffdd ZdejfddZZ S)HoulsbyAdapterLinearaThis class implements the Houlsby Adapter as described in: 'Parameter-Efficient Transfer Learning for NLP' https://arxiv.org/abs/1902.00751 Arguments --------- target_linear: nn.Module Module corresponding to the pretrained Linear that will be wrapped with this adapter. projection_size: int Size of the projection layer (usually smaller). activation: nn.Module The activation function. Default is Swish. bias: bool Whether to use biases in the linear projections. Example ------- >>> import torch >>> x = torch.rand((8, 60, 64)) >>> base_linear = nn.Linear(64, 64) >>> adapt = HoulsbyAdapterLinear(base_linear, 8) >>> output = adapt(x) >>> output.shape torch.Size([8, 60, 64]) Tcstt|tjstdt|d|jjj d}|jj }||_ d|j _ tj||||d|_ tj||||d|_||_|rS|j jjd|jjjddSdS)NzEHoulsbyLinear currently only supports linear layers, but instead got rrFbiasdevice)rrr'r)r^rdtypeweightdatashaperkpretrained_linearr adapter_down_projadapter_up_proj activationrjfill_)r1 target_linearprojection_sizertrj output_sizerkr6rrr s,  zHoulsbyAdapterLinear.__init__xcCs$||}|||||S)zApplies the HoulsbyAdapter to an input tensor `x`. Arguments --------- x: torch.Tensor Input tensor to the adapter module. Shape: [B, Time, X] Returns ------- The linear outputs )rqrsrtrr)r1ry x_pretrainedrrrr<-s zHoulsbyAdapterLinear.forward) rRrSrTrUrrr(Tensorr<r]rrr6rrhs  rhcs2eZdZdZd fdd ZdejfddZZS) LoRAaThis class implements the LoRA Adapter as described in: 'LoRA: Low-Rank Adaptation of Large Language Models' https://arxiv.org/abs/2106.09685 Arguments --------- target_module: nn.Module Module corresponding to the pretrained layer that will be wrapped with this adapter. Works with nn.Linear and nn.Conv rank: int Size of the projection layer or rank (usually smaller). alpha : float Value used to control the scaling in LoRA. Default is one. Example ------- >>> import torch >>> x = torch.rand((8, 60, 64)) >>> base_linear = nn.Linear(64, 64) >>> adapt = LoRA(base_linear, 64, 4) >>> output = adapt(x) >>> output.shape torch.Size([8, 60, 64]) ?cst|jjjd}|jjjd}||_|jD]}d|_q|jj}t j ||d|d|_ t j ||d|d|_ |j jj d|||_dS)NrbrFrirl)rrrnrorppretrained_modulerr rkr)r^rrrsruscaling)r1 target_modulerankalpha input_sizerxr2rkr6rrr^s z LoRA.__init__rycCs(||}||||j}||S)zApplies the LoRA Adapter. Arguments --------- x: torch.Tensor Input tensor to the adapter module. Returns ------- The linear outputs )rrsrrr)r1ryrzx_lorarrrr<ts z LoRA.forward)r}r~) rRrSrTrUrr(r{r<r]rrr6rr|Dsr|)rUr+rr(torch.nnr)speechbrain.nnet.activationsrspeechbrain.utilsrr-register_checkpoint_hooksrVrr#strr8rhr|rrrrs    S