o 8wÖiy¥ã@süddlmZddlZddlZddlZddlmZddlmZm Z m Z ddl Z ddl m Z ddlm mZddlmZddlmZmZdaedd„ƒZd d „ZGd d „d eƒZGd d„de jƒZGdd„deƒZGdd„de jeƒZGdd„de jeƒZdS)é)Ú annotationsN)Úcontextmanager)ÚAnyÚOptionalÚUnion)ÚFunction)ÚBaseTunerLayerÚcheck_adapters_to_mergecksˆi}| ¡D]\}}| ¡}|tjvrtj|||<t|ƒtj|<qdV|D]}| ¡}||vr:||tj|<q(tj |d¡q(dS)aÈ A context manager that will add each keyword argument passed to `os.environ` and remove them when exiting. Will convert the values in `kwargs` to strings and upper-case all the keys. Example: ```python >>> import os >>> from accelerate.utils import patch_environment >>> with patch_environment(FOO="bar"): ... print(os.environ["FOO"]) # prints "bar" >>> print(os.environ["FOO"]) # raises KeyError ``` N)ÚitemsÚupperÚosÚenvironÚstrÚpop)ÚkwargsÚ existing_varsÚkeyÚvalue©rúS/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/peft/tuners/boft/layer.pyÚpatch_environment&s€ úrc CsÎtdurtSddlm}tj t¡}z/tddd|d|›d|›dgdd }Wdƒn 1s3wYW|atSW|atStyf}zt   d |›d ¡t   d ¡d}WYd}~|atSd}~ww) Nr)ÚloadÚgcc)ÚCCÚCXXÚfbd_cudaz/fbd/fbd_cuda.cppz/fbd/fbd_cuda_kernel.cuT)ÚnameÚsourcesÚverbosez#Failed to load the CUDA extension: z, check if ninja is available.zHSetting boft_n_butterfly_factor to 1 to speed up the finetuning process.) Ú _FBD_CUDAÚtorch.utils.cpp_extensionrr ÚpathÚdirnameÚ__file__rÚ ExceptionÚwarningsÚwarn)rÚcurr_dirrÚerrrÚ get_fbd_cudaJs2  ýÿ ò ú €úr)c@s(eZdZdZedd„ƒZedd„ƒZdS)Ú FastBlockDiaga  Implements a custom autograd Function for a fast block diagonal operation using CUDA. This function is optimized for 4D tensors where the last two dimensions are equal, representing block diagonal matrices for efficient computation on CUDA devices. cCstƒ |¡d}| |¡|S)a| The forward method for FastBlockDiag. Computes the block diagonal operation on the input tensor using a CUDA-optimized function. This method assumes that the input is a 4D tensor where the last two dimensions are equal, which represent the blocks to be diagonalized. Parameters: ctx: A context object that can be used to stash information for backward computation. input (Tensor): The input tensor of shape (N, D, H, H), where `N` is the batch size, `D` represents one additional dimension (In BOFT, the number of BOFT blocks), and `H` is the size of the square blocks along the last two dimensions (In BOFT, the block size). Returns: Tensor: The resulting tensor after applying the block diagonal operation, will have the shape (N, DxH, DxH). r)r)ÚforwardÚsave_for_backward)ÚctxÚinputÚoutputrrrr+os zFastBlockDiag.forwardcCs|j\}tƒ ||¡d}|S)Nr)Ú saved_tensorsr)Úbackward)r-Ú grad_outputr.Ú grad_inputrrrr1†szFastBlockDiag.backwardN)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú staticmethodr+r1rrrrr*gs  r*cs*eZdZdZd‡fdd„ Zdd„Z‡ZS)ÚMultiplicativeDropoutLayerz? Implements the multiplicative dropout layer for BOFT. çcstƒ ¡||_dS)z¡ Initializes the multiplicative dropout layer. Parameters: p (float): The probability of dropping out a block. Defaults to 0.0. N)ÚsuperÚ__init__Úp)Úselfr=©Ú __class__rrr<’s  z#MultiplicativeDropoutLayer.__init__c Csê|jrs|jd|jdkrtdƒ‚|j\}}}}t d|d¡ ¡}t|j|ƒ}||}t tj ||j dtj ||j dg¡} | t  |¡  d|dd¡} tj ||dd|j d} | | |<tj||j d ||dd¡} d| || | }|S)a‹ Applies multiplicative dropout to the input tensor. Parameters: x (Tensor): The input tensor of shape (N, D, H, H), where `N` is the batch size, `D` represents one additional dimension (In BOFT, the number of BOFT blocks), and `H` is the size of the square blocks along the last two dimensions (In BOFT, the block size). éÿÿÿÿéþÿÿÿz4The last two dimensions of input should be the same!r)é©ÚdevicerC)ÚtrainingÚshapeÚ ValueErrorÚtorchÚrandintÚitemÚintr=ÚcatÚonesrEÚzerosÚrandpermÚviewÚeyeÚrepeat) r>ÚxÚNÚDÚHÚ_Ún_randomÚnum_to_replaceÚ num_zerosÚmaskÚ full_maskÚ eye_matrixrrrr+œs &z"MultiplicativeDropoutLayer.forward)r:)r4r5r6r7r<r+Ú __classcell__rrr?rr9s r9c@sjeZdZdZdZdZddd „Zd d „Zd dd„Zd!d"dd„Z dd„Z dd„Z dd„Z d#dd„Z dd„ZdS)$Ú BOFTLayerz$ Implements the BOFT layer. )Úboft_RÚboft_s)Úboft_block_sizeÚboft_block_numÚ boft_dropoutÚ base_layerú nn.ModuleÚreturnÚNonecKs°||_i|_i|_t i¡|_t i¡|_t i¡|_d|_ g|_ d|_ ||_ |  ¡}t|tjƒr9|j|j}}nt|tjƒrG|j|j}}n tdt|ƒ›ƒ‚||_||_dS)zý Initializes the BOFT layer. Note, currently only support linear layer and convolutional layer, with further support for other layers to be added soon. Parameters: base_layer: the pretrained model layer FTzUnsupported layer type N)rfrcrdÚnnÚ ModuleDictreÚ ParameterDictrarbÚ_disable_adaptersÚmerged_adaptersÚcast_input_dtype_enabledrÚget_base_layerÚ isinstanceÚLinearÚ in_featuresÚ out_featuresÚConv2dÚ in_channelsÚ out_channelsrHÚtype)r>rfrrsrtrrrr<Ìs$       zBOFTLayer.__init__cCs||jvrdSt d¡dS)NúGScaling operation for BOFT not supported! Automatically set scale to 1.)Úscalingr%r&)r>ÚadapterÚscalerrrÚ set_scaleïs zBOFTLayer.set_scaler|ÚfloatcCs6|dkrdS|jD]}||j ¡vrq t d¡q dS)NrCry©Úactive_adaptersraÚkeysr%r&©r>r|Úactive_adapterrrrÚ scale_layerös  üzBOFTLayer.scale_layerNcCs*|jD]}||j ¡vr qt d¡qdS)Nz?Unscaling operation for BOFT not supported! Keeping scale to 1.rr‚rrrÚ unscale_layers  üzBOFTLayer.unscale_layerc Cstƒs d|_d}nd|_|d}|dkrtd|d›dƒ‚|dkr(t|d}nt ¡}|j t ||i¡¡|dkr‰|dkr‰|j |dkrRtd |j ›d |›d ƒ‚|dkr|t t   |¡ƒkrltd |d›d |›d ƒ‚|d|dkrtd|›d|d›d ƒ‚t |j |ƒ}na|dkræ|dkræ|j |dkr¤td |j ›d|›d ƒ‚|dkrÞ|j |d|krÂtd|j ›d|d›d|›d ƒ‚|j |d|dkrÞtd|j ›d|d›d|›d ƒ‚t |j |ƒ}ntdƒ‚|dkr |ddkrýtd|›dƒ‚|ddkr td|›dƒ‚t  |d|j |j f¡}t|dƒD]} | |j t |d| ƒt |dƒ|¡} | | ¡} | || <q|jd|ddt t  |d|||¡¡|j|<t t  t |jƒd¡¡|j|<| ||¡||j|<||j|<| |¡| |j¡dS)zf Update the linear layer with trainable BOFT weights. Override for other layer types. FrCTrú-You can only specify boft_n_butterfly_factor ú! to be a positive integer number.r:©r=z in_features (ú') must be divisible by boft_block_num (ú)!ú0Invalid combination of boft_n_butterfly_factor (ú) and boft_block_num (éúboft_block_num (úJ) must be a multiple of 2 raised to the power of boft_n_butterfly_factor (ú() must be divisible by boft_block_size (z$Invalid combination of in_features (ú), boft_n_butterfly_factor (ú) and boft_block_size (úZSomething went wrong, please report this error: https://github.com/huggingface/peft/issuesú) must be an even number!úboft_block_size (Úboft_P©Ú persistentN)r)Úfbd_cuda_availablerHr9rjÚIdentityreÚupdaterkrsrLÚmathÚlog2rIÚemptyÚrangeÚblock_butterfly_permÚperm2matÚregister_bufferÚ ParameterrOrarNrtrbÚreset_boft_parametersrcrdÚ%_move_adapter_to_device_of_base_layerÚ set_adapterr€) r>Ú adapter_namercrdÚboft_n_butterfly_factorreÚ init_weightsÚboft_dropout_layerÚPÚiÚpermÚperm_matrrrÚ update_layersˆÿ ÿÿÿÿÿÿÿ  ÿ   ÿ    zBOFTLayer.update_layercCsŒ|durtjj|j|dddtjj|j|ddddS||j ¡vrD|dur=tj |j|¡tj |j|¡dStd|›ƒ‚dS) z, Reset the BOFT parameters. Fr:çš™™™™™¹?)ÚmeanÚstdgð?NTz$Unknown initialization init_weights=) rjÚinitÚnormal_rarbrÚzeros_Úones_rH)r>r§r©rrrr¤msúzBOFTLayer.reset_boft_parameterscCs8t|ƒ}t ||f¡}t|ƒD] \}}d|||f<q|S)z” Convert permutation indices to permutation matrix. Args: indices: A list of indices representing the permutation. rC)ÚlenrIrOÚ enumerate)r>ÚindicesÚnr®r¬Úidxrrrr¡~s zBOFTLayer.perm2matérCc CsŒ|dkr t |¡S||d|krtdƒ‚t||ƒ}t |¡}dd„}|||ƒ}td||ƒD]} | |} || | …} | ||| | …<q/|S)a0 Define the permutation matrix for the block butterfly permutation. Args: n: size of the permutation matrix b: desired number of blocks after multiplying with the permutation matrix r: base block size of the block diagonal matrix, e.g. 2x2, 3x3, 5x5 etc. rrzInvalid number of blocks!c Ss¢||}t |¡}tj|tjd}t d|d¡}t d|d¡}tj||fdd}t|ƒD]"\}} |t| |ƒt| ||ƒ…|t||ƒt|||ƒ…<q,|S)N)ÚdtyperrrC)Údim)rIÚarangeržÚlongrMr¸rL) ÚbÚrÚstepÚ initial_orderÚ sorted_orderÚevensÚoddsÚ sorted_seqr¬ÚposrrrÚ sort_block¤s >z2BOFTLayer.block_butterfly_perm..sort_block)rIr¿rHrLrŸ) r>rºrÁrÂÚn_butterfly_factorÚ block_sizer¹rÊrÅr¬Ú block_endÚ tmp_indicesrrrr ‘s     zBOFTLayer.block_butterfly_permcCs^|j\}}}d|| dd¡}tj||jd d¡ |||¡}tjj||||dd}|S)z³ Perform the Cayley parametrization on a batch of skew-symmetric matrices. Args: data: A batch of skew-symmetric matrices of shape (b, r, c). gà?rCrrDrF)Úleft) rGÚ transposerIrRrEÚ unsqueezeÚexpandÚlinalgÚsolve)r>ÚdatarÁrÂÚcÚskew_matÚid_matÚQrrrÚ cayley_batch¸s  zBOFTLayer.cayley_batch)rfrgrhri)r|r~rhri©N©rhri)r¼rC)r4r5r6r7Úadapter_layer_namesÚother_param_namesr<r}r„r…r¯r¤r¡r rÚrrrrr`Âs #  f  'r`csjeZdZdZ       d,d-‡fdd„ Zd.d/dd„Zd0dd„Zd1d!d"„Zd2d(d)„Zd3‡fd*d+„ Z ‡Z S)4rrz, BOFT implemented in a dense layer. érr°FTr§rrcrLrdr¨rer~Úfan_in_fan_outÚboolr©úUnion[bool, str]Úis_target_conv_1d_layerrhric  sHtƒ ¡tj||fi| ¤Ž||_||_| ||||||¡| |_dSrÛ)r;r<r`ràÚ_active_adapterr¯rã) r>rfr§rcrdr¨reràr©rãrr?rrr<Ïs  ÿ zLinear.__init__NÚ safe_mergeÚ adapter_namesúOptional[list[str]]c Cs:t||ƒ}|s dS|D]}||j ¡vrš| ¡}|jj}|ra|jj ¡}| |¡\}}t   |dd¡}t   ||  |j¡¡}t   |dd¡}||}t   |¡ ¡sVtd|›dƒ‚| ¡  |¡|jj_n3| |¡\}}|jj ¡}t   |dd¡}t   ||  |j¡¡}t   |dd¡}||}| ¡  |¡|jj_|j |¡q dS)á^ Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. NrrCz1NaNs detected in the merged weights. The adapter z seems to be broken)r rarrpÚweightr½rÕÚcloneÚget_delta_weightrIrÐÚmmÚtoÚisfiniteÚallrHÚ contiguousrfrnÚappend© r>rårærƒrfÚ orig_dtypeÚ orig_weightÚbutterfly_oft_matrbrrrÚmergeçs:   ÿ  €âz Linear.mergecCsÄ|js t d¡dSt|jƒdkr`|j ¡}| ¡}|jj}||j   ¡vrW|  |¡\}}|jj   ¡}t |dd¡}t | ¡| |j¡¡}t |dd¡}|d| |¡|j_ t|jƒdksdSdS©zW This method unmerges all merged adapter layers from the base weights. z Already unmerged. Nothing to do.NrrC)Úmergedr%r&r·rnrrprér½rarrërÕrêrIrÐrìÚtrí©r>rƒrfrórõrbrôrrrÚunmerges   ôzLinear.unmergeú!tuple[torch.Tensor, torch.Tensor]cCsæ|j|}|j|}|j\}}}}| ||||¡}| |¡}| ||||¡}|jr0t |¡} n| d¡}t j t   |¡Ž} |   d¡} |j  | j¡} t  | |  ddd¡¡} t  | | ¡} | d} td| jdƒD]} | | | } qf| |fS)ú Compute the delta weight for the given adapter. Args: adapter (str): The name of the adapter for which the delta weight should be computed. rrrC)rarbrGrQrÚr™r*ÚapplyÚsqueezerIÚ block_diagÚunbindrÑr–rírEÚbmmÚpermuterŸ©r>r{rarbrUrVrWrXÚorth_rotate_butterflyÚblock_diagonal_butterflyr–Úbutterfly_oft_mat_batchrõr¬rrrrë/s$       zLinear.get_delta_weightrTú torch.TensorÚargsrrc OsZ|j}|jr|jr | ¡|j|g|¢Ri|¤Ž}n |jr+|j|g|¢Ri|¤Ž}nûtj|j|j|d}tj t |j ƒdf|j|d}|j D]Ž}||j  ¡vrPqF|j |} |j|} |j|} | j\} } }}|  | | ||¡} | | ¡}| | | ||¡}| |ƒ}|jr‰t |¡}n| d¡}tjt |¡Ž}| d¡}|j |¡}| |¡}t || ddd¡¡}t ||¡}|d}td|jdƒD]}|||}qÃ||}| |}qF| |  ¡j!j"j¡}|  ¡j!j"}t #|dd¡}| |¡}| |¡}t $||¡}t #|dd¡}||}| |¡}|jj%dur|jj% |¡|j_%t&j'|||jj%d}| |¡}|S)N©rEr½rCrr)r.réÚbias)(r½Údisable_adaptersrørûrfrIrRrsrErNrLrtr€rarrbrerGrQrÚr™r*rþrÿrrrÑr–rírrrŸrprérÕrÐrìr ÚFÚlinear)r>rTr rÚprevious_dtypeÚresultÚ boft_rotationÚ boft_scalerƒrarbÚdropoutrUrVrWrXrrr–rrõr¬rôÚrotated_weightÚscaled_rotated_weightrrrr+Os`                  zLinear.forwardcótƒ ¡}d|S©Nzboft.©r;Ú__repr__©r>Úrepr?rrró zLinear.__repr__)rßrrr°FTF)r§rrcrLrdrLr¨rLrer~ràrár©rârãrárhri©FN©rårárærçrhrirÜ©rhrü©rTrr rrrrhr©rhr) r4r5r6r7r<rörûrër+rr_rrr?rrrÊsö  2  @rrcsneZdZdZ     d.d/‡fdd„ Zdd„Zd0d1dd„Zd2d d!„Zd3d#d$„Zd4d*d+„Z d5‡fd,d-„ Z ‡Z S)6ruz- BOFT implemented in a Conv2d layer. rßrr°Trfrgr§rrcrLrdr¨rer~r©rârhric  s4tƒ ¡t ||¡||_| ||||||¡dSrÛ)r;r<r`rär¯) r>rfr§rcrdr¨rer©rr?rrr<™s  ÿzConv2d.__init__cCstƒs d|_d}nd|_|d}|dkrtd|d›dƒ‚|dkr(t|d}nt ¡}|j t ||i¡¡|  ¡}|j |j d|j d} |dkr—|dkr—| |dkratd | ›d |›d ƒ‚|dkr|t t  |¡ƒkr{td |d›d |›d ƒ‚|d|dkrtd|›d|d›d ƒ‚t | |ƒ}nZ|dkrí|dkrí| |dkr°td | ›d|›d ƒ‚|dkræ| |d|krÌtd| ›d|d›d|›d ƒ‚| |d|dkrætd| ›d|d›d|›d ƒ‚t | |ƒ}ntdƒ‚|dkr|ddkrtd|›dƒ‚|ddkrtd|›dƒ‚t |d| | f¡} t|dƒD]} | | t |d| ƒt |dƒ|¡} | | ¡} | | | <q$|jd| ddt t |d|||¡¡|j|<t t dt |jƒ¡¡|j|<| ||¡||j|<||j|<| |¡| |j ¡dS)zF Update the conv2d layer with trainable BOFT weights. FrCTrr†r‡r:rˆz Convolutional kernel dimension (r‰rŠr‹rŒrrŽrrz7Invalid combination of convolutional kernel dimension (r‘r’r“r”r•r–r—N)!r)r™rHr9rjršrer›rkrprsÚ kernel_sizerLrœrrIržrŸr r¡r¢r£rOrarNrtrbr¤rcrdr¥r¦r€)r>r§rcrdr¨rer©rªrfÚconv_filter_dimr«r¬r­r®rrrr¯¬sŒÿ  ÿÿÿ ÿÿÿÿ ÿ   ÿ    zConv2d.update_layerFNrårárærçc Cs¤t||ƒ}|s dS|D]Ä}||j ¡vrÏ| ¡}|jj}|rt|jj ¡}| |¡\}}|  |j |j |j d|j d¡}t  |dd¡}t  || |j¡¡}t  |dd¡}||}|  |j |j |j d|j d¡}| ¡ |¡|jj_nU| |¡\}}|jj ¡}|  |j |j |j d|j d¡}t  |dd¡}t  || |j¡¡}t  |dd¡}||}|  |j |j |j d|j d¡}| ¡ |¡|jj_|j |¡q dS)rèNrrC)r rarrprér½rÕrêrërQrtrsr"rIrÐrìrírðrfrnrñròrrrrösJ  ÿÿ ÿÿ €Ùz Conv2d.mergecCs |js t d¡dSt|jƒdkr„|j ¡}| ¡}|jj}||j   ¡vr{|  |¡\}}|jj   ¡}| |j|j|jd|jd¡}t |dd¡}t | ¡| |j¡¡}t |dd¡}|d|}| |j|j|jd|jd¡}| |¡|j_ t|jƒdksdSdSr÷)rør%r&r·rnrrprér½rarrërÕrêrQrtrsr"rIrÐrìrùrírúrrrrûSs4   þ üézConv2d.unmergerücCsî|j|}|j| dd¡}|j\}}}}| ||||¡}| |¡}| ||||¡}|jr4t |¡} n|  d¡}t j t   |¡Ž} |   d¡} |j | j¡} t  | |  ddd¡¡} t  | | ¡} | d} td| jdƒD]} | | | } qj| |fS)rýrrCr)rarbrÐrGrQrÚr™r*rþrÿrIrrrÑr–rírErrrŸrrrrrëss$      zConv2d.get_delta_weightrTrr rrc Osº|j}|jr|jr | ¡|j|g|¢Ri|¤Ž}n;|jr,|j|g|¢Ri|¤Ž}n*tj|j|jjd|jjd|j |jd}tj t |j ƒdf|j |jd}|j D]’}||j ¡vr_qU|j|} |j| dd¡} |j|} | j\} } }}|  | | ||¡} | | ¡}| | | ||¡}| |ƒ}|jrœt |¡}n| d¡}tjt |¡Ž}| d¡}|j |¡}| |¡}t ||  ddd¡¡}t ||¡}|d}t!d|jdƒD]}|||}qÖ||}| |}qU| |jj"j#j¡}|jj"j#}| |j |j|jjd|jjd¡}t |dd¡}t $||¡}t |dd¡}||}| |j |j|jjd|jjd¡}| %||j¡}| %|jj&|j¡}t'j(||||jj)d|jj*dd}| |¡}|S)Nrr rCr)r.rér ÚpaddingÚstride)+r½r rørûrfrIrRrsr"rErNrLrtr€rarrbrÐrerGrQrÚr™r*rþrÿrrrÑr–rírrrŸrérÕrìÚ_cast_input_dtyper r Úconv2dr$r%)r>rTr rrrrrrƒrarbrrUrVrWrXrrr–rrõr¬rôrrr rrrr+”s|ý            þ ÿ  û zConv2d.forwardcrrrrr?rrrárzConv2d.__repr__)rßrrr°T)rfrgr§rrcrLrdrLr¨rLrer~r©rârhrirrrÜrr r!) r4r5r6r7r<r¯rörûrër+rr_rrr?rru”sø l ; !Mru)Ú __future__rrœr r%Ú contextlibrÚtypingrrrrIÚtorch.nnrjÚtorch.nn.functionalÚ functionalr Útorch.autogradrÚpeft.tuners.tuners_utilsrr rrr)r*ÚModuler9r`rrrurrrrÚs.     #&5 K