o NÆÏiLã@s´ddlmZddlZddlmZddlZddlZddlm Z m Z ddl m Z ddl mZddlmZdd lmZmZe ƒrHGd d „d ejjeƒZe ƒrXGd d „d ejjeƒZdSdS)é)Ú annotationsN)ÚOptional)Úis_bnb_4bit_availableÚis_bnb_available)Úcheck_adapters_to_merge)Údequantize_bnb_weight)Ú transposeé)Ú RandLoraLayerÚUniqueBaseGradcóneZdZ     d(d)‡fdd„ Zd*d+dd„Zd,dd„Zd-d.dd„Zd/d!d"„Zd0d$d%„Zd1‡fd&d'„ Z ‡Z S)2Ú Linear8bitLtrçFTÚ base_layerútorch.nn.ModuleÚ adapter_nameÚstrÚrÚintÚrandlora_alphaÚrandlora_dropoutÚfloatÚfan_in_fan_outÚboolÚ init_weightsÚreturnÚNonec ó>tƒ ¡t ||¡||_||_|j||||||| ddS©N)rrr©ÚsuperÚ__init__r rÚ_active_adapterÚ update_layer© ÚselfrrÚ randlora_AÚ randlora_BrrrrrÚkwargs©Ú __class__©úL/home/ubuntu/.local/lib/python3.10/site-packages/peft/tuners/randlora/bnb.pyr!!s  ùzLinear8bitLt.__init__NÚ safe_mergeÚ adapter_namesúOptional[list[str]]c Csêt||ƒ}|s dS|D]g}||j ¡vrq t d¡| |¡}| ¡j}| ¡j}|j dur2|j |_ t ||ƒ}|  |j ¡  |j ¡|}|rTt |¡ ¡sTtd|›dƒ‚tjj|  d¡d|jd  |j ¡| ¡_| ¡|j |¡q dS)ᆠMerge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`list[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. Nz[Merge RandLora module to 8-bit linear may get different generations due to rounding errors.ú1NaNs detected in the merged weights. The adapter ú seems to be brokenÚcpuF©Ú requires_gradÚhas_fp16_weights)rÚrandlora_lambdaÚkeysÚwarningsÚwarnÚget_delta_weightÚget_base_layerÚweightÚstateÚSCBrÚtoÚdtypeÚdeviceÚtorchÚisfiniteÚallÚ ValueErrorÚbnbÚnnÚ Int8Paramsr6Ú reset_gradsÚmerged_adaptersÚappend) r%r-r.Úactive_adapterÚ randlora_datar=r>ÚoutputÚw_datar+r+r,Úmerge=s8 ÿ      ÿÿþæzLinear8bitLt.mergecCsà|js t d¡dSt|jƒdkrn|j ¡}||j ¡vrq t d¡| |¡}|  ¡j }|  ¡j }|j dur;|j |_ t ||d}| |j¡ |j¡|}tjj| d¡d|jd |j¡|  ¡_ | ¡t|jƒdksdSdS) ú_ This method unmerges all merged adapter layers from the base weights. úAlready unmerged. Nothing to doNrz]Unmerge randlora module to 8-bit linear may get different generations due to rounding errors.)r>r3Fr4)Úmergedr9r:ÚlenrKÚpopr7r8r;r<r=r>r?rr@rArBrGrHrIr6rJ)r%rMrNr=r>rOrPr+r+r,Úunmergeks0  ÿ     ÿþìzLinear8bitLt.unmergeú list[torch.Tensor, torch.Tensor]cCó,|j|}|j|}|dur|j}|j}|jdko"|tjkp"|tjk}|j|  |¡}|j |  |¡}|rE|  ¡}|  ¡}|  ¡}|  ¡}t |j |jƒt|j |jƒ} } |dd…d|j…d| …f  |¡} |d| …d|j…dd…f  |¡} | jdd} t | ||¡jdd}| |jkr|| fS| j|jfS©aL Performs scaling on the smallest random base (randlora_A) and returns randlora_A and randlora_B in the correct order to fit the target layers' dimensions Args: adapter (str): The name of the adapter for which the delta weight should be computed. Nr3r )Ú start_dim)Úend_dim©r&r'rBrAÚtyperCÚfloat16Úbfloat16r7r@Úrandlora_gammarÚminÚ out_featuresÚ in_featuresÚmaxÚ num_basesÚflattenr ÚapplyÚT©r%ÚadapterrBr&r'rAÚ cast_to_fp32r7raÚmin_dimÚmax_dimÚsliced_AÚsliced_BÚupdate_BÚupdate_Ar+r+r,Úget_scaled_bases‰s* ""   zLinear8bitLt.get_scaled_basesú torch.TensorcCó4| |¡\}}||}t||jƒ}|j|}||S©zÖ Compute the delta weight for the given adapter. Args: adapter (str): The name of the adapter for which the delta weight should be computed. ©rsrrÚscaling©r%rkrqrrÚupdateÚ output_tensorrxr+r+r,r;¹s    zLinear8bitLt.get_delta_weightÚxcOs*|jr|jr | ¡|j|g|¢Ri|¤Ž}nx|jr'|j|g|¢Ri|¤Ž}nh|j|g|¢Ri|¤Ž}|jD]X}||j ¡vr@q6|j||jd\}}t   ¡ }|ra|j } |j } |j | kra|  | ¡}|j |} | |  |j ¡ƒ} t jj t jj | |¡|¡} |rƒ|   | ¡} |j|}|| |}q6|  |j ¡S)a Perform the forward pass using the RandLora adapter. Args: x (torch.Tensor): Input tensor. Returns: torch.Tensor: Output tensor after applying the RandLora adaptation. Note: This method implements the RandLora-specific forward pass. It applies the shared projections (randlora_A and randlora_B) along with the per-layer trainable parameters (lambda and gamma) to compute the adapter output. ©rB)Údisable_adaptersrTrWrÚactive_adaptersr7r8rsrBrCÚis_autocast_enabledrAr@rrHÚ functionalÚlinearrx©r%r|Úargsr(ÚresultrMrqrrÚrequires_conversionÚexpected_dtypeÚ compute_dtypeÚdropoutÚx_tempÚadapter_outputrxr+r+r,ÚforwardËs2        zLinear8bitLt.forwardcótƒ ¡}d|S©Nz randlora.©r Ú__repr__©r%Úrepr)r+r,rýó zLinear8bitLt.__repr__©rrrFT©rrrrrrrrrrrrrrrr©FN©r-rr.r/rr©rr©N©rrX©rrt©r|rtrrt©rr© Ú__name__Ú __module__Ú __qualname__r!rQrWrsr;rŒrÚ __classcell__r+r+r)r,r sö  .  0 2r cr )2Ú Linear4bitrrFTrrrrrrrrrrrrrrc rrrr$r)r+r,r!s  ùzLinear4bit.__init__Nr-r.r/cCsÄt||ƒ}|s dS|D]T}||j ¡vrq t d¡| |¡}| ¡j}|j}t j   |j |j ¡|}|rCt |¡ ¡sCtd|›dƒ‚t jj| d¡fddi|¤Ž |j¡| ¡_|j |¡q dS)r0Nz[Merge RandLora module to 4-bit linear may get different generations due to rounding errors.r1r2r3r5F)rr7r8r9r:r;r<r=Ú__dict__rGrÚdequantize_4bitÚdataÚ quant_staterCrDrErFrHÚ Params4bitr@rBrKrL)r%r-r.rMrNr=r(rPr+r+r,rQ s, ÿ   ÿ ÿëzLinear4bit.mergecCs¸|js t d¡dSt|jƒdkrZ|j ¡}||j ¡vrq t d¡| |¡}|  ¡j }|j }t j  |j|j¡|}t jj| d¡fddi|¤Ž |j¡|  ¡_ t|jƒdksdSdS)rRrSNrz]Unmerge RandLora module to 4-bit linear may get different generations due to rounding errors.r3r5F)rTr9r:rUrKrVr7r8r;r<r=r¤rGrr¥r¦r§rHr¨r@rB)r%rMrNr=r(rPr+r+r,rWIs$  ÿ   ÿózLinear4bit.unmergerXcCrYrZr]rjr+r+r,rsbs* ""   zLinear4bit.get_scaled_basesrtcCrurvrwryr+r+r,r;s   zLinear4bit.get_delta_weightr|cOs2|jr|jr | ¡|j|g|¢Ri|¤Ž}n||jr'|j|g|¢Ri|¤Ž}nl|j|g|¢Ri|¤Ž}| ¡}|jD]X}||j ¡vrDq:|j||j d\}}t   ¡ }|re|j } |j } |j | kre|  | ¡}|j|} | |  |j ¡ƒ} t jj t jj | |¡|¡} |r‡|   | ¡} |j|}|| |}q:|  |j ¡S)Nr})r~rTrWrÚclonerr7r8rsrBrCr€rAr@rrHrr‚rxrƒr+r+r,rŒ¡s4        zLinear4bit.forwardcrrŽrr‘r)r+r,rÆr“zLinear4bit.__repr__r”r•r–r—r˜r™ršr›rœrržr+r+r)r,r£sö  )  . %r£)Ú __future__rr9ÚtypingrÚ bitsandbytesrGrCÚpeft.import_utilsrrÚpeft.tuners.tuners_utilsrÚpeft.utils.integrationsrÚpeft.utils.otherrÚlayerr r rHÚModuler r£r+r+r+r,Ús     cþ