o NÆÏi§Dã @sddlZddlmZmZmZddlZddlmZddlm Z m Z ddl m Z m Z Gdd„de ƒZGdd „d ejeƒZd e d ed ejd ejfdd„Zd e d ed ejd ejdeejejff dd„Zd e d ed ejd ejdejf dd„Zdejjdede deejjfdd„ZdS)éN)ÚAnyÚOptionalÚUnion)ÚBaseTunerLayerÚcheck_adapters_to_mergeé)Ú RoadConfigÚ RoadVariantc@s„eZdZUdZdZeedfed<dZeedfed<dde j d e d d fd d „Z e d eefdd„ƒZ dde fdd„Zdd„Zd S)Ú RoadLayeru¢ Road layer. Generally the idea of RoAD is to split the input vector into many 2D vectors and rotate each 2D vector with its own 2D rotation matrix. For additional flexibility, each rotation matrix is multiplied by a trainable scale. when applied to vector R @ x each pair of elements of x is transformed like this: `yâ‚€ = xâ‚€ * α * cosθ - xâ‚™ * α * sinθ` and `yâ‚™ = xâ‚€ * α * sinθ + xâ‚™ * α * cosθ` The scales α and angles θ are learned for each pair of elements and, moreover, each of the 4 instances in the rotation matrix may actually be different (when using variant 2 or 4). Note that instead of using two consecutive elements xâ‚€ xâ‚ we first split the whole vector into groups and pair elements from the first with the second half of the same group, which allows for more efficient inference implementation. The adapter needs to only store the angles θ and scales α, rather than the full matrix R and the inference implementation only needs to do elementwise vector multiplications. For merging the weights, we make use of the following formula: R @ (W @ x + b) = (R @ W) @ x + R @ b. The lhs part is how it is used in unmerged state (using efficient elementwise implementation instead of matrix multiplication) and the rhs part is how it is used in merged state where (R @ W) becomes the new weight matrix and R @ b becomes the new bias. )Ú road_thetaÚ road_alpha.Úadapter_layer_names)ÚvariantÚ group_sizeÚother_param_namesFÚ base_layerÚephemeral_gpu_offloadÚreturnNcKs~||_i|_i|_t i¡|_t i¡|_d|_g|_|  ¡}t |tj ƒr-|j |j }}n tdt|ƒ›dƒ‚||_ ||_ dS)NFzUnsupported layer type 'z)' encountered, cannot apply RoAd adapter.)rrrÚnnÚ ParameterDictr r Ú_disable_adaptersÚmerged_adaptersÚget_base_layerÚ isinstanceÚLinearÚ in_featuresÚ out_featuresÚ ValueErrorÚtype)ÚselfrrÚkwargsrr©r!úJ/home/ubuntu/.local/lib/python3.10/site-packages/peft/tuners/road/layer.pyÚ__init__8s    zRoadLayer.__init__cCs h|j£S©N)r )rr!r!r"Ú_available_adaptersJs zRoadLayer._available_adaptersÚinference_modecCsÐ||j|<||j|<|j|dkrtd|›dƒ‚|dkr#|jd}n|dkr+|j}n|dkr5|jd}ntd|›d ƒ‚t t |¡¡|j|<t t |¡¡|j |<|  ||¡|  |¡|j |j |d dS) NrzDThe out_features of the base layer must be divisible by group_size (z) when using RoadLayer.Úroad_1éÚroad_2Úroad_4úUnsupported variant úB for RoadLayer. Supported variants are road_1, road_2, and road_4.)r&)rrrrrÚ ParameterÚtorchÚemptyr r Úreset_parametersÚ%_move_adapter_to_device_of_base_layerÚ set_adapterÚactive_adapters)rÚ adapter_namerrÚ init_weightsr&Úsizer!r!r"Ú update_layerNs(   ÿ   ÿ  zRoadLayer.update_layercCsl|dur tjj|j|jdddtjj|j|jddddStj |j|j¡tj |j|j¡dS)NFggà?)ÚmeanÚstdgð?)rÚinitÚnormal_r Údatar Úzeros_Úones_)rr4r5r!r!r"r0qs zRoadLayer.reset_parameters)F)Ú__name__Ú __module__Ú __qualname__Ú__doc__r ÚtupleÚstrÚ__annotations__rrÚModuleÚboolr#ÚpropertyÚsetr%r7r0r!r!r!r"r s  ú ú #r c sÊeZdZ   ddedededeeefdd f ‡fd d „ Zd d „Z de j de de de j fdd„Z de j de deede de j f dd„Zddedeeedd fdd„Zd dd„Zdef‡fdd„ Z‡ZS)!rr'é@Tr4rrr5rNc s:tƒ ¡tj||fi|¤Ž||_|j||||ddS)N)r5)Úsuperr#r Ú_active_adapterr7)rrr4rrr5r ©Ú __class__r!r"r#|s  üzLinear.__init__cOs`| dd¡}|dur dSt|ƒt|ƒkr%dt|ƒ›dt|ƒ›d}t|ƒ‚|jr.d}t|ƒ‚dS)zMCheck if the arguments are compatible with the configs and state of the modelÚ adapter_namesNzNLength of `adapter_names` should be the same as the number of inputs, but got z and z respectively.z`Cannot pass `adapter_names` when there are merged adapters, please call `unmerge_adapter` first.)ÚgetÚlenrÚmerged)rÚxÚargsr rOÚmsgr!r!r"Ú_check_forward_args‘s ÿÿÿüzLinear._check_forward_argsrSrTr cOs|j|g|¢Ri|¤Ž| dd¡}|jr*|jr| ¡|j|g|¢Ri|¤Ž}|S|jr;|j|g|¢Ri|¤Ž}|S|durO|j|g|¢Rd|i|¤Ž}|S|j|g|¢Ri|¤Ž}|j}|jD]&}||j vriqa|  ||j |j¡}t |j ||j||j ||j||ƒ}qa| |¡}|S)NrO)rVÚpopÚdisable_adaptersrRÚunmergerÚ_mixed_batch_forwardÚdtyper3r%Ú_cast_input_dtyper Ú _apply_roadrrr Úto)rrSrTr rOÚresultÚtorch_result_dtypeÚactive_adapterr!r!r"Úforward¤s8 éëî  û zLinear.forwardrOc sÆ|j|g|¢Ri|¤Ž}t|ƒ}g}|D]‰| ‡fdd„t|ƒDƒ¡qt|ƒD]7\}} | dkr2q)| |jvr8q)|j| jj} ||| | ¡} t |j | |j | |j| |j | | ƒ|||<q)|S)Ncsg|] \}}|ˆkr|‘qSr!r!)Ú.0ÚindexÚitem©Úadapterr!r"Ú Ïsz/Linear._mixed_batch_forward..Ú__base__) rrIÚappendÚ enumerater%r r<r[r^r]rrr ) rrSrOrTr r_Úunique_adaptersÚsub_batch_indices_listÚirar[Ú sub_batchr!rfr"rZÅs( ûzLinear._mixed_batch_forwardFÚ safe_mergec Cs|t||ƒ}|s dS|D]°}||jvr»| ¡}|jj}t|j||j||j|j |j |j ƒ}|r„|jj   ¡}t   | |¡|¡}t  |¡ ¡sOtd|›dƒ‚| ¡ |¡|j_ |jdurƒ|j  ¡}t   | |¡|¡}t  |¡ ¡sztd|›dƒ‚| ¡ |¡|j_ n1|jj }t   | |¡|¡}| ¡ |¡|j_ |jdurµ|jj }t   | |¡|¡}| ¡ |¡|j_ |j |¡q dS)ab Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If `True`, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If `None`, all active adapters will be merged. Defaults to `None`. Nz1NaNs detected in the merged weights. The adapter z seems to be brokenz.NaNs detected in the merged bias. The adapter )rr%rÚweightr[Ú_get_delta_weightrrr r<r Úcloner.Úmatmulr^ÚisfiniteÚallrÚ contiguousÚbiasrrj) rrprOrarÚ orig_dtypeÚroad_RÚ orig_weightÚ orig_biasr!r!r"ÚmergeæsP    ü  ÿ   ÿ€  €Õz Linear.mergecCsê|js t d¡dSt|jƒdkrs|j ¡}||jvrj| ¡j}|j }t |j ||j ||j |j|j|jƒ}tj | tj¡¡ |¡}t ||j¡}| ¡|_| ¡jdurjt || ¡jj¡}| ¡| ¡j_t|jƒdksdSdS)zW This method unmerges all merged adapter layers from the base weights. z Already unmerged. Nothing to do.Nr)rRÚwarningsÚwarnrQrrWr%rrqr[rrrrr r<r r.ÚlinalgÚinvr^Úfloat32rtrwrx)rrarqryrzÚ inv_road_Rr{r|r!r!r"rY%s*      ü ìzLinear.unmergecstƒ ¡}d|S)Nzroad.)rKÚ__repr__)rÚreprMr!r"r„Bs zLinear.__repr__)r'rJT)FN)rN)r?r@rArDr ÚintrrGr#rVr.ÚTensorrrbÚlistrZrr}rYr„Ú __classcell__r!r!rMr"rzs>úýüû úø!ÿÿÿÿ þ ! ?rrrr r c CsÄt||||ƒ\}}t |¡}|jd}| dd|d¡dd…ddgdd…f ¡}t |¡ dd|d|¡dd…ddgdd…dd…f} | dd…ddd…dd…fd9<|  ||¡} || 7}|S)Nréÿÿÿÿr(r)Ú _prepare_colsr.ÚdiagÚshapeÚreshapeÚflatten) rrr r Ú first_colÚ second_colÚ output_tensorr6Úswapped_second_colÚrotated_diag_second_colr!r!r"rrGs  ,6$ rrrc CsX|dkr6| d|d¡jddd ¡}| d|d¡jddd ¡}| ¡}| ¡}||}||}||fS|dkrN| ¡}| ¡}||}||}||fS|dkr¤| dd|¡}|dd…ddd…f ¡ ¡}|dd…ddd…f ¡ ¡}| dd|¡}|dd…ddd…f ¡}|dd…ddd…f ¡} ||}| |}||fStd |›d ƒ‚) Nr'rŠr(r©Údimr)r*rr+r,)rŽÚrepeat_interleaverÚcosÚsinr) rrr r Ú theta_cosÚ theta_sinrr‘Úalpha_1Úalpha_2r!r!r"r‹[s6éðü ÿr‹rSc Cs€t||||ƒ\}}| dd|d¡}|dd…ddd…f}|dd…ddd…f} tj| |fdd |j¡} ||| |} | S)NrŠr(rrr•)r‹rŽr.Ústackr) rrr r rSrr‘Ú x_groupedÚx1Úx2Ú rotate_half_xr_r!r!r"r]ƒsr]Útargetr4Ú road_configcKs@d}t|tƒr | ¡}n|}t|tjjƒrt||fi|¤Ž}|Sr$)rrrr.rr)r£r4r¤r Ú new_moduleÚtarget_base_layerr!r!r"Údispatch_default’s  r§)r~Útypingrrrr.Útorch.nnrÚpeft.tuners.tuners_utilsrrÚconfigrr r rFrr†r‡rrrCr‹r]rDr§r!r!r!r"ÚsR `Nÿÿÿÿ þ(ÿÿÿÿ ÿÿþý û