o Ni$@szddlZddlZddlmZddlZddlmZddlmmZ ddl m Z m Z Gddde Z Gdddeje ZdS)N)Optional)BaseTunerLayercheck_adapters_to_mergec@sLeZdZdZdZdejfddZ  dded efd d Z d d Z ddZ dS) ShiraLayer) shira_weight)rscaling shira_indices base_layercKs|||_i|_i|_ti|_i|_|jj|_ d|_ g|_ | }t |tjr/|j|j}}ntd||_||_||_dS)NFz)Only nn.Linear layers supported currently)r rrnn ParameterDictrr weightshape weight_shape_disable_adaptersmerged_adaptersget_base_layer isinstanceLinear in_features out_featuresNotImplementedErrorkwargs)selfr rrrrK/home/ubuntu/.local/lib/python3.10/site-packages/peft/tuners/shira/layer.py__init__ s    zShiraLayer.__init__TF init_weightsinference_modec Ksd|dkr td|||j|<d|j|<||j|j}||j|jkr4td|d|j|jd|r;t|nt|}tj | |j j j  |j j jdd|j|<|durt|dk} t| dd| d dgd tj|j|<|j| |j j j|j|<|j|jd |j|jdkrtd |d |j |||j|j|d dS) Nrz?`r` should be a positive integer value but the value passed is g?z The set rank zP results in more shira params than the total number of params in the base layer z and this is not allowed.T) requires_gradzFThe SHiRA indices and weights are not the same dimensions for adapter z in layer )r) ValueErrorrrrrtorchzerosrandnr Parametertor r dtypedevicerwherecat unsqueezeintr r%_move_adapter_to_device_of_base_layer set_adapteractive_adapters) r adapter_namemaskrrrrnum_shira_weightshira_init_weight mask_indicesrrr update_layer6s:     zShiraLayer.update_layercCstj|j|dSN)r initzeros_r)rr0rrrreset_shira_parametersfsz!ShiraLayer.reset_shira_parameterscCs||jvrdS||j|<dSr6)r)radapterscalerrr set_scaleis zShiraLayer.set_scaleN)TF) __name__ __module__ __qualname__adapter_layer_namesother_param_namesr Modulerboolr5r9r<rrrrrs 0 rc seZdZ   ddededededd f fd d Zdd ed eeedd fddZ dddZ de j fddZ de j de j fddZdeffdd ZZS)rrFTr0rfan_in_fan_outrreturnNc sVttj||fi|||_|j|urtd||_|j||||ddS)Nz)SHiRA does not support nested base layers)r) superrrrDr rr!_active_adapterr5)rr r1r0rrDrr __class__rrrrs zLinear.__init__ safe_merge adapter_namescCst||}|s dS|D]A}||jvrL|}|r;|jj}|||7}t | s6t d|d||j_n |jj||7_|j |q dS)a^ Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. Nz1NaNs detected in the merged weights. The adapter z seems to be broken)rrkeysrr datacloneget_delta_weightr"isfiniteallr!rappend)rrJrKactive_adapterr orig_weightsrrrmerges$     z Linear.mergecCsj|js tddSt|jdkr3|j}||jvr*|j j | |8_ t|jdksdSdS)Nz Already unmerged. Nothing to do.r) mergedwarningswarnlenrpoprrLrr rMrO)rrSrrrunmerges  zLinear.unmergecCsD|j||j|j|j|<t|j||j||j||jS)z Compute the delta weight for the given adapter. Args: adapter (str): The name of the adapter for which the delta weight should be computed. )r r&rr(r"sparse_coo_tensorrr)rr:rrrrOs zLinear.get_delta_weightxcOs|jr|jr ||j|g|Ri|}|S|jr)|j|g|Ri|}|St|jjj}|jD]}||j vr>q4|| |7}q4t j |||jjd}|S)N)bias)disable_adaptersrVr[r copydeepcopyr rMr/rrLrOFlinearr^)rr]argsrresult new_weightrSrrrforwards   zLinear.forwardcst}d|S)Nzshira.)rF__repr__)rreprHrrrhs zLinear.__repr__)rFT)FN)rEN)r=r>r?strr,rCrrlistrUr[r"TensorrOrgrh __classcell__rrrHrrps(   ' r)r`rWtypingrr"torch.nnr torch.nn.functional functionalrbpeft.tuners.tuners_utilsrrrrBrrrrrs  V