o Ni& @sddlZddlmZddlZddZdejdedejfdd Zdejded edejfd d Z d&dejdededd edejf ddZ d'dejdeddejfddZ dejdejdejfddZ de ejdejdejfddZde ejdejdedejfddZ d'de ejdejdededdejf d d!Zde ejdejdedejfd"d#Z d'de ejdejdededdejf d$d%ZdS)(N)LiteralcCs(|jd||}||}|S)aI Reshapes `weights` to match the shape of `task_tensors` by unsqeezing in the remaining dimenions. Args: task_tensors (`torch.Tensor`): The tensors that will be used to reshape `weights`. weights (`torch.Tensor`): The tensor to be reshaped. Returns: `torch.Tensor`: The reshaped tensor. ))shapedimview) task_tensorsweights new_shaper J/home/ubuntu/.local/lib/python3.10/site-packages/peft/utils/merge_utils.pyreshape_weight_task_tensorss r tensordensityreturncCsVt|d}t||}tj|d|dd}d||d<|||jS)a^ Prune the smallest values of the task tensors and retain the top-k values based on the specified fraction `density`. Args: tensor (`torch.Tensor`):The tensor to prune. density (`float`):The fraction of values to preserve. Should be in [0,1]. Returns: `torch.Tensor`: The tensor with the pruned weights. T)klargestr)torch zeros_likereshapeintnumeltopkabsr)r rmaskrtop_kr r r magnitude_based_pruning%s   rrescalecCs2ttj||d}||}|rtj||d|S)a Prune random values based on the specified fraction `density`. Args: tensor (`torch.Tensor`):The tensor to prune. density (`float`):The fraction of values to preserve. Should be in [0,1]. rescale (`bool`):Whether to rescale the result to preserve the expected value of the original tensor. Returns: `torch.Tensor`: The pruned tensor. )input fill_value)rother)r bernoulli full_likediv)r rrr pruned_tensorr r r random_pruning8s  r%Fmethod) magnituderandomcCsj|dkrtd|d|S|dkrtd||dkr#t||S|dkr.t|||dStd |) a Prune the values of task tensors based on the `method`. Args: tensor (`torch.Tensor`):The tensor to prune. density (`float`):The fraction of values to preserve. Should be in [0,1]. method (`str`):The method to use to prune. Should be one of ["magnitude", "random"]. rescale (`bool`):Whether to rescale the result to preserve the expected value of the original tensor. Returns: `torch.Tensor`: The pruned tensor. rz The density z= is greater than or equal to 1, no pruning will be performed.rzDensity should be >= 0, got r'r()rzUnknown method )warningswarn ValueErrorrr%)r rr&rr r r pruneKs r,total)r- frequencycCs^|}|dkr|jdd}n|dkr|jdd}ntd|dt|dkdd}||kS) aZ Get the mask of the majority sign across the task tensors. Task tensors are stacked on dimension 0. Args: tensor (`torch.Tensor`):The tensor to get the mask from. method (`str`):The method to use to get the mask. Should be one of ["total", "frequency"]. Returns: `torch.Tensor`: The majority sign mask. r-rrr.zUnimplemented mask method ""rr)signsum RuntimeErrorrwhere)r r&r1sign_magnitude majority_signr r r calculate_majority_sign_maskgsr7rmajority_sign_maskcCs.||jdd}|jdd}|tj|ddS)a Merge the task tensors using disjoint merge. Args: task_tensors (`torch.Tensor`):The task tensors to merge. majority_sign_mask (`torch.Tensor`):The mask of the majority sign across the task tensors. Returns: `torch.Tensor`: The merged tensor. rr/g?)min)r2rclamp)rr8mixed_task_tensorsnum_params_preservedr r r disjoint_merges r=rcCs0tj|dd}t||}||}|jdd}|S)a Merge the task tensors using `task arithmetic`. Args: task_tensors(`List[torch.Tensor]`):The task tensors to merge. weights (`torch.Tensor`):The weights of the task tensors. Returns: `torch.Tensor`: The merged tensor. rr/rstackr r2)rrweighted_task_tensorsr;r r r task_arithmetics   rAcBfdd|D}tj|dd}t||}||}|jdd}|S)aX Merge the task tensors using `task arithmetic`. Args: task_tensors(`List[torch.Tensor]`):The task tensors to merge. weights (`torch.Tensor`):The weights of the task tensors. density (`float`): The fraction of values to preserve. Should be in [0,1]. Returns: `torch.Tensor`: The merged tensor. cg|] }t|ddqSr'r&r,.0r rr r z#magnitude_prune..rr/r>rrrr@r;r rIr magnitude_prune    rMmajority_sign_methodcLfdd|D}tj|dd}t||d}t||}||}t||}|S)a Merge the task tensors using `ties`. Args: task_tensors(`List[torch.Tensor]`):The task tensors to merge. weights (`torch.Tensor`):The weights of the task tensors. density (`float`):The fraction of values to preserve. Should be in [0,1]. majority_sign_method (`str`): The method to use to get the majority sign mask. Should be one of ["total", "frequency"]. Returns: `torch.Tensor`: The merged tensor. crCrDrFrGrIr r rJrKzties..rr/rErr?r7r r=rrrrOr8r@r;r rIr ties   rScrB)aS Merge the task tensors using `dare linear`. Args: task_tensors(`List[torch.Tensor]`):The task tensors to merge. weights (`torch.Tensor`):The weights of the task tensors. density (`float`):The fraction of values to preserve. Should be in [0,1]. Returns: `torch.Tensor`: The merged tensor. cg|] }t|dddqSr(T)r&rrFrGrIr r rJzdare_linear..rr/r>rLr rIr dare_linearrNrXcrP)a Merge the task tensors using `dare ties`. Args: task_tensors(`List[torch.Tensor]`):The task tensors to merge. weights (`torch.Tensor`):The weights of the task tensors. density (`float`):The fraction of values to preserve. Should be in [0,1]. majority_sign_method (`str`): The method to use to get the majority sign mask. Should be one of ["total", "frequency"]. Returns: `torch.Tensor`: The merged tensor. crUrVrFrGrIr r rJrWzdare_ties..rr/rErQrRr rIr dare_tiesrTrY)F)r-)r)typingrrr Tensorfloatrboolr%r,r7r=listrArMrSrXrYr r r r sl    $ $