o Ni@sddlmZddlZddlmZmZddlZddlmZddl m Z eej ej ej fZGdddejZGdd d ejZdS) ) annotationsN)OptionalUnion) XLoraConfigcs&eZdZdfdd ZddZZS)TemperatureScaledSoftmax?cs"t||_tjdd|_dS)N)dim)super__init__ temperaturennSoftmaxsoftmax)selfr  __class__P/home/ubuntu/.local/lib/python3.10/site-packages/peft/tuners/xlora/classifier.pyr s z!TemperatureScaledSoftmax.__init__cCs||j}||S)N)r r)rlogits scaled_logitsrrrforward"s  z TemperatureScaledSoftmax.forward)r)__name__ __module__ __qualname__r r __classcell__rrrrrsrcsVeZdZdZdfd d Z d d!ddZ d d!ddZd"ddZd#ddZZ S)$XLoraClassifierz7 A classifier to select LoRA layers for XLora. model nn.Moduleconfigr n_classesintn_layersdevice torch.devicec st||_||_||_g|_t|jjd|_|j |_ d|_ t | j|_|jdk}g}|jjdkr\|jrKtj|j||dd||j}ntj|j|dd||j}n|jjdkrftd|tj|j|jdd||j|t|r|tj|jd t|jd D])} |tj|j|jdd||j|t|r|tj|jd q|jrtj|j||dd||j}ntj|j|dd||j}tjg||R|_d S) z Construct an X-LoRA classifier from a model, config and some metadata. Note that n_layers is the number of LoRA adapter layers, not the number of model layers. )r FgrT)biasrz'X-LoRA depth must be strictly positive.)pN)r r r!r#r log_scalingsrsoftmax_temperaturerscaling_pass_valueoverride_scaling_pass_valuescalings_loggingnext parametersdtypexlora_dropout_p xlora_depthlayerwise_scalingsrLinear hidden_sizeto ValueErrorappend xlora_sizeReLUDropoutrange Sequentiallayers) rrr r!r#r$ add_dropoutr>last_rrrr .s>   &" ((& zXLoraClassifier.__init__N input_idsOptional[torch.LongTensor] inputs_embedsOptional[torch.FloatTensor]return torch.TensorcOsd|dur|jd}|j}|jd}n |jd}|j}|jd}t|||j|jf|jj||jdS)aP Make some dummy scalings for the scalings pass (the one to get the logits for the X-LoRA classifier). These are of shape (batch_size, seq_len, n_layers, n_classes) and filled with the override scalings pass value. Note that n_layers is the number of LoRA adapter layers, not the number of model layers. Nrr)r$r0) shaper$torchfullr#r!r,r6r0)rrBrDargskwargs batch_sizer$seq_lenrrrmake_dummy_scalingses    z#XLoraClassifier.make_dummy_scalingsc Os|dur|jd}|jd}n |jd}|jd}|j}|d} |j| } |jjs8| d} | dd|jd} | |||j|j } |jj rK| | } |j rT|j| | S)zt Using the hidden states of the model, predict `n_classes` LoRA alpha values. Returns the scalings. Nrrr r()rH hidden_statesr>rr r3 unsqueezeexpandr#reshaper!enable_softmaxrr-r)r8) rresultrBrDrKrLrMrNrP hidden_staterscalingsrrrrs"        zXLoraClassifier.forward/dict[int, tuple[list[int], list[torch.Tensor]]]cCsdi}t|jD](\}}|jd}||vr|g|gf||<q||d|||d|q|S)aL Returns bucketed scalings, bucketed by seq_len. Each value consists of the positions (the first) and the associated tensors. The positions are paired with the associated tensors and give the position in the scaling log. Each scaling is a tensor of shape (batch_size, seq_len, n_layers, n_classes)). rr) enumerater)rHr8)r seqlens_mapiscalingrNrrr_get_bucketed_scalingss z&XLoraClassifier._get_bucketed_scalingsvalueUnion[Number, None]cCs*|dur d|j|_n||_|j|j_dS)Nr)r!r,r r+)rr^rrr _set_override_scaling_pass_valuesz0XLoraClassifier._set_override_scaling_pass_value) rrr rr!r"r#r"r$r%)NN)rBrCrDrErFrG)rFrX)r^r_) rrr__doc__r rOrr]r`rrrrrr)s9  .r) __future__rbuiltinstypingrrrItorch.nnrr rr"floatboolNumberModulerrrrrrs