o ߥi@sPddlZddlmZddlmZeZd ddZ  d ddZ  d d d ZdS) N)nn) get_loggercCs|jd}tjdd}tj|tjdd|tjdd|ddtj|tjdd|tjdd|dd}|durKt||t||}|St||}|S)a Calclate two logits' the KL div value symmetrically. :param logits1: The first logit. :param logits2: The second logit. :param attention_mask: An optional attention_mask which is used to mask some element out. This is usually useful in token_classification tasks. If the shape of logits is [N1, N2, ... Nn, D], the shape of attention_mask should be [N1, N2, ... Nn] :return: The mean loss. none) reduction)dimN)shaper KLDivLosstorchsum LogSoftmaxSoftmaxmean)logits1logits2attention_mask labels_numKLDivlossr^/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/structbert/adv_utils.py_symmetric_kl_divs4 rh㈵>cKsj|dur|nd|}||j|d|}|dd|vr&|dd|vr,dn|d} |d} | s8d} d|vrA|d|di|d|i} | j} t|| | } tj | |dj}|j dd t d d j d d d d}t t|}|rtd|S||d}|||}t |||}t|||}|di|d|i} | j}t||| }||S)a2 Calculate the adv loss of the model. :param embedding: Original sentense embedding :param model: The model, or the forward function(including decoder/classifier), accept kwargs as input, output logits :param ori_logits: The original logits outputed from the model function :param ori_loss: The original loss :param adv_grad_factor: This factor will be multipled by the KL loss grad and then the result will be added to the original embedding. More details please check:https://arxiv.org/abs/1908.04577 The range of this value always be 1e-3~1e-7 :param adv_bound: adv_bound is used to cut the top and the bottom bound of the produced embedding. If not proveded, 2 * sigma will be used as the adv_bound factor :param sigma: The std factor used to produce a 0 mean normal distribution. If adv_bound not proveded, 2 * sigma will be used as the adv_bound factor :param kwargs: the input param used in model function :return: The original loss adds the adv loss Nr input_ids inputs_embedswith_attention_maskFrTinfrkeepdimprz'Nan occurred when calculating adv loss.gư>rdatanewsizenormal_poplogitsrr autogradgradnormfloatmaxanyisnanloggerwarningmin) embeddingmodel ori_logitsori_lossadv_grad_factor adv_boundsigmakwargs embedding_1rroutputs v1_logitsremb_grad emb_grad_normis_nan embedding_2 adv_logitsadv_lossrrrcompute_adv_loss3sN         rEcKsP|dur|nd|}||j|d|} |dd|vr&|d|d i|d| i} | j\} } t|| t|| } | d} tj | | dj}|j ddt ddj d dd d}t t|}|rptd |S||}| ||}t | ||}t| ||}|d i|d|i} | j\}}t||t||}||S) a Calculate the adv loss of the model. This function is used in the pair logits scenerio. :param embedding: Original sentense embedding :param model: The model, or the forward function(including decoder/classifier), accept kwargs as input, output logits :param start_logits: The original start logits outputed from the model function :param end_logits: The original end logits outputed from the model function :param ori_loss: The original loss :param adv_grad_factor: This factor will be multipled by the KL loss grad and then the result will be added to the original embedding. More details please check:https://arxiv.org/abs/1908.04577 The range of this value always be 1e-3~1e-7 :param adv_bound: adv_bound is used to cut the top and the bottom bound of the produced embedding. If not proveded, 2 * sigma will be used as the adv_bound factor :param sigma: The std factor used to produce a 0 mean normal distribution. If adv_bound not proveded, 2 * sigma will be used as the adv_bound factor :param kwargs: the input param used in model function :return: The original loss adds the adv loss NrrrrTrrr!r"z,Nan occurred when calculating pair adv loss.rr#)r4r5 start_logits end_logitsr7r8r9r:r;r<r=v1_logits_start v1_logits_endrr?r@rArBadv_logits_startadv_logits_endrDrrrcompute_adv_loss_pairosX       rL)N)Nr)r rmodelscope.utils.loggerrr1rrErLrrrrs     B