o eiU@sddlZddlmZddlmmZddlmZmZm Z ddl m Z m Z m Z mZmZer4ddlmZer=ddlmZdd ZGd d d ejZGd d d ejZ     dddZdS)N)is_scipy_availableis_vision_availablerequires_backends)box_iou dice_lossgeneralized_box_iounested_tensor_from_tensor_listsigmoid_focal_losslinear_sum_assignment)center_to_corners_formatcCsddt||DS)NcSsg|] \}}||dqS))logits pred_boxes).0abrr\/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/loss/loss_rt_detr.py 'sz!_set_aux_loss..)zip) outputs_class outputs_coordrrr _set_aux_loss&srcs0eZdZdZfddZeddZZS)RTDetrHungarianMatcheraThis class computes an assignment between the targets and the predictions of the network For efficiency reasons, the targets don't include the no_object. Because of this, in general, there are more predictions than targets. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). Args: config: RTDetrConfig csztt|dg|j|_|j|_|j|_|j |_ |j |_ |j |_ |j|jkr8|jkr8dkr;tddSdS)Nscipyrz#All costs of the Matcher can't be 0)super__init__rmatcher_class_cost class_costmatcher_bbox_cost bbox_costmatcher_giou_cost giou_costuse_focal_loss matcher_alphaalpha matcher_gammagamma ValueError)selfconfig __class__rrr5s  " zRTDetrHungarianMatcher.__init__cCs~|djdd\}}|ddd}tdd|D}td d|D}|jrft|ddd}|dd|f}d|j||jd|d  } |jd||j|d  } | | } n|ddd d }|dd|f } tj ||dd } t t |t | } |j| |j| |j| }|||d }d d|D}ddt||d D}dd|DS)aPerforms the matching Params: outputs: This is a dict that contains at least these entries: "logits": Tensor of dim [batch_size, num_queries, num_classes] with the classification logits "pred_boxes": Tensor of dim [batch_size, num_queries, 4] with the predicted box coordinates targets: This is a list of targets (len(targets) = batch_size), where each target is a dict containing: "class_labels": Tensor of dim [num_target_boxes] (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels "boxes": Tensor of dim [num_target_boxes, 4] containing the target box coordinates Returns: A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected targets (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes) rNrrrrcSg|]}|dqS class_labelsrrvrrrr^z2RTDetrHungarianMatcher.forward..cSr/boxesrr2rrrr_r4g:0yE>)pcSg|]}t|dqSr5lenr2rrrrucSsg|] \}}t||qSrr )ricrrrrvscSs0g|]\}}tj|tjdtj|tjdfqS)dtype)torch as_tensorint64)rr=jrrrrxs0)shapeflattenrAcatr%Fsigmoidr'r)logsoftmaxcdistr rr"r r$viewcpu enumeratesplit)r+outputstargets batch_size num_queriesout_bbox target_ids target_bboxout_probneg_cost_classpos_cost_classr r"r$ cost_matrixsizesindicesrrrforwardDs&&" zRTDetrHungarianMatcher.forward) __name__ __module__ __qualname____doc__rrAno_gradr^ __classcell__rrr-rr*s  rcseZdZdZfddZdddZdddZed d Z d d Z d dZ dddZ ddZ ddZdddZddZeddZddZZS) RTDetrLossah This class computes the losses for RTDetr. The process happens in two steps: 1) we compute hungarian assignment between ground truth boxes and the outputs of the model 2) we supervise each pair of matched ground-truth / prediction (supervise class and box). Args: matcher (`DetrHungarianMatcher`): Module able to compute a matching between targets and proposals. weight_dict (`Dict`): Dictionary relating each loss with its weights. These losses are configured in RTDetrConf as `weight_loss_vfl`, `weight_loss_bbox`, `weight_loss_giou` losses (`list[str]`): List of all the losses to be applied. See `get_loss` for a list of all available losses. alpha (`float`): Parameter alpha used to compute the focal loss. gamma (`float`): Parameter gamma used to compute the focal loss. eos_coef (`float`): Relative classification weight applied to the no-object category. num_classes (`int`): Number of object categories, omitting the special no-object category. cs|tt||_|j|_|j|j|jd|_ ddg|_ |j |_ t |jd}|j |d<|d||j|_|j|_dS)N)loss_vfl loss_bbox loss_giouvflr6rr7 empty_weight)rrrmatcher num_labels num_classesweight_loss_vflweight_loss_bboxweight_loss_giou weight_dictlosseseos_coefficienteos_coefrAonesregister_bufferfocal_loss_alphar'focal_loss_gammar))r+r,rjr-rrrs      zRTDetrLoss.__init__TcCstd|vrtdd|vrtd||}|d|}tjddt||Ddd}tt|t|\} } t| } |d} td dt||D} tj | j dd |j tj | j d } | | |<tj| |j d d dddf}tj| | jd}| |j||<|d|}t| }|j||jd ||}tj| ||dd}|d | j d |}d|iS)Nr#No predicted boxes found in outputsrz$No predicted logits found in outputscS g|] \}\}}|d|qSr5rr_target_r=rrrr z.RTDetrLoss.loss_labels_vfl..rdimcSrzr0rr{rrrrr~rr@devicerrm.r7r?none)weight reductionrf)KeyError_get_source_permutation_idxrArGrrrdetachdiagfullrErmrCrrHone_hot zeros_liker@to unsqueezerIr'powr) binary_cross_entropy_with_logitsmeansum)r+rQrRr] num_boxesrJidx src_boxes target_boxesiousr} src_logitstarget_classes_originaltarget_classestargettarget_score_original target_score pred_scorerlossrrrloss_labels_vfls0    zRTDetrLoss.loss_labels_vflc Csd|vrtd|d}||}tddt||D}tj|jdd|jtj|j d} || |<t | dd| |j } d | i} | S) zClassification loss (NLL) targets dicts must contain the key "class_labels" containing a tensor of dim [nb_target_boxes] rz#No logits were found in the outputscSrzr0rr{rrrrr~z*RTDetrLoss.loss_labels..Nrrrloss_ce)rrrArGrrrErmrCrrH cross_entropy transpose class_weight) r+rQrRr]rrJrrrrrrrrrr loss_labelss zRTDetrLoss.loss_labelsc Csb|d}|j}tjdd|D|d}|djdkd}tj | | } d| i} | S) z Compute the cardinality error, i.e. the absolute error in the number of predicted non-empty boxes. This is not really a loss, it is intended for logging purposes only. It doesn't propagate gradients. rcSr9r0r:r2rrrrr<z/RTDetrLoss.loss_cardinality..)rr7g?rcardinality_error) rrArBrImaxvaluesrnn functionall1_lossfloat) r+rQrRr]rrrtarget_lengths card_predcard_errrrrrrloss_cardinalityszRTDetrLoss.loss_cardinalityc Csd|vrtd||}|d|}tjddt||Ddd}i}tj||dd} | ||d <d tt t |t |} | ||d <|S) a; Compute the losses related to the bounding boxes, the L1 regression loss and the GIoU loss. Targets dicts must contain the key "boxes" containing a tensor of dim [nb_target_boxes, 4]. The target boxes are expected in format (center_x, center_y, w, h), normalized by the image size. rrycSrzr5r)rtr}r=rrrrr~z)RTDetrLoss.loss_boxes..rrrrrgrrh) rrrArGrrHrrrr r) r+rQrRr]rrrrrrrgrhrrr loss_boxess  zRTDetrLoss.loss_boxesc Csd|vrtd||}||}|d}||}dd|D}t|\} } | |} | |} tjj|dddf| j ddddd }|ddd f d }| d } | |j } t || |t || |d } | S) z Compute the losses related to the masks: the focal loss and the dice loss. Targets dicts must contain the key "masks" containing a tensor of dim [nb_target_boxes, h, w]. pred_masksz#No predicted masks found in outputscSr/)masksrrrrrrr r4z)RTDetrLoss.loss_masks..NbilinearF)sizemode align_cornersrr) loss_mask loss_dice)rr_get_target_permutation_idxr decomposerrr interpolaterErFrMr r) r+rQrRr]r source_idx target_idx source_masksr target_masksvalidrrrrr loss_maskss(       zRTDetrLoss.loss_masksc Cs|d}||}tddt||D}tj|jdd|jtj|jd} || |<t j | |jddddd f} t j || d d d } | d |jd|} d | iS)NrcSrzr0rr{rrrr r~z.RTDetrLoss.loss_labels_bce..rrrr.r7g?rrloss_bce)rrArGrrrErmrCrrHrrrr r+rQrRr]rrJrrrrrrrrrloss_labels_bces  zRTDetrLoss.loss_labels_bcecC4tddt|D}tdd|D}||fS)NcSs g|] \}\}}t||qSrrA full_like)rr=sourcer}rrrr-r~z:RTDetrLoss._get_source_permutation_idx..cSsg|]\}}|qSrr)rrr}rrrr.r4rArGrO)r+r] batch_idxrrrrr+z&RTDetrLoss._get_source_permutation_idxcCr)NcSs g|] \}\}}t||qSrr)rr=r}rrrrr3r~z:RTDetrLoss._get_target_permutation_idx..cSsg|]\}}|qSrr)rr}rrrrr4r4r)r+r]rrrrrr1rz&RTDetrLoss._get_target_permutation_idxc Csd|vrtd|d}||}tddt||D}tj|jdd|jtj|j d} || |<t j | |jddd dd f} t || |j |j} | d|jd|} d | iS) NrzNo logits found in outputscSrzr0rr{rrrr>r~z0RTDetrLoss.loss_labels_focal..rrrr.r7 loss_focal)rrrArGrrrErmrCrrHrr r'r)rrrrrrloss_labels_focal7s  zRTDetrLoss.loss_labels_focalcCsL|j|j|j|j|j|j|jd}||vrtd|d||||||S)N)labels cardinalityr6rbcefocalrizLoss z not supported)rrrrrrrr*)r+rrQrRr]rloss_maprrrget_lossIs zRTDetrLoss.get_lossc Cs|d|d}}dd|D}|ddj}g}t|D]A\}}|dkrItj|tj|d} | |} t||t| ks?J|||| fq|tjdtj|dtjdtj|dfq|S)Ndn_positive_idx dn_num_groupcSr9r0r:rrrrrZr<z6RTDetrLoss.get_cdn_matched_indices..rr1r) rrOrAarangerCtiler;appendzeros) dn_metarRrrnum_gtsrdn_match_indicesr=num_gtgt_idxrrrget_cdn_matched_indicesWs  z"RTDetrLoss.get_cdn_matched_indicesc sdd|D}||}tdd|D}tj|gtjtt|j d}tj |dd }i}j D]} |||||fddD|q7d |vrt|d D]8\}||}j D]*}|d krpqi |||||fd dDfd dD|qiq\d |vrd|vrtd|d|}||dd}t|d D]8\}j D]0}|d krqi} j |||||fi| fddDfddD|qq|S)a This performs the loss computation. Args: outputs (`dict`, *optional*): Dictionary of tensors, see the output specification of the model for the format. targets (`list[dict]`, *optional*): List of dicts, such that `len(targets) == batch_size`. The expected keys in each dict depends on the losses applied, see each loss' doc. cSsi|] \}}d|vr||qS)auxiliary_outputsrrkr3rrr ysz&RTDetrLoss.forward..css|] }t|dVqdS)r1Nr:rrrr sz%RTDetrLoss.forward..rr)minc*i|]}|jvr||j|qSrrqrrl_dictr+rrr*rrcrrrrrrrrrc i|] \}}|d|qS)_aux_rrr=rrrr~dn_auxiliary_outputsdenoising_meta_valuesz}The output must have the 'denoising_meta_values` key. Please, ensure that 'outputs' includes a 'denoising_meta_values' entry.rcrrrrrrrrrcr)_dn_rrrrrrr~)itemsrkrrArBrnextiterrrclampitemrrrupdaterOr*r) r+rQrRoutputs_without_auxr]rrrrrkwargsr)r=rr+rr^nsN "         zRTDetrLoss.forward)T)r_r`rarbrrrrArcrrrrrrrr staticmethodrr^rdrrr-rre{s"       rec Kst|} | |i} || d<|| d<d} |jrx| dur4tj|| ddd\}}tj|| ddd\}}t|ddddfdd|ddddfdd} | | d <| d t|g|g| durxt|dd|dd| d <| | d <| | |}t| }||| fS) Nrr dn_num_splitrrr7rrrrr) rerauxiliary_lossrArPrrextendrr)rrrrr,rrenc_topk_logitsenc_topk_bboxesrr criterion outputs_lossr dn_out_coord dn_out_class loss_dictrrrrRTDetrForObjectDetectionLosss* :   r )NNNNN)rAtorch.nnrtorch.nn.functionalrrHutilsrrrloss_for_object_detectionrrr r r scipy.optimizer transformers.image_transformsrrModulerrer rrrrs(   Q<