o i^@s8ddlZddlZddlmZddlmZmZddlZ ddl Z ddl m m Zddl mZm Z ddlmZddlmZmZmZddlmZdd lmZmZdd lmZdd lmZmZm Z dd l!m"Z"d dl#m$Z$errddl%m&Z&e rddl'm(Z(ddl)m*Z*eeddGdddeZ+ dWde jde jde jfddZ,dededefddZ-de jde jde jfd d!Z.Gd"d#d#e j/Z0deded$e1defd%d&Z2de jde jd$e1de jfd'd(Z3Gd)d*d*e j/Z4Gd+d,d,e j/Z5Gd-d.d.e j/Z6 /dXd0e j/d1e jd2e jd3e jd4ee jd5e7d6e7fd7d8Z8Gd9d:d:e j/Z9Gd;d<de7d?e;de jfd@dAZGdFdGdGe j/Z?GdHdIdIeZ@GdJdKdKe jAZBGdLdMdMe j/ZCGdNdOdOe j/ZDGdPdQdQe j/ZEeGdRdSdSeZFedTdGdUdVdVeFZGdSdVgZHdS)ZN) dataclass)CallableOptional)Tensornn)ACT2FN) ModelOutputis_scipy_availablerequires_backends)GradientCheckpointingLayer)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringis_accelerate_available)check_model_inputs) EomtConfig)linear_sum_assignment) PartialState)reducea Class for outputs of [`EomtForUniversalSegmentationOutput`]. This output can be directly passed to [`~EomtImageProcessor.post_process_semantic_segmentation`] or [`~EomtImageProcessor.post_process_instance_segmentation`] or [`~EomtImageProcessor.post_process_panoptic_segmentation`] to compute final segmentation maps. Please, see [`~EomtImageProcessor] for details regarding usage. ) custom_introc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejed<dZee ejed<dZeeejed <dS) "EomtForUniversalSegmentationOutputa* loss (`torch.Tensor`, *optional*): The computed loss, returned when labels are present. class_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, num_labels + 1)` representing the proposed classes for each query. Note the `+ 1` is needed because we incorporate the null class. masks_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, height, width)` representing the proposed masks for each query. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Last hidden states (final feature map) of the last layer. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states all layers of the model. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tuple(torch.FloatTensor)` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Self and Cross Attentions weights from transformer decoder. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segmentation. Nlossclass_queries_logitsmasks_queries_logitslast_hidden_state hidden_states attentions patch_offsets)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrtupler r!listrr+r+c/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/eomt/modeling_eomt.pyr2s  rFinput_featurespoint_coordinatesreturncKsL|dkr d}|d}tjjj|d|dfi|}|r$|d}|S)a( A wrapper around `torch.nn.functional.grid_sample` to support 3D point_coordinates tensors. Args: input_features (`torch.Tensor` of shape (batch_size, channels, height, width)): A tensor that contains features map on a height * width grid point_coordinates (`torch.Tensor` of shape (batch_size, num_points, 2) or (batch_size, grid_height, grid_width,: 2)): A tensor that contains [0, 1] * [0, 1] normalized point coordinates add_dim (`bool`): boolean value to keep track of added dimension Returns: point_features (`torch.Tensor` of shape (batch_size, channels, num_points) or (batch_size, channels, height_grid, width_grid): A tensor that contains features for points in `point_coordinates`. rTg@?)dim unsqueezer&r functional grid_samplesqueeze)r-r.add_dimkwargspoint_featuresr+r+r, sample_point]s    r:inputslabelscCsd|d}dt||j}|ddddf|ddddf}d|d|d}|S)a A pair wise version of the dice loss, see `dice_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: `torch.Tensor`: The computed loss between each pairs. rr0N)sigmoidflattenr&matmulTsum)r;r< numerator denominatorrr+r+r,pair_wise_dice_loss}s ,rEc Csj|jd}tjdd}||t|}||t|}t|||j}t||d|j}||}|S)a A pair wise version of the cross entropy loss, see `sigmoid_cross_entropy_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss between each pairs. rnone reduction)shaperBCEWithLogitsLossr& ones_like zeros_liker@rA) r;r<height_and_width criterioncross_entropy_loss_poscross_entropy_loss_negloss_posloss_negrr+r+r,$pair_wise_sigmoid_cross_entropy_losss  rSc sleZdZdZ ddedededeffdd Zed ej d ej d ej d ej de e e f ddZ Z S)EomtHungarianMatcheraqThis class computes an assignment between the labels and the predictions of the network. For efficiency reasons, the labels don't include the no_object. Because of this, in general, there are more predictions than labels. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). r11 cost_class cost_mask cost_dice num_pointscsFt|dkr|dkr|dkrtd||_||_||_||_dS)aHCreates the matcher Params: cost_class (`float`, *optional*, defaults to 1.0): Relative weight of the classification error in the matching cost. cost_mask (`float`, *optional*, defaults to 1.0): This is the relative weight of the focal loss of the binary mask in the matching cost. cost_dice (`float`, *optional*, defaults to 1.0): This is the relative weight of the dice loss of the binary mask in the matching cost. num_points (`int`, *optional*, defaults to 12544): No. of points to sample on which the mask loss will be calculated. The same set of K points are uniformly sampled for all prediction and ground truth masks to construct the cost matrix for bipartite matching. rzAll costs can't be 0N)super__init__ ValueErrorrYrVrWrX)selfrVrWrXrY __class__r+r,r[s  zEomtHungarianMatcher.__init__rr mask_labels class_labelsr/cCsfg}|jd}t|D]}||d}||} |dd||f } ||| } | dddf} | dddf} tjd|jd| jd} | | jddd} t | | dd d} | | jddd}t | |dd d} t | | }t | | }|j ||j| |j|}t|td }t|td }t|d}t|}||q d d |D}|S) ao Params: masks_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, num_labels` with the classification logits. class_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, height, width` with the predicted masks. class_labels (`torch.Tensor`): A tensor of dim `num_target_boxes` (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels. mask_labels (`torch.Tensor`): A tensor of dim `num_target_boxes, height, width` containing the target masks. Returns: matched_indices (`list[tuple[Tensor]]`): A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected labels (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes). rr=Nrr0deviceF align_cornersg _Bg _cSs0g|]\}}tj|tjdtj|tjdfqS)dtype)r& as_tensorint64).0ijr+r+r, s$z0EomtHungarianMatcher.forward..)rIrangesoftmaxtor&randrYrcrepeatr:r6rSrErWrVrXminimumtensormaximum nan_to_numrcpuappend)r]rrr`raindices batch_sizerk pred_probs pred_maskrV target_maskr.target_coordinatespred_coordinatesrWrX cost_matrixassigned_indicesmatched_indicesr+r+r,forwards4       zEomtHungarianMatcher.forward)r1r1r1rU)r"r#r$r%floatintr[r&no_gradrr*r)r __classcell__r+r+r^r,rTs0 rT num_maskscCsX|d}d||d}|d|d}d|d|d}||}|S)a4 Compute the DICE loss, similar to generalized IOU for masks as follows: $$ \mathcal{L}_{\text{dice}(x, y) = 1 - \frac{2 * x \cap y }{x \cup y + 1}} $$ In practice, since `labels` is a binary mask, (only 0s and 1s), dice can be computed as follow $$ \mathcal{L}_{\text{dice}(x, y) = 1 - \frac{2 * x * y }{x + y + 1}} $$ Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). num_masks (`int`): The number of masks present in the current batch, used for normalization. Returns: `torch.Tensor`: The computed loss. rr0r=)r>r?rB)r;r<rprobsrCrDrr+r+r, dice_losss  rcCs,tjdd}|||}|d|}|S)a| Args: inputs (`torch.Tensor`): A float tensor of arbitrary shape. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss. rFrGr)rrJmeanrB)r;r<rrNcross_entropy_lossrr+r+r,sigmoid_cross_entropy_loss7s rcs~eZdZdedeeefffdd Zdeee dee fddZ d ee de e e ffd d Z d e d ee de ejdeee ffddZdej deej de ejde deeej ff ddZddZddZdej dej fddZdej de de dedej f d d!Z "d)dej d ej deej d eej d#eeeej fdeeej ff d$d%Zd ej d&ejdej fd'd(ZZS)*EomtLossconfig weight_dictcstt|dg|j|_||_|j|_t|jd}|j|d<| d||j |_ |j |_ |j |_ t|j|j|j|j d|_dS)aH The Eomt Loss. The loss is computed very similar to DETR. The process happens in two steps: 1) we compute hungarian assignment between ground truth masks and the outputs of the model 2) we supervise each pair of matched ground-truth / prediction (supervise class and mask) Args: config (`EomtConfig`): The configuration for Eomt model also containing loss calculation specific parameters. weight_dict (`dict[str, float]`): A dictionary of weights to be applied to the different losses. scipyrr= empty_weight)rVrXrWrYN)rZr[r num_labelsrno_object_weighteos_coefr&onesregister_buffertrain_num_pointsrYoversample_ratioimportance_sample_ratiorT class_weight dice_weight mask_weightmatcher)r]rrrr^r+r,r[Ls"    zEomtLoss.__init__sizesr/cCsB|d}|ddD]}t|D] \}}t|||||<qq |S)Nrr) enumeratemax)r]rmaxessublistindexitemr+r+r, _max_by_axisos zEomtLoss._max_by_axistensorscCs|dd|D}t|g|}|\}}}}|dj}|dj} tj||| d} tj|||ftj| d} t|| | D].\} } }| d| j dd| j dd| j df | d|d| j dd| j df<q;| | fS)NcSsg|]}t|jqSr+)r*rI)rjrtr+r+r,rmysz8EomtLoss._pad_images_to_max_in_batch..rrgrcrr0F) rlenrgrcr&zerosrboolziprIcopy_)r]rmax_size batch_shaperz_heightwidthrgrcpadded_tensors padding_masksrt padded_tensor padding_maskr+r+r,_pad_images_to_max_in_batchws   2"z$EomtLoss._pad_images_to_max_in_batchrrarycCs|}|j\}}}tj|jd}||} tddt||D} tj||f|j tj |j d} | | | <| dd} || | } d| i}|S)aCompute the losses related to the labels using cross entropy. Args: class_queries_logits (`torch.Tensor`): A tensor of shape `batch_size, num_queries, num_labels` class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. Returns: `dict[str, Tensor]`: A dict of `torch.Tensor` containing the following key: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. )weightcSsg|] \}\}}||qSr+r+)rjtargetrrlr+r+r,rmsz(EomtLoss.loss_labels..) fill_valuergrcrr0loss_cross_entropy) rIrCrossEntropyLossr$_get_predictions_permutation_indicesr&catrfullrrirc transpose)r]rrary pred_logitsrz num_queriesrrNidxtarget_classes_otarget_classespred_logits_transposedloss_celossesr+r+r, loss_labelss    zEomtLoss.loss_labelsrr`rcs|}|}||}|\}} ||}|dddf}|dddf}t"|fddjjj} t || dd d} Wdn1sRwYt || dd d} t | | |t | | |d} ~~| S)aCompute the losses related to the masks using sigmoid_cross_entropy_loss and dice loss. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. num_masks (`int)`: The number of masks, used for normalization. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing two keys: - **loss_mask** -- The loss computed using sigmoid cross entropy loss on the predicted and ground truth. masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth, masks. Ncs |SN)calculate_uncertainty)logitsr]r+r,s z%EomtLoss.loss_masks..Frdr) loss_mask loss_dice) r _get_targets_permutation_indicesrr&rsample_points_using_uncertaintyrYrrr:r6rr)r]rr`ryrsrc_idxtgt_idx pred_masks target_masksrr. point_labels point_logitsrr+rr, loss_maskss0      zEomtLoss.loss_maskscC4tddt|D}tdd|D}||fS)NcSs g|] \}\}}t||qSr+r& full_like)rjrksrcrr+r+r,rm zAEomtLoss._get_predictions_permutation_indices..cSsg|]\}}|qSr+r+)rjrrr+r+r,rmr&rr)r]ry batch_indicespredictions_indicesr+r+r,rz-EomtLoss._get_predictions_permutation_indicescCr)NcSs g|] \}\}}t||qSr+r)rjrkrtgtr+r+r,rmrz=EomtLoss._get_targets_permutation_indices..cSsg|]\}}|qSr+r+)rjrrr+r+r,rmrr)r]ryrtarget_indicesr+r+r,rrz)EomtLoss._get_targets_permutation_indicesrcCst| }|S)a In Eomt paper, uncertainty is estimated as L1 distance between 0.0 and the logit prediction in 'logits' for the foreground class in `classes`. Args: logits (`torch.Tensor`): A tensor of shape (R, 1, ...) for class-specific or class-agnostic, where R is the total number of predicted masks in all images and C is: the number of foreground classes. The values are logits. Returns: scores (`torch.Tensor`): A tensor of shape (R, 1, ...) that contains uncertainty scores with the most uncertain locations having the highest uncertainty score. )r&abs)r]runcertainty_scoresr+r+r,rs zEomtLoss.calculate_uncertaintyrYrrcCs|jd}t||}tj||d|jd}t||dd} || } t||} || } tj| dddddf| ddd} |tj|tj|jd }| |dddf7} | d d| d ddf || d}| dkr|tj |tj|| d|jdgdd }|S) a This function is meant for sampling points in [0, 1] * [0, 1] coordinate space based on their uncertainty. The uncertainty is calculated for each point using the passed `uncertainty function` that takes points logit prediction as input. Args: logits (`float`): Logit predictions for P points. uncertainty_function: A function that takes logit predictions for P points and returns their uncertainties. num_points (`int`): The number of points P to sample. oversample_ratio (`int`): Oversampling parameter. importance_sample_ratio (`float`): Ratio of points that are sampled via importance sampling. Returns: point_coordinates (`torch.Tensor`): Coordinates for P sampled points. rr0rbFrdNr)kr2rr=r2) rIrr&rqrcr:topkarangelongviewr)r]runcertainty_functionrYrr num_boxesnum_points_sampledr.rpoint_uncertaintiesnum_uncertain_pointsnum_random_pointsrshiftr+r+r,rs"   &(z(EomtLoss.sample_points_using_uncertaintyNauxiliary_predictionsc s|||||}|j||djd}i|||||||||}|durPt|D]$\} | d}| d}|||||} fdd| D} || q+|S)a This performs the loss computation. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. class_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, num_labels)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. auxiliary_predictions (`dict[str, torch.Tensor]`, *optional*): if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], then it contains the logits from the inner layers of the EomtMaskedAttentionDecoder. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing three keys: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. - **loss_mask** -- The loss computed using sigmoid cross_entropy loss on the predicted and ground truth masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth masks. if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], the dictionary contains additional losses for each auxiliary predictions. rrbNrrcs i|] \}}|d|qS)rr+)rjkeyvaluerr+r, nrz$EomtLoss.forward..) r get_num_masksrcrrrritemsupdate) r]rrr`rarryrr aux_outputs loss_dictr+rr,r<s$  zEomtLoss.forwardrccCs^tdd|D}tj|tj|d}d}tr$tjikr$t|}tj}tj ||dd}|S)zk Computes the average number of target masks across the batch, for normalization purposes. css|]}t|VqdSr)r)rjclassesr+r+r, wsz)EomtLoss.get_num_masks..rr)min) rBr&rhrrr _shared_stater num_processesclamp)r]rarcr world_sizer+r+r,rss zEomtLoss.get_num_masksr)r"r#r$rdictstrrr[r*rrrr)rnparrayrr&rrrrrrrrcrrr+r+r^r,rKsj#  "  > =  $7rcs6eZdZdZfddZdejdejfddZZS)EomtPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr kernel_sizestride)rZr[ image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesrConv2d projection)r]rrrrrrr^r+r,r[s  zEomtPatchEmbeddings.__init__ pixel_valuesr/cCsH|jd}||jkrtd|jd|d||ddd}|S)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r0)rIrr\rr?r)r]rr embeddingsr+r+r,rs  zEomtPatchEmbeddings.forward) r"r#r$r%r[r&rrrr+r+r^r,r s r cs@eZdZdZdeddffdd Zdejdejfdd ZZ S) EomtEmbeddingszM Construct the CLS token, mask token, position and patch embeddings. rr/Ncst||_|j|_ttdd|j|_ tt d|j |j|_ t ||_|jj}t|j|_d|j |_t||j|_|jdt|ddddS)Nr position_ids)rr=F) persistent)rZr[rrr Parameterr&randnr cls_tokenrnum_register_tokensregister_tokensr patch_embeddingsrDropouthidden_dropout_probdropoutnum_prefix_tokens Embeddingposition_embeddingsrrexpand)r]rrr^r+r,r[s    zEomtEmbeddings.__init__rcCs~|j\}}}}|jjjj}||j|d}|j|dd}|j|dd}|| |j }t j |||gdd}| |}|S)Nrfr=rr)rIr$rrrgrpr!r+r#r*rr&rr')r]rrzr target_dtyper cls_tokensr#r+r+r,rs  zEomtEmbeddings.forward) r"r#r$r%rr[r&rrrr+r+r^r,rsrmodulequeryrrattention_maskscalingr'c Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr=)r2rg)ptrainingrr0) r&r@rrr4rofloat32rprgr'r5 contiguous) r/r0rrr1r2r'r8 attn_weights attn_outputr+r+r,eager_attention_forwards  r:c sReZdZdZfddZ d dejdeejdeejeejffdd Z Z S) EomtAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)rZr[rr embed_dimnum_attention_heads num_headshead_dimr\scaleattention_dropoutr' is_causalrLineark_projv_projq_projout_projr]rr^r+r,r[s$   zEomtAttention.__init__Nrr1r/c Ks|j\}}}||}||}||} ||||j|jdd}||||j|jdd}| |||j|jdd} t} |j j dkrMt |j j } | |||| ||j |j |js\dn|jd\} } | |||} || } | | fS)z#Input shape: Batch x Time x Channelrr0eagerr.)rBr2r')rIrFrDrErr>r?rr:r_attn_implementationr rBr@r5r'reshaper7rG) r]rr1r8rz seq_lengthr<querieskeysvaluesattention_interfacer9r8r+r+r,rs.        zEomtAttention.forwardr) r"r#r$r%r[r&rrr)rrr+r+r^r,r;s r;c4eZdZdfdd ZdejdejfddZZS) EomtLayerScaler/Ncs(tt|jt|j|_dSr) rZr[rrlayerscale_valuer&rrlambda1rHr^r+r,r[s zEomtLayerScale.__init__ hidden_statecCs ||jSr)rTr]rUr+r+r,r#s zEomtLayerScale.forwardr/Nr"r#r$r[r&rrrr+r+r^r,rRsrRinput drop_probr5cCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. r.rr)rr)rIndimr&rqrgrcfloor_div)rYrZr5 keep_probrI random_tensoroutputr+r+r, drop_path's racsTeZdZdZd deeddffdd Zdejdejfdd Z de fd d Z Z S) EomtDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).NrZr/cst||_dSr)rZr[rZ)r]rZr^r+r,r[>s  zEomtDropPath.__init__rcCst||j|jSr)rarZr5r]rr+r+r,rBszEomtDropPath.forwardcCs d|jS)Nzp=)rZrr+r+r, extra_reprE zEomtDropPath.extra_reprr) r"r#r$r%rrr[r&rrrrdrr+r+r^r,rb;s rbcrQ) EomtMLPr/Ncsnt|j}}t|j|j}tj||dd|_t|j t r(t |j |_ n|j |_ tj||dd|_ dS)NTbias)rZr[rr mlp_ratiorrCfc1r hidden_actrr activationfc2r]r in_features out_featureshidden_featuresr^r+r,r[Js   zEomtMLP.__init__rUcCs"||}||}||}|Sr)rjrlrmrVr+r+r,rUs   zEomtMLP.forwardrWrXr+r+r^r,rfIs rfcrQ) EomtSwiGLUFFNr/Ncslt|j}}t|j|j}t|ddddd}tj|d|dd|_tj||dd|_dS)Nr0rTrg) rZr[rrrirrC weights_in weights_outrnr^r+r,r[]s  zEomtSwiGLUFFN.__init__rUcCs6||}|jddd\}}tj||}||S)Nr0r=r)ruchunkrr4silurv)r]rUx1x2hiddenr+r+r,rfs  zEomtSwiGLUFFN.forwardrWrXr+r+r^r,rr\s rrcsNeZdZdZdeddffdd Z d dejdeejdejfd d Z Z S) EomtLayerzCThis corresponds to the Block class in the original implementation.rr/Ncsttj|j|jd|_t||_t ||_ |j dkr#t |j nt |_tj|j|jd|_|jr;t||_nt||_t ||_dS)Nepsr.)rZr[r LayerNormrlayer_norm_epsnorm1r; attentionrR layer_scale1drop_path_raterbIdentityranorm2use_swiglu_ffnrrmlprf layer_scale2rHr^r+r,r[ps     zEomtLayer.__init__r head_maskcCsb||}|||\}}||}|||}||}||}||}|||}|Sr)rrrrarrr)r]rrhidden_states_normself_attention_outputr layer_outputr+r+r,rs     zEomtLayer.forwardr) r"r#r$r%rr[r&rrrrr+r+r^r,r|msr|cs4eZdZd fdd ZdejdejfddZZS) EomtLayerNorm2dư>Tcstj|||ddS)N)r~elementwise_affine)rZr[)r]rr~affiner^r+r,r[szEomtLayerNorm2d.__init__rUr/cCs>|dddd}t||j|j|j|j}|dddd}|S)Nrr0rr)permuteF layer_normnormalized_shaperrhr~rVr+r+r,rszEomtLayerNorm2d.forward)rTrXr+r+r^r,rsrc8eZdZdeffdd ZdejdejfddZZS)EomtScaleLayerrcsVt|j}tj||ddd|_t|j|_tj ||dd|dd|_ t ||_ dS)Nr0r rrF)r paddinggroupsrh) rZr[rrConvTranspose2dconv1rrkrlrconv2r layernorm2dr]rrr^r+r,r[s   zEomtScaleLayer.__init__rr/cCs,||}||}||}||}|Sr)rrlrrrcr+r+r,rs    zEomtScaleLayer.forward r"r#r$rr[r&rrrr+r+r^r,rsrcr)EomtScaleBlockrcs6tj|_tfddt|jD|_dS)Ncg|]}tqSr+)rrjrrr+r,rmrz+EomtScaleBlock.__init__..)rZr[num_upscale_blocks num_blocksr ModuleListrnblockrHr^rr,r[s $zEomtScaleBlock.__init__rr/cCs|jD]}||}q|Sr)r)r]rrr+r+r,rs  zEomtScaleBlock.forwardrr+r+r^r,rsrcr) EomtMaskHeadrcsJt|j}t|||_t|||_t|||_t|j |_ dSr) rZr[rrrCrjrmfc3rrkrlrr^r+r,r[s zEomtMaskHead.__init__rr/cCs.|||}|||}||}|Sr)rlrjrmrrcr+r+r,rs zEomtMaskHead.forwardrr+r+r^r,rs rc@sNeZdZUdZeed<dZdZdZdgZ dZ e e dZ d ejd d fd d Zd S)EomtPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. reomtrFr|T)rr r/r/NcCsl|jj}t|tjtjtjfrDtjj|j t dd|j durBtj |j \}}|dkr4dt |nd}tj|j | |dSdSt|tjrY|j jd|j jdSt|tjrz|j jjddd|jdurx|j j|jdSdSt|trt|dr|jj|jjdSdSt|trtjj|jjtjd|d|jj |j_|j!jdSdS) N)arrr1r.)rstdrT)"rinitializer_rangerrrCrrinitkaiming_uniform_rmathsqrtrh_calculate_fan_in_and_fan_outuniform_rdatafill_zero_r)normal_ padding_idxrRhasattrrTrSr trunc_normal_r!rpr&r6rgr#)r]r/rfan_inrboundr+r+r, _init_weightss8        z!EomtPreTrainedModel._init_weights)r"r#r$r%rr(base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpar|r;_can_record_outputsrModulerr+r+r+r,rs rzV The EoMT Model with head on top for instance/semantic/panoptic segmentation. cseZdZdZdeffdd Zdedededed eeefd eeeff d d Z d eeefd efddZ e e   ddede eede eede eedeed ef ddZddZdejfddZeddZZS)EomtForUniversalSegmentationrrcst|_j|_t|_tjjj d|_ t j j|_ tfddtjD|_t|_t|_tjjd|_jjjjf|_jjjd|_t|jd|_ |!dt"#j$|%dS)Nr}crr+)r|rrr+r,rmrz9EomtForUniversalSegmentation.__init__..r)rrr)rrattn_mask_probs)&rZr[rnum_hidden_layersrrrrrr layernormr)rr0rrnlayersr upscale_blockr mask_headrCrclass_predictorrr grid_sizerrrrrrNrr&rr post_initrHr^rr,r[ s$      z%EomtForUniversalSegmentation.__init__rrr`rarr/c CsN|j|||||d}|jD]\}}|D] \} } || vr#| |9} qq|S)Nrrr`rar)rNrr) r]rrr`rarrrrloss_keyrr+r+r, get_loss_dict's z*EomtForUniversalSegmentation.get_loss_dictrcCs t|Sr)rBrO)r]rr+r+r,get_loss?rez%EomtForUniversalSegmentation.get_lossNr!r8cKsrd\}}d}|durtd||} t|jD]\} } | |j|jjkrG|jjdddddf | j ddd | j } t j| | fdd} | |j|jjkr|jsa|j| |j|jjdkr|| } || \}}||f7}||f7}t j| j d| j d| j d| j t jd}tj||jd d }||d|dd}|jj}||jj}|dk|ddd||df<|j||j| |j|jj|||j d }|dddd f d|jjdd}||d }| | |} q|| }||\}}||f7}||f7}d}|dur0|dur0d}t ||D]\}}|j!||||dd}||"|7}qt#|||||dS)ag mask_labels (`list[torch.Tensor]`, *optional*): list of mask labels of shape `(num_labels, height, width)` to be fed to a model class_labels (`list[torch.LongTensor]`, *optional*): list of target class labels of shape `(num_labels, height, width)` to be fed to a model. They identify the labels of `mask_labels`, e.g. the label of `mask_labels[i][j]` if `class_labels[i][j]`. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segmentation. )r+r+Nz You have to specify pixel_valuesrr=rr)rcrgbilinear)sizemode)probnum_query_tokensencoder_start_tokensrc.ger.r)rrrrr!)$r\rrrrrrr0rr+rIrprcr&rr5rrpredictrrr interpolaterrrrr(_disable_attention_maskr=r masked_fillrrrr)r]rr`rar!r8masks_queries_logits_per_layerclass_queries_logits_per_layerr1rr layer_moduler0norm_hidden_statesrrinterpolated_logitsrrsequence_outputrrr+r+r,rBs 2    "      z$EomtForUniversalSegmentation.forwardcCs|jjSr)rr$rr+r+r,get_input_embeddingssz1EomtForUniversalSegmentation.get_input_embeddingsrcCs|ddd|jjddf}||}|dd|jj|jjdddf}|dd}|j|jddg|jR}| |}| |}t d||}||fS)Nrr0rr=zbqc, bchw -> bqhw) rrrrr(rrKrIrrrr&einsum)r]r query_tokens class_logits prefix_tokens mask_logitsr+r+r,rs &   z$EomtForUniversalSegmentation.predictcCsD|dkr tj|jd||d|k}d|ddd||df|<|S)Nrrrb)r&rqrI) attn_maskrrrrcrandom_queriesr+r+r,rsz4EomtForUniversalSegmentation._disable_attention_mask)NNN)r"r#r$rrr[rrrrrrrrr*rrrrrr&r staticmethodrrr+r+r^r,rsN      gr)F)r.)r.F)Icollections.abcrr dataclassesrtypingrrnumpyrr&torch.nn.functionalrr4rr activationsr file_utilsr r r modeling_layersr modeling_utilsr rprocessing_utilsrutilsrrr utils.genericrconfiguration_eomtrscipy.optimizer accelerateraccelerate.utilsrrr:rErSrrTrrrrr rrr:r;rRrrarbrfrrr|rrrrrrr__all__r+r+r+r,s          !  j :!, > *  * C