o ei@sLddlZddlZddlmZddlmZddlZddlZddl m m Z ddlm Z m Z ddlmZddlmZddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'er|ddl(m)Z)e!rddl*m+Z+ddl,m-Z-ee ddGdddeZ. dYdej dej dej fddZ/de de de fd d!Z0dej dej dej fd"d#Z1Gd$d%d%e j2Z3de de d&e4de fd'd(Z5dej dej d&e4dej fd)d*Z6Gd+d,d,e j2Z7Gd-d.d.e j2Z8Gd/d0d0e j2Z9 1dZd2e j2d3ej d4ej d5ej d6ej dBd7e:d8e:fd9d:Z;Gd;d<dd>e j2Z=d[d?ej d@e:dAe>dej fdBdCZ?GdDdEdEe j2Z@GdFdGdGe j2ZAGdHdIdIe j2ZBGdJdKdKeZCGdLdMdMe jDZEGdNdOdOe j2ZFGdPdQdQe j2ZGGdRdSdSe j2ZHe GdTdUdUeZIe dVdGdWdXdXeIZJdUdXgZKdS)\N)Callable) dataclass)Tensornn)initialization)ACT2FN) ModelOutputis_scipy_availablerequires_backends)GradientCheckpointingLayer)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringis_accelerate_available)merge_with_config_defaults)capture_outputs) EomtConfig)linear_sum_assignment) PartialState)reducea Class for outputs of [`EomtForUniversalSegmentationOutput`]. This output can be directly passed to [`~EomtImageProcessor.post_process_semantic_segmentation`] or [`~EomtImageProcessor.post_process_instance_segmentation`] or [`~EomtImageProcessor.post_process_panoptic_segmentation`] to compute final segmentation maps. Please, see [`~EomtImageProcessor] for details regarding usage. ) custom_introc@seZdZUdZdZejdBed<dZejdBed<dZ ejdBed<dZ ejdBed<dZ e ejdBed<dZ e ejdBed<dZeejdBed <dS) "EomtForUniversalSegmentationOutputa* loss (`torch.Tensor`, *optional*): The computed loss, returned when labels are present. class_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, num_labels + 1)` representing the proposed classes for each query. Note the `+ 1` is needed because we incorporate the null class. masks_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, height, width)` representing the proposed masks for each query. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Last hidden states (final feature map) of the last layer. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states all layers of the model. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tuple(torch.FloatTensor)` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Self and Cross Attentions weights from transformer decoder. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segmentation. Nlossclass_queries_logitsmasks_queries_logitslast_hidden_state hidden_states attentions patch_offsets)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrr tupler!r"listrr,r,d/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/eomt/modeling_eomt.pyr3s  rFinput_featurespoint_coordinatesreturncKsL|dkr d}|d}tjjj|d|dfi|}|r$|d}|S)a( A wrapper around `torch.nn.functional.grid_sample` to support 3D point_coordinates tensors. Args: input_features (`torch.Tensor` of shape (batch_size, channels, height, width)): A tensor that contains features map on a height * width grid point_coordinates (`torch.Tensor` of shape (batch_size, num_points, 2) or (batch_size, grid_height, grid_width,: 2)): A tensor that contains [0, 1] * [0, 1] normalized point coordinates add_dim (`bool`): boolean value to keep track of added dimension Returns: point_features (`torch.Tensor` of shape (batch_size, channels, num_points) or (batch_size, channels, height_grid, width_grid): A tensor that contains features for points in `point_coordinates`. rTg@?)dim unsqueezer'r functional grid_samplesqueeze)r.r/add_dimkwargspoint_featuresr,r,r- sample_point^s    r;inputslabelscCsd|d}dt||j}|ddddf|ddddf}d|d|d}|S)a A pair wise version of the dice loss, see `dice_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: `torch.Tensor`: The computed loss between each pairs. rr1N)sigmoidflattenr'matmulTsum)r<r= numerator denominatorrr,r,r-pair_wise_dice_loss~s ,rFc Csj|jd}tjdd}||t|}||t|}t|||j}t||d|j}||}|S)a A pair wise version of the cross entropy loss, see `sigmoid_cross_entropy_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss between each pairs. rnone reduction)shaperBCEWithLogitsLossr' ones_like zeros_likerArB) r<r=height_and_width criterioncross_entropy_loss_poscross_entropy_loss_negloss_posloss_negrr,r,r-$pair_wise_sigmoid_cross_entropy_losss  rTc sleZdZdZ ddedededeffdd Zed ej d ej d ej d ej de e e f ddZ Z S)EomtHungarianMatcheraqThis class computes an assignment between the labels and the predictions of the network. For efficiency reasons, the labels don't include the no_object. Because of this, in general, there are more predictions than labels. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). r21 cost_class cost_mask cost_dice num_pointscsFt|dkr|dkr|dkrtd||_||_||_||_dS)aHCreates the matcher Params: cost_class (`float`, *optional*, defaults to 1.0): Relative weight of the classification error in the matching cost. cost_mask (`float`, *optional*, defaults to 1.0): This is the relative weight of the focal loss of the binary mask in the matching cost. cost_dice (`float`, *optional*, defaults to 1.0): This is the relative weight of the dice loss of the binary mask in the matching cost. num_points (`int`, *optional*, defaults to 12544): No. of points to sample on which the mask loss will be calculated. The same set of K points are uniformly sampled for all prediction and ground truth masks to construct the cost matrix for bipartite matching. rzAll costs can't be 0N)super__init__ ValueErrorrZrWrXrY)selfrWrXrYrZ __class__r,r-r\s  zEomtHungarianMatcher.__init__rr mask_labels class_labelsr0cCsfg}|jd}t|D]}||d}||} |dd||f } ||| } | dddf} | dddf} tjd|jd| jd} | | jddd} t | | dd d} | | jddd}t | |dd d} t | | }t | | }|j ||j| |j|}t|td }t|td }t|d}t|}||q d d |D}|S) ao Params: masks_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, num_labels` with the classification logits. class_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, height, width` with the predicted masks. class_labels (`torch.Tensor`): A tensor of dim `num_target_boxes` (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels. mask_labels (`torch.Tensor`): A tensor of dim `num_target_boxes, height, width` containing the target masks. Returns: matched_indices (`list[tuple[Tensor]]`): A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected labels (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes). rr>Nrr1deviceF align_cornersg _Bg _cSs0g|]\}}tj|tjdtj|tjdfqS)dtype)r' as_tensorint64).0ijr,r,r- s$z0EomtHungarianMatcher.forward..)rJrangesoftmaxtor'randrZrdrepeatr;r7rTrFrXrWrYminimumtensormaximum nan_to_numrcpuappend)r^rrrarbindices batch_sizerl pred_probs pred_maskrW target_maskr/target_coordinatespred_coordinatesrXrY cost_matrixassigned_indicesmatched_indicesr,r,r-forwards4       zEomtHungarianMatcher.forward)r2r2r2rV)r#r$r%r&floatintr\r'no_gradrr+r*r __classcell__r,r,r_r-rUs0 rU num_maskscCsX|d}d||d}|d|d}d|d|d}||}|S)a4 Compute the DICE loss, similar to generalized IOU for masks as follows: $$ \mathcal{L}_{\text{dice}(x, y) = 1 - \frac{2 * x \cap y }{x \cup y + 1}} $$ In practice, since `labels` is a binary mask, (only 0s and 1s), dice can be computed as follow $$ \mathcal{L}_{\text{dice}(x, y) = 1 - \frac{2 * x * y }{x + y + 1}} $$ Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). num_masks (`int`): The number of masks present in the current batch, used for normalization. Returns: `torch.Tensor`: The computed loss. rr1r>)r?r@rC)r<r=rprobsrDrErr,r,r- dice_losss  rcCs,tjdd}|||}|d|}|S)a| Args: inputs (`torch.Tensor`): A float tensor of arbitrary shape. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss. rGrHr)rrKmeanrC)r<r=rrOcross_entropy_lossrr,r,r-sigmoid_cross_entropy_loss8s rcs~eZdZdedeeefffdd Zdeee dee fddZ d ee de e e ffd d Z d e d ee de ejdeee ffddZdej deej de ejde deeej ff ddZddZddZdej dej fddZdej de de dedej f d d!Z "d)dej d ej deej d eej d#eeej fd"Bdeeej ff d$d%Zd ej d&ejdej fd'd(ZZS)*EomtLossconfig weight_dictcstt|dg|j|_||_|j|_t|jd}|j|d<| d||j |_ |j |_ |j |_ t|j|j|j|j d|_dS)aH The Eomt Loss. The loss is computed very similar to DETR. The process happens in two steps: 1) we compute hungarian assignment between ground truth masks and the outputs of the model 2) we supervise each pair of matched ground-truth / prediction (supervise class and mask) Args: config (`EomtConfig`): The configuration for Eomt model also containing loss calculation specific parameters. weight_dict (`dict[str, float]`): A dictionary of weights to be applied to the different losses. scipyrr> empty_weight)rWrYrXrZN)r[r\r num_labelsrno_object_weighteos_coefr'onesregister_buffertrain_num_pointsrZoversample_ratioimportance_sample_ratiorU class_weight dice_weight mask_weightmatcher)r^rrrr_r,r-r\Ms"    zEomtLoss.__init__sizesr0cCsB|d}|ddD]}t|D] \}}t|||||<qq |S)Nrr) enumeratemax)r^rmaxessublistindexitemr,r,r- _max_by_axisps zEomtLoss._max_by_axistensorscCs|dd|D}t|g|}|\}}}}|dj}|dj} tj||| d} tj|||ftj| d} t|| | D].\} } }| d| j dd| j dd| j df | d|d| j dd| j df<q;| | fS)NcSsg|]}t|jqSr,)r+rJ)rkrur,r,r-rnzsz8EomtLoss._pad_images_to_max_in_batch..rrhrdrr1F) rlenrhrdr'zerosrboolziprJcopy_)r^rmax_size batch_shaper{_heightwidthrhrdpadded_tensors padding_masksru padded_tensor padding_maskr,r,r-_pad_images_to_max_in_batchxs   2"z$EomtLoss._pad_images_to_max_in_batchrrbrzcCs|}|j\}}}tj|jd}||} tddt||D} tj||f|j tj |j d} | | | <| dd} || | } d| i}|S)aCompute the losses related to the labels using cross entropy. Args: class_queries_logits (`torch.Tensor`): A tensor of shape `batch_size, num_queries, num_labels` class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. Returns: `dict[str, Tensor]`: A dict of `torch.Tensor` containing the following key: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. )weightcSsg|] \}\}}||qSr,r,)rktargetrrmr,r,r-rnsz(EomtLoss.loss_labels..) fill_valuerhrdrr1loss_cross_entropy) rJrCrossEntropyLossr$_get_predictions_permutation_indicesr'catrfullrrjrd transpose)r^rrbrz pred_logitsr{ num_queriesrrOidxtarget_classes_otarget_classespred_logits_transposedloss_celossesr,r,r- loss_labelss    zEomtLoss.loss_labelsrrarcs|}|}||}|\}} ||}|dddf}|dddf}t"|fddjjj} t || dd d} Wdn1sRwYt || dd d} t | | |t | | |d} ~~| S)aCompute the losses related to the masks using sigmoid_cross_entropy_loss and dice loss. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. num_masks (`int)`: The number of masks, used for normalization. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing two keys: - **loss_mask** -- The loss computed using sigmoid cross entropy loss on the predicted and ground truth. masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth, masks. Ncs |SN)calculate_uncertainty)logitsr^r,r-s z%EomtLoss.loss_masks..Frer) loss_mask loss_dice) r _get_targets_permutation_indicesrr'rsample_points_using_uncertaintyrZrrr;r7rr)r^rrarzrsrc_idxtgt_idx pred_masks target_masksrr/ point_labels point_logitsrr,rr- loss_maskss0      zEomtLoss.loss_maskscC4tddt|D}tdd|D}||fS)NcSs g|] \}\}}t||qSr,r' full_like)rkrlsrcrr,r,r-rn zAEomtLoss._get_predictions_permutation_indices..cSsg|]\}}|qSr,r,)rkrrr,r,r-rnr'rr)r^rz batch_indicespredictions_indicesr,r,r-rz-EomtLoss._get_predictions_permutation_indicescCr)NcSs g|] \}\}}t||qSr,r)rkrlrtgtr,r,r-rnrz=EomtLoss._get_targets_permutation_indices..cSsg|]\}}|qSr,r,)rkrrr,r,r-rnrr)r^rzrtarget_indicesr,r,r-rrz)EomtLoss._get_targets_permutation_indicesrcCst| }|S)a In Eomt paper, uncertainty is estimated as L1 distance between 0.0 and the logit prediction in 'logits' for the foreground class in `classes`. Args: logits (`torch.Tensor`): A tensor of shape (R, 1, ...) for class-specific or class-agnostic, where R is the total number of predicted masks in all images and C is: the number of foreground classes. The values are logits. Returns: scores (`torch.Tensor`): A tensor of shape (R, 1, ...) that contains uncertainty scores with the most uncertain locations having the highest uncertainty score. )r'abs)r^runcertainty_scoresr,r,r-rs zEomtLoss.calculate_uncertaintyrZrrcCs|jd}t||}tj||d|jd}t||dd} || } t||} || } tj| dddddf| ddd} |tj|tj|jd }| |dddf7} | d d| d ddf || d}| dkr|tj |tj|| d|jdgdd }|S) a This function is meant for sampling points in [0, 1] * [0, 1] coordinate space based on their uncertainty. The uncertainty is calculated for each point using the passed `uncertainty function` that takes points logit prediction as input. Args: logits (`float`): Logit predictions for P points. uncertainty_function: A function that takes logit predictions for P points and returns their uncertainties. num_points (`int`): The number of points P to sample. oversample_ratio (`int`): Oversampling parameter. importance_sample_ratio (`float`): Ratio of points that are sampled via importance sampling. Returns: point_coordinates (`torch.Tensor`): Coordinates for P sampled points. rr1rcFreNr)kr3rr>r3) rJrr'rrrdr;topkarangelongviewr)r^runcertainty_functionrZrr num_boxesnum_points_sampledr/rpoint_uncertaintiesnum_uncertain_pointsnum_random_pointsrshiftr,r,r-rs"   &(z(EomtLoss.sample_points_using_uncertaintyNauxiliary_predictionsc s|||||}|j||djd}i|||||||||}|durPt|D]$\} | d}| d}|||||} fdd| D} || q+|S)a This performs the loss computation. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. class_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, num_labels)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. auxiliary_predictions (`dict[str, torch.Tensor]`, *optional*): if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], then it contains the logits from the inner layers of the EomtMaskedAttentionDecoder. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing three keys: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. - **loss_mask** -- The loss computed using sigmoid cross_entropy loss on the predicted and ground truth masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth masks. if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], the dictionary contains additional losses for each auxiliary predictions. rrcNrrcs i|] \}}|d|qS)rr,)rkkeyvaluerr,r- orz$EomtLoss.forward..) r get_num_masksrdrrrritemsupdate) r^rrrarbrrzrr aux_outputs loss_dictr,rr-r=s$  zEomtLoss.forwardrdcCs^tdd|D}tj|tj|d}d}tr$tjikr$t|}tj}tj ||dd}|S)zk Computes the average number of target masks across the batch, for normalization purposes. css|]}t|VqdSr)r)rkclassesr,r,r- xsz)EomtLoss.get_num_masks..rr)min) rCr'rirrr _shared_stater num_processesclamp)r^rbrdr world_sizer,r,r-rts zEomtLoss.get_num_masksr)r#r$r%rdictstrrr\r+rrrr*rnparrayrr'rrrrrrrdrrr,r,r_r-rLsj#  "  > =  $7rcs6eZdZdZfddZdejdejfddZZS)EomtPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr kernel_sizestride)r[r\ image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesrConv2d projection)r^rrrrrrr_r,r-r\s  zEomtPatchEmbeddings.__init__ pixel_valuesr0cCsH|jd}||jkrtd|jd|d||ddd}|S)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r1)rJrr]rr@r)r^rr embeddingsr,r,r-rs  zEomtPatchEmbeddings.forward) r#r$r%r&r\r'rrrr,r,r_r-r s r cs@eZdZdZdeddffdd Zdejdejfdd ZZ S) EomtEmbeddingszM Construct the CLS token, mask token, position and patch embeddings. rr0Ncst||_|j|_ttdd|j|_ tt d|j |j|_ t ||_|jj}t|j|_d|j |_t||j|_|jdt|ddddS)Nr position_idsrr>F) persistent)r[r\rrr Parameterr'randnr cls_tokenrnum_register_tokensregister_tokensr patch_embeddingsrDropouthidden_dropout_probdropoutnum_prefix_tokens Embeddingposition_embeddingsrrexpand)r^rrr_r,r-r\s    zEomtEmbeddings.__init__rcCs~|j\}}}}|jjjj}||j|d}|j|dd}|j|dd}|| |j }t j |||gdd}| |}|S)Nrgr>rr)rJr&rrrhrqr#r-r%r,rr'rr))r^rr{r target_dtyper cls_tokensr%r,r,r-rs  zEomtEmbeddings.forward r#r$r%r&rr\r'rrrr,r,r_r-rsrmodulequeryrrattention_maskscalingr)c Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr>)r3rh)ptrainingrr1) r'rArrr5rpfloat32rqrhr)r8 contiguous) r2r3rrr4r5r)r9 attn_weights attn_outputr,r,r-eager_attention_forwards  r=c sReZdZdZfddZ d dejdejdBdeejejdBffdd ZZ S) EomtAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)r[r\rr embed_dimnum_attention_heads num_headshead_dimr]scaleattention_dropoutr) is_causalrLineark_projv_projq_projout_projr^rr_r,r-r\s$   zEomtAttention.__init__Nr r4r0c Ks|j\}}}||}||}||} ||||j|jdd}||||j|jdd}| |||j|jdd} t |j j t } | |||| ||j |j|jsVdn|jd\} } | |||} || } | | fS)z#Input shape: Batch x Time x Channelrr1r1)rEr5r))rJrIrGrHrrArBrr get_interfacer_attn_implementationr=rErCr8r)reshaper:rJ) r^r r4r9r{ seq_lengthr?querieskeysvaluesattention_interfacer<r;r,r,r-rs.      zEomtAttention.forwardr) r#r$r%r&r\r'rr*rrr,r,r_r-r>s r>c4eZdZdfdd ZdejdejfddZZS) EomtLayerScaler0Ncs(tt|jt|j|_dSr) r[r\rr!layerscale_valuer'rrlambda1rKr_r,r-r\ s zEomtLayerScale.__init__ hidden_statecCs ||jSr)rWr^rXr,r,r-r$s zEomtLayerScale.forwardr0Nr#r$r%r\r'rrrr,r,r_r-rUsrUinput drop_probr8cCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)zc Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). r1rr)rr)rJndimr'rrrhrdfloor_div)r\r]r8 keep_probrJ random_tensoroutputr,r,r- drop_path(s rdcsTeZdZdZd dedBddffdd Zdejdejfdd Zde fd d Z Z S) EomtDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nr]r0cst||_dSr)r[r\r])r^r]r_r,r-r\:s  zEomtDropPath.__init__r cCst||j|jSr)rdr]r8r^r r,r,r-r>szEomtDropPath.forwardcCs d|jS)Nzp=)r]rr,r,r- extra_reprA zEomtDropPath.extra_reprr) r#r$r%r&rr\r'rrrrgrr,r,r_r-re7s recrT) EomtMLPr0Ncsnt|j}}t|j|j}tj||dd|_t|j t r(t |j |_ n|j |_ tj||dd|_ dS)NTbias)r[r\rr mlp_ratiorrFfc1r hidden_actrr activationfc2r^r in_features out_featureshidden_featuresr_r,r-r\Fs   zEomtMLP.__init__rXcCs"||}||}||}|Sr)rmrorprYr,r,r-rQs   zEomtMLP.forwardrZr[r,r,r_r-riEs ricrT) EomtSwiGLUFFNr0Ncslt|j}}t|j|j}t|ddddd}tj|d|dd|_tj||dd|_dS)Nr1rTrj) r[r\rrrlrrF weights_in weights_outrqr_r,r-r\Ys  zEomtSwiGLUFFN.__init__rXcCs6||}|jddd\}}tj||}||S)Nr1r>r)rxchunkrr5silury)r^rXx1x2hiddenr,r,r-rbs  zEomtSwiGLUFFN.forwardrZr[r,r,r_r-ruXs rucsNeZdZdZdeddffdd Z d dejdejdBdejfd d ZZ S) EomtLayerzCThis corresponds to the Block class in the original implementation.rr0Ncsttj|j|jd|_t||_t ||_ |j dkr#t |j nt |_tj|j|jd|_|jr;t||_nt||_t ||_dS)Nepsr1)r[r\r LayerNormrlayer_norm_epsnorm1r> attentionrU layer_scale1drop_path_ratereIdentityrdnorm2use_swiglu_ffnrumlpri layer_scale2rKr_r,r-r\ls     zEomtLayer.__init__r r4cCsb||}|||\}}||}|||}||}||}||}|||}|Sr)rrrrdrrr)r^r r4hidden_states_normself_attention_outputr layer_outputr,r,r-r|s     zEomtLayer.forwardrr0r,r,r_r-risrcs4eZdZd fdd ZdejdejfddZZS) EomtLayerNorm2dư>Tcstj|||ddS)N)relementwise_affine)r[r\)r^rraffiner_r,r-r\szEomtLayerNorm2d.__init__rXr0cCs>|dddd}t||j|j|j|j}|dddd}|S)Nrr1rr)permuteF layer_normnormalized_shaperrkrrYr,r,r-rszEomtLayerNorm2d.forward)rTr[r,r,r_r-rsrc8eZdZdeffdd ZdejdejfddZZS)EomtScaleLayerrcsVt|j}tj||ddd|_t|j|_tj ||dd|dd|_ t ||_ dS)Nr1r rrF)r paddinggroupsrk) r[r\rrConvTranspose2dconv1rrnrorconv2r layernorm2dr^rrr_r,r-r\s   zEomtScaleLayer.__init__r r0cCs,||}||}||}||}|Sr)rrorrrfr,r,r-rs    zEomtScaleLayer.forward r#r$r%rr\r'rrrr,r,r_r-rsrcr)EomtScaleBlockrcs6tj|_tfddt|jD|_dS)Ncg|]}tqSr,)rrkrrr,r-rnrz+EomtScaleBlock.__init__..)r[r\num_upscale_blocks num_blocksr ModuleListroblockrKr_rr-r\s $zEomtScaleBlock.__init__r r0cCs|jD]}||}q|Sr)r)r^r rr,r,r-rs  zEomtScaleBlock.forwardrr,r,r_r-rsrcr) EomtMaskHeadrcsJt|j}t|||_t|||_t|||_t|j |_ dSr) r[r\rrrFrmrpfc3rrnrorr_r,r-r\s zEomtMaskHead.__init__r r0cCs.|||}|||}||}|Sr)rormrprrfr,r,r-rs zEomtMaskHead.forwardrr,r,r_r-rs rc@sZeZdZUdZeed<dZdZdZdZ dgZ dZ e e d Zed ejd d fd dZd S)EomtPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. reomtr)imageFrT)r r!r2r0NcCs|jj}t|tjtjtjfrCtj|j t dd|j durAt jj|j \}}|dkr4dt |nd}t|j | |dSdSt|tjrWt|j t|j dSt|tjrtj|j ddd|jdur}t|j ddst|j |jdSdSdSt|trt|d rt|j|jjdSdSt|trtj|jd|dt|jt |j!t "|j!j#d $d dSt|t%rt &|j'd}|j(|d <t |j)|dSt|t*rt|j+dSdS) N)arrr1)rstd_is_hf_initializedFrWr>r),rinitializer_rangerrrFrrinitkaiming_uniform_rmathsqrtrkr'_calculate_fan_in_and_fan_outuniform_rones_zeros_r+normal_ padding_idxgetattrrUhasattr constant_rWrVr trunc_normal_r#r%rrrrJr-rrrrrEomtForUniversalSegmentationattn_mask_probs)r^r2rfan_inrboundrr,r,r- _init_weightss>        &   z!EomtPreTrainedModel._init_weights)r#r$r%r&rr)base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdparr>_can_record_outputsr'rrModulerr,r,r,r-rs rzV The EoMT Model with head on top for instance/semantic/panoptic segmentation. cseZdZdZdeffdd Zdedededed eeefd eeeff d d Z d eeefd efddZ e e e   ddedeedBdeedBdeedBdeed ef ddZddZdejfddZeddZZS)rrrcst|_j|_t|_tjjj d|_ t j j|_ tfddtjD|_t|_t|_tjjd|_jjjjf|_jjjd|_t|jd|_ |!dt"#j$|%dS)Nrcrr,)rrrr,r-rnrz9EomtForUniversalSegmentation.__init__..r)rrr)rrr)&r[r\rnum_hidden_layersrrrrrr layernormr+rr3rrolayersr upscale_blockr mask_headrFrclass_predictorrr grid_sizerrrrrrOrr'rr post_initrKr_rr-r\s$      z%EomtForUniversalSegmentation.__init__rrrarbrr0c CsN|j|||||d}|jD]\}}|D] \} } || vr#| |9} qq|S)Nrrrarbr)rOrr) r^rrrarbrrrrloss_keyrr,r,r- get_loss_dict+s z*EomtForUniversalSegmentation.get_loss_dictrcCs t|Sr)rCrR)r^rr,r,r-get_lossCrhz%EomtForUniversalSegmentation.get_lossNr"r9cKsrd\}}d}|durtd||} t|jD]\} } | |j|jjkrG|jjdddddf | j ddd | j } t j| | fdd} | |j|jjkr|jsa|j| |j|jjdkr|| } || \}}||f7}||f7}t j| j d| j d| j d| j t jd}tj||jd d }||d|dd}|jj}||jj}|dk|ddd||df<|j||j| |j|jj|||j d }|dddd f d|jjdd}||d }| | |} q|| }||\}}||f7}||f7}d}|dur0|dur0d}t ||D]\}}|j!||||dd}||"|7}qt#|||||dS)ag mask_labels (`list[torch.Tensor]`, *optional*): list of mask labels of shape `(num_labels, height, width)` to be fed to a model class_labels (`list[torch.LongTensor]`, *optional*): list of target class labels of shape `(num_labels, height, width)` to be fed to a model. They identify the labels of `mask_labels`, e.g. the label of `mask_labels[i][j]` if `class_labels[i][j]`. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segmentation. )r,r,Nz You have to specify pixel_valuesrr>rr)rdrhbilinear)sizemode)probnum_query_tokensencoder_start_tokensrd.ger1r)rrrrr")$r]rrrrrrr3rr-rJrqrdr'rr8rrpredictrrr interpolaterrrrr*_disable_attention_maskr@r masked_fillrrrr)r^rrarbr"r9masks_queries_logits_per_layerclass_queries_logits_per_layerr4r r layer_moduler3norm_hidden_statesrrinterpolated_logitsrrsequence_outputrrr,r,r-rFs 2    "      z$EomtForUniversalSegmentation.forwardcCs|jjSr)rr&rr,r,r-get_input_embeddingssz1EomtForUniversalSegmentation.get_input_embeddingsrcCs|ddd|jjddf}||}|dd|jj|jjdddf}|dd}|j|jddg|jR}| |}| |}t d||}||fS)Nrr1rr>zbqc, bchw -> bqhw) rrrrr*rrNrJrrrr'einsum)r^r query_tokens class_logits prefix_tokens mask_logitsr,r,r-rs &   z$EomtForUniversalSegmentation.predictcCsD|dkr tj|jd||d|k}d|ddd||df|<|S)Nrrrc)r'rrrJ) attn_maskrrrrdrandom_queriesr,r,r-rsz4EomtForUniversalSegmentation._disable_attention_mask)NNN)r#r$r%rrr\rrrrrrrrr+rrrrrr'r staticmethodrrr,r,r_r-rsP      gr)F)r1)r1F)Lcollections.abcrrr dataclassesrnumpyr r'torch.nn.functionalrr5rrrr activationsr file_utilsr r r modeling_layersr modeling_utilsr rprocessing_utilsrutilsrrr utils.genericrutils.output_capturingrconfiguration_eomtrscipy.optimizer accelerateraccelerate.utilsrrr;rFrTrrUrrrrr rrr=r>rUrrdrerirurrrrrrrr__all__r,r,r,r-s             !  j :!, > *  2 D