o iF@sTddlmZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZmZdd lmZdd lmZmZmZdd lmZdd lmZddlmZddlm Z eeddGdddeZ!Gddde j"Z#ddZ$dIddZ%dej&de'dej&fdd Z( !dJd"e j"d#ej&d$ej&d%ej&d&eej&d'e)d(e)d)eefd*d+Z*Gd,d-d-e j"Z+Gd.d/d/e j"Z,Gd0d1d1e j"Z-d2ej&d3ej&d4ej&dej&fd5d6Z.Gd7d8d8e j"Z/Gd9d:d:e j"Z0eGd;d<de)de2ej&ej&ffd?d@Z3dAej&dBe'dCe'dej&fdDdEZ4edFdGdGdHdHe1Z5d)r&stackflatten)xx1x2rot_xr+r+r, rotate_halfesrTc Csj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. dtype)rVfloat unsqueezerTto) qkrArB position_ids unsqueeze_dimrVq_embedk_embedr+r+r,apply_rotary_pos_embms  r`r n_repr<cCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)shapeexpandreshape)r rabatchnum_key_value_headsslenhead_dimr+r+r, repeat_kvs 0rimodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nrr rMr=)r?rV)ptrainingr)rinum_key_value_groupsr&matmul transposerbr functionalsoftmaxfloat32rYrVrqrt contiguous) rkrlrmrnrorprqrr key_states value_states attn_weights causal_mask attn_outputr+r+r,eager_attention_forwards & rcseZdZdZdedeffdd Zedddd  dd ej d e e ej ej fd e ej de ej de ej de e de ej e ej ffddZZS)LightGlueAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr. layer_idxcst||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_dS)NrhgTr/)r1r2r.rgetattr hidden_sizer5rhrfrurpattention_dropout is_causalrr3attention_biasq_projk_projv_projo_projr8r.rr9r+r,r2s(   zLightGlueAttention.__init__past_key_valuepast_key_valuesz4.58)new_nameversionNr position_embeddingsroencoder_hidden_statesencoder_attention_maskrrr<cKs"|jdd}g|d|jR}|||dd} |du} | r%|n|} | r+|n|} || |dd} || |dd}|durV|\}}t| | ||\} } t}|j j dkrdt |j j }||| | || f|j spdn|j |jd|\}}|jg|dR}||}||fS)Nr=rreagerrj)rqrp)rbrhrviewrwrrr`rr._attn_implementationr rtrrprdr{r)r8r rrorrrr input_shape hidden_shape query_statesis_cross_attentioncurrent_statescurrent_attention_maskr|r}rArBattention_interfacerr~r+r+r,rHs:       zLightGlueAttention.forward)NNNN)r"r#r$r%rintr2rr&rJrr*rr rHrLr+r+r9r,rs. rc8eZdZdeffdd ZdejdejfddZZS) LightGlueMLPr.csVt||_t|j|_t|j|j|_ t|j|j |_ tj |jdd|_ dS)NT)elementwise_affine)r1r2r.r hidden_act activation_fnrr3intermediate_sizefc1rfc2 LayerNorm layer_normr7r9r+r,r2s  zLightGlueMLP.__init__r r<cCs,||}||}||}||}|SN)rrrr)r8r r+r+r,rHs    zLightGlueMLP.forward r"r#r$rr2r&rJrHrLr+r+r9r,rsrcs|eZdZdedeffdd Z  ddejdejdejd ee d ee d e ejee ejee ejff d d Z Z S)LightGlueTransformerLayerr.rcs:tt|||_t||_t|||_t||_dSr)r1r2rself_attentionrself_mlpcross_attention cross_mlprr9r+r,r2 s    z"LightGlueTransformerLayer.__init__F descriptorsrror;output_attentionsr<cCsX|rdnd}|r dnd}|r||f}|j\}} } |j||||d\} } tj|| gdd} || }||}|r<| |f}|dd| | d|| | }|durb|dddd| d|dd| nd}|j||||d\}}tj||gdd}||}||}|r||f}|||| | f|||| | f|}|r|| f|f}|||fS)Nr+)rrorr=r>rr)rrr) rbrr&catrrdfliprr)r8rrror;rall_hidden_statesall_attentions batch_size num_keypointsr4attention_outputself_attentionsintermediate_states output_statesself_attention_descriptorsself_attention_hidden_statesrrcross_attention_outputcross_attentionscross_intermediate_statescross_output_statescross_attention_hidden_statesr+r+r,rHs`        $   z!LightGlueTransformerLayer.forward)FF) r"r#r$rrr2r&rJrrKr*rHrLr+r+r9r,r s" "r similarity matchability0 matchability1c Cs|j\}}}tj|tj|dd}tj|d}tj|ddddd}|||d|dfd} |||| ddd|d|f<tj|d | dddddf<tj|d | dddddf<| S)z;create the log assignment matrix from logits and similarityrrr=rMrN) rbrrx logsigmoidrw log_softmaxr{new_fullsqueeze) rrrrnum_keypoints_0num_keypoints_1 certaintiesscores0scores1scoresr+r+r,sigmoid_log_double_softmax]s  ""&&rcsTeZdZdeffdd ZdejdejdejfddZdejdejfd d ZZ S) LightGlueMatchAssignmentLayerr.cs@t|j|_tj|j|jdd|_tj|jddd|_dS)NTr/r)r1r2r4rr3final_projection matchabilityr7r9r+r,r2ms z&LightGlueMatchAssignmentLayer.__init__rrr<cCs2|j\}}}||}|tj|j|jdd}||dd||}|dddf}|dddf}||dd} |durr||dd|}|dddfd} |dddfddd} | | }| |dkt | j j } | |} | |dd|d} | dddf} | dddf}t| | |}|S)Ndeviceg?rrrr=rM)rbrr&tensorr4rrdrwrX masked_fillfinforVminrr)r8rrrrr4 m_descriptorsm_descriptors0m_descriptors1rmask0mask1rmatchability_0matchability_1rr+r+r,rHts&    z%LightGlueMatchAssignmentLayer.forwardcCs ||}tj|d}|S)z0Get matchability of descriptors as a probabilityr=)rrrxsigmoidr)r8rrr+r+r,get_matchabilitys z.LightGlueMatchAssignmentLayer.get_matchability) r"r#r$rr2r&rJrHrrLr+r+r9r,rlsrcr)LightGlueTokenConfidenceLayerr.cstt|jd|_dSNr)r1r2rr3r4tokenr7r9r+r,r2s z&LightGlueTokenConfidenceLayer.__init__rr<cCs$||}tj|d}|S)Nr=)rdetachrrxrr)r8rrr+r+r,rHsz%LightGlueTokenConfidenceLayer.forwardrr+r+r9r,rsrc@s.eZdZUdZeed<dZdZdZdZ dZ dS)LightGluePreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r. lightglue pixel_valuesFTN) r"r#r$r%rr(base_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_supports_sdpar+r+r+r,rs rr thresholdcCsh|j\}}}|ddddddfd}|ddddddfd}|j}|j}tj|jd|jdd}tj|jd|jdd} ||d|k} | |d|k} |j}| d} t | || } t | | d|| }| | |k@}| |d|@}t ||d}t ||d}t ||g dd |dd}t | |g dd |dd}||fS)z1obtain matches from a score matrix [Bx M+1 x N+1]Nr=rrrr)rbmaxindicesr&arangergathervaluesexp new_tensorwhererNrwrd)rrr_max0max1matches0matches1indices0indices1mutual0mutual1zeromatching_scores0matching_scores1valid0valid1rrr+r+r,get_matches_from_scoress(      ""rrheightwidthcCsVtj||g|j|jdd}|d}|djd}||ddddf|d}|S)a Normalize keypoints locations based on image image_shape Args: keypoints (`torch.Tensor` of shape `(batch_size, num_keypoints, 2)`): Keypoints locations in (x, y) format. height (`int`): Image height. width (`int`): Image width. Returns: Normalized keypoints locations of shape (`torch.Tensor` of shape `(batch_size, num_keypoints, 2)`). rrVNrr=.).NN)r&rrrVrr)rrr sizeshiftscaler+r+r,normalize_keypointss rzV LightGlue model taking images as inputs and outputting the matching of them. cseZdZdZdeffdd ZdedefddZ d/d e j d e j d e e de e j e e j e j fffd dZde j dede j de j de j f ddZd0ddZde j de j dede j fddZd e j d e j de j de j de j de j defddZdd Zde j d!e j d"e j d#e j de e j e j ff d$d%Z   d1d e j d e j d&ed'ede e j d(e e d e e de e j e j e j e e ffd)d*Zee   d1d+e jd,e e jd(e e d e e dee eff d-d.ZZS)2LightGlueForKeypointMatchingar LightGlue is a model matching keypoints in images by leveraging detections from a keypoint detector such as SuperPoint. It is based on the SuperGlue architecture and is designed to be lightweight and efficient. It consists of : 1. Keypoint Encoder 2. A Graph Neural Network with self and cross attention layers 3. Matching Assignment layers The correspondence ids use -1 to indicate non-matching points. Philipp Lindenberger, Paul-Edouard Sarlin and Marc Pollefeys. LightGlue: Local Feature Matching at Light Speed. In ICCV 2023. https://huggingface.co/papers/2306.13643 r.csttjjjd|_jj|_j |_ j |_ j |_ j |_ j|_|j |jkr;tj|j|j dd|_nt|_t|_tfddtj D|_tfddtj D|_tfddtj dD|_|dS) N)trust_remote_codeTr/csg|]}t|dqS))r)r).0ir.r+r, z9LightGlueForKeypointMatching.__init__..cg|]}tqSr+)rrrrr+r,rcrr+)rrrr+r,rrr)r1r2r from_configkeypoint_detector_configrkeypoint_detectordescriptor_decoder_dim keypoint_detector_descriptor_dimr4num_hidden_layers num_layersfilter_thresholddepth_confidencewidth_confidencerr3input_projectionIdentityr-positional_encoder ModuleListrangetransformer_layersmatch_assignment_layerstoken_confidence post_initr7r9rr,r2s0      z%LightGlueForKeypointMatching.__init__ layer_indexr<cCs*ddtd||j}t|ddS)z-scaled confidence threshold for a given layerg?g?grr)nprrclip)r8r,rr+r+r,_get_confidence_thresholdsz6LightGlueForKeypointMatching._get_confidence_thresholdFrrr;cCs,|}||}|j||d}||fS)Nr;)rr{r#r%)r8rrr;projected_descriptorskeypoint_encoding_outputr+r+r,_keypoint_processings  z1LightGlueForKeypointMatching._keypoint_processingkeypoint_confidencesr num_pointsc Cs|j\}}||jdkr6||dkd}||dd}||}d||kjdd|}||jk} | Stj |tj d} | S)zRevaluate whether we should stop inference based on the confidence of the keypointsrrrr=g?r>rU) rbrrrdr/rWsumr!r&onesrK) r8r4r,rr5rrrratio_confidentearly_stopped_pairsr+r+r,_get_early_stopped_image_pairs's   z;LightGlueForKeypointMatching._get_early_stopped_image_pairsNcCs@|dur ||}||}|j|||}t||j\}}||fSr)r)rr )r8rrr, early_stopsrrrr+r+r,_get_keypoint_matching:s z3LightGlueForKeypointMatching._get_keypoint_matching confidencesrcCs,|d|jk}|dur||||kO}|S)z#mask points which should be removedrN)r"r/)r8r=rr,keepr+r+r,_get_pruning_maskBsz.LightGlueForKeypointMatching._get_pruning_maskr prune_outputcs|j\}} } |j||} ||| ||dktdfdd||d|d|fD\} } } }}t|D]}||||fd7<q;dd| | | |fD\} } } }| | f}t|ddd }| ||||fS) z For a given layer, prune keypoints based on the confidence of the keypoints and the matchability of the descriptors. rFc3s$|] }ddt|DVqdS)cSsg|]\}}||qSr+r+)rtrr+r+r,r^rzULightGlueForKeypointMatching._do_layer_keypoint_pruning...N)ziprrpruned_keypoints_maskr+r, ]s  zJLightGlueForKeypointMatching._do_layer_keypoint_pruning..rcss|] }t|ddVqdS)T) batch_firstNr)r pruned_tensorr+r+r,rFes  Tr=rG padding_value) rbr)rr?rr&rr'r)r8rrrrr@r4r,rrdescriptors_matchabilitypruned_descriptorspruned_keypoints_0pruned_keypoints_1 pruned_maskpruned_indicesrpruned_keypointsr+rDr,_do_layer_keypoint_pruningIs    z7LightGlueForKeypointMatching._do_layer_keypoint_pruningcsttjd}|}|dd||fD\}}dd||fD\}}fdd||||fD\}}}}||||fS)Nrcs|] }t|dddVqdS)Tr=rINrrCr+r+r,rF{  zMLightGlueForKeypointMatching._concat_early_stopped_outputs..csrS)TrrINrrCr+r+r,rFrTc3s|]}|VqdSrr+rCearly_stops_indicesr+r,rFs  )r&rNrrb)r8rVfinal_pruned_keypoints_indices!final_pruned_keypoints_iterationsrrids order_indicesr+rUr,_concat_early_stopped_outputsns$    z:LightGlueForKeypointMatching._concat_early_stopped_outputsrrrc s|j\}fdd|||fD\}}}|dddf}|dddf}|dddf}|dddf} |dddf} |dddf} tjdd|fd|j|jd} tjdd|f|j|jd} tdD]T}t||dkd||d||j dd| |d||f<t| |dkd||d| |j dd| |d||f<| || |d||f<| || |d||f<qi| | fS) Nc3s"|] }|dddVqdS)rr=N)rdrCrr+r,rFs zJLightGlueForKeypointMatching._do_final_keypoint_pruning..rrrr=r )r) rbr&fullrrVzerosr'rrclamp)r8rrrrrrrrrrr_matches_matching_scoresrr+r\r,_do_final_keypoint_prunings0   &&z7LightGlueForKeypointMatching._do_final_keypoint_pruningrr rc( s|rdnd}|r dnd} |jddkr.|jdd} |j| dtjd|| || || fS|j} |j\} } }} tj|| ddd}|| d|d}|durX|| d|nd}|| d||j}tj | d| d}t |||}|j |||d \}}|d}|j dk}|j dk}g}g}g}g}g}tj d|| d| dd}t|}t|jD]}|}|dur|||}n tj| |d f|jd}|j||||||d }|\}}} |r||}|r| | } |r||jdkr|j||}!|j|!|||d }"ntj| tjd}"t|"r|"d|}#|j|||d \}$}%|t|#|t|$|t|%|rO|t||t|||"}tfdd||d|d||fD\}}&}'}}|&|'f}|rtfdd|||!fD\}}}!t|"rn|r| ||||||!|\}}}}}q|r|r|!|||||\}}}}|"||||\}}n||||jd\}}t||j}|| d|}||||| fS)Nr+rrr=rUrr>rr0rM)ror;r)r5r;c3|]}|VqdSrr+rCrcr+r,rF  zALightGlueForKeypointMatching._match_image_pair..c3rdrr+rCrcr+r,rF#re)#rbrr&r new_zerosrr6rdrrrr3r!r"rc ones_liker'rr get_extended_attention_maskr7r(r*r:rKanyr@r<extendlistr*allrRr[rb)(r8rrrr rrr;rrrbrrrinitial_num_keypointsnum_points_per_pair image_indicesr2 do_early_stopdo_keypoint_pruningrVrrrWrXpruned_keypoints_indicespruned_keypoints_iterationsr,rextended_attention_mask layer_outputr attentionr4r9early_stopped_image_indicesearly_stopped_matchesearly_stopped_matching_scores keypoints_0 keypoint_1r+rcr,_match_image_pairs                   z.LightGlueForKeypointMatching._match_image_pairrlabelsc Csd}|dur td|dur|n|jj}|dur|n|jj}|jdks*|ddkr.td|j\}}}} } ||d|| | }||} | dd\} }} }| |ddd |} | |dd|j  |} ||dd}| }|dddddddf| |dddddddf<|dddddddf| |dddddddf<|j || | | |||d \}}}}}t |||| ||||d S) Nz9LightGlue is not trainable, no labels should be provided.rrzOInput must be a 5D tensor of shape (batch_size, 2, num_channels, height, width)r=r)rrr;)rrrrrrr r!) ValueErrorr.rr;ndimr rbrdrrYrcloner|r)r8rr}rr;rrrchannelsrr keypoint_detectionsrrrabsolute_keypointsrrrr r!r+r+r,rH`sJ  88 z$LightGlueForKeypointMatching.forwardrIr)NNN)r"r#r$r%rr2rrWr/r&rJrrKr*r3r:r<r?rRr[rbr|rrr' LongTensorrrrHrLr+r+r9r,rs     %  + . rr)rj)7 dataclassesrtypingrrrnumpyr-r&rtorch.nn.utils.rnnr activationsr modeling_flash_attention_utilsr modeling_utilsr r processing_utilsrutilsrrrutils.deprecationr utils.genericrauto.modeling_autorconfiguration_lightgluerrModuler-rTr`rJrrirWrrrrrrrrr*rrr__all__r+r+r+r,s           #  HS ) $  0