o oij @sUddlZddlZddlmZmZmZmZddlZddlm Z m Z m Z m Z m Z mZmZmZddlmZddlmZmZmZmZddlmZddlmZmZmZdd lmZiZ ee!e!fe"d <d e d <Gd dde Z#GdddZ$de de de fddZ%dde de&de&de'de f ddZ(dS)N)AnyDictOptionalTuple)ModuleTensor concatenatesinstacktensorwherezeros)KORNIA_CHECK_SHAPE) DetectorCfgHeatMapRefineCfgJunctionRefineCfgLineDetectorCfg)nms)dataclass_to_dictdict_to_dataclasstorch_meshgrid)SOLD2Neturlsz6https://cvg-data.inf.ethz.ch/SOLD2/sold2_wireframe.tar wireframecspeZdZdZddedeeddffdd Zd ee e fdee e ffd d Z d e dee e ffd dZ ZS)SOLD2_detectoruModule, which detects line segments in an image. This is based on the original code from the paper "SOLD²: Self-supervised Occlusion-aware Line Detector and Descriptor". See :cite:`SOLD22021` for more details. Args: config (DetectorCfg): Configuration object containing all parameters. None will load the default parameters, which are tuned for images in the range 400~800 px. Using a dataclass ensures type safety and clearer parameter management. pretrained (bool): If True, download and set pretrained weights to the model. Returns: The raw junction and line heatmaps, as well as the list of detected line segments (ij coordinates convention). Example: >>> img = torch.rand(1, 1, 128, 128) >>> sold2_detector = SOLD2_detector(pretrained=False) >>> line_segments = sold2_detector(img)["line_segments"] TN pretrainedconfigreturncst|trtjdtddt|t}t|dur|nt|_ |j j |_ |j j |_ |j j |_ tt|j |_|rTtjjtdtdd}||d}|j||t|j j|_dS)NzUsage of config as a plain dictionary is deprecated in favor of`kornia.features.sold2.structures.DetectorCfg`. The support of plaindictionaries as config will be removed in kornia v0.8.0 (December 2024).)category stacklevelrcpu) map_locationmodel_state_dict) isinstancedictwarningswarnDeprecationWarningrrsuper__init__r grid_sizedetection_threshjunc_detect_threshmax_num_junctionsrrmodeltorchhubload_state_dict_from_urlrdeviceadapt_state_dictload_state_dictevalLineSegmentDetectionModuleline_detector_cfg line_detector)selfrrpretrained_dict state_dict __class__W/home/ubuntu/.local/lib/python3.10/site-packages/kornia/feature/sold2/sold2_detector.pyr+:s&       zSOLD2_detector.__init__r=cCs:|d=|d=|d=|d|d<|d|d<|d=|d=|S)Nw_junc w_heatmapw_descz'heatmap_decoder.conv_block_lst.2.weightz)heatmap_decoder.conv_block_lst.2.0.weightz%heatmap_decoder.conv_block_lst.2.biasz'heatmap_decoder.conv_block_lst.2.0.biasr@)r;r=r@r@rAr5Ws  zSOLD2_detector.adapt_state_dictimgc Cst|gdi}||}|d|d<|d|d<g}t|d|dD] \}}t||j|j|j}|j||\}}} | t ||q%||d<|S)arRun forward. Args: img: batched images with shape :math:`(B, 1, H, W)`. Returns: line_segments: list of N line segments in each of the B images :math:`List[(N, 2, 2)]`. junction_heatmap: raw junction heatmap of shape :math:`(B, H, W)`. line_heatmap: raw line heatmap of shape :math:`(B, H, W)`. )B1HW junctionsjunction_heatmapheatmap line_heatmap line_segments) rr0zipprob_to_junctionsr,r.r/r:detectappendline_map_to_segments) r;rEoutputs net_outputslines junc_probrLrJline_map_r@r@rAforwardas    zSOLD2_detector.forward)TN)__name__ __module__ __qualname____doc__boolrrr+rstrrr5rrZ __classcell__r@r@r>rAr$s  "" rc@s2eZdZdZd+deeddfddZdededeeeeffd d Z d,ded e de defddZ   d-dede de d e de def ddZ dededefddZdededede de dejdeeeffdd Zded!edefd"d#Zded$ed%edefd&d'Zded$ed%ede de d(edejdefd)d*ZdS).r8aModule extracting line segments from junctions and line heatmaps. Args: config (LineDetectorCfg): Configuration dataclass containing all settings required for line segment detection. - detect_thresh (float): Probability threshold for mean activation (0. ~ 1.). - num_samples (int): Number of sampling locations along the line segments. - inlier_thresh (float): Minimum inlier ratio to satisfy (0. ~ 1.) => 0. means no threshold. - heatmap_low_thresh (float): Lowest threshold for pixel considered as a candidate in junction recovery. - heatmap_high_thresh (float): Higher threshold for NMS in junction recovery. - max_local_patch_radius (float): Maximum patch to be considered in local maximum search. - lambda_radius (float): Lambda factor in linear local maximum search formulation. - use_candidate_suppression (bool): Apply candidate suppression to break long segments into sub-segments. - nms_dist_tolerance (float): Distance tolerance for NMS. Decides whether the junctions are on the line. - use_heatmap_refinement (bool): Whether to use heatmap refinement methods. - heatmap_refine_cfg: Configuration for heatmap refinement methods. - use_junction_refinement (bool): Whether to use junction refinement methods. - junction_refine_cfg: Configuration for junction refinement methods. Example: >>> config = LineDetectorCfg(detect_thresh=0.6, use_heatmap_refinement=True) >>> module = LineSegmentDetectionModule(config) >>> junctions, heatmap = torch.rand(10, 2), torch.rand(256, 256) >>> line_map, junctions, _ = module.detect(junctions, heatmap) NrrcCs|durt}||_|jj|_|jj|_|jj|_|jj|_|jj|_|jj|_ |jj |_ t dd|j|_|jj|_|jj|_|jj|_|jj|_|jrV|jdurVtd|jj|_|jj|_|jrl|jdurntddSdS)Nrrz*[Error] Missing heatmap refinement config.z+[Error] Missing junction refinement config.)rr detect_thresh num_samples inlier_threshmax_local_patch_radiuslocal_patch_radius lambda_radiusheatmap_low_thresh low_threshheatmap_high_thresh high_threshr1linspace torch_sampleruse_candidate_suppressionnms_dist_toleranceuse_heatmap_refinementheatmap_refine_cfg ValueErroruse_junction_refinementjunction_refine_cfg)r;rr@r@rAr+s,             z#LineSegmentDetectionModule.__init__rJrLc" Cst|ddg|j\}}|j}|jrAt|jtrA|jjdkr*|||jj |jj }n|jjdkrA| ||jj |jj |jj |jj }t|}t||g|tjd}|dkrX|||fStjtj||g|tjddd}|jrq|||}t|} t| d d | dd gd } || d d d f} || d d df} |j|d } | d d d df| | d d d dfd| }| d d ddf| | d d ddfd| }tj|d |dd }tj|d |dd }ttj| tj| tjdd d}||d|dd}t|}d}||krt||}g}t |D]g}||dksX||||d|d d f}||||d|d d f}||||d|}n |||d d d f}|||d d d f}|||d }|!|||||||}|"|q"t|d }n |!|||||||}tj#|d d|j$k}|j%d krtj||j$kd d|j&|j'}||j%k} || }| |}!d||!d d d f|!d d dff<d||!d d df|!d d d ff<|j(rt|!d kr|)||||||\}}|||fS)zPerform line segment detection.rHrIgloballocalr4dtyperrdiagonalr.NNminmaxdim?i')*rshaper4rpr%rqrmoderefine_heatmapratio valid_threshrefine_heatmap_local num_blocks overlap_ratiolenr r1int32triuonesrncandidate_suppressionr rrmtoclampsqrtsumfloat32mathceilrangedetect_local_maxrRmeanrbrdrxrcrsrefine_junction_perturb)"r;rJrLrHrIr4 num_junctions line_map_pred candidate_mapcandidate_indicescandidate_index_mapcandidate_junc_startcandidate_junc_endsamplercand_samples_hcand_samples_wcand_hcand_wsegments_lengthnormalized_seg_lengthnum_cand group_sizenum_itersampled_feat_lstiter_idxcand_h_cand_w_normalized_seg_length_ sampled_feat_ sampled_featdetection_results inlier_ratiodetection_results_inlierdetected_junc_indexesr@r@rArQs      44$      " $$ z!LineSegmentDetectionModule.detect皙?{Gz?rrcCs\|||k}tj|ddd}t|jd|}t|d|}tj||ddd}|S)z!Global heatmap refinement method.T) descendingrN?r})r1sortrrrrr)r;rLrrheatmap_values sorted_values top10_lenmax20r@r@rAr-s z)LineSegmentDetectionModule.refine_heatmaprMb`?rrcCsV|j\}}d|}t|d|d|} t|d|d|} t|jtj|jd} t|jtj|jd} t|D]e} t|D]^}t| | |}t|| |}| |dkr[|| n|}||dkrg|| n|}|||||f}||kr|j |||d}| ||||f|7<| ||||fd7<q?q9tj | | ddd} | S)z Local heatmap refinement method.r)rxr4)rrr)rr~) rroundr r1intr4floatrrrr)r;rLrrrrrHrIincrease_ratioh_blockw_block count_mapheatmap_outputh_idxw_idxh_startw_starth_endw_end subheatmapr@r@rAr7s(    z/LineSegmentDetectionModule.refine_heatmap_localrcCs|j}tjtj|dd|ddddd}ttj|dd}|d}|d}||d d f}||d d f} |||f} | |tj| |ddd } |d|jdd} tj| dd} td | | d | d }|dk|dk}ttd | | d | d }| d t |}||k}||}t |}tj|ddgd}||d t d||f tj 8}||d t d||f tj 8}|dk}d|||||f<|S)z:Suppress overlapping long lines in the candidate segments.rr)N.rr|rryrNr{z bij,bjk->bik).NN.r)ror1r unsqueezer rnormeinsumacosr rarangerr)r;rJrdist_tolerance line_dist_map seg_indexesstart_point_idxsend_point_idxs start_points end_points line_distsdir_vecs cand_vecscand_vecs_normproj proj_mask cand_angles cand_distsjunc_dist_mask junc_masknum_segs junc_counts final_maskr@r@rAr\s4( $$z0LineSegmentDetectionModule.candidate_suppressionrXrHrIr4c'Cst|jtstdt|j|jj}|jj}|dd} tj| | || d||d} t | | | | gdd\} } } }t | d| d| d|dgd}| ddd}t tj |dd }||d }||d}t||gd}|jdd |d }tj|d d |dd|d <tj|dd |dd|d<g}t|}t|D]}||}|d d d f}|d d df}|j|d }|d d d df||d d d dfd|}|d d ddf||d d ddfd|}tj|d |dd}tj|d |dd} |||| }!tj|!dd }"t|"}#|||#d qt |d }$t |$d d d d d f|$d d dd d fgd }%tj|%d d }%||%|$}&|%|&fS)z5Refine the line endpoints in a similar way as in LSD.zIExpected to have dataclass of type JunctionRefineCfg for junction.Gotcha rr)startendstepr4ij)indexingr{r|ryrrNrr}).r)r%rtr TypeErrortype num_perturbsperturb_intervalr1rrrviewr rr rrrrrmrdetect_bilinearrargmaxrRuniquesegments_to_line_map)'r;rJrXrLrHrIr4rr side_perturbs perturb_vech1_gridw1_gridh2_gridw2_gridperturb_tensorperturb_tensor_flatdetected_seg_indexesrrrNline_segment_candidatesrefined_segment_lst num_segmentsidxsegmentrrrrrrr segment_featsegment_resultsmax_idxrefined_segments junctions_new line_map_newr@r@rArsb       44  2 z2LineSegmentDetectionModule.refine_junction_perturbsegmentscCst|}t||g|jd}ttj|d|ddddfkdd\}}ttj|d|ddddfkdd\}}d|||f<d|||f<|S)z)Convert the list of segments to line map.r4Nrrrr)rr r4r r1all)r;rJrrrXrY idx_junc1 idx_junc2r@r@rArs,,  z/LineSegmentDetectionModule.segments_to_line_maprrc Cst|tj}t|tj}t|tj}t|tj}|||f|||||||f|||||||f|||||||f||||}|S)zDetect by bilinear sampling.)r1floorrlongr) r;rLrr cand_h_floor cand_h_ceil cand_w_floor cand_w_ceilcand_samples_featr@r@rArsz*LineSegmentDetectionModule.detect_bilinearrcCs4d|j|}tj|d|jdd}t|d|dgd} t| } ttd|jdtd|jdg|d} t |j|jgg|tj d} t | d k\} }t| d|dgd}t tj || ddd}|||jkd d f}||j}tj| dd|d }t tj tj| dd|ddd}||dk}tj|d d d d d d d fd |dd tj}tj|d d d d d d dfd |dd tj}t|d|dgd}||d d d d d d d f|d d d d d d dff}||tj }t|d krtd |j}|Stj|ddd }|S) zDetect by local maximum search.g;f?r{r|rrrrrwrN)NNr})rgr1repeat_interleavercrrr rrfr rr rrrrrrremptyr)r;rLrrrHrIrr4 dist_thresh cand_pointscand_points_round patch_mask patch_centerH_patch_pointsW_patch_points patch_pointspatch_center_distpatch_points_shifted patch_distpatch_dist_maskpoints_Hpoints_Wpointsrsampled_feat_lmaxr@r@rArs0  , & 44<z+LineSegmentDetectionModule.detect_local_max)N)rr)rrrr)r[r\r]r^rrr+rrrQrrrrrr1r4rrrrr@r@r@rAr8sr 'h  %/  L r8rJrXrcCs,tt|\}}t||||gd}|S)zBConvert a junction connectivity map to a Nx2x2 tensor of segments.r)r r1rr )rJrX junc_loc1 junc_loc2rr@r@rArS.srSrprobdist prob_threshtop_kc Cstt||kd}t|dkr|St||d||dgd}|||k}t||d}||}|dkrCtt||}|d|}|S)zVExtract junctions from a probability map, apply NMS, and extract the top k candidates.r|rrrgMbP?N)r r rrrrr~) r.r/r0r1rJboxesscores remainingskr@r@rArP5s    rP)rr))rr'typingrrrrr1 kornia.corerrrr r r r r kornia.core.checkrkornia.feature.sold2.structuresrrrrkornia.geometry.bboxr kornia.utilsrrr backbonesrrr`__annotations__rr8rSrrrPr@r@r@rAs& (   _.$