o oi`@sddlmZddlmZddlZddlmmZddl m Z m Z m Z m Z mZmZmZmZmZddlmZddlmZmZddlmZmZddlmZdd lmZd d lm Z m!Z!d d l"m#Z#dIdJddZ$dIdJddZ%dIdKddZ&dIdKddZ'Gddde Z(Gdd d e Z) ! " " # $ % &dLdMd4d5Z* 6 7 7 # & % $ 8dNdOd;d<Z+ $dPdQd>d?Z,Gd@dAdAe Z-dRdSdEdFZ.GdGdHdHe Z/dS)T) annotations)OptionalN) ModuleTensor concatenaterandstacktensorwherezeros zeros_like)spatial_gradient3d)normalize_pixel_coordinatesnormalize_pixel_coordinates3d)create_meshgridcreate_meshgrid3d)torch_version_ge)safe_solve_with_mask)spatial_expectation2dspatial_softmax2d)nms3dhintwdeviceOptional[torch.device]returnrcCsB|dur td}t||d|d}t|||}|dddd}|S) zGenerate a kernel to with window coordinates, residual to window center. Args: h: kernel height. w: kernel width. device: device, on which generate. Returns: conv_kernel [2x1xhxw] NcpuFrrr)torchrrrpermute)rrr window_grid2d conv_kernelr&^/home/ubuntu/.local/lib/python3.10/site-packages/kornia/geometry/subpix/spatial_soft_argmax.py_get_window_grid_kernel2d$s   r(cCs|dur td}tdd|||d}|ddkr#|d}|dd}n |dd}|dd}|ddkr@|d}|dd}n |dd}|dd}dt|||||dd||||f<|S) zGenerate a kernel to return center coordinates, when applied with F.conv2d to 2d coordinates grid. Args: h: kernel height. w: kernel width. device: device, on which generate. Returns: conv_kernel [2x2xhxw]. Nrr!rrr?)rrr"rr float)rrr center_kernelh_i1h_i2w_i1w_i2r&r&r'_get_center_kernel2d8s       ,r1dc Cs|dur tdtdd||||d}|ddkr$|d}|dd}n |dd}|dd}|ddkrA|d}|dd}n |dd}|dd}|ddkr^|d} |dd} n |dd} |dd} t||||| | } d| |d d | | ||||f<|S) aGenerate a kernel to return center coordinates, when applied with F.conv2d to 3d coordinates grid. Args: d: kernel depth. h: kernel height. w: kernel width. device: device, on which generate. Returns: conv_kernel [3x3xdxhxw]. Nrr rr!rrr))rrr!r*) r2rrrr,r-r.r/r0d_i1d_i2 center_numr&r&r'_get_center_kernel3dYs*          "r6cCs|dur td}t||d|d}|dkr%tjdd||d|ddd}n tdddd|d}t|d||d||dddgd}| dddd  d}|S) zGenerate a kernel to return coordinates, residual to window center. Args: d: kernel depth. h: kernel height. w: kernel width. device: device, on which generate. Returns: conv_kernel [3x1xdxhxw] NrTrrr rr!) r"rrlinspaceviewr rrepeat contiguousr# unsqueeze)r2rrrgrid2dzgrid3dr%r&r&r'_get_window_grid_kernel3ds  *r@csFeZdZdZ       dd fdd Zd!ddZd"ddZZS)#ConvSoftArgmax2dzModule that calculates soft argmax 2d per window. See :func: `~kornia.geometry.subpix.conv_soft_argmax2d` for details. r r rrr)T:0yE>F kernel_sizetuple[int, int]stridepadding temperatureTensor | floatnormalized_coordinatesboolepsr+ output_valuerNonecs8t||_||_||_||_||_||_||_dSN) super__init__rErGrHrIrKrMrN)selfrErGrHrIrKrMrN __class__r&r'rRs  zConvSoftArgmax2d.__init__strcCsF|jjd|jd|jd|jd|jd|jd|jd|jdS) N (kernel_size= , stride= , padding=, temperature=, normalized_coordinates=, eps=, output_value=)) rU__name__rErGrHrIrKrMrNrSr&r&r'__repr__s zConvSoftArgmax2d.__repr__xrTensor | tuple[Tensor, Tensor]c Cs$t||j|j|j|j|j|j|jSrP)conv_soft_argmax2drErGrHrIrKrMrNrSrbr&r&r'forwardszConvSoftArgmax2d.forwardrBrCrCr)TrDF)rErFrGrFrHrFrIrJrKrLrMr+rNrLrrOrrVrbrrrcr_ __module__ __qualname____doc__rRrarf __classcell__r&r&rTr'rAs  rAcsHeZdZdZ        d!d"fdd Zd#ddZd$dd ZZS)%ConvSoftArgmax3dzModule that calculates soft argmax 3d per window. See :func: `~kornia.geometry.subpix.conv_soft_argmax3d` for details. r r r rrrr)FrDTrEtuple[int, int, int]rGrHrIrJrKrLrMr+rNstrict_maxima_bonusrrOc s>t||_||_||_||_||_||_||_||_ dSrP) rQrRrErGrHrIrKrMrNrt) rSrErGrHrIrKrMrNrtrTr&r'rRs  zConvSoftArgmax3d.__init__rVcCsN|jjd|jd|jd|jd|jd|jd|jd|jd|j d S) NrWrXrYrZr[r\z, strict_maxima_bonus=r]r^) rUr_rErGrHrIrKrMrtrNr`r&r&r'ras$ zConvSoftArgmax3d.__repr__rbrrcc Cs(t||j|j|j|j|j|j|j|j SrP) conv_soft_argmax3drErGrHrIrKrMrNrtrer&r&r'rfszConvSoftArgmax3d.forwardrprqrqr)FrDTrr)rErsrGrsrHrsrIrJrKrLrMr+rNrLrtr+rrOrhrirjr&r&rTr'ros  rorBrCr)TrDFinputrErFrGrHrIrJrKrLrMr+rNrcc Cst|stdt|t|jdkstd|j|dkr(td||j\}} } } |\} } |j}|j}| || d| | }t | | | |}t | | | |}t |d}|||}t| | }|t j||||d|}|t j|||||d|}| || |d |d }t| | d | |dd dd }t j||||d}t j||||d}|||}|||}|rt|dd d d| | }|dd dd }| || d |d |d }|r||fS|S) aCompute the convolutional spatial Soft-Argmax 2D over the windows of a given heatmap. .. math:: ij(X) = \frac{\sum{(i,j)} * exp(x / T) \in X} {\sum{exp(x / T) \in X}} .. math:: val(X) = \frac{\sum{x * exp(x / T) \in X}} {\sum{exp(x / T) \in X}} where :math:`T` is temperature. Args: input: the given heatmap with shape :math:`(N, C, H_{in}, W_{in})`. kernel_size: the size of the window. stride: the stride of the window. padding: input zero padding. temperature: factor to apply to input. normalized_coordinates: whether to return the coordinates normalized in the range of :math:`[-1, 1]`. Otherwise, it will return the coordinates in the range of the input shape. eps: small value to avoid zero division. output_value: if True, val is output, if False, only ij. Returns: Function has two outputs - argmax coordinates and the softmaxpooled heatmap values themselves. On each window, the function computed returns with shapes :math:`(N, C, 2, H_{out}, W_{out})`, :math:`(N, C, H_{out}, W_{out})`, where .. math:: H_{out} = \left\lfloor\frac{H_{in} + 2 \times \text{padding}[0] - (\text{kernel\_size}[0] - 1) - 1}{\text{stride}[0]} + 1\right\rfloor .. math:: W_{out} = \left\lfloor\frac{W_{in} + 2 \times \text{padding}[1] - (\text{kernel\_size}[1] - 1) - 1}{\text{stride}[1]} + 1\right\rfloor Examples: >>> input = torch.randn(20, 16, 50, 32) >>> nms_coords, nms_val = conv_soft_argmax2d(input, (3,3), (2,2), (1,1), output_value=True) Input type is not a Tensor. Got z-Invalid input shape, we expect BxCxHxW. Got: r5Temperature should be positive float or tensor. Got: rrCrGrHr!r F)r" is_tensor TypeErrortypelenshape ValueErrorrdtyper9r1tor(Fadaptive_max_pool2ddetachexpr+ avg_pool2dsizerr#conv2d expand_asr)rwrErGrHrIrKrMrNbcrrkykxrrr, window_kernelx_maxx_exp pool_coefden x_softmaxpool grid_globalgrid_global_pooled coords_maxr&r&r'rds> 3   rdrprqrrrsrtc ! Cst|stdt|t|jdkstd|j|dkr(td||j\} } } } } |\}}}|j}|j}| | | d| | | }t |||| |}t |||| |}t |d}|||}t|||}|t j|||||d|}t| | | d |d  |dd dd d }t j||||d}t j||||d}|||}|||}|rt|dd d d d| | | }|dd dd d }| | | d |d |d |d }|s|S|t j| |||||d|}|dkr0|d }|d }||d }t t||d|d} | dddd|||f} |d|| 9}| | | |d |d |d }||fS)a&Compute the convolutional spatial Soft-Argmax 3D over the windows of a given heatmap. .. math:: ijk(X) = \frac{\sum{(i,j,k)} * exp(x / T) \in X} {\sum{exp(x / T) \in X}} .. math:: val(X) = \frac{\sum{x * exp(x / T) \in X}} {\sum{exp(x / T) \in X}} where ``T`` is temperature. Args: input: the given heatmap with shape :math:`(N, C, D_{in}, H_{in}, W_{in})`. kernel_size: size of the window. stride: stride of the window. padding: input zero padding. temperature: factor to apply to input. normalized_coordinates: whether to return the coordinates normalized in the range of :math:[-1, 1]`. Otherwise, it will return the coordinates in the range of the input shape. eps: small value to avoid zero division. output_value: if True, val is output, if False, only ij. strict_maxima_bonus: pixels, which are strict maxima will score (1 + strict_maxima_bonus) * value. This is needed for mimic behavior of strict NMS in classic local features Returns: Function has two outputs - argmax coordinates and the softmaxpooled heatmap values themselves. On each window, the function computed returns with shapes :math:`(N, C, 3, D_{out}, H_{out}, W_{out})`, :math:`(N, C, D_{out}, H_{out}, W_{out})`, where .. math:: D_{out} = \left\lfloor\frac{D_{in} + 2 \times \text{padding}[0] - (\text{kernel\_size}[0] - 1) - 1}{\text{stride}[0]} + 1\right\rfloor .. math:: H_{out} = \left\lfloor\frac{H_{in} + 2 \times \text{padding}[1] - (\text{kernel\_size}[1] - 1) - 1}{\text{stride}[1]} + 1\right\rfloor .. math:: W_{out} = \left\lfloor\frac{W_{in} + 2 \times \text{padding}[2] - (\text{kernel\_size}[2] - 1) - 1}{\text{stride}[2]} + 1\right\rfloor Examples: >>> input = torch.randn(20, 16, 3, 50, 32) >>> nms_coords, nms_val = conv_soft_argmax3d(input, (3, 3, 3), (1, 2, 2), (0, 1, 1)) rx/Invalid input shape, we expect BxCxDxHxW. Got: rrzrrqr{Frryr!r Nr))r"r|r}r~rrrrrr9r6rr@radaptive_max_pool3drrr+ avg_pool3dview_asrr#conv3drrrr)!rwrErGrHrIrKrMrNrtrrr2rrkzrrrrr,rrrrrrrrr in_levels out_levels skip_levels strict_maximar&r&r'ruvsN :   &&&    $ruOptional[Tensor]cCs(|durtd}t||}t||}|S)aCompute the Spatial Soft-Argmax 2D of a given input heatmap. Args: input: the given heatmap with shape :math:`(B, N, H, W)`. temperature: factor to apply to input. normalized_coordinates: whether to return the coordinates normalized in the range of :math:`[-1, 1]`. Otherwise, it will return the coordinates in the range of the input shape. Returns: the index of the maximum 2d coordinates of the give map :math:`(B, N, 2)`. The output order is x-coord and y-coord. Examples: >>> input = torch.tensor([[[ ... [0., 0., 0.], ... [0., 10., 0.], ... [0., 0., 0.]]]]) >>> spatial_soft_argmax2d(input, normalized_coordinates=False) tensor([[[1.0000, 1.0000]]]) Nr))r rr)rwrIrK input_softoutputr&r&r'spatial_soft_argmax2ds   rcs8eZdZdZddfd d Zdd dZdddZZS)SpatialSoftArgmax2dzCompute the Spatial Soft-Argmax 2D of a given heatmap. See :func:`~kornia.geometry.subpix.spatial_soft_argmax2d` for details. NTrIrrKrLrrOcs*t|dur td}||_||_dS)Nr))rQrRr rIrK)rSrIrKrTr&r'rRs  zSpatialSoftArgmax2d.__init__rVcCs|jjd|jd|jdS)Nz temperature=r[r^)rUr_rIrKr`r&r&r'ra#s zSpatialSoftArgmax2d.__repr__rwrcCt||j|jSrP)rrIrK)rSrwr&r&r'rf*zSpatialSoftArgmax2d.forwardNT)rIrrKrLrrOrh)rwrrrrjr&r&rTr'rs  r$@Hz>tuple[Tensor, Tensor]c Cst|stdt|t|jdkstd|j|j\}}}}}t|||d|jd dddd d }| |j }t |dd d } | ddd ddd  d d d} t |d d d } | ddd ddd  d d} | d} | d} | d} d| d}d| d}d| d}t| |||| |||| g d d d d }tdds|t|d|jdd|7}t|dd}t| }t| |d ||d \}}}|||}|||t|d ddd<| }|jdddddk}|||ddt| dd d|}|||||||}|dkr!||| |j 7}|d |||||d  dddd d d}|dddddf|dddddf<||ddddd}||}||fS)aYCompute the single iteration of quadratic interpolation of the extremum (max or min). Args: input: the given heatmap with shape :math:`(N, C, D_{in}, H_{in}, W_{in})`. strict_maxima_bonus: pixels, which are strict maxima will score (1 + strict_maxima_bonus) * value. This is needed for mimic behavior of strict NMS in classic local features eps: parameter to control the hessian matrix ill-condition number. Returns: the location and value per each 3x3x3 window which contains strict extremum, similar to one done is SIFT. :math:`(N, C, 3, D_{out}, H_{out}, W_{out})`, :math:`(N, C, D_{out}, H_{out}, W_{out})`, where .. math:: D_{out} = \left\lfloor\frac{D_{in} + 2 \times \text{padding}[0] - (\text{kernel\_size}[0] - 1) - 1}{\text{stride}[0]} + 1\right\rfloor .. math:: H_{out} = \left\lfloor\frac{H_{in} + 2 \times \text{padding}[1] - (\text{kernel\_size}[1] - 1) - 1}{\text{stride}[1]} + 1\right\rfloor .. math:: W_{out} = \left\lfloor\frac{W_{in} + 2 \times \text{padding}[2] - (\text{kernel\_size}[2] - 1) - 1}{\text{stride}[2]} + 1\right\rfloor Examples: >>> input = torch.randn(20, 16, 3, 50, 32) >>> nms_coords, nms_val = conv_quad_interp3d(input, 1.0) rxrrFrrryrr!r diff)ordermoder7).r).r).r!g?).r ).ry).r NrpT)dimkeepdimgffffff?g?)r!r)rr!) r"r|r}r~rrrrrr#rrr reshaperr9rrrabsrr rmasked_scatterr max masked_fill_rbmmflipr:r<)rwrtrMBCHDHWrrAdxxdyydssdxydysdxsHesnms_maskx_solvedx_solved_masked_solved_correctly new_nms_maskdxmask1dyy_maxdx_resrr&r&r'conv_quad_interp3d.sJ "    & & $  *(rcs8eZdZdZddfd d Zdd d ZdddZZS)ConvQuadInterp3dzwCalculate soft argmax 3d per window. See :func: `~kornia.geometry.subpix.conv_quad_interp3d` for details. rrrtr+rMrrOcst||_||_dSrP)rQrRrtrM)rSrtrMrTr&r'rRs  zConvQuadInterp3d.__init__rVcCs|jjd|jdS)Nz(strict_maxima_bonus=r^)rUr_rtr`r&r&r'raszConvQuadInterp3d.__repr__rbrrcCrrP)rrtrMrer&r&r'rfrzConvQuadInterp3d.forwardrr)rtr+rMr+rrOrh)rbrrrrjr&r&rTr'rs  rrP)rrrrrrrr) r2rrrrrrrrrrg)rwrrErFrGrFrHrFrIrJrKrLrMr+rNrLrrcrv)rwrrErsrGrsrHrsrIrJrKrLrMr+rNrLrtr+rrcr)rwrrIrrKrLrrr)rwrrtr+rMr+rr)0 __future__rtypingrr"torch.nn.functionalnn functionalr kornia.corerrrrrr r r r kornia.filters.sobelr kornia.geometry.conversionsrr kornia.utilsrrkornia.utils._compatrkornia.utils.helpersrdsntrrnmsrr(r1r6r@rArordrurrrrr&r&r&r'sR  ,       ! (39 t   X