o oiO @sddlZddlmZmZmZddlZddlmZmZm Z ddl m Z ddl m Z ddlmZddlmZGdd d ZGd d d eZGd d d eZGdddeZ d!dedededejdedef ddZdedededefddZdedefdd ZdS)"N)DictListTuple)DeviceTensortensor) PinholeCamera)cameras_for_ids)torch_meshgrid)_torch_inverse_castc @szeZdZUdZeed<eed<eed<eed<eed<eed<ded ed ed ed e j d df ddZ e d efddZ e d efddZe d efddZe d efddZd efddZdeded efddZGdd d Zded!eeefd dfd"d#Zded eeeffd$d%ZGd&d'd'Zed(ed)ed*ed+ed,eeefd df d-d.Zed,eeefd eeeffd/d0ZdS)1 RaySampleraFClass to manage spatial ray sampling. Args: min_depth: sampled rays minimal depth from cameras: float max_depth: sampled rays maximal depth from cameras: float ndc: convert ray parameters to normalized device coordinates: bool device: device for ray tensors: Union[str, torch.device] _origins _directions_directions_cam _origins_cam _camera_ids _points_2d min_depth max_depthndcdevicedtypereturnNcCs"||_||_||_||_||_dSN) _min_depth _max_depth_ndc_device_dtypeselfrrrrrr!H/home/ubuntu/.local/lib/python3.10/site-packages/kornia/nerf/samplers.py__init__0s  zRaySampler.__init__cC|jSr)r r r!r!r"origins7zRaySampler.originscCr$r)rr%r!r!r" directions;r'zRaySampler.directionscCr$rrr%r!r!r" camera_ids?r'zRaySampler.camera_idscCr$rrr%r!r!r" points_2dCr'zRaySampler.points_2dcCs|jdurdS|jjdS)Nr)r&shaper%r!r!r"__len__Gs  zRaySampler.__len__camerasr,c Csr|j}|j}|j}|j}|d|d|d}|d|d|d}t|} tj||| gdd} | ddS)N.r.N.dim)fxfycxcytorch ones_likestackreshape) r r/r,r8r9r:r; directions_x directions_y directions_zdirections_camr!r!r"_calc_ray_directions_camLs  z#RaySampler._calc_ray_directions_camc@sReZdZdZdedeeddfddZedefdd Z edeefd d Z dS) zRaySampler.Points2DaAA class to hold ray 2d pixel coordinates and a camera id for each. Args: points_2d: tensor with ray pixel coordinates (the coordinates in the image plane that correspond to the ray):math:`(B, 2)` camera_ids: list of camera ids for each pixel coordinates: List[int] r,r*rNcCs||_||_dSr)rr)r r,r*r!r!r"r#cs zRaySampler.Points2D.__init__cCr$rr+r%r!r!r"r,gr'zRaySampler.Points2D.points_2dcCr$rr)r%r!r!r"r*kr'zRaySampler.Points2D.camera_ids) __name__ __module__ __qualname____doc__rrintr#propertyr,r*r!r!r!r"Points2DYs rKpoints_2d_camerac Csg}g}g}g}g}g}|D]} | jjdd\} } tj| d| d|j|jd|j} |j| dd| df<t || j } | | j ddd| }| |dd| ddfdd| |d| dddf|dd| ddfdd| || | j| |d|j| t| j  | dddddd| | jddqt||_t||_t||_t||_t||_|jr||\|_|_t||_dS) a,Calculate ray parameters: origins, directions. Also stored are camera ids for each ray, and its pixel coordinates. Args: cameras: scene cameras: PinholeCamera points_2d_camera: a dictionary that groups Point2D objects by total number of casted rays Nr7rrr3.r4r)valuesrr-r<onesrrrrr r* unprojectrepeatappendr?rDrpermutesqueezerIcatr rrrrr!transform_ray_params_world_to_ndc)r r/rLr&r(rC origins_camr*r,objnum_cams_groupnum_points_per_cam_groupdepthscams points_3dr!r!r"_calc_ray_paramsosF   $,&     zRaySampler._calc_ray_paramscCs,t||j}|j}|j}|j}|j}d||d}d||d}|jd|jd} |jd|jd} || } || } dd|j|jd} tj | | | gdd }t |j }||j d j dd }|d|d}|d|d}||| }||| }d| }tj |||gdd }|}|}||fS) zTransform ray parameters to normalized coordinate device (camera) system (NDC). Args: cameras: scene cameras: PinholeCamera g@?r0).rMr2r3rMr4r5r1)r rr8r9widthheightr rr<r>r rotation_matrixrrU)r r/r]r8r9widthsheights fx_widths fy_heightsoxozoyoz origins_ndc_x origins_ndc_y origins_ndc_z origins_ndcRt_invdirections_rotated_worlddxdzdydzdirections_ndc_xdirections_ndc_ydirections_ndc_zdirections_ndcorigins_ndc_worlddirections_ndc_worldr!r!r"rWs0    z,RaySampler.transform_ray_params_world_to_ndcc@seZdZdZdddZdS)zRaySampler.Points2D_FlatTensorszJClass to hold x/y pixel coordinates for each ray, and its scene camera id.rNcCs||g|_dSrr)r%r!r!r"r#s z(RaySampler.Points2D_FlatTensors.__init__)rN)rErFrGrHr#r!r!r!r"Points2D_FlatTensorssrxnxy camera_idpoints2d_as_flat_tensorscCs||vrt||<|||_|||_nt||j|f||_t||j|f||_||j|dS)zAdd x/y pixel coordinates for all rays casted by a scene camera to dictionary of pixel coordinates grouped by total number of rays. N) r rxflatten_x_yr<rVrrS)ryrzr{r|r}r!r!r"-_add_points2d_as_flat_tensors_to_num_ray_dicts z8RaySampler._add_points2d_as_flat_tensors_to_num_ray_dictcCsZi}|D]$\}}t|j}t|j|jfdd|dd}t ||j||<q|S)aNBuild a dictionary of ray pixel points, by total number of rays as key. The dictionary groups rays by the total amount of rays, which allows the case of casting different number of rays from each scene camera. Args: points2d_as_flat_tensors: dictionary of pixel coordinates grouped by total number of rays: Dict[int, Points2D_FlatTensors] Returns: dictionary of Points2D objects that holds information on pixel 2d coordinates of each ray and the camera id it was casted by: Dict[int, Points2D] r3rr4rM) itemslenrr<r>rrrTr?r rK)r}num_ray_dict_of_points2drypoints2d_as_flat_tensornum_camsr,r!r!r"_build_num_ray_dict_of_points2ds  z*RaySampler._build_num_ray_dict_of_points2d)rErFrGrHr__annotations__floatboolrr<rr#rJr&r(r*r,rIr.rrDrKrr_rrWrx staticmethodrrr!r!r!r"r sV  $ 15    r c sveZdZdZdededededejddf fd d Z d e d e d e de e e jffddZded e ddfddZZS)RandomRaySamplera>Class to manage random ray spatial sampling. Args: min_depth: sampled rays minimal depth from cameras: float max_depth: sampled rays maximal depth from cameras: float ndc: convert to normalized device coordinates: bool device: device for ray tensors: Union[str, torch.device] rrrrrrNct|||||dSrsuperr#r __class__r!r"r#zRandomRaySampler.__init__rerd num_img_raysc Cs|}i}tt|||D].\}\}}}ttj||j|jd|} ttj||j|jd|} t || | ||qt |S)aCRandomly sample pixel points in 2d. Args: heights: tensor that holds scene camera image heights (can vary between cameras): math: `(B)`. widths: tensor that holds scene camera image widths (can vary between cameras): math: `(B)`. num_img_rays: tensor that holds the number of rays to randomly cast from each scene camera: math: `(B)`. Returns: dictionary of Points2D objects that holds information on pixel 2d coordinates of each ray and the camera id it was casted by: Dict[int, Points2D] rN) rI enumerateziptolistr<truncrandrrr rr) r rerdrr}r|rbraryy_randx_randr!r!r"sample_points_2ds *  z!RandomRaySampler.sample_points_2dr/cCsP|j}||jdkrtd|d|jd||j|j|}|||dS)a?Calculate ray parameters: origins, directions. Also stored are camera ids for each ray, and its pixel coordinates. Args: cameras: scene cameras: PinholeCamera num_img_rays: tensor that holds the number of rays to randomly cast from each scene camera: int math: `(B)`. rzNumber of cameras zR does not match size of tensor to define number of rays to march from each camera N) batch_sizer- ValueErrorrrbrar_)r r/rrrLr!r!r"calc_ray_params0s z RandomRaySampler.calc_ray_params)rErFrGrHrrrr<rr#rrrIr rKrrr __classcell__r!r!rr"r s ( $rc s`eZdZdZdededededejddf fd d Z d e d e d e de e e jffddZZS)RandomGridRaySampleraClass to manage random ray spatial sampling. Sampling is done on a regular grid of pixels by randomizing column and row values, and casting rays for all pixels along the selected ones. Args: min_depth: sampled rays minimal depth from cameras: float max_depth: sampled rays maximal depth from cameras: float ndc: convert to normalized device coordinates: bool device: device for ray tensors: Union[str, torch.device] rrrrrrNcrrrrrr!r"r#RrzRandomGridRaySampler.__init__rerdrcCs|}i}tt|||D]M\}\}}}tt|} tjt||j|j ddt t|| } tjt||j|j ddt t|| } t | | gdd\} } t | | | | ||qt |S)aRandomly sample pixel points in 2d over a regular row-column grid. Args: heights: tensor that holds scene camera image heights (can vary between cameras): math: `(B)`. widths: tensor that holds scene camera image widths (can vary between cameras): math: `(B)`. num_img_rays: tensor that holds the number of rays to randomly cast from each scene camera. Number of rows and columns is the square root of this value: int math: `(B)`. Returns: dictionary of Points2D objects that holds information on pixel 2d coordinates of each ray and the camera id it was casted by: Dict[int, Points2D] rNNijindexing)rIrrrmathsqrtr<randpermrrminr r rr)r rerdrr}r|rbraryn_sqrtrry_gridx_gridr!r!r"rUs*** z%RandomGridRaySampler.sample_points_2d)rErFrGrHrrrr<rr#rrrIr rKrrr!r!rr"rDs( ,rc sveZdZdZdededededejddf fd d Z dd e d e de de e e jffddZdeddfddZZS)UniformRaySamplera[Class to manage uniform ray spatial sampling for all camera scene pixels. Args: min_depth: sampled rays minimal depth from cameras: float max_depth: sampled rays maximal depth from cameras: float ndc: convert to normalized device coordinates: bool device: device for ray tensors: Union[str, torch.device] rrrrrrNcrrrrrr!r"r#{rzUniformRaySampler.__init__r3rerd sampling_stepc Cs|}|}i}tt||D]1\}\}}||}ttjd|||j|jdtjd|||j|jdgdd\} } t || | ||qt |S)a=Uniformly sample pixel points in 2d for all scene camera pixels. Args: heights: tensor that holds scene camera image heights (can vary between cameras): math: `(B)`. widths: tensor that holds scene camera image widths (can vary between cameras): math: `(B)`. sampling_step: defines uniform strides between rows and columns: int. Returns: dictionary of Points2D objects that holds information on pixel 2d coordinates of each ray and the camera id it was casted by: Dict[int, Points2D] rrNrr) rIrrrr r<arangerrr rr) r rerdrr}r|rbraryrrr!r!r"r~s"   z"UniformRaySampler.sample_points_2dr/cCs ||j|j}|||dSr)rrbrar_)r r/rLr!r!r"rsz!UniformRaySampler.calc_ray_params)r3)rErFrGrHrrrr<rr#rrIrr rKrrrrr!r!rr"rps(    rFnum_raysnum_ray_pointsrr irregularrcCst|dkrtd|stjdd|||d}||d}|Stjdd|d||d}tj|||d||dd}|S) z'Sample points along the length of rays.r3z+Number of ray points must be greater than 1gr`rN)rNr4)rr<linspacerRr)rrrrr zero_to_onelengthsr!r!r"sample_lengthss  rr&r(rcCs0|ddddf|d|ddddf}|S)aSample points along ray. Args: origins: tensor containing ray origins in 3d world coordinates. Tensor shape :math:`(*, 3)`. directions: tensor containing ray directions in 3d world coordinates. Tensor shape :math:`(*, 3)`. lengths: tensor containing sampled distances along each ray. Tensor shape :math:`(*, num_ray_points)`. Returns: points_3d: Points along rays :math:`(*, num_ray_points, 3)` .Nr1r!)r&r(rr^r!r!r"sample_ray_pointss,rr^cCs,tjj||ddddfddd}|S)zCalculate t values along rays. Args: points_3d: Points along rays :math:`(*, num_ray_points, 3)` Returns: t values along rays :math:`(*, num_ray_points)` .rNr4r5)r<linalgnorm unsqueeze)r^t_valsr!r!r"calc_ray_t_valss( r)F)rtypingrrrr< kornia.corerrrkornia.geometry.camerarkornia.nerf.camera_utilsr kornia.utils._compatr kornia.utils.helpersr r rrrrIrrrrrr!r!r!r"sJ    n9,4