o €o™ir#ã@sÊddlmZddlZddlmZddlmZddlmZm Z ddl m Z ddl m Z ddlmZdd lmZmZdd lmZmZdd lmZdd lmZGd d„deƒZGdd„deƒZGdd„dƒZdS)é)Ú annotationsN)Únn)Ú functional)ÚModuleÚTensor)Ú PinholeCamera)ÚRay)ÚPositionalEncoder)Úsample_lengthsÚsample_ray_points)ÚIrregularRendererÚRegularRenderer)Útorch_inference_mode)Úcreate_meshgridcs.eZdZdZdd‡fd d „ Zddd„Z‡ZS)ÚMLPaXClass to represent a multi-layer perceptron. The MLP represents a deep NN of fully connected layers. The network is build of user defined sub-units, each with a user defined number of layers. Skip connections span between the sub-units. The model follows: Ben Mildenhall et el. (2020) at https://arxiv.org/abs/2003.08934. ééé€Únum_dimsÚintÚ num_unitsÚnum_unit_layersÚ num_hiddenÚreturnÚNonec s†tƒ ¡||_g}t|ƒD],}|dkr|n||}t|ƒD]}|dkr&|n|} t | |¡} | t | t ¡¡¡qqt  |¡|_ dS)aConstruct MLP. Args: num_dims: Number of input dimensions (channels). num_units: Number of sub-units. num_unit_layers: Number of fully connected layers in each sub-unit. num_hidden: Layer hidden dimensions. rN) ÚsuperÚ__init__Ú_num_unit_layersÚrangerÚLinearÚappendÚ SequentialÚReLUÚ ModuleListÚ_mlp) ÚselfrrrrÚlayersÚiÚnum_unit_inp_dimsÚjÚnum_layer_inp_dimsÚlayer©Ú __class__©úJ/home/ubuntu/.local/lib/python3.10/site-packages/kornia/nerf/nerf_model.pyr,s    ýz MLP.__init__ÚxrcCsP|}|}t|jƒD]\}}|dkr!||jdkr!tj||fdd}||ƒ}q |S)Nréÿÿÿÿ©Údim)Ú enumerater$rÚtorchÚcat)r%r0ÚoutÚx_skipr'r+r.r.r/ÚforwardAs z MLP.forward)rrr) rrrrrrrrrr)r0rrr©Ú__name__Ú __module__Ú __qualname__Ú__doc__rr9Ú __classcell__r.r.r,r/r"s rcs<eZdZdZ       dd‡fdd„ Zddd„Z‡ZS)Ú NerfModela-Class to represent NeRF model. Args: num_ray_points: Number of points to sample along rays. irregular_ray_sampling: Whether to sample ray points irregularly. num_pos_freqs: Number of frequencies for positional encoding. num_dir_freqs: Number of frequencies for directional encoding. num_units: Number of sub-units. num_unit_layers: Number of fully connected layers in each sub-unit. num_hidden: Layer hidden dimensions. log_space_encoding: Whether to apply log spacing for encoding. Té rrrÚnum_ray_pointsrÚirregular_ray_samplingÚboolÚ num_pos_freqsÚ num_dir_freqsrrrÚlog_space_encodingrrc s tƒ ¡||_||_|jrtƒntƒ|_td||d|_td||d|_ t |jj |||ƒ|_ t  ||¡|_t  t  ||j j |d¡t  ¡¡|_t j|ddd|_tj j |jjj¡t dg¡ ¡|jj_t  t  |dd¡t  ¡¡|_t gd¢¡ ¡|jd j_dS) Né)Ú log_spaceréT)Úbiasçš™™™™™¹?)ç{®Gáz”?rMrMr)rrÚ_num_ray_pointsÚ_irregular_ray_samplingr r Ú _rendererr Ú _pos_encoderÚ _dir_encoderrÚnum_encoded_dimsr$rrÚ_fc1r!r"Ú_fc2Ú_sigmar5ÚinitÚxavier_uniform_ÚweightÚdataÚtensorÚfloatrKÚSigmoidÚ_rgb) r%rBrCrErFrrrrGr,r.r/rZs ÿ zNerfModel.__init__ÚoriginsrÚ directionsc CsÚ|jd}t||j|j|j|jd}t|||ƒ}| |¡}| t j |dd¡}|  |¡}|  |¡}|  |¡} | t | ¡d} t | ¡} tj||dddd…f d|jd¡fdd}| |¡}| |¡} | | | |¡} | S)zêForward method. Args: origins: Ray origins with shape :math:`(B, 3)`. directions: Ray directions with shape :math:`(B, 3)`. Returns: Rendered image pixels :math:`(B, 3)`. r)ÚdeviceÚdtypeÚ irregularr1r2rL.N)Úshaper rNrarbrOr rQrRÚFÚ normalizer$rTrVr5Ú randn_likeÚrelur6ÚexpandrUr^rP) r%r_r`Ú batch_sizeÚlengthsÚ points_3dÚpoints_3d_encodedÚdirections_encodedÚyÚdensities_ray_pointsÚrgbs_ray_pointsÚrgbsr.r.r/r9ys* û      ,  zNerfModel.forward)TrArrrrT)rBrrCrDrErrFrrrrrrrrGrDrr)r_rr`rrrr:r.r.r,r/r@Ks÷r@c@s8eZdZdZdd d „Zddd„Zddd„Zddd„ZdS)ÚNerfModelRendererzHRenders a novel synthesis view of a trained NeRF model for given camera.Ú nerf_modelr@Ú image_sizeútuple[int, int]raútorch.device | Nonerbútorch.dtype | NonerrcCs,||_||_||_||_| ¡\|_|_dS)zÛConstruct NerfModelRenderer. Args: nerf_model: NeRF model. image_size: image size. device: device to run the model on. dtype: dtype to run the model on. N)Ú _nerf_modelÚ _image_sizeÚ_deviceÚ_dtypeÚ_create_pixels_gridÚ _pixels_gridÚ_ones)r%rtrurarbr.r.r/r«s  zNerfModelRenderer.__init__útuple[Tensor, Tensor]cCsP|j\}}t||d|j|jd}| dd¡}tj|jdd|j|j d}||fS)zãCreate the pixels grid to unproject to plane z=1. Args: image_size: image size: tuple[int, int] Returns: - Pixels grid: Tensor (1, H, W, 2) - Ones: Tensor (H*W, 1) F)Únormalized_coordinatesrarbr1rrrJ)rarb) rzrr{r|Úreshaper5Úonesrdrarb)r%ÚheightÚwidthÚ pixels_gridrƒr.r.r/r}¾s ÿ z%NerfModelRenderer._create_pixels_gridÚcamerarrcCs^| |¡}tƒ| |j|j¡}Wdƒn1swY| |jd|jdd¡}|S)zêRender a novel synthesis view of a trained NeRF model for given camera. Args: camera: camera for image rendering: PinholeCamera. Returns: Rendered image with shape :math:`(H, W, 3)`. NrrJrH)Ú _create_raysrryÚoriginÚ directionÚviewrz)r%r‡ÚraysÚ rgb_modelÚ rgb_imager.r.r/Ú render_viewÓs ÿzNerfModelRenderer.render_viewrcCsJ|j\}}|jddd…df}| ||d¡}| |j|j¡}t ||¡S)zwCreate rays for a given camera. Args: camera: camera for image rendering: PinholeCamera. .NrHr1rJ)rzÚ extrinsicsÚrepeatÚ unprojectr~rrÚthrough)r%r‡r„r…r‰Ú destinationr.r.r/rˆès  zNerfModelRenderer._create_raysN) rtr@rurvrarwrbrxrr)rr€)r‡rrr)r‡rrr)r;r<r=r>rr}rrˆr.r.r.r/rs¨s    rs)Ú __future__rr5rÚtorch.nnrreÚ kornia.corerrÚkornia.geometry.camerarÚkornia.geometry.rayrÚkornia.nerf.positional_encoderr Úkornia.nerf.samplersr r Úkornia.nerf.volume_rendererr r Úkornia.utils._compatrÚkornia.utils.gridrrr@rsr.r.r.r/Ús        )]