o }o™i»2ã@s˜dZddlZddlmmZddlmZe ddg¡e ddg¡dœZ dZ Gdd „d ejj ƒZ Gd d „d e ƒZ Gd d „d ejj ƒZGdd„deƒZdS)a The patcher and unpatcher implementation for 2D and 3D data. The idea of Haar wavelet is to compute LL, LH, HL, HH component as two 1D convolutions. One on the rows and one on the columns. For example, in 1D signal, we have [a, b], then the low-freq compoenent is [a + b] / 2 and high-freq is [a - b] / 2. We can use a 1D convolution with kernel [1, 1] and stride 2 to represent the L component. For H component, we can use a 1D convolution with kernel [1, -1] and stride 2. Although in principle, we typically only do additional Haar wavelet over the LL component. But here we do it for all as we need to support downsampling for more than 2x. For example, 4x downsampling can be done by 2x Haar and additional 2x Haar, and the shape would be. [3, 256, 256] -> [12, 128, 128] -> [48, 64, 64] éN)Ú rearrangegÍ;fž æ?gð?)ÚhaarrFcóDeZdZdZd‡fdd„ Zdd„Zdd d „Zd d „Zdd„Z‡Z S)ÚPatcheraeA module to convert image tensors into patches using torch operations. The main difference from `class Patching` is that this module implements all operations using torch, rather than python or numpy, for efficiency purpose. It's bit-wise identical to the Patching module outputs, with the added benefit of being torch.jit scriptable. ércó‚tƒ ¡||_||_|jdt|tdttt   t   |j¡¡  ¡ƒƒ|_|jdt   t|jd¡td| ¡D]}d|_q9dS©NÚwavelets©Ú persistentÚ_arangerF©ÚsuperÚ__init__Ú patch_sizeÚ patch_methodÚregister_bufferÚ _WAVELETSÚ _PERSISTENTÚrangeÚintÚtorchÚlog2ÚtensorÚitemÚarangeÚshapeÚ parametersÚ requires_grad©ÚselfrrÚparam©Ú __class__©úm/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/video_tokenizers/modules/patching.pyr3ó  ý ÿzPatcher.__init__cCó6|jdkr | |¡S|jdkr| |¡Std|jƒ‚©NrrzUnknown patch method: )rÚ_haarÚ_arrangeÚ ValueError©r Úxr$r$r%ÚforwardAó    zPatcher.forwardÚreflectFcCs`|j}|j}|jd}t|jdƒ}| d¡ ddd¡ |dd¡}|d|j ddd¡ |dd¡} | j|d} |j|d}t j ||d|d|d|df|d |¡}t j ||  d¡|dd} t j ||   d¡|dd} t j | |  d ¡|d d} t j | |   d ¡|d d} t j | |  d ¡|d d}t j | |   d ¡|d d}t j| | ||gdd }|r®|d}|S) Nrréÿÿÿÿ©Údtypeé©ÚpadÚmode©rr4©ÚgroupsÚstrideé©r4r©Údim)r3r rrÚflipÚreshapeÚrepeatr ÚtoÚFr6Úconv2dÚ unsqueezerÚcat)r r-r7Úrescaler3ÚhÚnÚgÚhlÚhhÚxlÚxhÚxllÚxlhÚxhlÚxhhÚoutr$r$r%Ú_dwtIs& "  .z Patcher._dwtcCs|jD] }|j|dd}q|S)NT©rH)rrU©r r-Ú_r$r$r%r)as z Patcher._haarcCst|d|j|jd ¡}|S)Nz$b c (h p1) (w p2) -> b (c p1 p2) h w©Úp1Úp2)rrÚ contiguousr,r$r$r%r*fsüûzPatcher._arrange©rr©r0F) Ú__name__Ú __module__Ú __qualname__Ú__doc__rr.rUr)r*Ú __classcell__r$r$r"r%r)s  rcó<eZdZdZd‡fdd„ Zddd „Zd d „Zd d „Z‡ZS)Ú Patcher3DzZA 3D discrete wavelet transform for video data, expects 5D tensor, i.e. a batch of videos.rrcs6tƒj||d|jd|tjdgtjdtddS)N©rrÚpatch_size_bufferrr2r )rrrrÚonesÚint32r©r rrr"r$r%rss  ýzPatcher3D.__init__r0Fc Cs |j}|j}|jd}t|jdƒ}| d¡ ddd¡ |dd¡} |d|j ddd¡ |dd¡} | j|d} | j|d} t j |t d|dƒ|d|d|d|d|df|d |¡}t j ||   d¡  d¡|d d } t j ||   d¡  d¡|d d } t j | |   d¡  d¡|d d } t j | |   d¡  d¡|d d }t j | |   d¡  d¡|d d }t j | |   d¡  d¡|d d }t j | |   d¡  d¡|d d }t j | |   d¡  d¡|d d }t j ||   d¡  d¡|d d }t j ||   d¡  d¡|d d }t j ||   d¡  d¡|d d }t j ||   d¡  d¡|d d }t j ||   d¡  d¡|d d }t j ||   d¡  d¡|d d }tj||||||||gdd }|rN|dt t d¡¡}|S)Nrrr1r2r4r5r<é©r4rrr9©rr4r©rrr4r>ç@)r3r rrr@rArBr rCrDr6ÚmaxÚconv3drFrrGÚsqrtr)r r-Úwaveletr7rHr3rIrJrKrLrMrNrOrPrQrRrSÚxlllÚxllhÚxlhlÚxlhhÚxhllÚxhlhÚxhhlÚxhhhrTr$r$r%rU{s6 "  @zPatcher3D._dwtcCs`tj|d|jddgdd\}}tj|j|jdd|gdd}|jD] }|j|ddd}q#|S)Nrr4r>rTrV)rÚsplitrrGÚrepeat_interleaverrrU)r r-ÚxiÚxvrXr$r$r%r)Ÿs " zPatcher3D._haarcCs`tj|d|jddgdd\}}tj|j|jdd|gdd}t|d|j|j|jd ¡}|S)Nrr4r>z0b c (t p1) (h p2) (w p3) -> b (c p1 p2 p3) t h w©rZr[Úp3)rr|rrGr}rrr\)r r-r~rr$r$r%r*¦s"ûúzPatcher3D._arranger]r^) r_r`rarbrrUr)r*rcr$r$r"r%reps  $recr)Ú UnPatcherahA module to convert patches into image tensorsusing torch operations. The main difference from `class Unpatching` is that this module implements all operations using torch, rather than python or numpy, for efficiency purpose. It's bit-wise identical to the Unpatching module outputs, with the added benefit of being torch.jit scriptable. rrcrrr rr"r$r%r½r&zUnPatcher.__init__cCr'r()rÚ_ihaarÚ _iarranger+r,r$r$r%r.Ër/zUnPatcher.forwardr0Fc Cs¢|j}|j}|jd}t|jddƒ}| dg¡ ddd¡ |ddg¡} |d|j ddd¡ |dd¡} | j|d} | j|d} t j | |¡ddd\} } } }t j j j | |  d¡|d|d dfd }|t j j j | |  d¡|d|d dfd 7}t j j j | |  d¡|d|d dfd }|t j j j ||  d¡|d|d dfd 7}t j j j ||  d ¡|d d|d fd }|t j j j ||  d ¡|d d|d fd 7}|rÏ|d }|S) Nrrrkr1r2r>r<r=r4)r:r;Úpaddingr8)r3r rrr@rArBr rCrÚchunkÚnnÚ functionalÚconv_transpose2drF)r r-rsr7rHr3rIrJrKrLrMrPrQrRrSÚylÚyhÚyr$r$r%Ú_idwtÓs$ ""  &*&*&*zUnPatcher._idwtcCs |jD] }|j|ddd}q|S)NrTrV)rrrWr$r$r%rƒìs zUnPatcher._ihaarcCst|d|j|jd}|S)Nz$b (c p1 p2) h w -> b c (h p1) (w p2)rY©rrr,r$r$r%r„ñsüzUnPatcher._iarranger]©rr0F) r_r`rarbrr.rrƒr„rcr$r$r"r%r‚³s  r‚crd)Ú UnPatcher3DzIA 3D inverse discrete wavelet transform for video wavelet decompositions.rrcstƒj||ddS)Nrf)rrrjr"r$r%rþszUnPatcher3D.__init__r0FcCs||j}|j}t|jddƒ}| dg¡ ddd¡ |ddg¡}|d|j ddd¡ |dd¡} |j|d}| j|d} t j |ddd\} } } } }}}}t j | |  d¡  d¡|d d }|t j | |   d¡  d¡|d d 7}t j | |  d¡  d¡|d d }|t j | |   d¡  d¡|d d 7}t j ||  d¡  d¡|d d }|t j ||   d¡  d¡|d d 7}t j ||  d¡  d¡|d d }|t j ||   d¡  d¡|d d 7}t j ||  d¡  d ¡|d d }|t j ||   d¡  d ¡|d d 7}t j ||  d¡  d ¡|d d }|t j ||   d¡  d ¡|d d 7}t j ||  d¡  d ¡|d d }|t j ||   d¡  d ¡|d d 7}|r<|dt  t  d¡¡}|S)Nrérr1r2r>r4r<rnr9rkrmrlro)r3r rrr@rArBr rCrr†rDÚconv_transpose3drFrrr)r r-rsr7rHr3rIrKrLrMrtrurvrwrxryrzr{rPrQrRrSrNrOr$r$r%rs2""   """""""zUnPatcher3D._idwtcCsB|jD] }|j|ddd}q|dd…dd…|jdd…df}|S)NrTrVr.)rrrrWr$r$r%rƒ(s "zUnPatcher3D._ihaarcCs>t|d|j|j|jd}|dd…dd…|jdd…df}|S)Nz0b (c p1 p2 p3) t h w -> b c (t p1) (h p2) (w p3)r€r.rŽr,r$r$r%r„.sû"zUnPatcher3D._iarranger]r) r_r`rarbrrrƒr„rcr$r$r"r%rûs  'r)rbrÚtorch.nn.functionalr‡rˆrDÚeinopsrrrrÚModulerrer‚rr$r$r$r%Ús    þGCH