o ¡¿¯iHã@s²dZddlZddlZ d+dd „Zd,d d „Zd-d d„Zd.dd„Z   d/dd„Zdd„Zdd„Z dd„Z dd„Z d0dd„Z dd„Z d d!„Zd"d#„Zd$d%„Zd&d'„Z )d1d*d(„ZdS)2a This implementation is modified from https://github.com/zcaceres/spec_augment MIT License Copyright (c) 2019 Zach Caceres Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETjjHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. éNééé(éFcCs$ttt||d|||d|||dS)a¥SpecAugment Reference: SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (https://arxiv.org/pdf/1904.08779.pdf) This implementation modified from https://github.com/zcaceres/spec_augment :param torch.Tensor spec: input tensor with the shape (T, dim) :param int W: time warp parameter :param int F: maximum width of each freq mask :param int T: maximum width of each time mask :param int num_freq_masks: number of frequency masks :param int num_time_masks: number of time masks :param bool replace_with_zero: if True, masked parts will be filled with 0, if False, filled with mean )ÚW)ÚFÚ num_masksÚreplace_with_zero)ÚTrr )Ú time_maskÚ freq_maskÚ time_warp)Úspecrrr Únum_freq_masksÚnum_time_masksr ©rúM/home/ubuntu/.local/lib/python3.10/site-packages/espnet/utils/spec_augment.pyÚspecaug"s ü÷rc CsÒ| d¡}|jd}|jd}|j}|d}|ddd…|f}t|ƒ|ks'J‚|t |||¡}t|tjƒs9J‚t | |¡}tj ||ggg|dtj |||ggg|d} } t || | ƒ\} } |   d¡  d¡S)zwTime warping :param torch.Tensor spec: input tensor with shape (T, dim) :param int W: time warp parameter rérN©Údeviceé) Ú unsqueezeÚshaperÚlenÚrandomÚ randrangeÚ isinstanceÚtorchÚTensorÚtensorÚsparse_image_warpÚsqueeze) rrÚspec_lenÚnum_rowsrÚyÚhorizontal_line_at_ctrÚ point_to_warpÚ dist_to_warpÚsrc_ptsÚdest_ptsÚwarped_spectroÚ dense_flowsrrrr Cs   þr rc Cs²| d¡ ¡}|jd}td|ƒD]B}t d|¡}t d||¡}|||kr.| d¡St |||¡} |rEd|ddd…|| …f<q| ¡|ddd…|| …f<q| d¡S)a Frequency masking :param torch.Tensor spec: input tensor with shape (T, dim) :param int F: maximum width of each mask :param int num_masks: number of masks :param bool replace_with_zero: if True, masked parts will be filled with 0, if False, filled with mean rrN©rÚclonerÚrangerrr"Úmean) rrrr ÚclonedÚnum_mel_channelsÚiÚfÚf_zeroÚmask_endrrrr _ó    r c Cs²| d¡ ¡}|jd}td|ƒD]B}t d|¡}t d||¡}|||kr.| d¡St |||¡} |rEd|d|| …dd…f<q| ¡|d|| …dd…f<q| d¡S)aTime masking :param torch.Tensor spec: input tensor with shape (T, dim) :param int T: maximum width of each mask :param int num_masks: number of masks :param bool replace_with_zero: if True, masked parts will be filled with 0, if False, filled with mean rrNr-) rr rr r1Ú len_spectror3ÚtÚt_zeror6rrrr {r7r çcCsV|j}||}|j\}} } t| | |ƒ} t||| ||ƒ} t| || | ƒ} t|| ƒ}|| fS©N)rrÚget_flat_grid_locationsÚinterpolate_splineÚcreate_dense_flowsÚdense_image_warp)Ú img_tensorÚsource_control_point_locationsÚdest_control_point_locationsÚinterpolation_orderÚregularization_weightÚnum_boundaries_pointsrÚcontrol_point_flowsÚ batch_sizeÚ image_heightÚ image_widthÚflattened_grid_locationsÚflattened_flowsr,Ú warped_imagerrrr!—s$ ÿûÿ r!cCsLtjd|d||d}tjd|d||d}t ||¡\}}t ||fd¡S)Nrrréÿÿÿÿ)rÚlinspaceÚmeshgridÚstack©rIrJrÚy_rangeÚx_rangeÚy_gridÚx_gridrrrÚget_grid_locations¸srWcCst |||dg¡S©Nr©rÚreshape)Úgrid_locationsrIrJrrrÚflatten_grid_locations¿sr\cCsZtjd|d||d}tjd|d||d}t ||¡\}}t ||fd¡ ||dg¡S)NrrrrNr)rrOrPrQrZrRrrrr=Ãsr=cCst ||||dg¡SrXrY)rLrHrIrJrrrr?Êsr?cCs&t||||ƒ\}}t|||||ƒ}|Sr<)Úsolve_interpolationÚapply_interpolation)Ú train_pointsÚ train_valuesÚ query_pointsÚorderrEÚwÚvÚ query_valuesrrrr>Ïsr>cCsF|j}|j\}}}|jd}|} | ¡} tt| | ƒ|ƒ d¡} tjd|j|d  gd¢¡} t  | | fd¡ ¡} t  | t  | dd¡fd¡}| jd}tj |||f|dd}t  | |fd¡}t  ||fd¡}tj ||d|f|j|d ¡}t  | |fd¡}t ||¡\}}|dd…d|…dd…f}|dd…|d…dd…f}||fS) NrNrr©Údtyper)rNrrrrg _ B)rrÚfloatÚphiÚcross_squared_distance_matrixrrÚonesrgÚviewÚcatÚ transposeÚrandnÚzerosÚgesv)r_r`rbrErÚbÚnÚdÚkÚcr4Úmatrix_arkÚmatrix_bÚ left_blockÚ num_b_colsÚ lhs_zerosÚ right_blockÚlhsÚ rhs_zerosÚrhsÚXÚLUrcrdrrrr]Þs.   ÿþr]cCs\t t ||¡¡}t t ||¡¡}t | d¡| d¡ dd¡¡}|d||}| ¡S)akPairwise squared distance between two (batch) matrices' rows (2nd dim). Computes the pairwise distances between rows of x and rows of y Args: x: [batch_size, n, d] float `Tensor` y: [batch_size, m, d] float `Tensor` Returns: squared_dists: [batch_size, n, m] float `Tensor`, where squared_dists[b,i,j] = ||x[b,i,:] - y[b,j,:]||^2 rrr)rÚsumÚmulÚmatmulr"rnrh)Úxr%Úx_norm_squaredÚy_norm_squaredÚ x_y_transposeÚ squared_distsrrrrjs   rjcCsÎtjd|jd}|dkrt ||¡}t |¡}|S|dkr*d|t t ||¡¡S|dkr>dt |¡t t ||¡¡S|ddkrYt ||¡}dt |d|¡t |¡St ||¡}t |d|¡S)aCoordinate-wise nonlinearity used to define the order of the interpolation. See https://en.wikipedia.org/wiki/Polyharmonic_spline for the definition. Args: r: input op order: interpolation order Returns: phi_k evaluated coordinate-wise on r, for k = r g»½×Ùß|Û=rrrgà?ér)rr rÚmaxÚsqrtÚlogÚsquareÚpow)ÚrrbÚEPSILONrrrris       ric Csp| d¡}t| ¡| ¡ƒ}t||ƒ}t ||¡}t |ddd…f¡}t ||fd¡ ¡} t | |¡} || S)a¥Apply polyharmonic interpolation model to data. Notes: Given coefficients w and v for the interpolation model, we evaluate interpolated function values at query_points. Args: query_points: `[b, m, d]` x values to evaluate the interpolation at train_points: `[b, n, d]` x values that act as the interpolation centers ( the c variables in the wikipedia article) w: `[b, n, k]` weights on each interpolation center v: `[b, d, k]` weights on each input dimension order: order of the interpolation Returns: Polyharmonic interpolation evaluated at points defined in query_points. r.Nrr)rrjrhrirr„Ú ones_likerm) rar_rcrdrbÚpairwise_distsÚphi_pairwise_distsÚrbf_termrkÚquery_points_padÚ linear_termrrrr^9s  ÿ   r^cCs®| d¡}|j\}}}}|j}t tj||dtj||d¡\}}tj||fdd ¡} |  d¡ dddd¡} | |} t  | |||dg¡} t || ƒ} t  | ||||g¡} | S)aImage warping using per-pixel flow vectors. Apply a non-linear warp to the image, where the warp is specified by a dense flow field of offset vectors that define the correspondences of pixel values in the output image back to locations in the source image. Specifically, the pixel value at output[b, j, i, c] is images[b, j - flow[b, j, i, 0], i - flow[b, j, i, 1], c]. The locations specified by this formula do not necessarily map to an int index. Therefore, the pixel value is obtained by bilinear interpolation of the 4 nearest pixels around (b, j - flow[b, j, i, 0], i - flow[b, j, i, 1]). For locations outside of the image, we use the nearest pixel values at the image boundary. Args: image: 4-D float `Tensor` with shape `[batch, height, width, channels]`. flow: A 4-D float `Tensor` with shape `[batch, height, width, 2]`. name: A name for the operation (optional). Note that image and flow can be of type tf.half, tf.float32, or tf.float64, and do not necessarily have to be the same type. Returns: A 4-D float `Tensor` with shape`[batch, height, width, channels]` and same type as input image. Raises: ValueError: if height < 2 or width < 2 or the inputs have the wrong number of dimensions. rrr)ÚdimrNrr) rrrrrPÚarangerQrhÚpermuterZÚinterpolate_bilinear)ÚimageÚflowrHÚheightÚwidthÚchannelsrÚgrid_xÚgrid_yÚ stacked_gridÚ batched_gridÚquery_points_on_gridÚquery_points_flattenedÚ interpolatedrrrr@]s ÿÿ r@r›Úijc#sf|dkr |dkr tdƒ‚|j}t|ƒdkr d}t|t|jƒƒ‚|j\‰}‰‰ˆ|ˆˆg}|j}|j}|j} |jd‰g} g} g} |dkrIddgnddg} | d¡}| D]n}||}||d}tj|d|| d }tjd || d }t  |t  |¡¡}t  ||¡}|  ¡}|   |¡|d}|   |¡tj|||| d }tjd || d }tjd || d }t  t  ||¡|¡}t |d¡}|   |¡qTt |ˆ|ˆˆg¡‰t tjˆ| d |ˆˆdg¡‰‡‡‡‡‡‡fd d„}|| d| ddƒ}|| d| ddƒ}|| d| ddƒ}|| d| ddƒ}| d|||} | d|||}!| d|!| | }"|"S)a¡Similar to Matlab's interp2 function. Notes: Finds values for query points on a grid using bilinear interpolation. Args: grid: a 4-D float `Tensor` of shape `[batch, height, width, channels]`. query_points: a 3-D float `Tensor` of N points with shape `[batch, N, 2]`. name: a name for the operation (optional). indexing: whether the query points are specified as row and column (ij), or Cartesian coordinates (xy). Returns: values: a 3-D `Tensor` with shape `[batch, N, channels]` Raises: ValueError: if the indexing mode is invalid, or if the shape of the inputs invalid. r¨Úxyz"Indexing mode must be 'ij' or 'xy'rŠz+Grid must be 4 dimensional. Received size: rrrrfr;gð?rcs4ˆ|ˆ|}t ˆ ¡d|¡}t |ˆˆˆg¡S)Nr)rÚgatherr9rZ)Úy_coordsÚx_coordsÚnameÚlinear_coordinatesÚgathered_values©Ú batch_offsetsrHr Úflattened_gridÚ num_queriesrŸrrrªåsz$interpolate_bilinear..gatherÚtop_leftÚ top_rightÚ bottom_leftÚ bottom_right)Ú ValueErrorrrÚstrrgrÚunbindrr r‹ÚfloorÚminÚlongÚappendrrZr™)#Úgridrar­ÚindexingrÚmsgržÚ query_typeÚ grid_typeÚ grid_deviceÚalphasÚfloorsÚceilsÚ index_orderÚunstacked_query_pointsr˜ÚqueriesÚsize_in_indexing_dimensionÚ max_floorÚ min_floorÚmaxxr»Ú int_floorÚceilÚalphaÚ min_alphaÚ max_alpharªr´rµr¶r·Ú interp_topÚ interp_bottomÚinterprr°rr›sb      ÿ     ÿ)rrrrrF)r)rrF)rrF)rr;r)r;)r›r¨)Ú__doc__rrrr r r r!rWr\r=r?r>r]rjrir^r@r›rrrrÚs2 ÿ !    ú! û($4ÿ