o iH[@sdZddlZddlmZmZmZddlZddlZddZ ddZ dd Z d;d d Z dd!ed(ed)ed*ejd"ejf d+d,Zd-ejd"ejfd.d/Z d?d0ejd1ejd2ed3ed"eejejejejff d4d5Z!d6ejd7ed8efd9d:Z"dS)@zNetwork related utility tools.N)DictListTuplecCsNt|tjjrt|j}nt|tjr|j}n tdt || |S)zSend tensor into the device of the module. Args: m (torch.nn.Module): Torch module. x (Tensor): Torch tensor. Returns: Tensor: Torch tensor located in the same place as torch module. z3Expected torch.nn.Module or torch.tensor, bot got: ) isinstancetorchnnModulenext parametersdeviceTensor TypeErrortypeto)mxr r^/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/transformer/utils/nets_utils.py to_device s    rcCsxt|}tdd|D}|dj||g|dddR|}t|D]}||||d||df<q(|S)Perform padding for the list of tensors. Args: xs (List): List of Tensors [(T_1, `*`), (T_2, `*`), ..., (T_B, `*`)]. pad_value (float): Value for padding. Returns: Tensor: Padded tensor (B, Tmax, `*`). Examples: >>> x = [torch.ones(4), torch.ones(2), torch.ones(1)] >>> x [tensor([1., 1., 1., 1.]), tensor([1., 1.]), tensor([1.])] >>> pad_list(x, 0) tensor([[1., 1., 1., 1.], [1., 1., 0., 0.], [1., 0., 0., 0.]]) cs|]}|dVqdSrNsize.0rrrr 5zpad_list..rN)lenmaxnewrfill_range)xs pad_valuen_batchmax_lenpadirrrpad_list s .  r*c s&t|}t|dj}g}t|D]|tfdd|Dq|dj|g|R|}t|D]\|dkrJ||d|df<q4|dkrf||d|dd|df<q4|dkr||d|dd|dd|df<q4td ||S) rrc3s|]}|VqdSNrrr)rrrVrz#pad_list_all_dim..rNzApad_list_all_dim only support 1-D, 2-D and 3-D tensors, not {}-D.) rshaper#appendr r!r"r ValueErrorformat)r$r%r&num_dimmax_len_all_dimr(rr,rpad_list_all_dim>s"   0@r5c s.dkr tdt|ts|}tt|}|dur/|dur)tt|}n|}n|dus5J|tt|ks?Jt j d|t j d}| d ||}|| d}||k}|dur|d|ksqJ|d|fdkr{|tfddt|D} || ||j}|S)a Make mask tensor containing indices of padded part. Args: lengths (LongTensor or List): Batch of lengths (B,). xs (Tensor, optional): The reference tensor. If set, masks will be the same shape as this tensor. length_dim (int, optional): Dimension indicator of the above tensor. See the example. Returns: Tensor: Mask tensor containing indices of padded part. dtype=torch.uint8 in PyTorch 1.2- dtype=torch.bool in PyTorch 1.2+ (including 1.2) Examples: With only lengths. >>> lengths = [5, 3, 2] >>> make_pad_mask(lengths) masks = [[0, 0, 0, 0 ,0], [0, 0, 0, 1, 1], [0, 0, 1, 1, 1]] With the reference tensor. >>> xs = torch.zeros((3, 2, 4)) >>> make_pad_mask(lengths, xs) tensor([[[0, 0, 0, 0], [0, 0, 0, 0]], [[0, 0, 0, 1], [0, 0, 0, 1]], [[0, 0, 1, 1], [0, 0, 1, 1]]], dtype=torch.uint8) >>> xs = torch.zeros((3, 2, 6)) >>> make_pad_mask(lengths, xs) tensor([[[0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 1]], [[0, 0, 0, 1, 1, 1], [0, 0, 0, 1, 1, 1]], [[0, 0, 1, 1, 1, 1], [0, 0, 1, 1, 1, 1]]], dtype=torch.uint8) With the reference tensor and dimension indicator. >>> xs = torch.zeros((3, 6, 6)) >>> make_pad_mask(lengths, xs, 1) tensor([[[0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1]], [[0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]], [[0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]]], dtype=torch.uint8) >>> make_pad_mask(lengths, xs, 2) tensor([[[0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 1]], [[0, 0, 0, 1, 1, 1], [0, 0, 0, 1, 1, 1], [0, 0, 0, 1, 1, 1], [0, 0, 0, 1, 1, 1], [0, 0, 0, 1, 1, 1], [0, 0, 0, 1, 1, 1]], [[0, 0, 1, 1, 1, 1], [0, 0, 1, 1, 1, 1], [0, 0, 1, 1, 1, 1], [0, 0, 1, 1, 1, 1], [0, 0, 1, 1, 1, 1], [0, 0, 1, 1, 1, 1]]], dtype=torch.uint8) rzlength_dim cannot be 0: {}Ndtyper6c3s(|]}|dfvrtdndVqdSr)slice)rr) length_dimrrrs&z make_pad_mask..)r1r2rlisttolistintrr rrarangeint64 unsqueezeexpandr!dimtupler# expand_asrr ) lengthsr$r;maxlenbs seq_rangeseq_range_expandseq_length_expandmaskindrr:r make_pad_maskhs,V      rNcCst|||S)a Make mask tensor containing indices of non-padded part. Args: lengths (LongTensor or List): Batch of lengths (B,). xs (Tensor, optional): The reference tensor. If set, masks will be the same shape as this tensor. length_dim (int, optional): Dimension indicator of the above tensor. See the example. Returns: ByteTensor: mask tensor containing indices of padded part. dtype=torch.uint8 in PyTorch 1.2- dtype=torch.bool in PyTorch 1.2+ (including 1.2) Examples: With only lengths. >>> lengths = [5, 3, 2] >>> make_non_pad_mask(lengths) masks = [[1, 1, 1, 1 ,1], [1, 1, 1, 0, 0], [1, 1, 0, 0, 0]] With the reference tensor. >>> xs = torch.zeros((3, 2, 4)) >>> make_non_pad_mask(lengths, xs) tensor([[[1, 1, 1, 1], [1, 1, 1, 1]], [[1, 1, 1, 0], [1, 1, 1, 0]], [[1, 1, 0, 0], [1, 1, 0, 0]]], dtype=torch.uint8) >>> xs = torch.zeros((3, 2, 6)) >>> make_non_pad_mask(lengths, xs) tensor([[[1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 0]], [[1, 1, 1, 0, 0, 0], [1, 1, 1, 0, 0, 0]], [[1, 1, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0]]], dtype=torch.uint8) With the reference tensor and dimension indicator. >>> xs = torch.zeros((3, 6, 6)) >>> make_non_pad_mask(lengths, xs, 1) tensor([[[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0]], [[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]], [[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]]], dtype=torch.uint8) >>> make_non_pad_mask(lengths, xs, 2) tensor([[[1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 0]], [[1, 1, 1, 0, 0, 0], [1, 1, 1, 0, 0, 0], [1, 1, 1, 0, 0, 0], [1, 1, 1, 0, 0, 0], [1, 1, 1, 0, 0, 0], [1, 1, 1, 0, 0, 0]], [[1, 1, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0]]], dtype=torch.uint8) )rN)rFr$r;rrrmake_non_pad_masksVrOcCs^|dt|ks J|jj||}t|D]\}}||d|f||d|f<q|S)a}Mask tensor according to length. Args: xs (Tensor): Batch of input tensor (B, `*`). lengths (LongTensor or List): Batch of lengths (B,). fill (int or float): Value to fill masked part. Returns: Tensor: Batch of masked input tensor (B, `*`). Examples: >>> x = torch.arange(5).repeat(3, 1) + 1 >>> x tensor([[1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5]]) >>> lengths = [5, 3, 2] >>> mask_by_length(x, lengths) tensor([[1, 2, 3, 4, 5], [1, 2, 3, 0, 0], [1, 2, 0, 0, 0]]) rN)rrdatar!r" enumerate)r$rFfillretr)lrrrmask_by_length6s rUcCst|tjr|jjdkrddlm}||St|St|t r@ddlm}d|vs.d|vr7t d t |||d|dSt|tj rH|Sd t|}zddlm}Wn tybt |wt||rj|St |)a{Change to torch.Tensor or ComplexTensor from numpy.ndarray. Args: x: Inputs. It should be one of numpy.ndarray, Tensor, ComplexTensor, and dict. Returns: Tensor or ComplexTensor: Type converted inputs. Examples: >>> xs = np.ones(3, dtype=np.float32) >>> xs = to_torch_tensor(xs) tensor([1., 1., 1.]) >>> xs = torch.ones(3, 4, 5) >>> assert to_torch_tensor(xs) is xs >>> xs = {'real': xs, 'imag': xs} >>> to_torch_tensor(xs) ComplexTensor( Real: tensor([1., 1., 1.]) Imag; tensor([1., 1., 1.]) ) cr) ComplexTensorrealimagzhas 'real' and 'imag' keys: {}zox must be numpy.ndarray, torch.Tensor or a dict like {{'real': torch.Tensor, 'imag': torch.Tensor}}, but got {})rnpndarrayr8kindtorch_complex.tensorrWr from_numpydictr1r2r<r r Exception)rrWerrorrrrto_torch_tensorUs.          rbcCs|dkr tdgS|dkr3|dkr3tj|jdtjd}tdtddd d |D|S|d kr;|d vsK|dkrC|dksK|d kr|dkrtj|jdtjd}|j dr|j ds|j d}tt|jdt|D] }t||||<qtntdtdddd |D|S|d kr|dkrtj|j|jdtjd}|j dr|j ds|j d}tt|j|jdt|D] }t||||<qntdtdddd |D|S|d krd|dkrdg}t|jD]a}tj|j|dtjd}|j | drD|j | dsD|j | d}tt|j|dt|D] }t||||<q7ntd|dtdddd |D||q|Std||)a\Parse the subsampling factors from the args for the specified `mode` and `arch`. Args: train_args: argument Namespace containing options. mode: one of ('asr', 'mt', 'st') arch: one of ('rnn', 'rnn-t', 'rnn_mix', 'rnn_mulenc', 'transformer') Returns: np.ndarray / List[np.ndarray]: subsampling factors. transformerrmtrnnr7z5Subsampling is not performed for machine translation.z subsample:  cSg|]}t|qSrstrrrrr z!get_subsample..asr)rezrnn-tstpvgg_zTSubsampling is not performed for vgg*. It is performed in max pooling layers at CNN.cSrgrrhrrrrrjrkrnn_mixcSrgrrhrrrrrjrk rnn_mulencz`Encoder %d: Subsampling is not performed for vgg*. It is performed in max pooling layers at CNN.cSrgrrhrrrrrjrkz!Invalid options: mode={}, arch={})rZarrayoneselayersint32loggingwarninginfojoinetypeendswith startswith subsamplesplitr#minrr> elayers_sdnum_encsr0r1r2) train_argsmodearchr~ssjsubsample_listidxrrr get_subsamplesd    "$ r old_prefix new_prefix state_dictcs`fdd|D}t|dkrtdd||D]}||}||}|||<qdS)z9Replace keys of old prefix with new prefix in state dict.csg|] }|r|qSr)r})rkrrrrjsz%rename_state_dict..rzRename: z -> N)rrwrxpopreplace)rrrold_keysrvnew_krrrrename_state_dicts    rcsFeZdZdZd dededdffdd Zd ejdejfd d Z Z S)SwishaSwish activation definition. Swish(x) = (beta * x) * sigmoid(x) where beta = 1 defines standard Swish activation. References: https://arxiv.org/abs/2108.12943 / https://arxiv.org/abs/1710.05941v1. E-swish variant: https://arxiv.org/abs/1801.07145. Args: beta: Beta parameter for E-Swish. (beta >= 1. If beta < 1, use standard Swish). use_builtin: Whether to use PyTorch function if available. ?Fbeta use_builtinreturnNcsLt|_|dkrfdd_dS|rtj_dSdd_dS)Nrcsj|t|Sr+)rrsigmoidrselfrrrkz Swish.__init__..cSs|t|Sr+)rrrrrrr s)super__init__rswishrrSiLU)rrr __class__rrrs zSwish.__init__rcCs ||S)zForward computation.)r)rrrrrforwards z Swish.forward)rF) __name__ __module__ __qualname____doc__floatboolrrr r __classcell__rrrrrs rcCs*tjjtjjtjjtjjtd}||S)zReturn activation function.)hardtanhtanhreluselur)rrHardtanhTanhReLUSELUr)actactivation_funcsrrrget_activations rcs2eZdZdZdedededdffdd ZZS) TooShortUttErrorzRaised when the utt is too short for subsampling. Args: message: Error message to display. actual_size: The size that cannot pass the subsampling. limit: The size limit for subsampling. message actual_sizelimitrNcst|||_||_dS)z$Construct a TooShortUttError module.N)rrrr)rrrrrrrr+s  zTooShortUttError.__init__)rrrrrir>rrrrrrr!s& r sub_factorrrcCs@|dkr |dkr dS|dkr|dkrdS|dkr|dkrdSd S) a Check if the input is too short for subsampling. Args: sub_factor: Subsampling factor for Conv2DSubsampling. size: Input size. Returns: : Whether an error should be sent. : Size limit for specified subsampling factor. r-r.)Tr )Tr)Fr6r)rrrrrcheck_short_utt3s r input_sizecCsj|dkrdd|dddfS|dkr dd|ddddfS|dkr1dd|ddddfStd)a-Get conv2D second layer parameters for given subsampling factor. Args: sub_factor: Subsampling factor (1/X). input_size: Input size. Returns: : Kernel size for second convolution. : Stride for second convolution. : Conv2DSubsampling output size. r-r.rrrz?subsampling_factor parameter should be set to either 2, 4 or 6.)r1)rrrrrsub_factor_to_paramsIs r chunk_sizeleft_chunk_sizer cCsrtj|||tjd}t|D]'}|dkrd}n t||||d}t||d||}d||||f<q|S)aCreate chunk mask for the subsequent steps (size, size). Reference: https://github.com/k2-fsa/icefall/blob/master/icefall/utils.py Args: size: Size of the source mask. chunk_size: Number of frames in chunk. left_chunk_size: Size of the left context in chunks (0 means full context). device: Device for the mask tensor. Returns: mask: Chunk mask. (size, size) )r r8rrT)rzerosrr#r r)rrrr rLr)startendrrrmake_chunk_mask`s rrFcCs8|}|d}t||||}||dkS)zCreate source mask for given lengths. Reference: https://github.com/k2-fsa/icefall/blob/master/icefall/utils.py Args: lengths: Sequence lengths. (B,) Returns: : Mask for the sequence lengths. (B, max_len) rr)r rrr?rBrrA)rFr' batch_sizeexpanded_lengthsrrrmake_source_masks rlabelsencoder_out_lens ignore_idblank_idc sd dttjdtfdd}|j}fdd|D}|d|g|fdd|D||}|||tj|}t t t|}t ||} t d d|D|} ||| | fS) aGet Transducer loss I/O. Args: labels: Label ID sequences. (B, L) encoder_out_lens: Encoder output lengths. (B,) ignore_id: Padding symbol ID. blank_id: Blank symbol ID. Returns: decoder_in: Decoder inputs. (B, U) target: Target label ID sequences. (B, U) t_len: Time lengths. (B,) u_len: Label lengths. (B,) rr padding_valuecSstt|}|dj|tdd|Dg|dddR|}t|D]}||||d||df<q&|S)zCreate padded batch of labels from a list of labels sequences. Args: labels: Labels sequences. [B x (?)] padding_value: Padding value. Returns: labels: Batch of padded labels sequences. (B,) rcsrrrrrrrrrz;get_transducer_task_io..pad_list..rN)rr!r rr"r#)rrrpaddedr)rrrr*s   z(get_transducer_task_io..pad_listcsg|]}||kqSrrry)rrrrjsz*get_transducer_task_io..csg|] }tj|gddqS)rrC)rcat)rlabel)blankrrrjscSsg|]}|dqSrrrrrrrjsNr) rrr r>r r!rrrvr<map IntTensor) rrrrr*r labels_unpad decoder_intargett_lenu_lenr)rrrget_transducer_task_ios rtpad_lenrCcCsR|||kr |St|j}|||||<tj|tj||j|jdg|dS)zMPad the tensor `t` at `dim` to the length `pad_len` with right padding zeros.)r8r r)rr<r/rrrr8r )rrrCpad_sizerrr pad_to_lens  $r)Nr6N)Nr6rr)r6r)#rrwtypingrrrnumpyrZrrr*r5rNrOrUrbrrir rrrrrr`rr>rrrr rrrrrrrrs^ * u YD K #  " A