o %ݫi@sdZddlZddlZddlZddlmZmZmZddlZ ddl Z ddl m m ZddlmmZddl mZm Z ddlmZmZddlmZeeZGddde jZd d Zd e jd e jfd dZde fddZ!Gddde jZ"Gddde jZ#Gddde jZ$Gddde j%j&Z'ddZ(Gddde jZ)Gddde jZ*Gd d!d!e jZ+d"e jd dfd#d$Z,Gd%d&d&Z-dS)'a]This lobe enables the integration of pretrained BEATs: Audio Pre-Training with Acoustic Tokenizers. Reference: https://arxiv.org/abs/2212.09058 Based on Github source: https://github.com/microsoft/unilm/tree/master/beats You could download the checkpoints from: https://github.com/microsoft/unilm/tree/master/beats Author * Pooneh Mousavi 2024 N)DictOptionalTuple)Tensornn) LayerNorm Parameter)length_to_maskc seZdZdZ   ddedededdffd d Zd ejd ejdejfd dZ  ddejde de dejfddZ   ddejde ejde de fddZ   ddejde ejde de dejf ddZZS)BEATsai BEATs: Audio Pre-Training with Acoustic Tokenizers. This class implements the BEATs model, which processes audio signals for feature extraction or downstream tasks. The model supports loading from a checkpoint, applying normalization, and optionally freezing parameters. Arguments --------- ckp_path : str, optional Path to the checkpoint file. If None, the model initializes without pre-trained weights. You could download the checkpoints from : https://github.com/microsoft/unilm/tree/master/beats freeze : bool, optional (default: False) If True, the model parameters are frozen and the model is set to evaluation mode. output_all_hiddens : bool, optional (default: False) If True, the forward function outputs hidden states from all transformer layers. For example BEATs_iter3 has 12 transformer layers and the output is of shape (13, B, T, C), where a projection of the CNN output is added to the beginning. If False, the forward function outputs the hidden states only from the last transformer layer. Example ------- >>> audio = torch.randn(4, 10000) # Batch of 4 audio signals >>> length = torch.tensor([1.0, 0.5, 0.75, 1.0]) >>> model = BEATs() >>> outputs = model.extract_features(audio, length)[0] >>> outputs.shape torch.Size([4, 24, 768]) NTFckp_pathfreezeoutput_all_hiddensreturncsjtd\}}|r$tj|std|dt|}|dd}t ||_ t d|j j ||_||_|j j|_|j|j jkrNt|j|j jnd|_|j j|_tjd|j|j|j|j jd|_t|j j|_|j jrz|j jrzJdt|j |_t |j|_!|j j"rt|j j#|_#t|j j|j j$|_%nd|_%|r|&|d |jr|'dSdS) NNNzCheckpoint file 'z' does not exist.cfgzBEATs Config: ) kernel_sizestridebiaszLConfiguration error: 'deep_norm' and 'layer_norm_first' cannot both be True.model)(super__init__ospathexistsFileNotFoundErrortorchloadget BEATsConfigrloggerinfo__dict__r r embed_dimembedencoder_embed_dimrLinearpost_extract_projinput_patch_sizeConv2d conv_biaspatch_embeddingDropout dropout_input deep_normlayer_norm_firstTransformerEncoderencoderr layer_normfinetuned_modelpredictor_dropoutpredictor_class predictorload_state_dicteval)selfr r r r checkpoint __class__R/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/beats.pyr=s\            zBEATs.__init__features padding_maskcCsV|d|d}|dkr|ddd| f}||d|dd}|dS)ak Adjusts the padding mask for the given features. Arguments --------- features : torch.Tensor Input features after patch embedding. padding_mask : torch.Tensor Original padding mask for input signals. Returns ------- torch.Tensor Adjusted padding mask. rrN)sizeviewall)r9r?r@extrar=r=r>forward_padding_masks zBEATs.forward_padding_maskP.@(9@source fbank_mean fbank_stdcCsXg}|D]}|dd}tj|ddddd}||qtj|dd}||d |S) a Preprocesses the input waveform by extracting filter banks and applying normalization. Arguments --------- source : torch.Tensor Input waveform signals. fbank_mean : float, optional Mean value for filter bank normalization (default: 15.41663). fbank_std : float, optional Standard deviation for filter bank normalization (default: 6.55582). Returns ------- torch.Tensor Normalized filter banks. rii> ) num_mel_binssample_frequency frame_length frame_shiftdim) unsqueezeta_kaldifbankappendrstack)r9rIrJrKfbankswaveformrXr=r=r> preprocesss zBEATs.preprocesswavwav_lenscCsN|jrt|||||WdS1swY|||||S)aYTakes an input waveform and return its corresponding beats encoding. Arguments --------- wav : torch.Tensor A batch of audio signals to transform to features. wav_lens : torch.Tensor The relative length of the wav given in SpeechBrain format. fbank_mean : float, optional Mean value for filter bank normalization (default: 15.41663). fbank_std : float, optional Standard deviation for filter bank normalization (default: 6.55582). Returns ------- BEATs encoded features. N)r rno_gradextract_features)r9r^r_rJrKr=r=r>forwards  z BEATs.forwardcCs||||}|dur|d}t||||jd}|dur'|||}|d}||}||j d|j dd dd}| |}|durQ|||}|j dur[| |}| |}|j|||jd\} } |jdur|| } || } |dur|rd| |<| jdd} | |jddd| } n| jdd} t| } |jrtj| dd} | | |fS|jrtj| dd} | fS) ar Extracts features from the input waveform. Arguments --------- wav : torch.Tensor A batch of audio signals to transform to features. wav_lens : torch.Tensor The relative length of the wav given in SpeechBrain format. fbank_mean : float, optional Mean value for filter bank normalization (default: 15.41663). fbank_std : float, optional Standard deviation for filter bank normalization (default: 6.55582). Returns ------- torch.Tensor Extracted features from the BEATs model. NrAdevicerrrU)r@r rS)r]rBr rdboolrFrVr+reshapeshape transposer2r'r-r1r r6r4anysum expand_asmeanrsigmoidrZ)r9r^r_rJrKrXmax_lenr@r?x layer_resultsx_dlogitslprobsr=r=r>ras^                 zBEATs.extract_features)NTF)rGrH)NrGrH)__name__ __module__ __qualname____doc__strrerrrrFfloatr]rrbra __classcell__r=r=r;r>r st E  ( %r c CsHttdstdtjt_d|dttj|dt|dS)aB Applies the Gaussian Error Linear Unit (GELU) activation function using an accurate approximation. Arguments --------- x: torch.Tensor Input tensor on which to apply the GELU activation. Returns ------- torch.Tensor: Tensor with GELU activation applied element-wise. _arUg?rgHm?) hasattr gelu_accuratemathsqrtpir{rtanhpowror=r=r>r~6s "r~rorcCstjj||S)a Applies the Gaussian Error Linear Unit (GELU) activation function. Arguments --------- x: torch.Tensor Input tensor to apply the GELU activation. Returns ------- torch.Tensor Tensor with GELU activation applied element-wise. )rr functionalgelurytype_asrr=r=r>rNsr activationcCsx|dkrtjS|dkr tS|dkrtdtS|dkrtS|dkr%tjS|dkr-dd S|d kr5d d Std |) aE Returns the activation function corresponding to the provided activation name. Arguments --------- activation : str Name of the activation function. Supported values: - "relu": Applies ReLU activation. - "gelu": Applies the GELU activation. - "gelu_fast": Alias for `gelu_accurate` with a deprecation warning. - "gelu_accurate": Applies the accurate GELU activation. - "tanh": Applies the Tanh activation. - "linear": Applies the identity function. - "glu": Applies the identity function (GLU placeholder). Returns ------- Callable[[torch.Tensor], torch.Tensor] The corresponding activation function to apply to input tensors. Raises ------ RuntimeError If the specified activation function is not supported. relur gelu_fastz;--activation-fn=gelu_fast has been renamed to gelu_accurater~rlinearcS|SNr=rr=r=r>z#get_activation_fn..glucSrrr=rr=r=r>rrz --activation-fn {} not supported) Frrr warnr~rr RuntimeErrorformat)rr=r=r>get_activation_fn_s(rcs*eZdZdZdfdd ZddZZS)SamePada Implements a module that adjusts the padding of a tensor after convolution to maintain its original size, with an option for causal padding. This is particularly useful for handling padding in convolutional layers where the kernel size or causality affects the output size. Arguments --------- kernel_size : int The size of the convolutional kernel. causal : bool, optional (default=False) If True, applies causal padding by removing `(kernel_size - 1)` elements from the end of the tensor. If False, removes elements to center-align the padding, ensuring the output size matches the input size. Fcs6t|r|d|_dS|ddkrdnd|_dS)NrrUr)rrremove)r9rcausalr;r=r>rs zSamePad.__init__cCs,|jdkr|ddddd|j f}|S)a Adjusts the padding of the input tensor `x`. If `self.remove > 0`, the method slices the tensor along the last dimension to remove excess padding based on the `kernel_size` and `causal` settings. Arguments --------- x : torch.Tensor The input tensor to adjust padding for. Returns ------- torch.Tensor The tensor with adjusted padding. rN)rr9ror=r=r>rbs zSamePad.forward)Frtrurvrwrrbrzr=r=r;r>rsrcs(eZdZdZfddZddZZS)Swisha% Implements the Swish activation function as a PyTorch module. Swish is a smooth, non-monotonic activation function defined as: Swish(x) = x * sigmoid(x) It is often used in deep learning for its ability to improve training performance in certain architectures. cstt|tj|_dSr)rrrrrSigmoidactr9r;r=r>rszSwish.__init__cCs|||S)aI Applies the Swish activation function to the input tensor. Arguments --------- x : torch.Tensor The input tensor to which the Swish activation is applied. Returns ------- torch.Tensor The input tensor after applying the Swish activation. )rrr=r=r>rbsz Swish.forwardrr=r=r;r>rs rcs$eZdZdZ dfdd ZZS) GLU_Lineara Implements a Gated Linear Unit (GLU) combined with a linear transformation. Arguments --------- input_dim : int The dimensionality of the input features. output_dim : int The dimensionality of the output features. glu_type : str, optional (default="sigmoid") The type of activation function used for gating. Supported values are: - "sigmoid": Uses the sigmoid activation function. - "swish": Uses the Swish activation function. - "relu": Uses the ReLU activation function. - "gelu": Uses the GELU activation function. bias_in_glu : bool, optional (default=True) Whether to include a bias term in the linear transformation. rmTcstt|||_||_|dkrtj|_n|dkr!t |_n|dkr,tj |_n |dkr6tj |_|rDt ||dd|_ dSt ||dd|_ dS)NrmswishrrrUTF)rrrglu_type output_dimrrrglu_actrReLUGELUr&r)r9 input_dimrr bias_in_glur;r=r>rs  zGLU_Linear.__init__)rmT)rtrurvrwrrzr=r=r;r>rsrc@s(eZdZdZeddZeddZdS) GradMultiplya; A custom autograd function that scales gradients during the backward pass. This is useful for scenarios where gradient scaling is required without affecting the forward pass output. The forward pass returns the input as-is, while the backward pass scales the gradients by a specified factor. cCs||_||}|S)a Performs the forward pass of the GradMultiply function. Arguments --------- ctx : torch.autograd.Function The context object to store information for the backward computation. x : torch.Tensor The input tensor to be forwarded unchanged. scale : float The factor by which the gradients will be scaled during the backward pass. Returns ------- torch.Tensor A new tensor identical to the input tensor. )scalenew)ctxrorresr=r=r>rbs zGradMultiply.forwardcCs||jdfS)a Performs the backward pass, scaling the gradients by the stored factor. Arguments --------- ctx : torch.autograd.Function The context object containing the stored scaling factor. grad : torch.Tensor The gradient tensor from the subsequent layer. Returns ------- Tuple[torch.Tensor, None] The scaled gradient tensor and None (for the scale input, which has no gradient). N)r)rgradr=r=r>backward/szGradMultiply.backwardN)rtrurvrw staticmethodrbrr=r=r=r>rs  rcCs|dkr|St|tjtjtjfsJ|jjdk}|s+|jd|dks)JddS|jdkr=|j |dks;JddS|jd|jd}||dksQJddS) a  Wraps modules and applies quantization noise to their weights for subsequent quantization using Iterative Product Quantization (iPQ). This approach is described in the paper: "Training with Quantization Noise for Extreme Model Compression." It introduces quantization noise during training to improve model robustness for extreme weight compression scenarios. Arguments --------- module : nn.Module The module to which quantization noise will be applied. Supported modules are Linear, Embedding, and Conv2d. p : float The amount of quantization noise to apply. Typically a probability or scaling factor. block_size : int The size of the blocks for subsequent quantization with iPQ. Returns ------- None rrz0Input features must be a multiple of block sizes)rrz0Input channels must be a multiple of block sizesz,Kernel size must be a multiple of block sizeN) isinstancerr& Embeddingr)weightndimrBr in_channels)modulep block_sizeis_convkr=r=r> quant_noiseCs   rcs4eZdZdZfddZd ddZd ddZZS) r0z Implements the Transformer Encoder module. Arguments --------- args : Namespace or dict A collection of model hyperparameters and configurations. csTtj_j_tjjjjjdjd_ d}t dd|jj}tj j j jd|dtj j jdtjjj ddd_ tj tjt_ td roj_j_j_n d _d_d_tfd d tjD_jrtd jD]}j|j`jdjjj|j_qj _ t!j_"j#_$%t&j'r!t (djd}tjD]T}tj j)j|jj*jd dtj j)j|jj+j|dtj j)j|jj,jd dtj j)j|jj-j|dtj j)j|j.j|dtj j)j|j/j|dqt0dd _1dS)NrU)rpaddinggroupsrr?rlstdr)namerTrelative_position_embeddingFcsLg|]"}tjjjjjjjjj j j j j jdqS)) embedding_dimffn_embedding_dimnum_attention_headsdropoutattention_dropoutactivation_dropout activation_fnr/r.has_relative_attention_bias num_buckets max_distance gru_rel_posencoder_layers)TransformerSentenceEncoderLayerrencoder_ffn_embed_dimencoder_attention_headsrrrrr/r.rrrrr).0iargsr9r=r> s&z/TransformerEncoder.__init__..rgпgainlayer_wise_gradient_decay_ratio)2rrrr%rrConv1dconv_posconv_pos_groupspos_convrrinitnormal_r constant_rutils weight_norm Sequentialrrr}rrr ModuleListrangerlayers self_attnrelative_attention_biasr/rr2encoder_layerdrop layerdropapplyinit_bert_paramsr.rxavier_normal_k_projv_projq_projout_projfc1fc2getattrr)r9rrrrdeep_norm_betar;rr>rs        zTransformerEncoder.__init__NcCs.||||\}}|jr|r||}||fS)aN Processes the input sequence through the Transformer Encoder layers. Arguments --------- x : torch.Tensor The input tensor of shape `(seq_len, batch_size, embed_dim)` containing the input embeddings. padding_mask : torch.Tensor, optional A binary mask of shape `(batch_size, seq_len)` indicating which positions are padding and should be ignored in attention computations. Default is `None`. output_all_hiddens : bool, optional If True, returns the hidden states from all encoder layers in addition to the final output. Default is `None`. Returns ------- Tuple[torch.Tensor, List[torch.Tensor]] - The final output tensor of shape `(seq_len, batch_size, embed_dim)`. )rar/r2)r9ror@r rpr=r=r>rbs   zTransformerEncoder.forwardc Cs |durd||<||dd}|dd}||}|js#||}tj||j|jd}|dd}g}d}|r>||d}d}t|j D]-\} } |j dkrWt ||j }t j} |jrd| |jkro| ||d|d\}}}||qG|dur{|}|dd}||fS) a Extracts features from the input sequence using positional convolution, layer normalization, dropout, and a series of Transformer Encoder layers. Arguments --------- x : torch.Tensor The input tensor of shape `(batch_size, seq_len, embed_dim)` containing the input embeddings. padding_mask : torch.Tensor, optional A binary mask of shape `(batch_size, seq_len)` indicating which positions are padding and should be ignored in computations. Default is `None`. output_all_hiddens : bool, optional If True, collects and returns the hidden states from all encoder layers in addition to the final output. Default is `None`. Returns ------- Tuple[torch.Tensor, List[torch.Tensor]] - The final output tensor of shape `(batch_size, seq_len, embed_dim)`. NrrrU)rtrainingrF)self_attn_padding_mask need_weightspos_bias)rrhr/r2rrrrY enumeraterrrrnprandomr) r9ror@r x_convrpzrrrlayerdropout_probabilityr=r=r>ras>        z#TransformerEncoder.extract_featuresr)rtrurvrwrrbrarzr=r=r;r>r0|s  a r0c!seZdZdZ               d"d ed ed ed ed edededededededededededdf fdd Z    d#de j de j de j defd d!Z Z S)$ra Implements a single Transformer Sentence Encoder layer. Arguments --------- embedding_dim : float, optional (default=768) The dimensionality of input embeddings. ffn_embedding_dim : float, optional (default=3072) The dimensionality of the feed-forward network's hidden layer. num_attention_heads : float, optional (default=8) The number of attention heads for self-attention. dropout : float, optional (default=0.1) The dropout rate applied to the output of the feed-forward network and attention layers. attention_dropout : float, optional (default=0.1) The dropout rate applied within the attention mechanism. activation_dropout : float, optional (default=0.1) The dropout rate applied after the activation function in the feed-forward network. activation_fn : str, optional (default="relu") The activation function used in the feed-forward network. Supported values include "relu" and "gelu". layer_norm_first : bool, optional (default=False) If True, applies layer normalization before attention and feed-forward layers; otherwise, applies it afterward. deep_norm : bool, optional (default=False) If True, uses deep normalization scaling for residual connections. has_relative_attention_bias : bool, optional (default=False) If True, includes relative position bias in the attention mechanism. num_buckets : int, optional (default=0) The number of buckets used for relative attention bias (if enabled). max_distance : int, optional (default=0) The maximum distance for relative attention bias (if enabled). rescale_init : bool, optional (default=False) If True, rescales parameter initialization for improved stability. gru_rel_pos : bool, optional (default=False) If True, incorporates GRU-style relative position encoding. encoder_layers : int, optional (default=0) The number of encoder layers in the Transformer.  r皙?rFrrrrrrrrr/r.rrr rescale_initrrrNc st||_||_||_||_t||_t|j||d| | | | |d |_ t ||_ t |j|_ t ||_||_t|j|_|jdkrOt|j|d|_nt |j||_t ||j|_t|j|_| |_|jrvtd|d|_dSd|_dS)NT)rself_attentionrrrrrrrrUg?r)rrrrractivation_namerrMultiheadAttentionrrr,dropout1dropout2dropout3r/rself_attn_layer_normrrr&rfinal_layer_normr.rrdeep_norm_alpha)r9rrrrrrrr/r.rrrrrrr;r=r>rqsB        z(TransformerSentenceEncoderLayer.__init__roself_attn_maskrrc CsJ|}|jrP||}|j||||d||d\}}}||}||}|}||}|jdkr4||}n|||}||}| |}| |}||}nP|j|||||||d\}}}||}||j |}||}|}|jdkr}||}n|||}||}| |}| |}||j |}||}|||fS)ai Processes the input tensor through the Transformer sentence encoder layer. Arguments --------- x : torch.Tensor Input tensor of shape `(seq_len, batch_size, embed_dim)`. self_attn_mask : torch.Tensor, optional Mask for the self-attention mechanism, typically used for causal or padding masking. Default is `None`. self_attn_padding_mask : torch.Tensor, optional Padding mask of shape `(batch_size, seq_len)`, indicating which tokens should be ignored in attention computations. Default is `None`. need_weights : bool, optional (default=False) Whether to return attention weights. If `True`, attention weights are included in the output. pos_bias : optional Positional bias for relative attention, if applicable. Default is `None`. Returns ------- Tuple[torch.Tensor, torch.Tensor, optional] - `x` (torch.Tensor): The output tensor of shape `(seq_len, batch_size, embed_dim)` after applying the encoder layer. F)querykeyvaluekey_padding_maskr attn_mask position_biasr) r/r rr rrrrr rr r)r9rorrrrresidualattnr=r=r>rbsZ"                  z'TransformerSentenceEncoderLayer.forward)rrrrrrrFFFrrFFr)NNFN) rtrurvrwryrxreintrrrrbrzr=r=r;r>rKs~'     ArcseZdZdZ               d7fd d Zd d Z d8d dZdededej fddZ        d9de de e de e de e de e e e e e e ffdedede e dedede e dee e e e e ffdd Zd!d"Zd#d$Zd:d%d&Z   d;d'd(Zede e d)e e d*ed+edede e f d,d-Zde e e e e e e ffde e e e ffd.d/Zde e e e e e ffd0e e e e ffd1d2Zd3ed+ed4efd5d6ZZS)r5sf      zMultiheadAttention.__init__cCs|jr1tjj|jjdtddtjj|jjdtddtjj|j jdtddntj|jjtj|jjtj|j jtj|j j|j j dur`tj |j j d|j durltj|j |jdurxtj|j|jrtj|jjdSdS)ze Initializes the weights for the projection layers and relative position embeddings. rrUrNr)rrrxavier_uniform_rrrrrrrrrr&rr'rrrr=r=r>r,s"    z#MultiheadAttention.reset_parametersc Cs|j}|j}d}|r |d}||dktj|7}t|}n t|t| }|d}||k}|t| |t ||||tj}t|t ||d}|t |||7}|S)aComputes bucket indices for relative positions for relative attention bias. Arguments --------- relative_positions : torch.Tensor A tensor of relative positions, where negative values indicate positions to the left and positive values indicate positions to the right. bidirectional : bool, optional, (default: True) If True, separate buckets are used for positive and negative positions. Returns ------- torch.Tensor A tensor of the same shape as `relative_positions`, where each value is the bucket index corresponding to the relative position. rrUr) rrtorlongabsmin zeros_likelogryr full_likewhere) r9relative_positions bidirectionalrrrelative_buckets max_exactis_smallrelative_position_if_larger=r=r>_relative_positions_buckets@    z-MultiheadAttention._relative_positions_bucket query_length key_lengthrcCsztj|tjddddf}tj|tjddddf}||}|j|dd}||jjj}||}|gd}|S)a Computes relative position bias for attention scores. Arguments --------- query_length : int The length of the query sequence. key_length : int The length of the key sequence. Returns ------- torch.Tensor A tensor of shape `(num_heads, query_length, key_length)` containing the relative position bias values for each attention head. )dtypeNT)r:)rUrr) raranger2r?r1rrrdpermute)r9r@rAcontext_positionmemory_positionrelative_positionrelative_position_bucketvaluesr=r=r> compute_biass zMultiheadAttention.compute_biasrrrrincremental_stater static_kvrbefore_softmaxneed_head_weightsrc  Cs | rd}|\} } }| }||jksJt|| | |gks!J|durJ|\}}}tjsJ|| ks7J|dus=J|sJJ| |jddk|jrj| durj|| |} | d | ddd | |j | |} |dur| |}|durd|vr|r|jr|jrJd}}nd}d}|j|||| | ||dd\}}}}}|durRd|vr|d}|dusJ| | |j d |j}|r|}n|dusJtj||gdd }|d}d |vr|d }|dusJ| | |j d |j}|r|}n|dusJtj||gdd }d}d |vr|d }|dur|dusJtj||| |d|d }| | |j d |j|d<| | |j d |j|d <||d <|dusLJ|||}|dusYJ|d|kscJ||||||| | || \}}| rz||| fS||||| | | |||| | \}}||| fS)a Forward pass for multi-head attention with support for relative position embeddings, caching, and optional dropout. This method implements the core functionality of multi-head attention with optional features such as relative position bias, incremental decoding, and support for various masking options. Arguments --------- query : torch.Tensor Query tensor of shape `(target_length, batch_size, embed_dim)`. key : torch.Tensor, optional Key tensor of shape `(source_length, batch_size, embed_dim)`. Defaults to `None`. value : torch.Tensor, optional Value tensor of shape `(source_length, batch_size, embed_dim)`. Defaults to `None`. key_padding_mask : torch.Tensor, optional Mask to exclude padding keys, of shape `(batch_size, source_length)`, where padding elements are indicated by 1s. Defaults to `None`. incremental_state : dict, optional Stores cached key and value tensors for incremental decoding. Defaults to `None`. need_weights : bool, optional If True, returns the attention weights. Defaults to `True`. static_kv : bool, optional If True, the key and value tensors remain static for incremental decoding. Defaults to `False`. attn_mask : torch.Tensor, optional Attention mask to prevent certain positions from attending, typically for causal attention. Shape: `(target_length, source_length)`. Defaults to `None`. before_softmax : bool, optional If True, returns raw attention scores before softmax. Defaults to `False`. need_head_weights : bool, optional If True, returns attention weights for each head. Implies `need_weights=True`. Defaults to `False`. position_bias : torch.Tensor, optional Precomputed position bias tensor. If `None`, it is computed during the forward pass. Returns ------- attn : torch.Tensor Attention output of shape `(target_length, batch_size, embed_dim)`. attn_weights : torch.Tensor, optional Attention weights of shape `(batch_size, num_heads, target_length, source_length)`, averaged across heads if `need_head_weights=False`. position_bias : torch.Tensor, optional Computed or passed relative position bias of shape `(num_heads, target_length, source_length)`. TNrUrrprev_keyr)alpharArS prev_valueprev_key_padding_mask)rrR batch_sizesrc_lenrL)rBr#listrjit is_scriptingrgrrJrVrepeatrCr_get_input_bufferr%r_prepare_attention_inputsr!catr _append_prev_key_padding_mask_set_input_buffer_process_attention_weights_compute_attention_output)r9rrrrrKrrLrrMrNrtgt_lenbszr#rTkey_bsz_ saved_staterPqrv _prev_keyrO _prev_valuerQrR attn_weightsrr=r=r>rbs@                  zMultiheadAttention.forwardc  Csx|dur_|} |jdkrT|||j||j| |j} | \}}}}t|| |||ddj dddj ddd\}}|||j d d }|||j|d|} | |} || }t j |dd}||}||}|dusvJt||}t|||j||jgksJ|d d|||}||}d}| r|||j||dd }| s|jd d}||fS) a- Computes the final attention output, including relative position bias adjustments, attention weight computation, and attention projection. Arguments --------- q : torch.Tensor Query tensor. v : torch.Tensor Value tensor. attn_weights : torch.Tensor Attention weights tensor. position_bias : Optional[torch.Tensor] Relative position bias tensor. bsz : int Batch size. tgt_len : int Target sequence length. src_len : int Source sequence length. embed_dim : int Embedding dimension. need_weights : bool Whether to return attention weights. need_head_weights : bool Whether to return head-specific weights. alpha : float Scaling factor for relative position. Returns ------- Tuple[torch.Tensor, Optional[torch.Tensor]] Final attention output and optional attention weights. NrrUrrAF)keepdimrSrg@r)rrCrr"r$rBrrmr)rjchunkr+rsoftmaxrr bmmrUr!rh contiguousrrl)r9rerfrirrar`rTr#rrNrPattn_mask_rel_pos query_layer_B_H_L__gate_agate_bgate_a_1attn_weights_float attn_probsrattn_weights_outr=r=r>r_s^1           z,MultiheadAttention._compute_attention_outputc Cs&|jjdk} |dur|dkrd}|dur(|d|ksJ|d|ks(J|jr|dus1J|d7}tj|||ddf|ddgdd}tj|||ddf|ddgdd}|dur{tj|||ddgdd}|durtj|t|dd |gdd}t || dd} | | j ddd d| } | | |||} t| ||j||gksJ|dur|d}| |7} |dur| ||j||} | s| |ddtjtd } n| dd} | |td } | dd} | ||j||} | |fS) ao Processes attention weights, including handling key padding masks, adding zero attention if required, and computing the attention weights with masking. Arguments --------- q : torch.Tensor Query tensor. k : torch.Tensor Key tensor. v : torch.Tensor Value tensor. attn_mask : torch.Tensor Attention mask key_padding_mask : torch.Tensor Key padding mask. bsz : int Batch size. tgt_len : int Target sequence length. src_len : int Source sequence length. alpha : float Scaling factor for relative position. Returns ------- Tuple[torch.Tensor, Optional[torch.Tensor]] Computed attention weights and the updated attention mask. xlaNrrrUrSrAT)rTrjz-inf)rdtyperTrBr(rr[ new_zeroszerosrrmrhmaxapply_sparse_maskrUrrVrC masked_fillr1rery) r9rerrfrrrar`rTrPis_tpurir=r=r>r^%s| ! ((         z-MultiheadAttention._process_attention_weightscCs|jdurT|jdusJdtj||jd|dgdd}tj||jd|dgdd}|dur@tj|||ddgdd}|durTtj|||ddgdd}||||fS)a Applies bias_k and bias_v to the key and value tensors, updating the attention mask and key padding mask accordingly. Arguments --------- k : torch.Tensor Key tensor. v : torch.Tensor Value tensor. bsz : int Batch size. attn_mask : torch.Tensor Attention mask key_padding_mask : torch.Tensor Key padding mask. Returns ------- Tuple[torch.Tensor, torch.Tensor, Optional[torch.Tensor], Optional[torch.Tensor]]: Updated key, value, attention mask, and key padding mask. Nz(bias_k and bias_v must both be provided.rrrS)r&r'rr[rXr}rB)r9rrfrarrr=r=r> apply_biass&   zMultiheadAttention.apply_biasc Cs6|jr||} ||} ||} n;|jr5||} |dur*|dus%Jd} } n$||} ||} n|dur=|dus?J||} ||} ||} | |j9} | d|9} | |||j|j  dd} | dur| d||j|j  dd} | dur| d||j|j  dd} | | | ||fS)a Prepares and scales the projections, applies biases, and reshapes the query, key, and value tensors for multi-head attention. Arguments --------- query : torch.Tensor Query tensor. key : torch.Tensor Key tensor. value : torch.Tensor Value tensor. bsz : int Batch size. tgt_len : int Target sequence length. key_padding_mask : torch.Tensor Key padding mask. attn_mask : torch.Tensor Attention mask alpha : float, optional Scaling factor for relative position. Default is 32. Returns ------- Tuple[torch.Tensor, torch.Tensor, torch.Tensor, Optional[torch.Tensor], Optional[torch.Tensor]] Scaled and reshaped query, key, and value tensors, along with updated attention and key padding masks. NrrrA) rrrrr%r$rnrCrr"rhr#r!) r9rrrrar`rrrPrerrfr=r=r>rZs@(             z,MultiheadAttention._prepare_attention_inputsrRrSrTcCs|dur |r |}|S|dur!|dur!tj||gdd}|S|durP||dkrJtj|||df|jd}tj||gdd}|S|}|S|dur||dkrytj|||df|jd}tj||gdd}|S|}|S|}|S)a Combines the previous and current key padding masks to create a unified mask. Arguments --------- key_padding_mask : Optional[torch.Tensor] The current key padding mask of shape `(batch_size, seq_len)`, or `None`. prev_key_padding_mask : Optional[torch.Tensor] The previous key padding mask of shape `(batch_size, seq_len)`, or `None`. batch_size : int The batch size of the input. src_len : int The source sequence length to which the masks need to align. static_kv : bool If `True`, indicates that the key-value pairs are static and only the previous key padding mask should be used. Returns ------- Optional[torch.Tensor] The combined key padding mask of shape `(batch_size, src_len)`, or `None` if both input masks are `None`. NrrSrc)rr[ryrBr~rd)rrRrSrTrLnew_key_padding_maskfillerr=r=r>r\sD ! z0MultiheadAttention._append_prev_key_padding_maskcCs ||d}|dur |Si}|S)aU Retrieves the input buffer for incremental decoding. Arguments --------- incremental_state : Optional[Dict[str, Dict[str, Optional[Tensor]]]] The state dictionary used for incremental decoding. It stores intermediate computation states, such as attention states, for efficient sequential processing. Returns ------- Dict[str, Optional[Tensor]] The attention state dictionary containing keys and values for incremental decoding. If no state exists, an empty dictionary is returned. attn_stateN)get_incremental_state)r9rKresult empty_resultr=r=r>rY_s z$MultiheadAttention._get_input_bufferbuffercCs||d|S)a Updates the input buffer for incremental decoding. Arguments --------- incremental_state : Dict[str, Dict[str, Optional[Tensor]]] The state dictionary used for incremental decoding. It stores intermediate computation states, such as attention states. buffer : Dict[str, Optional[Tensor]] The attention state dictionary containing keys and values to be stored for incremental decoding. Returns ------- None r)set_incremental_state)r9rKrr=r=r>r]zsz$MultiheadAttention._set_input_bufferr`racCs|S)aI Applies a sparse mask to the attention weights. Arguments --------- attn_weights : torch.Tensor The attention weights tensor of shape `(batch_size * num_heads, tgt_len, src_len)`. tgt_len : int The target sequence length. src_len : int The source sequence length. bsz : int The batch size. Returns ------- torch.Tensor The (potentially modified) attention weights tensor. By default, this is the same as the input tensor. r=)r9rir`rTrar=r=r>rsz$MultiheadAttention.apply_sparse_mask)NNrTFFFFrrFrrLFF)T)NNTFNFFNr)NNr)rtrurvrwrr,r?rrrrJrrrxrerrbr_r^rrZrr\rYr]rrzr=r=r;r>r  s/W 50      <e p: S C  r rcCsdtjddfdd}t|tjr%||jj|jdur#|jjdSdSt|tj rC||jj|j durA|jj|j dSdSt|t r_||j jj||j jj||jjjdSdS)z Initializes weights and biases for modules in the BERT model. Arguments --------- module : nn.Module The module to initialize. Can be one of `nn.Linear`, `nn.Embedding`, or `MultiheadAttention`. datarNcSs$||jddd|jdS)z Initializes a tensor with values drawn from a normal distribution. Arguments --------- data : torch.Tensor The tensor to initialize. rg{Gz?rN)copy_cpurr1rd)rr=r=r>rs$ z!init_bert_params..normal_)rrrrr&rrrzero_r padding_idxr rrr)rrr=r=r>rs       rc@s(eZdZdZdddZdefddZdS) ra Configuration class for the BEATs model. This class defines the configuration for the BEATs model. It provides a default configuration that can be updated with custom settings via the `update` method. Arguments --------- cfg : dict, optional A dictionary containing custom configuration values. If provided, it will override the default settings. NcCsd|_d|_d|_d|_d|_d|_d|_d|_d|_d|_ d|_ d |_ d |_ d |_ d |_d |_d |_d|_d|_d |_d |_d|_d|_d |_d|_|durV||dSdS)NiF rrrrrrrLi@ii)r(r#r*rr%rrrrr/r.rrrrr-rrrrrrr3r4r5updater9rr=r=r>rsVzBEATsConfig.__init__rcCs|j|dS)a Updates the instance's attributes with key-value pairs from a given configuration dictionary. Arguments --------- cfg : dict A dictionary containing the configuration values to update the instance with. N)r"rrr=r=r>r*s zBEATsConfig.updater)rtrurvrwrdictrr=r=r=r>rs Fr).rwloggingrrtypingrrrnumpyrrtorch.nn.functionalrrrtorchaudio.compliance.kaldi compliancekaldirWrtorch.nnrrspeechbrain.dataio.dataior getLoggerrtr Moduler r~rrxrrrrautogradFunctionrrr0rr rrr=r=r=r>sL   20!,59P?**