o }oi @s@ddlZddlmZgdZdZd ddZd dd Zd d ZdS)N)NEG_INFform_attention_masktransformer_weights_initmask_padded_tokensgcCs~|durdSd|jd|jdf}|jtdd}|dur0ttj|tj|jd|}||@}d|tjt }|dS)a Build attention mask with optional masking of future tokens we forbid to attend to (e.g. as it is in Transformer decoder). Args: input_mask: binary mask of size B x L with 1s corresponding to valid tokens and 0s corresponding to padding tokens diagonal: diagonal where triangular future mask starts None -- do not mask anything 0 -- regular translation or language modeling future masking 1 -- query stream masking as in XLNet architecture Returns: attention_mask: mask of size B x 1 x L x L with 0s corresponding to tokens we plan to attend to and -10000 otherwise N)dtype)rdevice) shapetobool unsqueezetorchtrilonesrfloatr) input_maskdiagonal attn_shape attn_mask future_maskattention_maskrc/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/parts/transformer_utils.pyrs r{Gz?TcCst|tjr+|rtj|jn tjj|jd|d|jdur)tj|jddSdSt|tj r=tjj|jd|ddSt|tj rUtj|jdtj|jddSdS)a{ Initialize different weights in Transformer model. Args: module: torch.nn.Module to be initialized std_init_range: standard deviation of normal initializer xavier: if True, xavier initializer will be used in Linear layers as was proposed in AIAYN paper, otherwise normal initializer will be used (like in BERT paper) g)meanstdNg?) isinstancennLinearinitxavier_uniform_weightnormal_bias constant_ Embedding LayerNorm)modulestd_init_rangexavierrrrr3s    rcCs ||k}|SNr)tokenspad_idmaskrrrrMsrr*)rT)r torch.nnr__all__rrrrrrrrs