o ̳id/@sddlZddlmZmZmZddlZddlmmZ ddlmZm Z ddl m Z ddl mZGdddejZGdd d ejZGd d d ejZGd d d ejZdededede fddZdS)N)ListOptionalUnion)nnTensor)MultiHeadAttention) _get_clonescszeZdZdZdddejdeejeejej fdejde de d e d e d e e d dffd dZ de d e fddZZS) T5Encodera The T5 encoder module. T5 paper: https://arxiv.org/abs/1910.10683 Args: token_embedding (nn.Embedding): PyTorch embedding layer to place tokens in an embedding space. layers (Union[nn.Module, List[nn.Module], nn.ModuleList]): A single encoder layer. final_norm (nn.Module): Module that applies normalization to the output of the encoder num_heads (int): The number of attention heads. rel_pos_num_buckets (int): Number of discrete buckets to divide the relative positions into. See: :class:`~torchtune.models.t5._encoder.T5EncoderRelativePositionBias` rel_pos_max_dist (int): Maximum distance for relative positions. Distances beyond this are grouped into the last bucket. See: :class:`~torchtune.models.t5._encoder.T5EncoderRelativePositionBias` max_seq_len (int): The maximum sequence length (context length) of the model. num_layers (Optional[int]): Number of encoder layers, only define when layers is not a list. Raises: AssertionError: If ``num_layers`` is set and layer is a list, **or** ``num_layers`` is not set and layer is an ``nn.Module``. N) num_layerstoken_embeddinglayers final_norm num_headsrel_pos_num_bucketsrel_pos_max_dist max_seq_lenr returnc st||_||_||_t||||d|_d|_t|t j r%||_dSt|t r2t ||_dSt|t j sr?r@)rr;rr<r=r>r?r@rr!r"rs"    zT5EncoderSelfAttention.__init__r)r*rc Cs|j\}}}||}||}||}||||j|jdd}||||j|jdd}||||j|jdd}t ||dd} | |7} t j | dd | j} t | |} | dd|||} || S)r9)dim)r%r=r>r?viewrr< transposetorchmatmulFsoftmaxfloattodtypereshaper@) rr)r*r'r(r;qkv attn_score attn_weightattn_outr!r!r"r,s     zT5EncoderSelfAttention.forward) r-r.r/r0r2rrrrr,r3r!r!rr"r:s$r:cs@eZdZdZdedededeffdd Zdefd d ZZS) ra- Computes binned birectional relative position bias for the T5 encoder. It places relative positions into buckets and for each bucket, learns bias values for each attention head. Args: num_buckets (int): Number of discrete buckets to divide the relative positions into. max_dist (int): Maximum distance for relative positions (distances beyond this are grouped into the last bucket) num_heads (int): Number of attention heads in the transformer. max_seq_len (int): Maximum sequence length (context length). rrrrcs:t||_t|||_|jdt|||dddS)Nrelative_position_to_bucketF) persistent)rrrrr1 embeddingregister_buffer#_calc_birectional_rel_pos_to_bucket)rrrrrrr!r"rs   z&T5EncoderRelativePositionBias.__init__rcCs ||j}|gddS)z Returns: torch.Tensor: relative position bias tensor with shape [1, num_heads, max_seq_len, max_seq_len] )rBrrAr)rXrVpermute unsqueeze)rr)r!r!r"r, s z%T5EncoderRelativePositionBias.forward) r-r.r/r0r2rrr,r3r!r!rr"rs rrrrrc Cstj|tjddddf}tj|tjddddf}||}t|}|d}|d}||k} |t||t||||tj} t| t | |d} |dktj|t | || } | S)a Calculate the mapping from relative positions to bucket indices. NOTE: This is for the T5 encoder (birectional), not the decoder (unidirectional). Args: num_buckets (int): Number of discrete buckets to divide the relative positions into. max_dist (int): Maximum distance for relative positions (distances beyond this are grouped into the last bucket) max_seq_len (int): Maximum sequence length (context length). Returns: Tensor: shape=[max_seq_len, max_seq_len], range=[0, num_buckets] )rNNrBrAr) rHarangelongabslogrLmathrMmin full_likewhere) rrrquery_positions key_positionsrelative_positionsabs_relative_positionshalf_num_buckets max_exactis_exactrelative_position_if_not_exactrVr!r!r"rZs:  rZ)ratypingrrrrHtorch.nn.functionalr functionalrJrtorchtune.modulesrtorchtune.modules.transformerrrr r4r:rr2rZr!r!r!r"s(  ]'W)