o ̳iB @snddlmZddlmZmZmZddlmZddlm Z de de de de d e d e d e d e d e de fddZ dS))nn) T5EncoderT5EncoderLayerT5EncoderSelfAttention) FeedForward)RMSNorm embed_dimmlp_dim num_headshead_dim num_layersrel_pos_num_bucketsrel_pos_max_dist vocab_sizenorm_eps max_seq_lenc  Cst||} t} t|D]V} t|||tj||ddtj||ddtj||ddtj||ddd} ttj||ddtj||ddtj||ddtd}t| |t ||dt ||dd}| |qt ||d}t | | ||||| dS)aM Builder for the T5 encoder. T5 paper: https://arxiv.org/abs/1910.10683 Args: embed_dim (int): The model dimension. mlp_dim (int): The inner dimension of the feed forward layers. num_heads (int): The number of attention heads. head_dim (int): The dimension of the attention heads (should equal `embed_dim // num_heads`) num_layers (int): Number of encoder layers. rel_pos_num_buckets (int): Number of discrete buckets to divide the relative positions into. See: :class:`~torchtune.models.t5._encoder.T5EncoderRelativePositionBias` rel_pos_max_dist (int): Maximum distance for relative positions. Distances beyond this are grouped into the last bucket. See: :class:`~torchtune.models.t5._encoder.T5EncoderRelativePositionBias` vocab_size (int): Vocab size of the model's tokenizer. norm_eps (float): Small value added to denominator for numerical stability. max_seq_len (int): The maximum sequence length (context length) of the model. Returns: T5Encoder F)bias)rr r q_projk_projv_proj output_proj) gate_proj down_projup_proj activation)eps)attnmlpsa_normmlp_norm)token_embeddinglayers final_normr r rr) r Embedding ModuleListrangerLinearrGELUrrappendr)rr r r r r rrrrr r!_rrlayerr"r+[/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/t5/_component_builders.py t5_encodersF #      r-N) torchrtorchtune.models.t5._encoderrrrtorchtune.modules.feed_forwardrtorchtune.modules.rms_normrintfloatr-r+r+r+r,s2