o ̳i @s`ddlZddlmZddlZddlmZmZddlmZGdddejZ GdddejZ dS) N)Optional)nnTensor) _MaskTypecspeZdZdZdddddddejd ejd ed ed ed edeffddZdddede e defddZ Z S)CLIPTextEncodera Text encoder for CLIP. CLIP is a model that encodes text and images into a shared vector space. Blog post: https://openai.com/index/clip/ Paper: https://arxiv.org/abs/2103.00020 Args: layer (nn.Module): A single encoder layer. final_norm (nn.Module): Callable that applies normalization to the output of the encoder vocab_size (int): size of the vocabulary, default 49408 max_seq_len (int): context size, default 77 embed_dim (int): embedding/model dimension size, default 768 num_layers (int): number of transformer layers, default 12 eot_token (int): the id of the end-of-text token (for selecting the final output) iMi i) vocab_size max_seq_len embed_dim num_layers eot_tokenlayer final_normr r r r r cs`ttfddt|D|_||_||_||_t |||_ t t |||_dS)Ncsg|]}tqS)copydeepcopy).0irrW/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/clip/_text_encoder.py .sz,CLIPTextEncoder.__init__..)super__init__r ModuleListrangelayersrr r Embeddingtoken_embedding Parametertorchemptyposition_embedding)selfrrr r r r r  __class__rrr"s zCLIPTextEncoder.__init__Nmasktokensr'returncCs|j\}}||jkrtd|d|jd|||j}|jD]}|||d}q!||}||jkj dd}|j | dddddj dd}|S)a! Args: tokens (Tensor): input tensor with shape ``[b x s]`` mask (Optional[_MaskType]): Used to mask the scores after the query-key multiplication and before the softmax. Default is None. Returns: Tensor: output tensor with shape [b x d] Raises: ValueError: if seq_len of tokens is bigger than max_seq_len Shape notation: - b: batch size - s: token sequence length - d: token embed dim z seq_len (z6) of input tensor should be smaller than max_seq_len ()r&)dim) shaper ValueErrorrr"rrr intargmaxtake_along_dimviewsqueeze)r#r(r'bszseq_lenxreos_token_positionsrrrforward6s"     zCLIPTextEncoder.forward) __name__ __module__ __qualname____doc__rModuler0rrrrr9 __classcell__rrr$rrs< rc@s&eZdZdZdejdejfddZdS) QuickGELUz% Fast approximation of GELU. r7r)cCs|td|S)NgZd;?)r sigmoid)r#r7rrrr9nszQuickGELU.forwardN)r:r;r<r=r rr9rrrrr@isr@) rtypingrr rr!torchtune.modules.attention_utilsrr>rr@rrrrs  Y