o ̳i@sJddlmZmZddlmZmZddlmZddlm Z ddl m Z ddl m Z ddl mZddlmZ d efd d Zd'd edeedee d e fddZ      d(dee dedededededed efddZeedd Zd!e_d efd"d#Z      d(dee dedededededed efd$d%Zeedd Zd&e_d S)))ListOptional)gemma lora_gemma)TransformerDecoder)GemmaTokenizer)LORA_ATTN_MODULES) _TemplateType)_get_prompt_template)partialreturnc Cstddddddddd d d S) z Builder for creating a Gemma 2B model initialized w/ the default 2b parameter values from: https://blog.google/technology/developers/gemma-open-models/ Returns: TransformerDecoder: Instantiation of Gemma 2B model @ ư> vocab_size num_layers num_headshead_dim num_kv_heads embed_dimintermediate_dim max_seq_len attn_dropoutnorm_epsrr#r#Z/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/gemma/_model_builders.pygemma_2br%Npathrprompt_templatecCs"t|||dur t|dSddS)a Tokenizer for Gemma. Args: path (str): path to the tokenizer max_seq_len (Optional[int]): maximum sequence length for tokenizing a single list of messages, after which the input will be truncated. Default is None. prompt_template (Optional[_TemplateType]): optional specified prompt template. If a string, it is assumed to be the dotpath of a :class:`~torchtune.data.PromptTemplateInterface` class. If a dictionary, it is assumed to be a custom prompt template mapping role to the prepend/append tags. Returns: GemmaTokenizer: Instantiation of the Gemma tokenizer Nr'rr()rr r)r#r#r$gemma_tokenizer.s"r*Frrlora_attn_modulesapply_lora_to_mlp lora_rank lora_alpha lora_dropoutuse_dora quantize_basecCsptdid|d|ddddddd d d d d dddddddddd|d|d|d|d|S)a Builder for creating a Gemma 2B model with LoRA enabled. The Gemma defaults are the same as in :func:`~torchtune.models.gemma.gemma_2b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Gemma 2B model with LoRA applied r,r-rr rrrrrrrrrrrrrrr rr!rr.r/r0r1r2Nr#rr,r-r.r/r0r1r2r#r#r$ lora_gemma_2bBF      r5T)r2z Builder for creating a Gemma model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_gemma_2b` for full API arguments. c Cstdddddddddd d S) z Builder for creating a Gemma 7B model initialized w/ the default 7b parameter values from: https://blog.google/technology/developers/gemma-open-models/ Returns: TransformerDecoder: Instantiation of Gemma 7B model r r+r `rrrrr"r#r#r#r$gemma_7br&r:cCsptdid|d|ddddddd d d dd d ddddddddd|d|d|d|d|S)a Builder for creating a Gemma 7B model with LoRA enabled. The Gemma defaults are the same as in :func:`~torchtune.models.gemma.gemma_7b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Gemma 7B model with LoRA applied r,r-rr rr7rr+rrrrr8rr9rrr rr!rr.r/r0r1r2Nr#r3r4r#r#r$ lora_gemma_7br6r;z Builder for creating a Gemma model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_gemma_7b` for full API arguments. )NN)Frr+rFF)typingrr*torchtune.models.gemma._component_buildersrrtorchtune.modulesr!torchtune.models.gemma._tokenizerrtorchtune.modules.peftr torchtune.data._prompt_templatesr r functoolsr r%strintr*boolfloatr5qlora_gemma_2b__doc__r:r;qlora_gemma_7br#r#r#r$s|      $ 4 4