o ̳i'@sZddlmZddlmZddlmZmZddlmZddl m Z  defddZ d$d ee de de dedede de defddZeeddZde_defddZ d$d ee de de dedede de defddZeeddZde_defddZ d$d ee de de dedede de defd d!ZeeddZd"e_d#S)%)partial)List)gemma2 lora_gemma2)TransformerDecoder)LORA_ATTN_MODULESreturncC"tddddddddd d d d d d S)z Builder for creating a Gemma2 2B model initialized w/ the default 2b parameter values from: https://github.com/google/gemma_pytorch/blob/main/gemma/config.py Returns: TransformerDecoder: Instantiation of Gemma2 2B model  $ ư>>@I@ vocab_size num_layers num_headshead_dim num_kv_heads embed_dimintermediate_dim max_seq_len attn_dropoutnorm_epshidden_capping_valuefinal_capping_valuesliding_window_sizerr&r&[/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/gemma2/_model_builders.py gemma2_2br(Fr rlora_attn_modulesapply_lora_to_mlp lora_rank lora_alpha lora_dropoutuse_dora quantize_basecCtd#id|d|ddddddd d d d d dddddddddddddddd|d|d|d |d!|S)$a Builder for creating a Gemma2 2B model with LoRA enabled. The Gemma defaults are the same as in :func:`~torchtune.models.gemma.gemma_2b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Gemma2 2B model with LoRA applied r+r,rr rr rr rr rrrrrrrrr rr!rr"rr#rr$rr-r.r/r0r1Nr&rr+r,r-r.r/r0r1r&r&r'lora_gemma2_2b-R      r5T)r1z Builder for creating a Gemma2 model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_gemma2_2b` for full API arguments. cCr )z Builder for creating a Gemma2 9B model initialized w/ the default 9b parameter values from: https://github.com/google/gemma_pytorch/blob/main/gemma/config.py Returns: TransformerDecoder: Instantiation of Gemma 9B model r *r*r r 8rrrrrrrr%r&r&r&r' gemma2_9bnr)r:cCr2)$a Builder for creating a Gemma 9B model with LoRA enabled. The Gemma defaults are the same as in :func:`~torchtune.models.gemma.gemma_7b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Gemma2 9B model with LoRA applied r+r,rr rr7rr*rr rr rr8rr9rrr rr!rr"rr#rr$rr-r.r/r0r1Nr&r3r4r&r&r'lora_gemma2_9br6r;z Builder for creating a Gemma model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_gemma2_9b` for full API arguments. cCs$tddddddddd d d d d ddS)z Builder for creating a Gemma2 27B model initialized w/ the default 27b parameter values from: https://github.com/google/gemma_pytorch/blob/main/gemma/config.py Returns: TransformerDecoder: Instantiation of Gemma2 27B model r . r*rrrrrr)rrrrrrrrr r!r"r#r$query_pre_attn_scalarr%r&r&r&r' gemma2_27bs rCcCstd%id|d|ddddddd d d d d dddddddddddddddddd|d |d!|d"|d#|S)&a Builder for creating a Gemma2 27B model with LoRA enabled. The Gemma defaults are the same as in :func:`~torchtune.models.gemma.gemma_7b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Gemma2 27B model with LoRA applied r+r,rr rr<rr=rr>rr*rr?rr@rrr rr!rr"rr#rr$rrBrAr-r.r/r0r1Nr&r3r4r&r&r'lora_gemma2_27bsV      rDz Builder for creating a Gemma model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_gemma2_27b` for full API arguments. N)Fr r*rFF) functoolsrtypingr+torchtune.models.gemma2._component_buildersrrtorchtune.modulesrtorchtune.modules.peftrr(boolintfloatr5qlora_gemma2_2b__doc__r:r;qlora_gemma2_9brCrDqlora_gemma2_27br&r&r&r's     8 8 9