o ̳i*@sddlmZmZddlmZmZddlmZmZddl m Z m Z ddl m Z ddlmZddlmZ de fd d Zde fd d Zde fd dZ    d)dededeedeedeede f ddZ       d*deededededed ed!ed"ede fd#d$Z      d+deedededed ed!ed"ede fd%d&Z      d+deedededed ed!ed"ede fd'd(ZdS),)ListOptional)_get_prompt_template _TemplateType) lora_qwen2qwen2)QWEN2_SPECIAL_TOKENSQwen2Tokenizer)TransformerDecoder)LORA_ATTN_MODULES)parse_hf_tokenizer_jsonreturnc Cstdddddddddd d S) z Builder for creating a Qwen2 model initialized w/ the default 7B parameter values from https://huggingface.co/Qwen/Qwen2-7B-Instruct Returns: TransformerDecoder: Instantiation of Qwen2 7B model RJư>.A) vocab_size num_layers num_heads num_kv_heads embed_dimintermediate_dim max_seq_len attn_dropoutnorm_eps rope_baserr"r"Z/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/qwen2/_model_builders.pyqwen2_7bsr$c Ctddddddddd d d d S) a Builder for creating a Qwen2 model initialized w/ the default 0.5B parameter values from https://huggingface.co/Qwen/Qwen2-0.5B-Instruct Returns: TransformerDecoder: Instantiation of Qwen2 0.5B model Note: Qwen2 0.5B and Qwen2 1.5B model builders will enable `tie_word_embeddings` by default and returns an instance of `TransformerDecoder`. QrrrrT rrrrrrrrrr tie_word_embeddingsr!r"r"r"r# qwen2_0_5b- r.c Cr%) a Builder for creating a Qwen2 model initialized w/ the default 1.5B parameter values from https://huggingface.co/Qwen/Qwen2-1.5B-Instruct Returns: TransformerDecoder: Instantiation of Qwen2 1.5B model Note: Qwen2 0.5B and Qwen2 1.5B model builders will enable `tie_word_embeddings` by default and returns an instance of `TransformerDecoder`. r&r r)#rrrrTr,r!r"r"r"r# qwen2_1_5bHr/r3Npath merges_filespecial_tokens_pathrprompt_templatecKsB|durt|nt}|durt|nd}td|||||d|S)a Tokenizer for Qwen2. Args: path (str): path to the vocab.json file. merges_file (str): path to the merges.txt file. special_tokens_path (Optional[str]): Path to ``tokenizer.json`` from Hugging Face model files that contains all registered special tokens, or a local json file structured similarly. Default is None to use the canonical Qwen2 special tokens. max_seq_len (Optional[int]): A max sequence length to truncate tokens to. Default: None prompt_template (Optional[_TemplateType]): optional specified prompt template. If a string, it is assumed to be the dotpath of a :class:`~torchtune.data.PromptTemplateInterface` class. If a dictionary, it is assumed to be a custom prompt template mapping role to the prepend/append tags. Default is None. Returns: Qwen2Tokenizer: Instantiation of the Qwen2 tokenizer N)r4r5special_tokensrr7r")r rrr )r4r5r6rr7kwargsr8templater"r"r#qwen2_tokenizercsr;Frlora_attn_modulesapply_lora_to_mlpapply_lora_to_output lora_rank lora_alpha lora_dropoutuse_dora quantize_basecCsvtdid|d|d|ddddddd d d d d dddddddddd|d|d|d|d|S)a Builder for creating a Qwen2 7B model with LoRA enabled. The Qwen2 defaults are the same as in :func:`~torchtune.models.qwen2.qwen2_7b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False apply_lora_to_output (bool): whether to apply LoRA to the model's final output projection. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Qwen2 7B model with LoRA applied r>r?r@rrrrrrrrrrrrrrrrrr rrArBrCrDrENr"r)r>r?r@rArBrCrDrEr"r"r# lora_qwen2_7bsJ!     rGcC|td!id|d|ddddddd d d d d dddddddddddddd|d|d|d|d|S)"a Builder for creating a Qwen2 0.5B model with LoRA enabled. The Qwen2 defaults are the same as in :func:`~torchtune.models.qwen2.qwen2_0_5b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Qwen2 0.5B model with LoRA applied Note: Qwen2 0.5B and Qwen2 1.5B model builders will enable `tie_word_embeddings` by default and returns an instance of `TransformerDecoder`. r>r?r@Frr&rr'rr(rr)rr*rr+rrrrrrr rr-TrArBrCrDrENr"rFr>r?rArBrCrDrEr"r"r#lora_qwen2_0_5bN"     rJcCrH)"a Builder for creating a Qwen2 1.5B model with LoRA enabled. The Qwen2 defaults are the same as in :func:`~torchtune.models.qwen2.qwen2_1_5b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Qwen2 1.5B model with LoRA applied Note: Qwen2 0.5B and Qwen2 1.5B model builders will enable `tie_word_embeddings` by default and returns an instance of `TransformerDecoder`. r>r?r@Frr&rrrr0rr)rr1rr2rrrrrrr rr-TrArBrCrDrENr"rFrIr"r"r#lora_qwen2_1_5brKrL)NNNN)FFr<r=rFF)Fr<r=rFF)typingrr torchtune.data._prompt_templatesrr*torchtune.models.qwen2._component_buildersrr!torchtune.models.qwen2._tokenizerrr torchtune.modulesr torchtune.modules.peftr 'torchtune.modules.transforms.tokenizersr r$r.r3strintr;boolfloatrGrJrLr"r"r"r#s    / 9 ;