o ̳i;@sddlmZddlmZmZddlmZmZddlm Z m Z ddl m Z ddl mZddlmZddlmZ d efd d Z d%d ededeedeedeed e f ddZ       d&deedededededededed efd d!Zeed"d#Zd$e_d S)')partial)ListOptional)_get_prompt_template _TemplateType) lora_phi3phi3) Phi4Tokenizer)TransformerDecoder)LORA_ATTN_MODULES)parse_hf_tokenizer_jsonreturnc Cstdddddddddd S) z Builder for creating the Phi4 (14B) Instruct Model. Returns: TransformerDecoder: Instantiation of Phi4 (14B) Instruct Model ( F@h㈵>) vocab_size num_layers num_heads num_kv_heads embed_dimintermediate_dim max_seq_len attn_dropoutnorm_eps)rrrY/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/phi4/_model_builders.pyphi4_14bsr!N vocab_path merges_pathspecial_tokens_pathrprompt_templatecCs:|durt|nd}|durt|nd}t|||||dS)a Phi4 tokenizer. Args: vocab_path (str): Path to vocab.json. merges_path (str): Path to merges.txt. special_tokens_path (Optional[str]): Path to ``tokenizer.json`` from Hugging Face model files that contains all registered special tokens, or a local json file structured similarly. Default is None to use the canonical Phi4 special tokens. max_seq_len (Optional[int]): maximum sequence length for tokenizing a single list of messages, after which the input will be truncated. Default is None. prompt_template (Optional[_TemplateType]): optional specified prompt template. If a string, it is assumed to be the dotpath of a :class:`~torchtune.data.PromptTemplateInterface` class. If a dictionary, it is assumed to be a custom prompt template mapping role to the prepend/append tags. Returns: Phi4Tokenizer: Instantiation of the Phi-4 (14B) tokenizer. N)r"r#special_tokensrr%)r rr )r"r#r$rr%r&templaterrr phi4_tokenizer(sr(Frlora_attn_modulesapply_lora_to_mlpapply_lora_to_output lora_rank lora_alpha lora_dropoutuse_dora quantize_basecCsptdid|d|d|ddddddd d d d d dddddddd|d|d|d|d|S)a Builder for creating a Phi4 (14b) model with LoRA enabled. The Phi4 defaults are the same as in :func:`~torchtune.models.phi4.phi4`. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False apply_lora_to_output (bool): whether to apply LoRA to the model's final output projection. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Phi4 (14B) model with LoRA applied r+r,r-rrrrrrrrrrrrrrrrrr.r/r0r1r2Nr)r)r+r,r-r.r/r0r1r2rrr lora_phi4_14bSsF!     r3T)r2z Builder for creating a Phi4 (14B) model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_phi4_14b` for full API arguments. )NNNNN)FFr)r*rFF) functoolsrtypingrr torchtune.data._prompt_templatesrr)torchtune.models.phi3._component_buildersrr torchtune.models.phi4._tokenizerr torchtune.modulesr torchtune.modules.peftr torchtune.modules.tokenizersr r!strintr(boolfloatr3qlora_phi4_14b__doc__rrrr sp      - 6