o ©Ì³i’ã@s¼ddlmZddlZddlmZ ddddddd d d d d dddddœZ    ddeeejfdededededeeejff dd„Z    ddeeejfdedededef dd„Z dS) é)ÚDictN)Úget_mapped_keyztok_embeddings.weightzlayers.{}.attn.q_proj.weightzlayers.{}.attn.k_proj.weightzlayers.{}.attn.v_proj.weightz!layers.{}.attn.output_proj.weightzlayers.{}.mlp.w1.weightzlayers.{}.mlp.w3.weightzlayers.{}.mlp.w2.weightzlayers.{}.sa_norm.scalezlayers.{}.sa_scale.scalezlayers.{}.mlp_norm.scalezlayers.{}.mlp_scale.scaleznorm.rms_norm.scalez output.weight)zmodel.embed_tokens.weightz'model.layers.{}.self_attn.q_proj.weightz'model.layers.{}.self_attn.k_proj.weightz'model.layers.{}.self_attn.v_proj.weightz'model.layers.{}.self_attn.o_proj.weightz-model.layers.{}.self_attn.rotary_emb.inv_freqz$model.layers.{}.mlp.gate_proj.weightz"model.layers.{}.mlp.up_proj.weightz$model.layers.{}.mlp.down_proj.weightz&model.layers.{}.input_layernorm.weightz/model.layers.{}.post_attention_layernorm.weightz1model.layers.{}.post_feedforward_layernorm.weightz0model.layers.{}.pre_feedforward_layernorm.weightzmodel.norm.weightzlm_head.weighté éÚ state_dictÚ num_headsÚ num_kv_headsÚdimÚhead_dimÚreturnc sxi}ˆdur ˆ|‰‡‡fdd„}| ¡D]$\}}d|vr9t|tƒ} d|vr,|||ƒ}n d|vr5|||ƒ}||| <q|S)a( Convert a state dict from HF's format to torchtune's format. State dicts from multiple checkpoint files should be consolidated into a single state dict before calling this function. Eg of HF-format state dict can be found in the ``meta-llama/Llama-2-7b-hf`` repo in HF (https://huggingface.co/meta-llama/Llama-2-7b-hf). Args: state_dict (Dict[str, torch.Tensor]): State dict in HF's format. num_heads (int): Number of heads in the model. num_kv_heads (int): Number of heads in the key/value projection layers. dim (int): Dimension of the model. head_dim (int): Dimension of the head. If not provided, it will be calculated as dim // num_heads. Returns: Dict[str, torch.Tensor]: State dict in torchtune's format. Ncs(| |dˆdˆ¡ dd¡ ˆ|ˆ¡S©Néé©ÚviewÚ transposeÚreshape©ÚtÚn_heads©r r ©ú\/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/gemma2/_convert_weights.pyÚ_permuteFó ýz#gemma2_hf_to_tune.._permutezrotary_emb.inv_freqÚq_projÚk_proj)ÚitemsrÚ_GEMMA2_FROM_HF) rrrr r Úconverted_state_dictrÚkeyÚvalueÚnew_keyrrrÚgemma2_hf_to_tune(s   €r#c s‚i}dd„t ¡Dƒ}ˆdurˆ|‰‡‡fdd„}| ¡D] \}} t||ƒ} d|vr1|| |ƒ} n d|vr:|| |ƒ} | || <q|S)an Convert a state dict from torchtune's format to HF's format. This function doesn't handle any sharding or splitting of state dicts. It follows the state_dict IN -> state_dict OUT pattern. Args: state_dict (Dict[str, torch.Tensor]): State dict in torchtune's format. num_heads (int): Number of heads in the model. num_kv_heads (int): Number of heads in the key/value projection layers. dim (int): Dimension of the model. head_dim (int): Dimension of model attention heads. Default None. Returns: Dict[str, torch.Tensor]: State dict in HF's format. cSsi|]\}}||“qSrr)Ú.0ÚkÚvrrrÚ psz%gemma2_tune_to_hf..Ncs(| |ˆddˆ¡ dd¡ ˆ|ˆ¡Sr rrrrrrurz#gemma2_tune_to_hf.._permuterr)rrr) rrrr r rÚinverted_mapping_dictrr r!r"rrrÚgemma2_tune_to_hfYs    r))rrrN) ÚtypingrÚtorchÚ torchtune.models.convert_weightsrrÚstrÚTensorÚintr#r)rrrrÚsh   ñû ÿþýüû  ú3û ÿþýüû