o ̳iL @sddlmZmZddlZddlmZdddddd d d d d Zdeeejfdee dee dee deeejff ddZ deeejfdeeejffddZ dS))DictOptionalN)get_mapped_keyztok_embeddings.weightzlayers.{}.attn.q_proj.weightz!layers.{}.attn.output_proj.weightzlayers.{}.mlp.w1.weightzlayers.{}.mlp.w2.weightzlayers.{}.sa_norm.scalezlayers.{}.mlp_norm.scalez norm.scalez output.weight) zmodel.embed_tokens.weightz)model.layers.{}.self_attn.qkv_proj.weightz'model.layers.{}.self_attn.o_proj.weightz'model.layers.{}.mlp.gate_up_proj.weightz$model.layers.{}.mlp.down_proj.weightz&model.layers.{}.input_layernorm.weightz/model.layers.{}.post_attention_layernorm.weightzmodel.norm.weightzlm_head.weight state_dict num_heads num_kv_headsdimreturncCsi}|dur!|dus|durtd|}|||}|||}nd\}}}|D]]\}} t|t} d|vri|durJtj| |||gdd\} } } n | jddd\} } } | || <| || dd <| || dd <q*d |vr| jd dd\}}||| <||| d d<q*| || <q*|S)z Convertor from HF state dict to torchtune state dict. This handles: - Splitting the fused q,k and v matrix - Splitting the fused gate and up projection matrix NzKPhi models with GQA require dim, num_heads and num_kv_heads to be specified)NNNqkvrrq_projk_projv_projgatew1w3) ValueErroritemsr _PHI3_MINItorchsplitchunkreplace)rrrrconverted_state_dictq_dimk_dimv_dimkeyvaluenew_keyqkvrrr%Z/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/phi3/_convert_weights.pyphi3_hf_to_tunes:      r'c Csi}ddtD}|D][\}}d|vsd|vsd|vr qt||}d|vrJ|}||dd}||dd}tj|||gdd} | ||<qd |vrf|} ||d d} tj| | gdd} | ||<q|||<q|S) z Convertor from torchtune state dict to HF state dict. This handles: - Fusing q,k and v matrix - Fusing gate and up projection matrix cSsi|]\}}||qSr%r%).0r#r$r%r%r& Psz#phi3_tune_to_hf..rrrr rr r)rrrrrcat) rrinverted_mapping_dictrr r!r"r#r$r gate_projup_proj gate_up_projr%r%r&phi3_tune_to_hfIs&    r/) typingrrr torchtune.models.convert_weightsrrstrTensorintr'r/r%r%r%r&s2    *.