o wio1@sFdZddlmZmZddlZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZd d lmZmZmZmZmZmZmZddlmZeeZdZ dZ!Gdddej"Z#d#ddZ$Gdddej"Z%GdddeZ&GdddeZ'Gdddee'Z(GdddeZ)Gd d!d!eZ*gd"Z+dS)$zPyTorch Phi-3 model.)CallableOptionalN)nn)ACT2FN)Cache)FlashAttentionKwargs)ALL_ATTENTION_FUNCTIONS)Unpack)logging)MistralDecoderLayerMistralForCausalLM MistralForSequenceClassificationMistralForTokenClassificationMistralPreTrainedModeleager_attention_forward rotate_half) Phi3Configz microsoft/Phi-3-mini-4k-instructrcs2eZdZfddZdejdejfddZZS)Phi3MLPcsPt||_tj|jd|jdd|_tj|j|jdd|_t |j |_ dS)Nr Fbias) super__init__configrLinear hidden_sizeintermediate_size gate_up_proj down_projr hidden_act activation_fn)selfr __class__b/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/phi3/modular_phi3.pyr1s zPhi3MLP.__init__ hidden_statesreturncCs4||}|jddd\}}|||}||S)Nr dim)rchunkr"r )r#r( up_statesgater&r&r'forward9s  zPhi3MLP.forward)__name__ __module__ __qualname__rtorch FloatTensorr0 __classcell__r&r&r$r'r0s rc Cs||}||}|jd}|dd|f|d|df}}|dd|f|d|df} } tj||t|||gdd} tj| |t| || gdd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. r*.Nr+) unsqueezeshaper4catr) qkcossin position_ids unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embedr&r&r'apply_rotary_pos_embBs   """"rGcseZdZdZddedeeffdd Z  ddej de ej ej fd eej d ee d eej d e ed e ej eej ee ej ffddZZS) Phi3Attentionz=Multi-headed attention from 'Attention Is All You Need' paperNr layer_idxcst||_||_t|d|j|j|_|j|j|_ |j|_|jd|_ |j |_ d|_ |j|jd|j|j}t j|j|j|jdd|_t j|j|dd|_dS)Nhead_dimgTr Fr)rrrrIgetattrrnum_attention_headsrJnum_key_value_headsnum_key_value_groupsscalingattention_dropout is_causalrro_projqkv_proj)r#rrIop_sizer$r&r'res  zPhi3Attention.__init__r(position_embeddingsattention_maskpast_key_valuecache_positionkwargsr)cKs~|jdd}g|d|jR}||} |jj|j} | dd| f} | d| | |j|jf} | d| |j|jdf} | |dd} | |dd} | |dd} |\}}t| | ||\} } |dur~|||d}| | | |j |\} } t }|jj dkrt |jj }||| | | |f|jsdn|j|jt|jddd |\}}|jg|dR}||}||fS) Nr*.rr )r=r<rXeagergsliding_window)dropoutrOr[)r8rJrSrrLrMview transposerGupdaterIr_attn_implementationr trainingrPrOrKreshape contiguousrR)r#r(rUrVrWrXrY input_shape hidden_shapeqkv query_pos query_states key_states value_statesr<r= cache_kwargsattention_interface attn_output attn_weightsr&r&r'r0tsD        zPhi3Attention.forward)N)NN)r1r2r3__doc__rrintrr4Tensortupler LongTensorr rr0r6r&r&r$r'rHbs(rHcseZdZdedeffdd Z       ddejdeejd eej d ee d ee d ee d eej dee ejejfde ede ejee ejejfffddZZS)Phi3DecoderLayerrrIcsLt||||_t||d|_t||_t|j |_ t|j |_ dS)N)rrI) rrrrH self_attnrmlprDropout resid_pdropresid_attn_dropoutresid_mlp_dropout)r#rrIr$r&r'rs  zPhi3DecoderLayer.__init__NFr(rVr>rWoutput_attentions use_cacherXrUrYr)c Ks|} ||}|jd||||||||d| \}} | ||}|} ||}||}| ||}|f} |r>| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_value (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r(rVr>rWr{r|rXrUNr&)input_layernormrurypost_attention_layernormrvrz) r#r(rVr>rWr{r|rXrUrYresidualself_attn_weightsoutputsr&r&r'r0s."      zPhi3DecoderLayer.forward)NNNFFNN)r1r2r3rrprr4rqrrsrboolrrr rr5r0r6r&r&r$r'rts<    rtc@seZdZdZdS)Phi3PreTrainedModelz0.0.5N)r1r2r3_versionr&r&r&r'rsrc@s$eZdZ       dddZdS)Phi3ForCausalLMNTc Ksb|r|jjr|jd|jjdkr|d} | |jjkrd}tjd||||||||d| } | S)Nrr) input_idspast_key_valuesrV inputs_embedsrXr>r|logits_to_keepr&)r rope_scalingr8 original_max_position_embeddingsrprepare_inputs_for_generation) r#rrrVrrXr>r|rrY past_length model_inputsr&r&r'rs*   z-Phi3ForCausalLM.prepare_inputs_for_generation)NNNNNTN)r1r2r3rr&r&r&r'rsrc@ eZdZdS)Phi3ForSequenceClassificationNr1r2r3r&r&r&r'rrc@r)Phi3ForTokenClassificationNrr&r&r&r'r!rr)r Phi3Modelrrr)Nr),rotypingrrr4torch.utils.checkpointr activationsr cache_utilsrmodeling_flash_attention_utilsrmodeling_utilsr processing_utilsr utilsr mistral.modeling_mistralr rrrrrrconfiguration_phi3r get_loggerr1logger_CHECKPOINT_FOR_DOC_CONFIG_FOR_DOCModulerrGrHrtrrrr__all__r&r&r&r's2       $   EI)