o 8wÖiªã@sfddlZddlZddlmZdejdejfdd„Zdd„Zdejdejfd d „Zd e de fd d „Z dS)éNÚxÚreturncCsH|dd|jdd…f}|d|jddd…f}tj| |fddS)aÈ Rotate half the hidden dims of the input. This function was duplicated verbatim from: https://github.com/huggingface/transformers/blob/1de8ce9ee1191ba761a593ac15d9ccbf5851bfc5/src/transformers/models/llama/modeling_llama.py#L126 This was done to eliminate the Llama transformers implementation as a dependency of this file. Note that some other functions were also adapted from the transformers implementation but were modified. .Néÿÿÿÿé)Údim)ÚshapeÚtorchÚcat)rÚx1Úx2©r ú^/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/peft/tuners/adaption_prompt/utils.pyÚllama_rotate_halfs rcCs¸t|jƒdkrB|dd…ddd…df}| d|jdd|jd¡}t | |jdddd¡d|¡}t | |jdddd¡d|¡}n|| d¡}|| d¡}||t|ƒ|}|S)a® Apply rotary position embedding to query states in the Llama model. This function was adapted from: https://github.com/huggingface/transformers/blob/1de8ce9ee1191ba761a593ac15d9ccbf5851bfc5/src/transformers/models/llama/modeling_llama.py#L133 It was modified to remove unnecessary processing of key states. The method is compatible with transformers <= 4.34.2 and also with the latest version (>=4.35). éNéérr)ÚlenrÚrepeatrÚgatherÚ unsqueezer)ÚqÚcosÚsinÚ position_idsÚgather_indicesÚq_embedr r r Úllama_apply_rotary_pos_emb#s  "rÚmodelcKs| d¡}| d¡}| d¡}| ¡\}}}t|dƒr|j}n|jj}| |¡ ||||j¡  dd¡} |j j |j j } |  |¡ |||| |j¡  dd¡} |} |durjt|tƒrb| |djd 7} n| | |j¡7} d |vrˆ|d \} }|  d¡} | d¡}| | t| ƒ|Sdt |jj¡jvr¢|j| | d \} }t| | ||ƒSd}|durÏ|dur¸tj|||| jd }n| ||j¡}tj|||| jd }| d¡}d|i}d t |jj¡jvrã|||d <|j| fi|¤Ž\} }t| jƒdkrÿ|  d¡} | d¡}| | t| ƒ|S)a Compute query states for Llama models specifically. They need to be recomputed as the forward() method of the original LlamaModel in the transformers library does not return them. See the related discussion in the PR: https://github.com/huggingface/peft/pull/268 Ú hidden_statesrÚpast_key_valueÚ num_headsrrNréþÿÿÿÚposition_embeddings)Úseq_len)Údevicer#r) ÚgetÚsizeÚhasattrr ÚconfigÚnum_attention_headsÚq_projÚviewÚhead_dimÚ transposeÚk_projÚ in_featuresÚ out_featuresÚv_projÚ isinstanceÚtuplerÚget_seq_lengthÚ layer_idxrrÚinspectÚ signatureÚ rotary_embÚforwardÚ parametersrrÚaranger$Úget_usable_lengthr)rÚkwargsrrrÚbszÚq_lenÚ_r Ú query_statesÚfactorÚ value_statesr#rrÚpast_seen_tokensÚnew_cache_positionsÚrotary_emb_kwargsr r r Úllama_compute_query_states<sL     $        rGÚparamscCs| d¡d d¡S)zEReturn True if module is trainable under adaption prompt fine-tuning.Ú.rÚ adaption_)ÚsplitÚ startswith)rHr r r Úis_adaption_prompt_trainable‚srM) r6rÚtorch.nnÚnnÚTensorrrÚModulerGÚstrÚboolrMr r r r Ús  F