o ¾e¦iJã@s¤ddlZddlZddlmZddlmZddlmZmZddl m Z m Z ddl m Z ddlmZd d lmZd d lmZmZmZmZmZmZmZmZmZd d lmZd dlmZe  e!¡Z"dZ#dZ$Gdd„deƒZ%dd„Z&Gdd„deƒZ'Gdd„dej(ƒZ)Gdd„de)ƒZ*Gdd„de)ƒZ+e)e*e+dœZ,Gdd„deƒZ-Gd d!„d!eƒZ.Gd"d#„d#eƒZ/Gd$d%„d%eƒZ0Gd&d'„d'eƒZ1Gd(d)„d)eƒZ2Gd*d+„d+eƒZ3gd,¢Z4dS)-éN)Únné)Úinitialization)ÚCacheÚ StaticCache)Ú_flash_attention_forwardÚ!flash_attn_supports_top_left_mask)ÚPreTrainedModel)Úloggingé)ÚGemmaForCausalLM) ÚLlamaDecoderLayerÚLlamaForQuestionAnsweringÚLlamaForSequenceClassificationÚLlamaForTokenClassificationÚ LlamaModelÚLlamaPreTrainedModelÚLlamaRotaryEmbeddingÚapply_rotary_pos_embÚ repeat_kv)Ú MistralMLPé)ÚDiffLlamaConfigzkajuma/DiffLlama-0.3B-handcutrc@ó eZdZdS)Ú DiffLlamaMLPN©Ú__name__Ú __module__Ú __qualname__©rrúm/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/diffllama/modular_diffllama.pyr1órcCsddt d|¡S)Ngš™™™™™é?g333333ã?g333333Ó¿)ÚmathÚexp)Ú layer_idxrrr Úlambda_init_fn5sr%c@r)ÚDiffLlamaRotaryEmbeddingNrrrrr r&9r!r&cs¦eZdZdZddededBf‡fdd„ Z     ddejd e ejejfd ejdBd ej dBd e dBd e dej dBde ejejdBe ejdBffdd„Z ‡ZS)ÚDiffLlamaAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNÚconfigr$csžtƒ ¡||_||_|durt d|jj›d¡|j|_|j |_ |j |_ t |d|j |j ƒ|_ |j|_|j |j|_|j|_d|_tj|j |j |j |jd|_tj|j |j|j |jd|_tj|j |j|j |jd|_tj|j |j |j |jd|_t|ƒ|_t tjd|j|j fd¡|_t tjd|j|j fd¡|_ t tjd|j|j fd¡|_!t tjd|j|j fd¡|_"tj#d|j |j$d d |_%dS) NzInstantiating z¹ without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Úhead_dimT)Úbiasr)Úsizer F)ÚepsÚelementwise_affine)&ÚsuperÚ__init__r(r$ÚloggerÚ warning_onceÚ __class__rÚattention_dropoutÚ hidden_sizeÚnum_attention_headsÚ num_headsÚgetattrr)Únum_key_value_headsÚnum_key_value_groupsÚmax_position_embeddingsÚ is_causalrÚLinearÚattention_biasÚq_projÚk_projÚv_projÚo_projr%Ú lambda_initÚ ParameterÚtorchÚnormalÚlambda_std_devÚ lambda_q1Ú lambda_k1Ú lambda_q2Ú lambda_k2ÚRMSNormÚ rms_norm_epsÚ groupnorm©Úselfr(r$©r2rr r/@s2 ÿ zDiffLlamaAttention.__init__FÚ hidden_statesÚposition_embeddingsÚattention_maskÚ position_idsÚpast_key_valuesÚ use_cacheÚcache_positionÚreturncKs:| ¡\} } } | } | |¡} | |¡}| |¡}|  | | |j|j¡ dd¡} | | | |j|j¡ dd¡}| | | |j|j¡ dd¡}|\}}t | |||ƒ\} }|durd|||dœ}|  |||j |¡\}}t ||j ƒ}t ||j ƒ}tjtj|ddddd}| dddd¡}t | | dd¡¡t |j¡}|dur||}tjj|dtjd | j¡}tjj||j|jd}t tj|j |j!dtjd¡ | j¡}t tj|j"|j#dtjd¡ | j¡}|||j$}t ||¡}tj|ddd\}}|||}d|j$| %|¡}| dd¡ &¡}| '| | d¡}| (|¡}||fS) Nrr ©ÚsinÚcosrW©Údiméÿÿÿÿr©r]Údtype)ÚpÚtraining))r+r>r?r@Úviewr6r)Ú transposer8rÚupdater$rr9rDÚcatÚchunkÚrepeatÚmatmulr"ÚsqrtrÚ functionalÚsoftmaxÚfloat32Útor`Údropoutr3rbr#ÚsumrGrHrIrJrBrMÚ contiguousÚreshaperA)rOrQrRrSrTrUrVrWÚkwargsÚbszÚ target_lenÚ_Úq_lenÚ query_statesÚ key_statesÚ value_statesr[rZÚ cache_kwargsÚ attn_weightsÚlambda_1Úlambda_2Ú lambda_fullÚ attn_outputÚ attn_output1Ú attn_output2rrr ÚforwardasJ         ÿ ÿ   zDiffLlamaAttention.forward©N©NNNFN)rrrÚ__doc__rÚintr/rDÚTensorÚtupleÚ LongTensorrÚboolrƒÚ __classcell__rrrPr r'=s2%øþýüûúùø ör'cs„eZdZdZ‡fdd„Z     ddejdeejejfdejdBd ejdBd e dBd e d ejdBd eejdffdd„Z ‡Z S)ÚDiffLlamaFlashAttention2aN DiffLlama flash attention module. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cstƒj|i|¤Žtƒ|_dSr„)r.r/rÚ_flash_attn_uses_top_left_mask)rOÚargsrsrPrr r/¦s z!DiffLlamaFlashAttention2.__init__NFrQrRrSrTrUrVrWrXc Csät|tƒr tdƒ‚| ¡\}} } | |¡} | |¡} | |¡} |  || |j|j ¡  dd¡} |  || |j |j ¡  dd¡} |  || |j |j ¡  dd¡} |\}}t | | ||ƒ\} } |durk|||dœ}|  | | |j|¡\} } |   dd¡} |   dd¡} |   dd¡} |jrƒ|jnd}| j}| jjdkr’| jjnd}|tjkrÌt |¡r¤t |¡}nt|jdƒr¯|jj}n|jjj}t d |›d ¡|  |¡} |  |¡} |  |¡} tj| ddd \}}| dddd¡}| dddd¡}t | | ||| ||t!|d dƒ|j"|j#d }t | | ||| ||t!|d dƒ|j"|j#d }tj$||gdd }tj|ddd \}}t %tj&|j'|j(dtjd¡ | j¡}t %tj&|j)|j*dtjd¡ | j¡}|||j+}|||}d|j+| ,|¡}| -|| d¡ .¡}| /|¡}|dfS)NzÈ`static` cache implementation is not compatible with `attn_implementation==flash_attention_2` make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformersrr rYçÚmpsÚcpuÚ _is_quantizedz¾The input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in Ú.r\Úsliding_window)rTror•Úuse_top_left_maskr;r^r_)0Ú isinstancerÚ ValueErrorr+r>r?r@rcr6r)rdr8rrer$rbr3r`ÚdeviceÚtyperDrmÚis_autocast_enabledÚget_autocast_dtypeÚhasattrr(Úweightr0r1rnrgrhrr7rŽr;rfr#rprGrHrIrJrBrMrrrqrA)rOrQrRrSrTrUrVrWrtrwrvrxryrzr[rZr{Ú dropout_rateÚ input_dtypeÚ device_typeÚ target_dtypeÚ value_states1Ú value_states2rr‚r€r}r~rrrr rƒ®sš ÿ             þÿ    ö  ö  ÿ ÿ  z DiffLlamaFlashAttention2.forwardr…) rrrr†r/rDrˆr‰rŠrr‹rƒrŒrrrPr rŸs2  øþýüûúùø ÷rc@s†eZdZdZ     ddejdeejejfdejdBdejdBdedBd e d ejdBd eejejdBeejdBffd d „Z dS)ÚDiffLlamaSdpaAttentiona DiffLlama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. NFrQrRrSrTrUrVrWrXcKs0| ¡\} } } | |¡} | |¡} | |¡}|  | | |j|j¡ dd¡} |  | | |j|j¡ dd¡} | | | |j|j¡ dd¡}|\}}t | | ||ƒ\} } |durb|||dœ}|  | ||j |¡\} }t | |j ƒ} t ||j ƒ}tjtj|ddddd}| dddd¡}|}|durœ|dd…dd…dd…d| jd…f}|duo£| dk}tjjj| | |||jr²|jnd|d}tj|ddd\}}t tj|j|jdtjd ¡ | j¡}t tj|j|j dtjd ¡ | j¡}|||j!}|||}d|j!| "|¡}| dd¡ #¡}| | | d¡}| $|¡}|dfS) Nrr rYr\r^éþÿÿÿr)Ú attn_maskÚ dropout_pr;r_)%r+r>r?r@rcr6r)rdr8rrer$rr9rDrfrgrhÚshaperrkÚscaled_dot_product_attentionrbr3r#rprGrHrmrnr`rIrJrBrMrqrA)rOrQrRrSrTrUrVrWrsrtrwrvrxryrzr[rZr{Ú causal_maskr;r€rr‚r}r~rrrr rƒ.sT      &ú ÿ ÿ  zDiffLlamaSdpaAttention.forwardr…) rrrr†rDrˆr‰rŠrr‹rƒrrrr r¥&s0 øþýüûúùø ör¥)ÚeagerÚflash_attention_2Úsdpacs&eZdZdedef‡fdd„ Z‡ZS)ÚDiffLlamaDecoderLayerr(r$cs&tƒ ||¡t|j||d|_dS)N)r(r$)r.r/ÚDIFFLLAMA_ATTENTION_CLASSESÚ_attn_implementationÚ self_attnrNrPrr r/{szDiffLlamaDecoderLayer.__init__)rrrrr‡r/rŒrrrPr r¯zsr¯c@s$eZdZdZdZe ¡dd„ƒZdS)ÚDiffLlamaPreTrainedModelFcCsnt ||¡t|tƒr5t |jd|jj¡t |j d|jj¡t |j d|jj¡t |j d|jj¡dSdS)Nr) r Ú _init_weightsr—r'ÚinitÚnormal_rGr(rFrHrIrJ)rOÚmodulerrr r´…s  üz&DiffLlamaPreTrainedModel._init_weightsN)rrrÚ_supports_flex_attnÚ_supports_attention_backendrDÚno_gradr´rrrr r³s r³c@r)ÚDiffLlamaModelNrrrrr r»r!r»c@r)ÚDiffLlamaForCausalLMNrrrrr r¼“r!r¼c@r)Ú"DiffLlamaForSequenceClassificationNrrrrr r½—r!r½c@r)ÚDiffLlamaForQuestionAnsweringNrrrrr r¾›r!r¾c@r)ÚDiffLlamaForTokenClassificationNrrrrr r¿Ÿr!r¿)r³r»r¼r½r¾r¿)5r"rDrÚrrµÚ cache_utilsrrÚmodeling_flash_attention_utilsrrÚmodeling_utilsr Úutilsr Úgemma.modeling_gemmar Úllama.modeling_llamar rrrrrrrrÚmistral.modeling_mistralrÚconfiguration_diffllamarÚ get_loggerrr0Ú_CHECKPOINT_FOR_DOCÚ_CONFIG_FOR_DOCrr%r&ÚModuler'rr¥r°r¯r³r»r¼r½r¾r¿Ú__all__rrrr ÚsD     ,  bNý