o ei$@szddlZddlmZddlmZddlZddlmZddlmZ ddl m Z ddl m Z mZmZdd lmZdd lmZmZdd lmZdd lmZmZdd lmZmZmZmZddlm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/m0Z0ddl1m2Z2ddl3m4Z4e-5e6Z7Gdddej8Z9Gdddej8Z:ddZ;edd@ddZd"ej=fd#d$Z?d%d&Z@Gd'd(d(ej8ZAGd)d*d*eAZBGd+d,d,eAZCed-Gd.d/d/ej8ZDeAeBeCd0ZEGd1d2d2eZFe+Gd3d4d4e&ZGe+Gd5d6d6eGZHe+Gd7d8d8eGeZIGd9d:d:eeGZJGd;d<dd>eeGZLgd?ZMdS)AN)Callable)Optional)nn)initialization)ACT2FN)Cache DynamicCache StaticCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hub)create_causal_mask)_flash_attention_forward!flash_attn_supports_top_left_mask)GenericForQuestionAnswering GenericForSequenceClassificationGenericForTokenClassificationGradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)PreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)capture_outputs)DiffLlamaConfigcs$eZdZfddZddZZS) DiffLlamaMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFbias) super__init__config hidden_sizeintermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnselfr* __class__n/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/diffllama/modeling_diffllama.pyr)9s zDiffLlamaMLP.__init__cCs$||||||}|SN)r0r2r.r/)r4xr0r7r7r8forwardCs zDiffLlamaMLP.forward)__name__ __module__ __qualname__r)r; __classcell__r7r7r5r8r$8s  r$c s~eZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZZS)DiffLlamaRotaryEmbeddinginv_freqNr*cst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dddS)N rope_typedefaultrAF) persistentoriginal_inv_freq) r(r)max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr*rope_parametersrBcompute_default_rope_parametersrattention_scalingregister_bufferclone)r4r*device rope_init_fnrAr5r7r8r)Ks   z!DiffLlamaRotaryEmbedding.__init__rNz torch.deviceseq_lenreturnz torch.TensorcCsZ|jd}t|ddp|j|j}d}d|tjd|dtjdj|tjd|}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rdtype)rNrV) rIgetattrr+num_attention_headstorcharangeint64tofloat)r*rNrPbasedimattention_factorrAr7r7r8rJ[s &z8DiffLlamaRotaryEmbedding.compute_default_rope_parametersc Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t |dd+|| dd}t j ||fdd }| |j}||j} Wdn1slwY|j|jd | j|jd fS) Nrr"mpscpuF) device_typeenabledrTr_rU)rAr]expandshaper\rN isinstancetypestrr transposerYcatcosrKsinrV) r4r: position_idsinv_freq_expandedposition_ids_expandedrdfreqsembrnror7r7r8r;ys0&z DiffLlamaRotaryEmbedding.forwardr9)NNN)r<r=r>rYTensor__annotations__r#r) staticmethodrinttupler]rJno_gradrr;r?r7r7r5r8r@Hs&   r@cCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..NrarTrf)rhrYrm)r:x1x2r7r7r8 rotate_halfsr}rotary_pos_embcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer})qkrnro unsqueeze_dimq_embedk_embedr7r7r8apply_rotary_pos_embs  r hidden_statesn_reprQcCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r"N)rhrgreshape)rrbatchnum_key_value_headsslenrSr7r7r8 repeat_kvs 0rcCsddtd|S)Ng?g333333?g333333ӿ)mathexp) layer_idxr7r7r8lambda_init_fnsrcseZdZdZddededBffdd Z     ddejd e ejejfd ejdBd ej dBd e dBd e dej dBde ejejdBe ejdBffddZ ZS)DiffLlamaAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNr*rcst||_||_|durtd|jjd|j|_|j |_ |j |_ t |d|j |j |_ |j|_|j |j|_|j|_d|_tj|j |j |j |jd|_tj|j |j|j |jd|_tj|j |j|j |jd|_tj|j |j |j |jd|_t||_ttjd|j|j fd|_ttjd|j|j fd|_ ttjd|j|j fd|_!ttjd|j|j fd|_"tj#d|j |j$d d |_%dS) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rSTr&r)sizerTF)epselementwise_affine)&r(r)r*rlogger warning_oncer6r<attention_dropoutr+rX num_headsrWrSrnum_key_value_groupsrF is_causalrr-attention_biasq_projk_projv_projo_projr lambda_init ParameterrYnormallambda_std_dev lambda_q1 lambda_k1 lambda_q2 lambda_k2RMSNorm rms_norm_eps groupnormr4r*rr5r7r8r)s2  zDiffLlamaAttention.__init__Frposition_embeddingsattention_maskrppast_key_values use_cachecache_positionrQcKs:|\} } } | } ||} ||}||}| | | |j|jdd} || | |j|jdd}|| | |j|jdd}|\}}t | |||\} }|durd|||d}| |||j |\}}t ||j }t ||j }tjtj|ddddd}|dddd}t| |ddt|j}|dur||}tjj|dtjd| j}tjj||j|jd}ttj|j |j!dtjd| j}ttj|j"|j#dtjd| j}|||j$}t||}tj|ddd\}}|||}d|j$|%|}|dd&}|'| | d}|(|}||fS) Nr"rTrornrrfrarr_rV)ptraining))rrrrviewrrSrlrrupdaterrrrYrmchunkrepeatmatmulrsqrtr functionalsoftmaxfloat32r\rVdropoutrrrsumrrrrrr contiguousrr)r4rrrrprrrkwargsbsz target_len_q_len query_states key_states value_statesrnro cache_kwargs attn_weightslambda_1lambda_2 lambda_full attn_output attn_output1 attn_output2r7r7r8r;sJ             zDiffLlamaAttention.forwardr9NNNFN)r<r=r>__doc__r#rxr)rYrury LongTensorrboolr;r?r7r7r5r8rs2% rcseZdZdZfddZ     ddejdeejejfdejdBd ejdBd e dBd e d ejdBd eejdffddZ Z S)DiffLlamaFlashAttention2aN DiffLlama flash attention module. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cstj|i|t|_dSr9)r(r)r_flash_attn_uses_top_left_mask)r4argsrr5r7r8r)#s z!DiffLlamaFlashAttention2.__init__NFrrrrprrrrQc Cst|tr td|\}} } ||} ||} ||} | || |j|j  dd} | || |j |j  dd} | || |j |j  dd} |\}}t | | ||\} } |durk|||d}| | | |j|\} } | dd} | dd} | dd} |jr|jnd}| j}| jjdkr| jjnd}|tjkrt|rt|}nt|jdr|jj}n|jjj}td |d | |} | |} | |} tj| ddd \}}|dddd}|dddd}t | | ||| ||t!|d d|j"|j#d }t | | ||| ||t!|d d|j"|j#d }tj$||gdd }tj|ddd \}}t%tj&|j'|j(dtjd| j}t%tj&|j)|j*dtjd| j}|||j+}|||}d|j+|,|}|-|| d.}|/|}|dfS)Nz`static` cache implementation is not compatible with `attn_implementation==flash_attention_2` make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformersr"rTrrbrc _is_quantizedzThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .rfsliding_window)rprruse_top_left_maskrrar)0rir ValueErrorrrrrrrrSrlrrrrrrrVrNrjrYris_autocast_enabledget_autocast_dtypehasattrr*weightrrr\rrrrWrrrmrrrrrrrrrrr)r4rrrrprrrrrrrrrrnror dropout_rate input_dtyperd target_dtype value_states1 value_states2rrrrrrr7r7r8r;+s                         z DiffLlamaFlashAttention2.forwardr) r<r=r>rr)rYruryrrrr;r?r7r7r5r8rs2   rc@seZdZdZ     ddejdeejejfdejdBdejdBdedBd e d ejdBd eejejdBeejdBffd d Z dS)DiffLlamaSdpaAttentiona DiffLlama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. NFrrrrprrrrQcKs0|\} } } ||} ||} ||}| | | |j|jdd} | | | |j|jdd} || | |j|jdd}|\}}t | | ||\} } |durb|||d}| | ||j |\} }t | |j } t ||j }tjtj|ddddd}|dddd}|}|dur|ddddddd| jdf}|duo| dk}tjjj| | |||jr|jnd|d}tj|ddd\}}ttj|j|jdtjd | j}ttj|j|j dtjd | j}|||j!}|||}d|j!|"|}|dd#}|| | d}|$|}|dfS) Nr"rTrrfrar) attn_mask dropout_prr)%rrrrrrrSrlrrrrrrrYrmrrrhrrscaled_dot_product_attentionrrrrrrrr\rVrrrrrr)r4rrrrprrrrrrrrrrrnror causal_maskrrrrrrrr7r7r8r;sT      &    zDiffLlamaSdpaAttention.forwardr) r<r=r>rrYruryrrrr;r7r7r7r8rs0  rrcsFeZdZd deddffdd Zdejdejfdd Zd d ZZ S) DiffLlamaRMSNormư>rrQNcs&ttt||_||_dS)z? DiffLlamaRMSNorm is equivalent to T5LayerNorm N)r(r)rrrYonesrvariance_epsilon)r4r+rr5r7r8r)s  zDiffLlamaRMSNorm.__init__rcCsJ|j}|tj}|djddd}|t||j}|j||S)NrTraT)keepdim) rVr\rYrpowmeanrsqrtrr)r4rrvariancer7r7r8r;s  zDiffLlamaRMSNorm.forwardcCst|jjd|jS)Nz, eps=)ryrrhr)r4r7r7r8 extra_reprszDiffLlamaRMSNorm.extra_repr)r) r<r=r>r]r)rYrur;rr?r7r7r5r8rsr)eagerflash_attention_2sdpacseZdZdedeffdd Z      ddejdejdBd ejdBd e dBd e dBd ejdBd e ejejfdBde e dejfddZZS)DiffLlamaDecoderLayerr*rcsXt|j|_t|j||d|_t||_t|j|j d|_ t|j|j d|_ dS)N)r*rr) r(r)r+DIFFLLAMA_ATTENTION_CLASSES_attn_implementation self_attnr$mlprrinput_layernormpost_attention_layernormrr5r7r8r) s  zDiffLlamaDecoderLayer.__init__NFrrrprrrrrrQc Ks^|} ||}|jd|||||||d|\}} | |}|} ||}||}| |}|S)N)rrrprrrrr7)rrrr) r4rrrprrrrrresidualrr7r7r8r;s&     zDiffLlamaDecoderLayer.forward)NNNFNN)r<r=r>r#rxr)rYrurrrryrrr;r?r7r7r5r8r s6   rcs`eZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZefdd ZZS) DiffLlamaPreTrainedModelr*modelTrrF)r attentionscsnt|t|tr5t|jd|jjt|j d|jjt|j d|jjt|j d|jjdSdS)Nr) r( _init_weightsririnitnormal_rr*rrrr)r4moduler5r7r8rKs  z&DiffLlamaPreTrainedModel._init_weights)r<r=r>r#rvbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsrYrzrr?r7r7r5r8r9s rcseZdZdeffdd Zeee       ddej dBdej dBdej dBde dBd ej dBd ej dBd e dBd eed efddZZS)DiffLlamaModelr*cstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr7)r).0rr*r7r8 ^sz+DiffLlamaModel.__init__..rrF)r(r) pad_token_id padding_idx vocab_sizer Embeddingr+ embed_tokens ModuleListrangenum_hidden_layerslayersrrnormr@ rotary_embgradient_checkpointing post_initr3r5rr8r)Ws   zDiffLlamaModel.__init__N input_idsrrpr inputs_embedsrrrrQc Ks|du|duAr td|dur||}|r!|dur!t|jd}|dur<|dur-|nd} tj|jd|jd| }|durE| d}t |j|||||d} |} |j | |d} |j d|jj D]} | | f| | ||||d|} qb|| } t| |d S) Nz:You must specify exactly one of input_ids or inputs_embedsrrr")rN)r*r&rrrrp)rp)rrrprrr)last_hidden_stater)rrr r*get_seq_lengthrYrZrhrNrrr"r rr!r)r4r%rrprr&rrrpast_seen_tokensrrr decoder_layerr7r7r8r;gsP      zDiffLlamaModel.forward)NNNNNNN)r<r=r>r#r)r r!rrYrrur FloatTensorrrrrr;r?r7r7r5r8rUs>  rcseZdZddiZddiZddgdgfiZfddZee dd e j d Bd e j d Bd e j d Bde d Bde j d Bde j d Bded Bde j d Bdee j BdeedefddZZS)DiffLlamaForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputrlogitscs@t|t||_|j|_tj|j|jdd|_| dSr%) r(r)rrrrr-r+r-r$r3r5r7r8r)s   zDiffLlamaForCausalLM.__init__Nrr%rrprr&labelsrrlogits_to_keeprrQc  Ks|jd|||||||d| } | j} t| trt| dn| } || dd| ddf}d}|durB|jd|||jjd| }t ||| j | j | j dS)a Example: ```python >>> from transformers import AutoTokenizer, DiffLlamaForCausalLM >>> model = DiffLlamaForCausalLM.from_pretrained("google/diffllama-7b") >>> tokenizer = AutoTokenizer.from_pretrained("google/diffllama-7b") >>> prompt = "What is your favorite condiment?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "What is your favorite condiment?" ```)r%rrprr&rrN)r/r0r)lossr/rrrr7) rr'rirxslicer- loss_functionr*rrrrr)r4r%rrprr&r0rrr1routputsr slice_indicesr/r2r7r7r8r;s0  zDiffLlamaForCausalLM.forward) NNNNNNNNr)r<r=r>_tied_weights_keys_tp_plan_pp_planr)rrrYrrurr+rrxrrrr;r?r7r7r5r8r,sN      r,c@ eZdZdS)"DiffLlamaForSequenceClassificationNr<r=r>r7r7r7r8r;r;c@seZdZdZdS)DiffLlamaForQuestionAnswering transformerN)r<r=r>r r7r7r7r8r>sr>c@r:)DiffLlamaForTokenClassificationNr<r7r7r7r8r@r=r@)rrr,r;r>r@)r")Nrcollections.abcrtypingrrYrrr activationsr cache_utilsrr r generationr integrationsr r masking_utilsrmodeling_flash_attention_utilsrrmodeling_layersrrrrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrprocessing_utilsrutilsrrrr utils.genericrr utils.output_capturingr!configuration_diffllamar# get_loggerr<rModuler$r@r}rrurxrrrrrrrrrrr,r;r>r@__all__r7r7r7r8sd            A  bM-PK