o i@sdZddlmZmZmZddlZddlmZddlmZddl m Z m Z ddl m Z dd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZmZddlmZmZddlm Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)e$rddl*m+Z+ddl,m-Z-e%.e/Z0Gdddej1Z2ddZ3d7ddZ4Gdddej1Z5 d8dej1d ej6d!ej6d"ej6d#eej6d$e7d%e7fd&d'Z8Gd(d)d)ej1Z9Gd*d+d+eZ:e"Gd,d-d-eZ;e"Gd.d/d/e;ZGd4d5d5ee;Z?gd6Z@dS)9zPyTorch Persimmon model.)CallableOptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs) GenericForSequenceClassificationGenericForTokenClassificationGradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)deprecate_kwarg)PersimmonConfig) BlockMask)make_flex_block_causal_maskcsDeZdZUejed<ddeffdd Zee ddZ Z S) PersimmonRotaryEmbeddinginv_freqNconfigcstt|drt|jtr|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultr!F) persistent)super__init__hasattr isinstancer#dictgetr$max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr"r rope_init_fnattention_scalingregister_bufferr!original_inv_freq)selfr"devicer! __class__m/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/persimmon/modeling_persimmon.pyr)>s   z!PersimmonRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) NrrmpscpuF) device_typeenableddim)dtype)r!floatexpandshapetor6r+r%strtorchautocast transposecatcosr2sinrC) r5x position_idsinv_freq_expandedposition_ids_expandedr>freqsembrMrNr9r9r:forwardOs0&z PersimmonRotaryEmbedding.forwardN) __name__ __module__ __qualname__rITensor__annotations__rr)no_gradrrU __classcell__r9r9r7r:r ;s r cCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr;r@rA)rFrIrL)rOx1x2r9r9r: rotate_half`sr`cCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer`)qkrMrNrP unsqueeze_dimq_embedk_embedr9r9r:apply_rotary_pos_embhs  rgcs$eZdZfddZddZZS) PersimmonMLPcs>tt|j|j|_t|j|j|_t|j |_ dSrV) r(r)rLinear hidden_sizeintermediate_size dense_h_to_4h dense_4h_to_hr hidden_actactr5r"r7r9r:r)s zPersimmonMLP.__init__cCs"||}||}||}|SrV)rlrorm)r5 hidden_statesr9r9r:rUs   zPersimmonMLP.forward)rWrXrYr)rUr]r9r9r7r:rhs rhmodulequerykeyvalueattention_maskscalingdropoutc Kst||dd|}|dur'|ddddddd|jdf} || }tjj|dtjd|j }tjj |||j d}t||} | dd } | |fS)Nr@rr;)rBrC)ptrainingr) rImatmulrKrFr functionalsoftmaxfloat32rGrCryr| contiguous) rsrtrurvrwrxrykwargs attn_weights causal_mask attn_outputr9r9r:eager_attention_forwards & rcseZdZdZddedeeffdd Zdej de ej ej ej ffd d Z e d d d d       ddej deej deej d eedededeej dee ej ej fdeede ej eej ee ej ffddZZS)PersimmonAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNr" layer_idxcsFt||_||_|durtd|jjd|j|_|j |_ |j|j |_ |j |_ t |j |j|_d|_|j |j |jkrOtd|jd|j dtj|jd|jdd|_tj|j |j |jdd|_|j|_|j d |_|jrtj|j|j |jdd |_tj|j|j |jdd |_t|j|_t|jd |_dS) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).rbiasg)epselementwise_affiner") r(r)r"rlogger warning_oncer8rWrjnum_attention_heads num_headshead_dim rope_thetaintpartial_rotary_factor rotary_ndims is_causal ValueErrorrriquery_key_valuedense qk_layernormrx LayerNormlayer_norm_eps q_layernorm k_layernormDropoutattention_dropoutr rotary_embr5r"rr7r9r:r)s@   zPersimmonAttention.__init__ fused_qkvreturncCsV|j\}}}||||jd|j}|ddddf|ddddf|ddddffS)a Split the last dimension into (num_heads, head_dim) without making any copies, results share same memory storage as `fused_qkv` Args: fused_qkv (`torch.tensor`): [batch_size, seq_length, num_heads * 3 * head_dim] Returns: query: [batch_size, seq_length, num_heads, head_dim] key: [batch_size, seq_length, num_heads, head_dim] value: [batch_size, seq_length, num_heads, head_dim] r.rNrr@)rFviewrr)r5r batch_size seq_lengththree_times_hidden_sizer9r9r: _split_headss 4zPersimmonAttention._split_headspast_key_valuepast_key_values4.58new_nameversionFrqrwrPoutput_attentions use_cachecache_positionposition_embeddingsrc Ks|\} } } ||} || \}}}|jr!||}||}|dd}|dd}|dd}|\}}|dd|jf|d|jdf}}|dd|jf|d|jdf}}t||||\}}t j ||fdd}t j ||fdd}|dur|||j|d}| |||j |\}}t }|jjdkrt|jj}||||||f|jsdn|jj|jd | \}}|| | d}||}|sd}||fS) Nrr@.r;rA)rNrMpartial_rotation_sizereagerrr)ryrx)sizerrrrrrKrrgrIrLupdaterrr"_attn_implementationrr|rrxreshaper)r5rqrwrPrrrrrrbszq_len_r query_states key_states value_statesrMrN query_rot query_passkey_rotkey_pass cache_kwargsattention_interfacerrr9r9r:rUs\          zPersimmonAttention.forwardrVNNNFFNN)rWrXrY__doc__rrrr)rIrZtuplerr LongTensorrboolrr rUr]r9r9r7r:rsB$&    rcseZdZdedeffdd Zedddd dd ejd e ejd e ej de e de e de e de ej de e ejejfdeede eje e ejejfffddZZS)PersimmonDecoderLayerr"rcsdt|j|_t||d|_t||_tj|j|j d|_ tj|j|j d|_ t |j |_dS)N)r"rr)r(r)rjr self_attnrhmlprrrinput_layernormpost_attention_layernormrhidden_dropoutryrr7r9r:r)5s  zPersimmonDecoderLayer.__init__rrrrNFrqrwrPrrrrrrc Ks~|} ||}|jd||||||||d| \}} | |}|} ||}||}||}|| }|f} |r=| | f7} | S)an Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )rqrwrPrrrrrNr9)rrrrry) r5rqrwrPrrrrrrresidualself_attn_weightsoutputsr9r9r:rU>s0%       zPersimmonDecoderLayer.forwardr)rWrXrYrrr)rrIrZrrrrrrr FloatTensorrUr]r9r9r7r:r4s>    rc@s@eZdZUeed<dZdZdgZdZdZ dZ dZ dZ ddZ dS) PersimmonPreTrainedModelr"modelTrrcCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rC|jjjd|d|j durA|jj|j  dSdSt|tj rX|jj d|jj dSdS)Nrr)meanstdg?)r"initializer_ranger+rriweightdatanormal_rzero_ Embedding padding_idxrfill_)r5rsrr9r9r: _init_weightss     z&PersimmonPreTrainedModel._init_weightsN)rWrXrYrr[base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraph_supports_sdpa_supports_flash_attn_supports_attention_backendrr9r9r9r:rs  rcseZdZdZdeffdd Zee         ddee j dee j dee j d ee d ee j d eed eed eedee j deedefddZ ddee j dfde j de j d e d ef ddZede j dedede jde j def ddZZS) PersimmonModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`PersimmonDecoderLayer`] Args: config: PersimmonConfig r"cstj|_j|_tjj|j|_t fddt j D|_ tj jjd|_td|_d|_|dS)Ncsg|]}t|qSr9)r).0rrr9r: sz+PersimmonModel.__init__..rrF)r(r) pad_token_idr vocab_sizerrrj embed_tokens ModuleListrangenum_hidden_layerslayersrrfinal_layernormr rgradient_checkpointing post_initrpr7rr:r)s   zPersimmonModel.__init__N input_idsrwrPr inputs_embedsrroutput_hidden_statesrrrc  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|du|duAr*td|jr9|jr9|r9tdd}|rE|durEt |jd}|durN| |}| durj|durZ| nd} t j | | |jd|jd} |durs| d}|||| ||} |} || |}|rdnd}|rdnd}|jD]&}|r|| f7}|| f| ||||| |d | }|d} |r||df7}q|| } |r|| f7}t| |||d S) Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrrr6r9)rwrPrrrrr)last_hidden_staterrq attentions)r"rrrrrr|rrr rget_seq_lengthrIarangerFr6ra_update_causal_maskrrrr)r5rrwrPrrrrrrrpast_seen_tokensrrqrall_hidden_statesall_self_attns decoder_layer layer_outputsr9r9r:rUsv               zPersimmonModel.forwardFr input_tensorc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) Nflash_attention_2rrflex_attentionrFsdpa)rpast_key_values_length is_trainingrr;)sequence_length target_lengthrCrr)cudaxpunpu)r"ranyr+rIrZrris_compileabler _ignore_causal_mask_sdpar|rCrFget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionr6r%finfomin_unmask_unattended) r5rwrrrrrusing_compilable_cacherCr rr min_dtyper9r9r:rsT           z"PersimmonModel._update_causal_maskr rrCrc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuerCr6r)diagonalrr;r)rBrIrrfullr6triurrrEclonerFrG masked_fill) rwr rrCrrrrr mask_length padding_maskr9r9r:r\s,  $ 6  zDPersimmonModel._prepare_4d_causal_attention_mask_with_cache_position) NNNNNNNNN)F)rWrXrYrrr)rrrrIrrZrrrrr rrUrr staticmethodrrCrr]r9r9r7r:rs~    a  DrcseZdZdgZfddZee           ddeej deej deej d ee d eej d eej d ee d ee dee deej deeej fdefddZZS)PersimmonForCausalLMzlm_head.weightcs@t|t||_|j|_tj|j|jdd|_| dS)NFr) r(r)rrrrrirjlm_headrrpr7r9r:r)s   zPersimmonForCausalLM.__init__NrrrwrPrrlabelsrrrrlogits_to_keeprc  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}d}|durX|j ||fd|jj i| }t ||| j | j | jdS)uk labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PersimmonForCausalLM >>> model = PersimmonForCausalLM.from_pretrained("adept/persimmon-8b-base") >>> tokenizer = AutoTokenizer.from_pretrained("adept/persimmon-8b-base") >>> prompt = "human: Hey, what should I eat for dinner?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'human: Hey, what should I eat for dinner?\n\ncat: 🐱\n\nhuman: 😐\n\n' ```N) rrwrPrrrrrrr)losslogitsrrqrr9)r"rrrrr+rslicer' loss_functionrrrrqr)r5rrwrPrrr(rrrrr)rrrq slice_indicesr+r*r9r9r:rUsH(  zPersimmonForCausalLM.forward) NNNNNNNNNNr)rWrXrY_tied_weights_keysr)rrrrIrrZrrrrrrrUr]r9r9r7r:r&sR      r&c@ eZdZdS)"PersimmonForSequenceClassificationNrWrXrYr9r9r9r:r1 r1c@r0)PersimmonForTokenClassificationNr2r9r9r9r:r4r3r4)r&rrr1r4)Nr)rr)ArtypingrrrrIr activationsr cache_utilsrr generationr modeling_attn_mask_utilsr modeling_flash_attention_utilsr modeling_layersr rrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrutils.deprecationrconfiguration_persimmonr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerrWrModuler r`rgrhrZrDrrrrrr&r1r4__all__r9r9r9r:sj           %   Qt_