o eik@sdZddlmZddlmZddlZddlmZddlmZddl m Z m Z dd l m Z dd lmZdd lmZdd lmZmZmZdd lmZmZddlmZmZddlmZmZddlm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(e$)e*Z+Gdddej,Z-ddZ.d6ddZ/Gdddej,Z0 d7dej,dej1d ej1d!ej1d"ej1dBd#e2d$e2fd%d&Z3Gd'd(d(ej,Z4Gd)d*d*eZ5e"Gd+d,d,eZ6e"Gd-d.d.e6Z7Gd/d0d0e6e Z8Gd1d2d2ee6Z9Gd3d4d4ee6Z:gd5Z;dS)8zPyTorch Persimmon model.)Callable)OptionalN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask)FlashAttentionKwargs) GenericForSequenceClassificationGenericForTokenClassificationGradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringcan_return_tuplelogging)maybe_autocast)PersimmonConfigc s~eZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZZS)PersimmonRotaryEmbeddinginv_freqNconfigcst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dddS)N rope_typedefaultrF) persistentoriginal_inv_freq) super__init__max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrrope_parametersrcompute_default_rope_parametersrattention_scalingregister_bufferclone)selfrdevice rope_init_fnr __class__n/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/persimmon/modeling_persimmon.pyr$;s   z!PersimmonRotaryEmbedding.__init__r.z torch.deviceseq_lenreturnz torch.Tensorc Cst|jd}|jdd}t|ddp|j|j}t||}d}d|tjd|dtjdj |tj d |}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorg?head_dimNrdtype)r.r;) r(getgetattr hidden_sizenum_attention_headsinttorcharangeint64tofloat) rr.r4baser7r8dimattention_factorrr2r2r3r)Ks  &z8PersimmonRotaryEmbedding.compute_default_rope_parametersc Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t |dd+|| dd}t j ||fdd }| |j}||j} Wdn1slwY|j|jd | j|jd fS) NrrmpscpuF) device_typeenabledr9rGr:)rrEexpandshaperDr. isinstancetypestrr transposerAcatcosr*sinr;) r-x position_idsinv_freq_expandedposition_ids_expandedrLfreqsembrVrWr2r2r3forwardls0&z PersimmonRotaryEmbedding.forwardN)NNN)__name__ __module__ __qualname__rATensor__annotations__rr$ staticmethodrr@tuplerEr)no_gradrr^ __classcell__r2r2r0r3r8s&   rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..NrIr9rN)rPrArU)rXx1x2r2r2r3 rotate_half}srkcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerk)qkrVrW unsqueeze_dimq_embedk_embedr2r2r3apply_rotary_pos_embs  rrcs$eZdZfddZddZZS) PersimmonMLPcs>tt|j|j|_t|j|j|_t|j |_ dSr_) r#r$rLinearr>intermediate_size dense_h_to_4h dense_4h_to_hr hidden_actactr-rr0r2r3r$s zPersimmonMLP.__init__cCs"||}||}||}|Sr_)rvryrw)r- hidden_statesr2r2r3r^s   zPersimmonMLP.forward)r`rarbr$r^rhr2r2r0r3rss rsmodulequerykeyvalueattention_maskscalingdropoutc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr9rrI)rGr;)ptrainingr) rAmatmulrTr functionalsoftmaxfloat32rDr;rr contiguous) r}r~rrrrrkwargs attn_weights attn_outputr2r2r3eager_attention_forwards  rcseZdZdZddededBffdd Zdejde ejejejffd d Z     dd ejd ejdBdej dBde dBde de dej dBde ejejfdBdeede ejejdBe ejdBffddZZS)PersimmonAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNr layer_idxcs4t||_||_|durtd|jjd|j|_|j |_ |j|j |_ t |j |j d|_d|_|j |j |jkrMtd|jd|j dtj|jd|jdd |_tj|j |j |jdd |_|j|_|j d |_|jrtj|j|j |jdd |_tj|j|j |jdd |_t|j|_dS) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.r7Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).rbiasg)epselementwise_affine)r#r$rrlogger warning_oncer1r`r>r? num_headsr8r@r( rotary_ndims is_causal ValueErrorrrtquery_key_valuedense qk_layernormr LayerNormlayer_norm_eps q_layernorm k_layernormDropoutattention_dropoutr-rrr0r2r3r$s<   zPersimmonAttention.__init__ fused_qkvr5cCsV|j\}}}||||jd|j}|ddddf|ddddf|ddddffS)a Split the last dimension into (num_heads, head_dim) without making any copies, results share same memory storage as `fused_qkv` Args: fused_qkv (`torch.tensor`): [batch_size, seq_length, num_heads * 3 * head_dim] Returns: query: [batch_size, seq_length, num_heads, head_dim] key: [batch_size, seq_length, num_heads, head_dim] value: [batch_size, seq_length, num_heads, head_dim] r.rNrr9)rPviewrr8)r-r batch_size seq_lengththree_times_hidden_sizer2r2r3 _split_headss 4zPersimmonAttention._split_headsFr{rrYpast_key_valuesoutput_attentions use_cachecache_positionposition_embeddingsrc Ks|\} } } ||} || \}}}|jr!||}||}|dd}|dd}|dd}|\}}|dd|jf|d|jdf}}|dd|jf|d|jdf}}t||||\}}t j ||fdd}t j ||fdd}|dur|||j|d}| |||j |\}}t |jjt}||||||f|jsdn|jj|jd| \}}|| | d}||}|sd}||fS) Nrr9.rIrN)rWrVpartial_rotation_sizerr|)rr)sizerrrrrrTrrrrArUupdaterr get_interfacer_attn_implementationrrrrreshaper)r-r{rrYrrrrrrbszq_len_r query_states key_states value_statesrVrW query_rot query_passkey_rotkey_pass cache_kwargsattention_interfacerrr2r2r3r^s\        zPersimmonAttention.forwardr_NNNFFNN)r`rarb__doc__rr@r$rArcrfr LongTensorrboolrr r^rhr2r2r0r3rs@$%   rcseZdZdedeffdd Z       ddejdejdBd ejdBd e dBd e dBd e dBd ejdBde ejejfdBde e de eje ejejfdBffddZZS)PersimmonDecoderLayerrrcsdt|j|_t||d|_t||_tj|j|j d|_ tj|j|j d|_ t |j |_dS)N)rrr)r#r$r>r self_attnrsmlprrrinput_layernormpost_attention_layernormrhidden_dropoutrrr0r2r3r$Ks  zPersimmonDecoderLayer.__init__NFr{rrYrrrrrrr5c Ks~|} ||}|jd||||||||d| \}} | |}|} ||}||}||}|| }|f} |r=| | f7} | S)an Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r{rrYrrrrrNr2)rrrrr) r-r{rrYrrrrrrresidualself_attn_weightsoutputsr2r2r3r^Ts0$       zPersimmonDecoderLayer.forwardr)r`rarbrr@r$rArcrrrrfrr FloatTensorr^rhr2r2r0r3rJs<    rc@s8eZdZUeed<dZdZdgZdZdZ dZ dZ dZ dS)PersimmonPreTrainedModelrmodelTrrN) r`rarbrrdbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraph_supports_sdpa_supports_flash_attn_supports_attention_backendr2r2r2r3rs rcseZdZdZdeffdd Zee         ddej dBdej dBdej dBd e dBd ej dBd e dBd e dBd e dBdej dBdeedefddZZS)PersimmonModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`PersimmonDecoderLayer`] Args: config: PersimmonConfig rcstj|_j|_tjj|j|_t fddt j D|_ tj jjd|_t|jd|_d|_|dS)Ncsg|]}t|qSr2)r).0rrr2r3 sz+PersimmonModel.__init__..rrF)r#r$ pad_token_id padding_idx vocab_sizer Embeddingr> embed_tokens ModuleListrangenum_hidden_layerslayersrrfinal_layernormrr rotary_embgradient_checkpointing post_initrzr0rr3r$s  zPersimmonModel.__init__N input_idsrrYr inputs_embedsrroutput_hidden_statesrrr5c  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|du|duAr*td|jr9|jr9|r9tdd}|rE|durEt |jd}|durN| |}| durj|durZ| nd} t j | | |jd|jd} |durs| d}t|j||| ||d} |} |j| |d }|rd nd}|rd nd}|jD]&}|r|| f7}|| f| ||||| |d | }|d} |r||df7}q|| } |r|| f7}t| |||d S) Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrr)r.)rrrrrrY)rYr2)rrYrrrrr)last_hidden_staterr{ attentions)rrrrrrrrrrrget_seq_lengthrArBrPr.rlr rrrr)r-rrrYrrrrrrrpast_seen_tokens causal_maskr{rall_hidden_statesall_self_attns decoder_layer layer_outputsr2r2r3r^s              zPersimmonModel.forward) NNNNNNNNN)r`rarbrrr$rrrArrcrrrrr rr^rhr2r2r0r3rsJ    rcseZdZddiZfddZee           ddejdBdej dBd ejdBd e dBd ej dBd ejdBd e dBde dBde dBdejdBde ej BdefddZZS)PersimmonForCausalLMzlm_head.weightzmodel.embed_tokens.weightcs@t|t||_|j|_tj|j|jdd|_| dS)NFr) r#r$rrrrrtr>lm_headrrzr0r2r3r$#s   zPersimmonForCausalLM.__init__NrrrrYrrlabelsrrrrlogits_to_keepr5c  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}d}|durX|j ||fd|jj i| }t ||| j | j | jdS)uk labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PersimmonForCausalLM >>> model = PersimmonForCausalLM.from_pretrained("adept/persimmon-8b-base") >>> tokenizer = AutoTokenizer.from_pretrained("adept/persimmon-8b-base") >>> prompt = "human: Hey, what should I eat for dinner?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'human: Hey, what should I eat for dinner?\n\ncat: 🐱\n\nhuman: 😐\n\n' ```N) rrrYrrrrrrr)losslogitsrr{rr2)rrrrrrQr@slicer loss_functionrrrr{r)r-rrrYrrrrrrrrrrr{ slice_indicesrrr2r2r3r^,sH(  zPersimmonForCausalLM.forward) NNNNNNNNNNr)r`rarb_tied_weights_keysr$rrrArrcrrrr@rr^rhr2r2r0r3rsR      rc@ eZdZdS)"PersimmonForSequenceClassificationNr`rarbr2r2r2r3r~ rc@r)PersimmonForTokenClassificationNrr2r2r2r3rrr)rrrrr)r)r|)sd           E  P w_