o ۷i=?@sfdZddlmZmZmZddlZddlmZddlmZddl m Z ddl m Z dd l mZmZdd lmZdd lmZdd lmZdd lmZmZddlmZddlmZmZmZmZm Z m!Z!ddl"m#Z#e$e%Z&Gdddej'Z(Gddde Z)ddZ*d&ddZ+GdddeZ,GdddeZ-Gdd d e Z.Gd!d"d"eZ/Gd#d$d$eZ0gd%Z1dS)'zPyTorch Cohere model.)CallableOptionalUnionN)nn)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)dynamic_rope_update)ALL_ATTENTION_FUNCTIONS)Unpack)TransformersKwargslogging)deprecate_kwarg)LlamaAttentionLlamaForCausalLMLlamaMLP LlamaModelLlamaRotaryEmbeddingeager_attention_forward) CohereConfigcs&eZdZdfdd ZddZZS) CohereLayerNormNh㈵>Fcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeepsbias __class___/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/cohere/modular_cohere.pyr6s  zCohereLayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)NT)keepdimr) dtypetor float32meanpowrsqrtr#r")r$ hidden_states input_dtyper1variancer*r*r+forward<s  zCohereLayerNorm.forward)NrF)__name__ __module__ __qualname__rr7 __classcell__r*r*r(r+r5src@s eZdZeeddZdS)CohereRotaryEmbeddingc Cs|jddddf|jddd}|dddddf}t|jjtr2|jjdkr2|jjnd}tj |dd*|| dd}tj |ddd }| |j }||j } Wdn1shwY|j|jd | j|jd fS) Nrr,rmpscpuF) device_typeenabledrdimr.)inv_freqfloatexpandshape isinstancedevicetypestrr autocast transposerepeat_interleavecosattention_scalingsinr/r.) r$x position_idsinv_freq_expandedposition_ids_expandedr?freqsembrOrQr*r*r+r7Gs(&zCohereRotaryEmbedding.forwardN)r8r9r:r no_gradr r7r*r*r*r+r<Fsr<cCsB|ddddf}|ddddf}tj| |gddd}|S)N.rrr,rA)r stackflatten)rRx1x2rot_xr*r*r+ rotate_halfWsr_c Csj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. rC)r.rE unsqueezer_r/) qkrOrQrS unsqueeze_dimr.q_embedk_embedr*r*r+apply_rotary_pos_emb_s  rfcseZdZfddZZS) CohereMLPcsRt|tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_dS)NF)r') rrrLinearr%intermediate_size gate_projup_proj down_projr$configr(r*r+r~s zCohereMLP.__init__)r8r9r:rr;r*r*r(r+rg}srgcseZdZdZddedeeffdd Zeddd d   dd e j d e e j e j fd ee j dee dee j deede e j ee j ffddZZS)CohereAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNrn layer_idxcsTt|||j|_|jr(t|j|jf|jd|_t|j|jf|jd|_ dSdS)Nr%r&) rr use_qk_normrnum_attention_headshead_dimlayer_norm_epsq_normnum_key_value_headsk_normr$rnrpr(r*r+rs zCohereAttention.__init__past_key_valuepast_key_values4.58new_nameversionr4position_embeddingsattention_maskcache_positionkwargsreturncKsJ|jdd}g|d|jR}|||} |||} |||} |jr6|| } || } | dd} | dd} | dd} |\} } t | | | | \} } |durj| | |d}| | | |j |\} } t }|jjdkrxt|jj}||| | | |f|jsdn|j|jd|\}}|jg|dR}||}||fS)Nr,rr)rQrOreagerg)dropoutscaling)rGrtq_projviewk_projv_projrrrvrxrMrfupdaterprrn_attn_implementationr trainingattention_dropoutrreshape contiguouso_proj)r$r4rrr{rr input_shape hidden_shape query_states key_states value_statesrOrQ cache_kwargsattention_interface attn_output attn_weightsr*r*r+r7sD           zCohereAttention.forwardN)NN)r8r9r:__doc__rrintrrr Tensortupler LongTensorrrr7r;r*r*r(r+ros* rocseZdZdedeffdd Zedddd dd ejd e ejd e ej de e de e de ej de e ejejfdeede eje e ejejfffddZZS)CohereDecoderLayerrnrpcs@t|j|_t||d|_t||_t|j|jd|_ dS)N)rnrprq) rrr%ro self_attnrgmlprruinput_layernormryr(r*r+rs  zCohereDecoderLayer.__init__rzr{r|r}NFr4rrS use_cacherrrrc KsL|} ||}|jd|||||||d|\} } ||} | | | }|S)ar Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r4rrSr{rrrNr*)rrr) r$r4rrSr{rrrrresidualhidden_states_attention_hidden_states_mlpr*r*r+r7s   zCohereDecoderLayer.forward)NNNFNN)r8r9r:rrrrr rrrrboolrrr FloatTensorr7r;r*r*r(r+rs8   rcs"eZdZdeffdd ZZS) CohereModelrncsNttfddtjD|_td|_t j j d|_ dS)Ncsg|]}t|qSr*)r).0rprnr*r+ sz(CohereModel.__init__..rrq) rrr ModuleListrangenum_hidden_layerslayersr< rotary_embrr%runormrmr(rr+rs  zCohereModel.__init__)r8r9r:rrr;r*r*r(r+rsrcseZdZfddZ           ddeejdeejdeejdeee e ej fd eej d eejd ee d ee d ee deejdee ejfdeedefddZZS)CohereForCausalLMcs*t|t||_|j|_|j|_dSr)rrrmodel logit_scaletie_word_embeddingsrmr(r*r+rs   zCohereForCausalLM.__init__Nr input_idsrrSr{ inputs_embedslabelsroutput_attentionsoutput_hidden_statesrlogits_to_keeprrc  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}||j }d}|dur]|j d|||jj d| }t ||| j | j| jdS)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, CohereForCausalLM >> model = CohereForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrSr{rrrrr)logitsr vocab_size)lossrr{r4 attentionsr*)rnrrrlast_hidden_staterHrslicelm_headr loss_functionrr r{r4r)r$rrrSr{rrrrrrrroutputsr4 slice_indicesrrr*r*r+r7s<%   zCohereForCausalLM.forward) NNNNNNNNNNr)r8r9r:rrr rrrrlistrrrrrr r7r;r*r*r(r+r sP      r)rrCoherePreTrainedModel)Nr)2rtypingrrrr r cache_utilsrmodeling_flash_attention_utilsrmodeling_layersr modeling_outputsr r modeling_rope_utilsr modeling_utilsr processing_utilsrutilsrrutils.deprecationrllama.modeling_llamarrrrrrconfiguration_coherer get_loggerr8loggerModulerr<r_rfrgrorrr__all__r*r*r*r+s4            D: R