o wiB@sdZddlmZmZmZddlZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZmZdd lmZdd lmZdd lmZdd lmZmZddlmZmZmZmZmZm Z m!Z!ddl"m#Z#e$e%Z&Gdddej'Z(Gddde Z)ddZ*d)ddZ+GdddeZ,GdddeZ-Gddde Z.Gd d!d!eZ/Gd"d#d#eZ0Gd$d%d%e eZ1Gd&d'd'eZ2gd(Z3dS)*zPyTorch Cohere model.)CallableOptionalUnionN)nn)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)dynamic_rope_update)ALL_ATTENTION_FUNCTIONS)Unpack) LossKwargslogging)LlamaAttentionLlamaForCausalLMLlamaMLP LlamaModelLlamaPreTrainedModelLlamaRotaryEmbeddingeager_attention_forward) CohereConfigcs&eZdZdfdd ZddZZS) CohereLayerNormNh㈵>Fcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeepsbias __class__f/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/cohere/modular_cohere.pyr7s  zCohereLayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)NT)keepdimr) dtypetor float32meanpowrsqrtr#r")r$ hidden_states input_dtyper1variancer*r*r+forward=s  zCohereLayerNorm.forward)NrF)__name__ __module__ __qualname__rr7 __classcell__r*r*r(r+r6src@s eZdZeeddZdS)CohereRotaryEmbeddingc Cs|jddddf|jddd}|dddddf}t|jjtr2|jjdkr2|jjnd}tj |dd*|| dd}tj |ddd }| |j }||j } Wdn1shwY|j|jd | j|jd fS) Nrr,rmpscpuF) device_typeenabledrdimr.)inv_freqfloatexpandshape isinstancedevicetypestrr autocast transposerepeat_interleavecosattention_scalingsinr/r.) r$x position_idsinv_freq_expandedposition_ids_expandedr?freqsembrOrQr*r*r+r7Hs(&zCohereRotaryEmbedding.forwardN)r8r9r:r no_gradr r7r*r*r*r+r<Gsr<cCsB|ddddf}|ddddf}tj| |gddd}|S)N.rrr,rA)r stackflatten)rRx1x2rot_xr*r*r+ rotate_halfXsr_c Csj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. rC)r.rE unsqueezer_r/) qkrOrQrS unsqueeze_dimr.q_embedk_embedr*r*r+apply_rotary_pos_emb`s  rfcseZdZfddZZS) CohereMLPcsRt|tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_dS)NF)r') rrrLinearr%intermediate_size gate_projup_proj down_projr$configr(r*r+rs zCohereMLP.__init__)r8r9r:rr;r*r*r(r+rg~srgcseZdZdZddedeeffdd Z  ddej de ej ej fd eej d ee d eej d e ed e ej eej ee ej ffddZZS)CohereAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNrn layer_idxcsTt|||j|_|jr(t|j|jf|jd|_t|j|jf|jd|_ dSdS)Nr%r&) rr use_qk_normrnum_attention_headshead_dimlayer_norm_epsq_normnum_key_value_headsk_normr$rnrpr(r*r+rs zCohereAttention.__init__r4position_embeddingsattention_maskpast_key_valuecache_positionkwargsreturncKsJ|jdd}g|d|jR}|||} |||} |||} |jr6|| } || } | dd} | dd} | dd} |\} } t | | | | \} } |durj| | |d}| | | |j |\} } t }|jjdkrxt|jj}||| | | |f|jsdn|j|jd|\}}|jg|dR}||}||fS)Nr,rr)rQrOr}eager)dropoutscaling)rGrtq_projviewk_projv_projrrrvrxrMrfupdaterprrn_attn_implementationr trainingattention_dropoutrreshape contiguouso_proj)r$r4rzr{r|r}r~ input_shape hidden_shape query_states key_states value_statesrOrQ cache_kwargsattention_interface attn_output attn_weightsr*r*r+r7sD           zCohereAttention.forwardN)NN)r8r9r:__doc__rrintrr Tensortupler LongTensorrrr7r;r*r*r(r+ros(rocseZdZdedeffdd Z       ddejdeejd eej d ee d ee d ee d eej dee ejejfde ede ejee ejejfffddZZS)CohereDecoderLayerrnrpcs@t|j|_t||d|_t||_t|j|jd|_ dS)N)rnrprq) rrr%ro self_attnrgmlprruinput_layernormryr(r*r+rs  zCohereDecoderLayer.__init__NFr4r{rSr|output_attentions use_cacher}rzr~rc  Ksb|} ||}|jd||||||||d| \} } ||} | | | }|f}|r/|| f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r4r{rSr|rrr}rzNr*)rrr)r$r4r{rSr|rrr}rzr~residualhidden_states_attentionself_attn_weightshidden_states_mlpoutputsr*r*r+r7s(   zCohereDecoderLayer.forward)NNNFFNN)r8r9r:rrrr rrrrboolrrr FloatTensorr7r;r*r*r(r+rs<    rc@seZdZddZdS)CoherePreTrainedModelcCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rC|jjjd|d|j durA|jj|j  dSdSt|t rQ|jj ddSdS)Nr)r1stdg?)rninitializer_rangerHrrhr"datanormal_r'zero_ Embedding padding_idxrfill_)r$modulerr*r*r+ _init_weightss     z#CoherePreTrainedModel._init_weightsN)r8r9r:rr*r*r*r+rs rcs"eZdZdeffdd ZZS) CohereModelrncsNttfddtjD|_td|_t j j d|_ dS)Ncsg|]}t|qSr*)r).0rprnr*r+ !sz(CohereModel.__init__..rrq) rrr ModuleListrangenum_hidden_layerslayersr< rotary_embrr%runormrmr(rr+rs  zCohereModel.__init__)r8r9r:rrr;r*r*r(r+rsrc@s eZdZdS)KwargsForCausalLMN)r8r9r:r*r*r*r+r's rcseZdZfddZ           ddeejdeejdeejdeee e ej fd eej d eejd ee d ee d ee deejdee ejfdeedefddZZS)CohereForCausalLMcs*t|t||_|j|_|j|_dSr)rrrmodel logit_scaletie_word_embeddingsrmr(r*r+r+s   zCohereForCausalLM.__init__Nr input_idsr{rSpast_key_values inputs_embedslabelsrroutput_hidden_statesr}logits_to_keepr~rc  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}||j }d}|dur]|j d|||jj d| }t ||| j | j| jdS)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, CohereForCausalLM >> model = CohereForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rr{rSrrrrrr})logitsr vocab_size)lossrrr4 attentionsr*)rnrrrlast_hidden_staterHrslicelm_headr loss_functionrr rr4r)r$rr{rSrrrrrrr}rr~rr4 slice_indicesrrr*r*r+r71s<%   zCohereForCausalLM.forward) NNNNNNNNNNr)r8r9r:rrr rrrrlistrrrrrr r7r;r*r*r(r+r*sP      r)rrr)Nr)4rtypingrrrr torch.utils.checkpointr cache_utilsrmodeling_flash_attention_utilsrmodeling_layersr modeling_outputsr r modeling_rope_utilsr modeling_utilsr processing_utilsrutilsrrllama.modeling_llamarrrrrrrconfiguration_coherer get_loggerr8loggerModulerr<r_rfrgrorrrrr__all__r*r*r*r+s8       $  CE  R