o ei=@s^dZddlmZddlZddlmZddlmZddlmZddl m Z dd l m Z m Z dd lmZdd lmZdd lmZdd lmZmZddlmZddlmZmZmZmZmZmZddl m!Z!e"e#Z$Gdddej%Z&GdddeZ'ddZ(d&ddZ)GdddeZ*GdddeZ+Gdd d e Z,Gd!d"d"eZ-Gd#d$d$eZ.gd%Z/dS)'zPyTorch Cohere model.)CallableN)nn)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)dynamic_rope_update)ALL_ATTENTION_FUNCTIONS)Unpack)TransformersKwargslogging)maybe_autocast)LlamaAttentionLlamaForCausalLMLlamaMLP LlamaModelLlamaRotaryEmbeddingeager_attention_forward) CohereConfigcs&eZdZdfdd ZddZZS) CohereLayerNormNh㈵>Fcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeepsbias __class__g/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/cohere/modular_cohere.pyr5s  zCohereLayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)NT)keepdimr) dtypetorfloat32meanpowrsqrtr!r )r" hidden_states input_dtyper/variancer(r(r)forward;s  zCohereLayerNorm.forward)NrF)__name__ __module__ __qualname__rr5 __classcell__r(r(r&r)r4src@s eZdZeeddZdS)CohereRotaryEmbeddingc Cs|jddddf|jddd}|dddddf}t|jjtr2|jjdkr2|jjnd}t|dd*|| dd}t j |ddd }| |j }||j } Wdn1sgwY|j|jd | j|jd fS) Nrr*rmpscpuF) device_typeenabledrdimr,)inv_freqfloatexpandshape isinstancedevicetypestrr transposerrepeat_interleavecosattention_scalingsinr-r,) r"x position_idsinv_freq_expandedposition_ids_expandedr=freqsembrLrNr(r(r)r5Fs(&zCohereRotaryEmbedding.forwardN)r6r7r8rno_gradr r5r(r(r(r)r:Esr:cCsB|ddddf}|ddddf}tj| |gddd}|S)N.rrr*r?)rstackflatten)rOx1x2rot_xr(r(r) rotate_halfVsr\cCsj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. rA)r,rC unsqueezer\r-)qkrLrN unsqueeze_dimr,q_embedk_embedr(r(r)apply_rotary_pos_emb^s  rccseZdZfddZZS) CohereMLPcsRt|tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_dS)NF)r%) rrrLinearr#intermediate_size gate_projup_proj down_projr"configr&r(r)r{s zCohereMLP.__init__)r6r7r8rr9r(r(r&r)rdzsrdcseZdZdZddededBffdd Z  ddejde ejejfd ejdBd e dBd ej dBd e e d e ejejdBffddZZS)CohereAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNrk layer_idxcsTt|||j|_|jr(t|j|jf|jd|_t|j|jf|jd|_ dSdS)Nr#r$) rr use_qk_normrnum_attention_headshead_dimlayer_norm_epsq_normnum_key_value_headsk_normr"rkrmr&r(r)rs zCohereAttention.__init__r2position_embeddingsattention_maskpast_key_valuescache_positionkwargsreturncKs>|jdd}g|d|jR}|||} |||} |||} |jr6|| } || } | dd} | dd} | dd} |\} } t | | | | \} } |durj| | |d}| | | |j |\} } t |jjt}||| | | |f|js~dn|j|jd|\}}|jg|dR}||}||fS)Nr*rr)rNrLrzg)dropoutscaling)rErqq_projviewk_projv_projrorsrurJrcupdatermr get_interfacerk_attn_implementationrtrainingattention_dropoutr~reshape contiguouso_proj)r"r2rwrxryrzr{ input_shape hidden_shape query_states key_states value_statesrLrN cache_kwargsattention_interface attn_output attn_weightsr(r(r)r5sD         zCohereAttention.forwardN)NN)r6r7r8__doc__rintrrTensortupler LongTensorr rr5r9r(r(r&r)rls(rlcseZdZdedeffdd Z      ddejdejdBd ejdBd e dBd e dBd ejdBd e ejejfdBde e de eje ejejfdBffddZZS)CohereDecoderLayerrkrmcs@t|j|_t||d|_t||_t|j|jd|_ dS)N)rkrmrn) rrr#rl self_attnrdmlprrrinput_layernormrvr&r(r)rs  zCohereDecoderLayer.__init__NFr2rxrPry use_cacherzrwr{r|c KsL|} ||}|jd|||||||d|\} } ||} | | | }|S)ar Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r2rxrPryrrzrwNr()rrr) r"r2rxrPryrrzrwr{residualhidden_states_attention_hidden_states_mlpr(r(r)r5s   zCohereDecoderLayer.forward)NNNFNN)r6r7r8rrrrrrrboolrr r FloatTensorr5r9r(r(r&r)rs6   rcs"eZdZdeffdd ZZS) CohereModelrkcsBttfddtjD|_tjj d|_ dS)Ncsg|]}t|qSr()r).0rmrkr(r) sz(CohereModel.__init__..rn) rrr ModuleListrangenum_hidden_layerslayersrr#rrnormrjr&rr)rs zCohereModel.__init__)r6r7r8rrr9r(r(r&r)rsrcseZdZfddZ           ddejdBdejdBdejdBdedBd ejdBd ejdBd e dBd e dBd e dBdejdBde ejBde e de fddZZS)CohereForCausalLMcs*t|t||_|j|_|j|_dSr)rrrmodel logit_scaletie_word_embeddingsrjr&r(r)rs   zCohereForCausalLM.__init__Nr input_idsrxrPry inputs_embedslabelsroutput_attentionsoutput_hidden_statesrzlogits_to_keepr{r|c  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}||j }d}|dur]|j d|||jj d| }t ||| j | j| jdS)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, CohereForCausalLM >> model = CohereForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrxrPryrrrrrz)logitsr vocab_size)lossrryr2 attentionsr()rkrrrlast_hidden_staterFrslicelm_headr loss_functionrr ryr2r)r"rrxrPryrrrrrrzrr{outputsr2 slice_indicesrrr(r(r)r5s<%   zCohereForCausalLM.forward) NNNNNNNNNNr)r6r7r8rrrrrrrrr r r r5r9r(r(r&r)rsP      r)rrCoherePreTrainedModel)r)0rcollections.abcrrr cache_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrr modeling_rope_utilsr modeling_utilsr processing_utilsr utilsr r utils.genericrllama.modeling_llamarrrrrrconfiguration_coherer get_loggerr6loggerModulerr:r\rcrdrlrrr__all__r(r(r(r)s4             C9 R