o eime@s,ddlmZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z dd l mZmZmZdd lmZdd lmZdd lmZmZdd lmZmZddlmZmZddlmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+e$,e-Z.ddZ/eddgd;Z?dS)>)Callable)OptionalN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)capture_outputs) GraniteConfigcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2r)j/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/granite/modeling_granite.py rotate_half/sr+rotary_pos_embcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer+)qkcossin unsqueeze_dimq_embedk_embedr)r)r*apply_rotary_pos_emb6s  r5 hidden_statesn_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r#expandreshape)r6r7batchnum_key_value_headsslenhead_dimr)r)r* repeat_kvPs 0r?modulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)Nr rr)r"dtype)ptrainingr)r?num_key_value_groupsr$matmul transposer functionalsoftmaxfloat32torIrGrK contiguous) rArBrCrDrErFrGrH key_states value_states attn_weights attn_outputr)r)r*eager_attention_forward\s  rXcseZdZdZddededBffdd Z    ddejde ejejfdBd ejdBd e dBd ej dBd e e d e ejejffddZZS)GraniteAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNconfig layer_idxcst||_||_t|d|j|j|_|j|j|_ |j |_ |j |_ d|_ tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_dS)Nr>Tbias)super__init__rZr[getattr hidden_sizenum_attention_headsr>r<rLattention_multiplierrFattention_dropout is_causalrLinearattention_biasq_projk_projv_projo_projselfrZr[ __class__r)r*r_ys(  zGraniteAttention.__init__r6position_embeddingsrEpast_key_valuescache_positionrHr8cKs|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } t| | | | \} } |durW| | |d}|| | |j |\} } t |j j t}||| | | |f|jskdn|j|jd|\}}|jg|dR}||}||fS)Nrrr )r1r0rrr@)rGrF)r#r>rhviewrNrirjr5updater[r get_interfacerZ_attn_implementationrXrKrdrFr:rSrk)rmr6rprErqrrrH input_shape hidden_shape query_statesrTrUr0r1 cache_kwargsattention_interfacerWrVr)r)r*forwards8    zGraniteAttention.forwardN)NNNN)__name__ __module__ __qualname____doc__rintr_r$Tensortupler LongTensorrrr| __classcell__r)r)rnr*rYus,rYRMSNormcsFeZdZd deddffdd Zdejdejfdd Zd d ZZ S) GraniteRMSNormư>epsr8Ncs&ttt||_||_dS)z= GraniteRMSNorm is equivalent to T5LayerNorm N)r^r_r Parameterr$onesweightvariance_epsilon)rmrarrnr)r*r_s  zGraniteRMSNorm.__init__r6cCsJ|j}|tj}|djddd}|t||j}|j||S)Nr rT)keepdim) rIrRr$rQpowmeanrsqrtrr)rmr6 input_dtypevariancer)r)r*r|s  zGraniteRMSNorm.forwardcCst|jjd|jS)Nz, eps=)rrr#r)rmr)r)r* extra_reprszGraniteRMSNorm.extra_repr)r) r~rrfloatr_r$rr|rrr)r)rnr*rsrcs$eZdZfddZddZZS) GraniteMLPcsxt||_|j|_|j|_tj|j|j|jd|_tj|j|j|jd|_ tj|j|j|jd|_ t |j |_ dS)Nr\)r^r_rZraintermediate_sizerrfmlp_bias gate_projup_proj down_projr hidden_actact_fnrmrZrnr)r*r_s zGraniteMLP.__init__cCs$||||||}|Sr})rrrr)rmr&rr)r)r*r|s zGraniteMLP.forward)r~rrr_r|rr)r)rnr*rs  rcseZdZdedeffdd Z       ddejdejdBd ejdBd e dBd e dBd e dBd ejdBde ejejfdBde ej e ej ej fdBffddZ ZS)GraniteDecoderLayerrZr[csZt|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ |j |_ dS)N)rZr[r) r^r_rarY self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormresidual_multiplierrlrnr)r*r_s   zGraniteDecoderLayer.__init__NFr6rE position_idsrqoutput_attentions use_cacherrrpr8c Ks|} ||}|jd||||||||d| \}} | ||j}|} ||}||}| ||j}|f} |r>| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r6rErrqrrrrrpNr))rrrrr) rmr6rErrqrrrrrprHresidualself_attn_weightsoutputsr)r)r*r|s."      zGraniteDecoderLayer.forward)NNNFFNN)r~rrrrr_r$rrrboolr FloatTensorr|rr)r)rnr*rs8   rc@sHeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS)GranitePreTrainedModelrZmodelTrrq)r6 attentionsN)r~rrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrrY_can_record_outputsr)r)r)r*r.s  rc s~eZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZZS)GraniteRotaryEmbeddinginv_freqNrZcst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dddS)N rope_typedefaultrF) persistentoriginal_inv_freq) r^r_max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrZrope_parametersrcompute_default_rope_parametersrattention_scalingregister_bufferclone)rmrZdevice rope_init_fnrrnr)r*r_Ds   zGraniteRotaryEmbedding.__init__rz torch.deviceseq_lenr8z torch.TensorcCsZ|jd}t|ddp|j|j}d}d|tjd|dtjdj|tjd|}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetar>Ng?rr rI)rrI) rr`rarbr$arangeint64rRr)rZrrbaser"attention_factorrr)r)r*rTs &z6GraniteRotaryEmbedding.compute_default_rope_parametersc Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t |dd+|| dd}t j ||fdd }| |j}||j} Wdn1slwY|j|jd | j|jd fS) NrrrmpscpuF) device_typeenabledr r!r)rrr9r#rRr isinstancetypestrrrNr$r%r0rr1rI) rmr&rinv_freq_expandedposition_ids_expandedrfreqsembr0r1r)r)r*r|rs0&zGraniteRotaryEmbedding.forwardr})NNN)r~rrr$rrrr_ staticmethodrrrrrno_gradrr|rr)r)rnr*rAs&   rcseZdZdeffdd Zeee         ddej dBdej dBdej dBde dBd ej dBd e dBd e dBd e dBd ej dBdeedefddZZS) GraniteModelrZcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_j|_|dS)Ncsg|]}t|qSr))r).0r[rZr)r* sz)GraniteModel.__init__..rrF)r^r_ pad_token_id padding_idx vocab_sizer Embeddingra embed_tokens ModuleListrangenum_hidden_layerslayersrrnormr rotary_embgradient_checkpointingembedding_multiplier post_initrrnrr*r_s   zGraniteModel.__init__N input_idsrErrq inputs_embedsrroutput_hidden_statesrrrHr8c  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|du|duAr*td|jr9|jr9|r9tdd}|durB| |}||j }|rS|durSt |jd}| duro|dur_| nd} t j| | |jd|jd} |durx| d}t|j||| ||d} |} |j| |d }|rd nd}|rd nd}|jd|jjD]&}|r|| f7}|| f| ||||| |d | }|d} |r||df7}q|| } |r|| f7}t| |r|nd||d S) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Frrr)r)rZrrErrrqr)rr))rErrqrrrrrp)last_hidden_staterqr6r)rZrrr ValueErrorrrKlogger warning_oncerrrget_seq_lengthr$rr#rr-r rrrrr)rmrrErrqrrrrrrrHpast_seen_tokens causal_maskr6rpall_hidden_statesall_self_attns decoder_layer layer_outputsr)r)r*r|s              zGraniteModel.forward) NNNNNNNNN)r~rrrr_rrrr$rrrrrrrrr|rr)r)rnr*rsJ    rcseZdZddiZddiZddgdgfiZfddZee dd e j d Bd e j d Bd e j d Bde d Bde j d Bde j d Bded Bded Bded Bde j d Bdee j BdeedefddZZS)GraniteForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr6logitscs@t|t||_|j|_tj|j|jdd|_| dS)NFr\) r^r_rrrrrfrarrrrnr)r*r_s   zGraniteForCausalLM.__init__NrrrErrqrlabelsrrrrrlogits_to_keeprHr8c  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}||jj }d}|dur^|j d|||jj d| }t ||| j | j| jdS)a Example: ```python >>> from transformers import AutoTokenizer, GraniteForCausalLM >>> model = GraniteForCausalLM.from_pretrained("meta-granite/Granite-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-granite/Granite-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrErrqrrrrrr)rrr)lossrrqr6rr))rZrrrrrrslicerlogits_scaling loss_functionrrrqr6r)rmrrErrqrrrrrrrrrHrr6 slice_indicesrrr)r)r*r|s<"   zGraniteForCausalLM.forward) NNNNNNNNNNr)r~rr_tied_weights_keys_tp_plan_pp_planr_rrr$rrrrrrrrrr|rr)r)rnr*rsZ       r)rrr)r)r@)@collections.abcrtypingrr$r activationsr cache_utilsrr generationr integrationsr r r masking_utilsr modeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_graniter get_loggerr~rr+r5rrr?ModulerrXrYrrrrrrr__all__r)r)r)r*sn             FMAu V