o ei@sddlmZddlZddlmZddlmZddlmZm Z m Z m Z m Z ddl mZdd lmZeeZd ZGd d d eZGd dde ZddZdddZGdddeZGddde ZGddde ZGddde ZgdZdS))OptionalN)logging)LlamaAttentionLlamaForCausalLMLlamaForSequenceClassificationLlamaForTokenClassificationLlamaRotaryEmbedding)Phi3MLP) GlmConfigzTHUDM/glm-4-9bc@ eZdZdS)GlmMLPN__name__ __module__ __qualname__rra/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/glm/modular_glm.pyr%rc @sFeZdZe   d dedBdeddedBdedeffdd Z dS) GlmRotaryEmbeddingNconfigdevicez torch.deviceseq_lenreturnz torch.Tensorc Cst|jd}|jdd}t|ddp|j|j}t||}d}d|tjd|dtjdj |tj d |}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorg?head_dimNrr)dtype)rr) rope_parametersgetgetattr hidden_sizenum_attention_headsinttorcharangeint64tofloat) rrrbaserrdimattention_factorinv_freqrrrcompute_default_rope_parameters*s  &z2GlmRotaryEmbedding.compute_default_rope_parameters)NNN) rrr staticmethodr rr%tupler*r/rrrrr)s rcCs>|ddddf}|ddddf}tj| |fdddS) z*Rotates half the hidden dims of the input..rNrr r,)r&stackflatten)xx1x2rrr rotate_halfKsr:c Cs||}||}|dd|jddfjddd}|dd|jddfjddd}|jd}|dd|f|d|df}}|dd|f|d|df}} ||t||} ||t||} tj| |gdd} tj| | gdd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Nr2rr3) unsqueezeshaperepeat_interleaver:r&cat) qkcossin unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embedrrrapply_rotary_pos_embRs  $$ ""rKcs,eZdZddededBffdd ZZS) GlmAttentionNr layer_idxcs.t||tj|j|j|jdd|_dS)NF)bias)super__init__nnLinearr$rr#o_proj)selfrrM __class__rrrP{s zGlmAttention.__init__)N)rrrr r%rP __classcell__rrrUrrLzs$rLc@r)GlmForCausalLMNrrrrrrXrrXc@r)GlmForSequenceClassificationNrrrrrrYrrYc@r)GlmForTokenClassificationNrrrrrrZrrZ)GlmPreTrainedModelGlmModelrXrYrZ)r )typingrr&torch.nnrQutilsrllama.modeling_llamarrrr r phi3.modeling_phi3r configuration_glmr get_loggerrlogger_CHECKPOINT_FOR_DOCrrr:rKrLrXrYrZ__all__rrrrs$      " (