o ei@s ddlZddlmZddlmZddlmZddlmZddl m Z ddl m Z m Z d d lmZmZmZmZd d lmZd d lmZd dlmZe eZdZGdddeZGdddeZGdddeZGdddeZ GdddeZ!GdddeZ"gdZ#dS)N)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)CausalLMOutputWithPast)Unpack)TransformersKwargslogging) GlmAttentionGlmForCausalLMGlmForSequenceClassificationGlmForTokenClassification)Phi3MLP) Glm4Config) Glm4RMSNormzTHUDM/GLM-4-9B-0414c@ eZdZdS)Glm4MLPN__name__ __module__ __qualname__rrc/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/glm4/modular_glm4.pyr#rcseZdZdedeffdd Z      ddejdejdBd ejdBd e dBd e dBd ejdBd e ejejfdBde e de eje ejejfdBffddZZS)Glm4DecoderLayerconfig layer_idxcsvt|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ t|j|jd|_ t|j|jd|_ dS)N)rr)eps) super__init__ hidden_size Glm4Attention self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernorm)selfrr __class__rrr!(s  zGlm4DecoderLayer.__init__NF hidden_statesattention_mask position_idspast_key_values use_cachecache_positionposition_embeddingskwargsreturnc Ksr|} ||}|jd|||||||d|\}} ||}| |}|} ||}||}||}| |}|S)N)r.r/r0r1r2r3r4r)r'r$r)r(r%r*) r+r.r/r0r1r2r3r4r5residual_rrrforward3s*     zGlm4DecoderLayer.forward)NNNFNN)rrrrintr!torchTensor LongTensorrbooltuplerr FloatTensorr9 __classcell__rrr,rr's6  rc@r)r#Nrrrrrr#Wrr#cs.eZdZdeedeeBffdd ZZS)Glm4ForCausalLM super_kwargsr6c stjdi|S)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Glm4ForCausalLM >>> model = Glm4ForCausalLM.from_pretrained("THUDM/GLM-4-9B-0414") >>> tokenizer = AutoTokenizer.from_pretrained("THUDM/GLM-4-9B-0414") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```Nr)r r9)r+rCr,rrr9\szGlm4ForCausalLM.forward) rrrrrr?rr9rArrr,rrB[s rBc@r)Glm4ForSequenceClassificationNrrrrrrDyrrDc@r)Glm4ForTokenClassificationNrrrrrrE}rrE)Glm4PreTrainedModel Glm4ModelrBrDrE)$r; cache_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrprocessing_utilsrutilsrr glm.modeling_glmr r r rphi3.modeling_phi3rconfiguration_glm4r modeling_glm4r get_loggerrlogger_CHECKPOINT_FOR_DOCrrr#rBrDrE__all__rrrrs(         0