o ۷i@s&ddlmZmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z dd lmZmZdd lmZd d lmZmZmZmZd d lmZddlmZddlmZeeZ dZ!GdddeZ"Gddde Z#GdddeZ$GdddeZ%GdddeZ&GdddeZ'gdZ(dS))OptionalUnionN)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)CausalLMOutputWithPast)Unpack)TransformersKwargslogging)deprecate_kwarg) GlmAttentionGlmForCausalLMGlmForSequenceClassificationGlmForTokenClassification)Phi3MLP) Glm4Config) Glm4RMSNormzTHUDM/GLM-4-9B-0414c@ eZdZdS)Glm4MLPN__name__ __module__ __qualname__rr[/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/glm4/modular_glm4.pyr&rcseZdZdedeffdd Zedddd dd ejd e ejd e ej de e de e de ej de e ejejfdeede eje e ejejfffddZZS)Glm4DecoderLayerconfig layer_idxcsvt|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ t|j|jd|_ t|j|jd|_ dS)N)r r!)eps) super__init__ hidden_size Glm4Attention self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernorm)selfr r! __class__rrr$+s  zGlm4DecoderLayer.__init__past_key_valuepast_key_valuesz4.58)new_nameversionNF hidden_statesattention_mask position_ids use_cachecache_positionposition_embeddingskwargsreturnc Ksr|} ||}|jd|||||||d|\}} ||}| |}|} ||}||}||}| |}|S)N)r5r6r7r2r8r9r:r)r*r'r,r+r(r-) r.r5r6r7r2r8r9r:r;residual_rrrforward6s*     zGlm4DecoderLayer.forward)NNNFNN)rrrrintr$r torchTensorr LongTensorrbooltupler r FloatTensorr? __classcell__rrr/rr*s8   rc@r)r&Nrrrrrr&[rr&cs2eZdZdeedeeefffdd ZZ S)Glm4ForCausalLM super_kwargsr<c stjdi|S)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Glm4ForCausalLM >>> model = Glm4ForCausalLM.from_pretrained("THUDM/GLM-4-9B-0414") >>> tokenizer = AutoTokenizer.from_pretrained("THUDM/GLM-4-9B-0414") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```Nr)r#r?)r.rIr/rrr?`szGlm4ForCausalLM.forward) rrrr r rrErr?rGrrr/rrH_s  rHc@r)Glm4ForSequenceClassificationNrrrrrrJ}rrJc@r)Glm4ForTokenClassificationNrrrrrrKrrK)Glm4PreTrainedModel Glm4ModelrHrJrK))typingrrrA cache_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrprocessing_utilsr utilsr r utils.deprecationr glm.modeling_glmrrrrphi3.modeling_phi3rconfiguration_glm4r modeling_glm4r get_loggerrlogger_CHECKPOINT_FOR_DOCrrr&rHrJrK__all__rrrrs,          1