o i @s"ddlmZddlmZddlmZddlmZmZm Z m Z m Z ddl m Z mZmZmZmZmZddlmZeeZGd d d eZGd d d eZGd dde ZGdddeZGddde ZGddde ZGdddeZGddde ZGdddeZ GdddeZ!gdZ"dS))CausalLMOutputWithPast)Unpack)logging)DeepseekV3DecoderLayer DeepseekV3MLP DeepseekV3MoEDeepseekV3PreTrainedModelDeepseekV3TopkRouter)Qwen3AttentionQwen3ForCausalLM Qwen3Model Qwen3RMSNormQwen3RotaryEmbeddingTransformersKwargs) Dots1Configc@ eZdZdS) Dots1RMSNormN__name__ __module__ __qualname__rrd/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/dots1/modular_dots1.pyr'rc@r)Dots1RotaryEmbeddingNrrrrrr+rrc@r)Dots1AttentionNrrrrrr/rrc@r)Dots1MLPNrrrrrr3rrc@r)Dots1MoENrrrrrr7rrc@r)Dots1TopkRouterNrrrrrr ;rr cs&eZdZdedeffdd ZZS)Dots1DecoderLayerconfig layer_idxcst|||j||_dS)N)super__init__ layer_typesattention_type)selfr"r# __class__rrr%@szDots1DecoderLayer.__init__)rrrrintr% __classcell__rrr)rr!?sr!c@r)Dots1PreTrainedModelNrrrrrr-Err-c@r) Dots1ModelNrrrrrr.Irr.cs*eZdZdeedeffdd ZZS)Dots1ForCausalLM super_kwargsreturnc stjdi|S)a~ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Dots1ForCausalLM >>> model = Dots1ForCausalLM.from_pretrained("rednote-hilab/dots1.llm1.inst") >>> tokenizer = AutoTokenizer.from_pretrained("rednote-hilab/dots1.llm1.inst") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```Nr)r$forward)r(r0r)rrr2NszDots1ForCausalLM.forward)rrrrrrr2r,rrr)rr/Ms r/)r-r.r/N)#modeling_outputsrprocessing_utilsrutilsr deepseek_v3.modeling_deepseek_v3rrrr r qwen3.modeling_qwen3r r r rrrconfiguration_dots1r get_loggerrloggerrrrrrr r!r-r.r/__all__rrrrs$