o ei@s*ddlZddlmZddlmZddlmZddlmZm Z m Z m Z m Z ddl mZmZmZmZmZmZd d lmZeeZGd d d eZGd ddeZGdddeZGddde ZGddde ZGddde ZGdddeZGddde Z GdddeZ!GdddeZ"gdZ#dS) N)CausalLMOutputWithPast)Unpack)logging)DeepseekV3DecoderLayer DeepseekV3MLP DeepseekV3MoEDeepseekV3PreTrainedModelDeepseekV3TopkRouter)Qwen3AttentionQwen3ForCausalLM Qwen3Model Qwen3RMSNormQwen3RotaryEmbeddingTransformersKwargs) Dots1Configc@ eZdZdS) Dots1RMSNormN__name__ __module__ __qualname__rre/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/dots1/modular_dots1.pyr(rc@r)Dots1RotaryEmbeddingNrrrrrr,rrc@r)Dots1AttentionNrrrrrr0rrc@r)Dots1MLPNrrrrrr4rrc@r)Dots1TopkRouterNrrrrrr 8rr c@seZdZddZdS)Dots1MoEc Cs|}||jj}|d|j|j|jjddddjdd}tj||j dddd}t |}| d|d| d d|j|j|jd|j}||d}tj||jdddd}|d|} |jrx| jdd d d } | | } | |j} || fS) Nr)dimrF)kr#sortedrgT)r#keepdimg#B ;)sigmoidgatee_score_correction_biasviewn_groupn_routed_expertstopksumtorch topk_group zeros_likescatter_ unsqueezeexpandreshape masked_fillbooltop_kgathernorm_topk_probrouted_scaling_factor) self router_logitsrouter_logits_for_choice group_scores group_idx group_mask score_maskscores_for_choice topk_indices topk_weights denominatorrrrroute_tokens_to_experts=s2     z Dots1MoE.route_tokens_to_expertsN)rrrrGrrrrr!<s r!cs&eZdZdedeffdd ZZS)Dots1DecoderLayerconfig layer_idxcst|||j||_dS)N)super__init__ layer_typesattention_type)r<rIrJ __class__rrrLXszDots1DecoderLayer.__init__)rrrrintrL __classcell__rrrOrrHWsrHc@seZdZdZdS)Dots1PreTrainedModelN)rrr"_keys_to_ignore_on_load_unexpectedrrrrrS]srSc@r) Dots1ModelNrrrrrrUarrUcs*eZdZdeedeffdd ZZS)Dots1ForCausalLM super_kwargsreturnc stjdi|S)a~ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Dots1ForCausalLM >>> model = Dots1ForCausalLM.from_pretrained("rednote-hilab/dots1.llm1.inst") >>> tokenizer = AutoTokenizer.from_pretrained("rednote-hilab/dots1.llm1.inst") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```Nr)rKforward)r<rWrOrrrYfszDots1ForCausalLM.forward)rrrrrrrYrRrrrOrrVes rV)rSrUrV)$r/modeling_outputsrprocessing_utilsrutilsr deepseek_v3.modeling_deepseek_v3rrr r r qwen3.modeling_qwen3r r rrrrconfiguration_dots1r get_loggerrloggerrrrrr r!rHrSrUrV__all__rrrrs&