o i@s"dZddlmZmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZd d lmZd dlmZmZmZmZmZmZmZddlmZee Z!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&GdddeZ'gdZ(dS)zPyTorch BitNet model.)CallableOptionalN)Cache)FlashAttentionKwargs)CausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONS)Unpack)logging)deprecate_kwarg)GemmaMLP)LlamaAttentionLlamaDecoderLayerLlamaForCausalLM LlamaModel LlamaRMSNormapply_rotary_pos_embeager_attention_forward) BitNetConfigc@ eZdZdS) BitNetRMSNormN__name__ __module__ __qualname__rrf/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/bitnet/modular_bitnet.pyr+rcs*eZdZdeffdd ZddZZS) BitNetMLPconfigcs"t|t|j|jd|_dSN)eps)super__init__rintermediate_size rms_norm_eps ffn_sub_norm)selfr! __class__rrr%0s zBitNetMLP.__init__c Cs*|||||||}|S)N) down_projr(act_fn gate_projup_proj)r)xr,rrrforward4s&zBitNetMLP.forward)rrrrr%r1 __classcell__rrr*rr /sr cseZdZdedeffdd Zedddd dd ejd e ejejfd e ejde e d e ej de ede eje ejffddZZS)BitNetAttentionr! layer_idxcs$t||t|j|jd|_dSr")r$r%r hidden_sizer' attn_sub_norm)r)r!r4r*rrr%:szBitNetAttention.__init__past_key_valuepast_key_valuesz4.58)new_nameversionN hidden_statesposition_embeddingsattention_maskcache_positionkwargsreturncKs.|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } t| | | | \} } |durW| | |d}|| | |j |\} } t }|j j dkret |j j }||| | | |f|jsqdn|j|jd|\}}|jg|dR}||}||}||fS)Nrr )sincosr>eagerg)dropoutscaling)shapehead_dimq_projview transposek_projv_projrupdater4rr!_attn_implementationrtrainingattention_dropoutrFreshape contiguousr6o_proj)r)r;r<r=r8r>r? input_shape hidden_shape query_states key_states value_statesrCrB cache_kwargsattention_interface attn_output attn_weightsrrrr1>s:       zBitNetAttention.forward)NN)rrrrintr%r torchTensortuplerr LongTensorr rr1r2rrr*rr39s( r3c@r)BitNetDecoderLayerNrrrrrrcmrrcc@r) BitNetModelNrrrrrrdqrrdcs0eZdZdgZdZdZdeffdd ZZS)BitNetForCausalLMzlm_head.weightNr@c stjdi|S)a$ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, transformers., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, transformers., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BitNetForCausalLM >>> model = BitNetForCausalLM.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> prompt = f'<|begin_of_text|>User: Hey, are you conscious? Can you talk to me?<|eot_id|>Assistant: ' >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=100) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "User: Hey, are you conscious? Can you talk to me?Assistant: No, I'm not conscious. I'm an artificial intelligence designed to assist with information and tasks. How can I help you today?" ```Nr)r$r1)r) super_kwargsr*rrr1zszBitNetForCausalLM.forward) rrr_tied_weights_keys_tp_plan_pp_planrr1r2rrr*rreusre)rerdBitNetPreTrainedModel))__doc__typingrrr_ cache_utilsrmodeling_flash_attention_utilsrmodeling_outputsrmodeling_utilsrprocessing_utilsr utilsr utils.deprecationr gemma.modeling_gemmar llama.modeling_llamarrrrrrrconfiguration_bitnetr get_loggerrloggerrr r3rcrdre__all__rrrrs*         $  4 "