o %ݫiW>@sxdZddlZddlmZddlmZddlmZmZm Z ddl m Z ddl m Z ddlmZeeZGdd d e ZdS) zThis lobe enables the integration of huggingface pretrained LLAMA2-chat model. Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html Authors * Pooneh Mousavi 2023 * Ha Nguyen 2023 N) Linear4bit) LoraConfigget_peft_modelprepare_model_for_kbit_training)BitsAndBytesConfig)HFTransformersInterface) get_loggercseZdZdZ          d-d ed ed ededededededededededededdffdd Zde j de j fdd Z d!gfd"d#Z d$d%Z &d.de j de j fd'd(Zd)d*Zd+d,ZZS)/LLAMA2u This lobe enables the integration of HuggingFace pretrained LLAMA2 model. Source paper LLAMA2: https://arxiv.org/abs/2307.09288 Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html The model can be finetuned. It will download automatically the model from HuggingFace or use a local path. Notes: - To use this model, you need to install the extra dependencies in recipes/MultiWOZ/response_generation/llama2/extra_requirements.txt - transformers and peft libraries should follow the versions mentioned in the extra_requirements. - Llama 2 is licensed under the LLAMA 2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Arguments --------- source : str HuggingFace hub name: e.g "meta-llama/Llama-2-7b-chat-hf" save_path : str Path (dir) of the downloaded model. freeze : bool (default: False) If True, the model is frozen. If False, the model will be trained alongside with the rest of the pipeline. max_new_tokens: int (default: 200) use_4bit: bool (default: True) bnb_4bit_compute_dtype: str (default: "float16") This sets the computational type which might be different than the input time. For example, inputs might be fp32, but computation can be set to bf16 for speedups. bnb_4bit_quant_type: str (default:"nf4") This sets the quantization data type in the bnb.nn.Linear4Bit layers. Options are FP4 and NF4 data types which are specified by fp4 or nf4. use_nested_quant: bool (default: False) You have set this to False, which means you're not using nested quantization. This seems reasonable, as nested quantization can be computationally expensive. min_length: int (default: 1) The minimum length of the sequence to be generated. Corresponds to the length of the input prompt + min_new_tokens. Its effect is overridden by min_new_tokens, if also set top_k: int (default: 45) The number of highest probability vocabulary tokens to keep for top-k-filtering. top_p: float (default: 0.9) If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation. num_beams: int (default: 8) Number of beams for beam search. 1 means no beam search. early_stopping: bool (default: True) Controls the stopping condition for beam-based methods, like beam-search. It accepts the following values: - True, where the generation stops as soon as there are num_beams complete candidates - False, where an heuristic is applied and the generation stops when is it very unlikely to find better candidates - "never", where the beam search procedure only stops when there cannot be better candidates (canonical beam search algorithm). with_peft: bool (default:False) If set to True, the peft model (model + adaptors) are loaded. If set to False, the original model is loaded. Example ------- >>> model_hub = "meta-llama/Llama-2-7b-chat-hf" >>> save_path = "savedir" >>> model = LLAMA2(model_hub, save_path) >>> tokens = torch.tensor([[1, 1]]) >>> attention_mask = torch.tensor([[1, 1]]) >>> outputs = model(tokens, attention_mask) FTfloat16nf4-?source save_pathfreezemax_new_tokensuse_4bitbnb_4bit_compute_dtypebnb_4bit_quant_typeuse_nested_quant min_lengthtop_ktop_p num_beamsearly_stopping with_peftreturnNcs"||_||_| |_| |_| |_| |_| |_||_||_d|_ t t |}d|_ |rRt ||||d|_ |t jkrR|rRt j\}}|dkrRtdtdtdtj|||d|j d|j|dddd |j_d |j_|r|j st|j|_td d d ddd}t|j||_||jdS)NF) load_in_4bitrrbnb_4bit_use_double_quantrzP================================================================================z>Your GPU supports bfloat16: accelerate training with bf16=TrueT)rrrwith_casual_lmquantization_config)r pad_tokenuse_fastzright皙?@none CAUSAL_LM lora_alpha lora_dropoutrbias task_type)rrrrrrrrris_sbgetattrtorch bnb_configrr cudaget_device_capabilityloggerinfosuper__init__load_tokenizer tokenizerr$ padding_sidermodelrrprint_trainable_parameters)selfrrrrrrrrrrrrrr compute_dtypemajor_config __class__l/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/huggingface_transformers/llama2.pyr;Ss`     zLLAMA2.__init__ input_idsattention_maskcCsDt|j |jj||d}Wd|S1swY|S)aTakes an input a history of conversation and returns its corresponding reply. Arguments --------- input_ids : torch.Tensor A batch of input-id to transform to features. attention_mask : torch.Tensor A batch of attention_mask. Returns ------- output : torch.Tensor Reply to conversation. )rKN)r4set_grad_enabledrr?forward)rArJrKoutputrHrHrIrMs zLLAMA2.forward base_modelc Cs2d|_tj|dd}tdd|Dd}||d|jjj}|dkr,|jj dd |j rdd l m }||jd g|j d |_dd lm}|j}|D]} tj|| jtjdd|| <qO||j||d|dditjdd\} } } ddlm} d|j_d|j_| j|j_d|j_d|j_d|j_i}d|d<d|d<d|d<d|d<d|d<d|d<d|d<d|d<d|d<d|d <||jj_dd!lm }ddiddd"d#}||jfi|t!|j|_t"d$d%d&d'd(d)}t#|j||_i}|$D]\} }|D]}|| vr| %d*||}|||<qq|S)+aFA custom loading ensures SpeechBrain compatibility for Pretrain and model de/serialization. Here, the scope is to remove '.wav2vec2' before loading. Arguments --------- path : str Checkpoint path, file name relative to the repo root. replaceables : List[str] State dict sub-keys that if found, shall be dropped (incl. the 'model.' parent key), elevating key structures. Returns ------- modified_state_dict : see torch.load SpeechBrain-valid deserialized pretrained model. Tcpu) map_locationcss|] }d|vr|VqdS)zembed_tokens.weightNrH).0keyrHrHrI sz,LLAMA2._modify_state_dict..Nri})new_num_tokens)replace_with_bnb_linearlm_head)modules_to_not_convertr#) _load_state_dict_into_meta_model)dtypedevice)r? state_dictloaded_state_dict_keys start_prefix expected_keys device_maprZ is_quantized)QuantizationMethodFr rr rr! llm_int8_enable_fp32_cpu_offloadllm_int8_has_fp16_weightllm_int8_skip_modulesg@llm_int8_thresholdr load_in_8bit bitsandbytes quant_method)dispatch_modelpast_key_values)ra offload_dir offload_index skip_keysr'r(r)r*r+r,zmodel.)&r2r4loadnextgetsizer?rE vocab_sizeresize_token_embeddingsrtransformers.integrationsrVr5transformers.modeling_utilsrYr]keysrandshaper &transformers.utils.quantization_configrc_is_quantized_training_enabledis_8bit_serializableBITS_AND_BYTESquantization_methodrbis_loaded_in_4bitis_loaded_in_8bitr# acceleraterkrrritemsreplace)rApath replaceablesorig_state_dict desired_keyrUrVrYr]rSnew_error_msgsrnstate_dict_indexrcr#rkdevice_map_kwargs lora_configmodified_state_dictparamstagsave_keyrHrHrI_modify_state_dicts            zLLAMA2._modify_state_dictc CsR|D]"\}}t|tjr!|dkr!t||t|j|j|jdq| |qdS)zModify the loaded module linear layers with Linear4bit to be compatible Arguments --------- module : nn.module llama2 model. rW)r0N) named_children isinstancennLinearsetattrr in_features out_featuresr0replace_linear)rAmodulenamechildrHrHrIr6s  zLLAMA2.replace_lineargreedycCst9|dkr#|jj||d|j|j|j|jd|jddd|j d }n|jj||j|d}Wd|SWd|S1s@wY|S)aTakes an input a history of conversation and returns its corresponding reply. Arguments --------- input_ids : torch.Tensor A batch of input-id which are dialogue context tokens attention_mask : torch.Tensor A batch of attention_mask. decoder_type : str It shows strategy for autoregressive decoding either beam search or greedy. Returns ------- hyp : torch.Tensor Reply to conversation input. beamTg?r ) rJrK do_samplerrrr temperaturernum_return_sequencesrepetition_penaltylength_penaltyr)rJrrKN) r4no_gradr?generaterrrrrr)rArJrK decoder_typehyprHrHrIrKs:   zLLAMA2.generatecCs |jr|j|j|j|jd}|S)aoverride config to include quantization config. Arguments --------- config : HuggingFace config object The original config. Returns ------- config : HuggingFace config object Overridden config. ) cache_dirr#)r5from_pretrainedrr)rArErHrHrIoverride_config}s zLLAMA2.override_configc Cs`d}d}|D]\}}||7}|jr||7}qtd|d|dd||dS)zI Prints the number of trainable parameters in the model. rztrainable params: z || all params: z || trainable%: dN)named_parametersnumel requires_gradr8r9)rAr?trainable_params all_paramrDparamrHrHrIr@s  z!LLAMA2.print_trainable_parameters) Fr Tr r Fr rrrTF)r)__name__ __module__ __qualname____doc__strboolintfloatr;r4TensorrMrrrrr@ __classcell__rHrHrFrIr sr=     M 2r )rr4torch.nnrbitsandbytes.nnrpeftrrr transformersr=speechbrain.lobes.models.huggingface_transformers.huggingfacerspeechbrain.utils.loggerrrr8r rHrHrHrIs