o Ti1Q@sddlZddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZmZmZmZmZmZddlmZddlmZd d lm Z d d l!m"Z"d d l#m$Z$m%Z%eroddl&m'Z'ervddl(Z(GdddeZ)dS)N)Path)CallableOptionalUnion)Dataset) DataLoader) AutoModelForCausalLM AutoTokenizerBaseImageProcessor DataCollatorDataCollatorForLanguageModelingDataCollatorForSeq2SeqFeatureExtractionMixinPreTrainedModelPreTrainedTokenizerBaseProcessorMixinTrainerTrainingArgumentsis_wandb_available)EvalLoopOutput)is_peft_available) PPODecorators)IterativeSFTConfig)generate_model_cardget_comet_experiment_url) PeftModelcseZdZdZddgZ       d(deeefdeee e fdee d eee e ee ffd eeeeeefd eejjejjjfd eeejejgejfd eeege fffdd Zdede defddZdejdejdejfddZedeej deej deej deedeef ddZ!e"#     d)deeej deeej deeej deeedeeef ddZ$dd Z%fd!d"Z&   d*d#eed$eed%eeeedffd&d'Z'Z(S)+IterativeSFTTrainerat The IterativeSFTTrainer can be used to finetune models with methods that requires some steps between optimization. Args: model (`Union[str, PreTrainedModel]`): Model to be trained. Can be either: - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a path to a *directory* containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded using [`~transformers.AutoModelForCausalLM.from_pretrained`] with the keyword arguments in `args.model_init_kwargs`. - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported. args ([`IterativeSFTConfig`], *optional*, defaults to `None`): Configuration for this trainer. If `None`, a default configuration is used. data_collator (`DataCollator`, *optional*): Function to use to form a batch from a list of elements of the processed `train_dataset` or `eval_dataset`. Will default to [`~transformers.default_data_collator`] if no `processing_class` is provided, an instance of [`~transformers.DataCollatorWithPadding`] otherwise if the processing_class is a feature extractor or tokenizer. eval_dataset (`datasets.Dataset`): The dataset to use for evaluation. processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*, defaults to `None`): Processing class used to process the data. If `None`, the processing class is loaded from the model's name with [`~transformers.AutoTokenizer.from_pretrained`]. optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`): The optimizer and scheduler to use for training. preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`): The function to use to preprocess the logits before computing the metrics. compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*): The function to use to compute the metrics. Must take a `EvalPrediction` and return a dictionary string to metric values. trlz iterative-sftNNNmodelargs data_collator eval_datasetprocessing_class optimizerspreprocess_logits_for_metricscompute_metricsc st|tr|n|jj} |dur| dd} t| d}nt|tr=t|ts=|} |j| d<| dtdi| }|durFt | }|j durUt|tsUt dt|tr`|||}trlt|trld|_nd|_||_t|jd d|_|dur|jrt|d d d |_n t|jdd |_n||_|j|_|j|_|j|_tj|||j|||||dt|jdr|j |j!|"|j#j$|j%&|j|j'|j(\|_|_'|_(|jdkrdnd|j_)t|dst*d|jt+_dS)N/z -IterativeSFT hub_tokenpush_to_hub_tokenzYou passed model_init_kwargs to the `IterativeSFTConfig`, but your model is already instantiated. The `model_init_kwargs` will be ignored.TFis_encoder_decoder)label_pad_token_idpad_to_multiple_of)mlm)r!r"r#r$r%r(r&r'add_model_tagskeep_endleftright acceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.), isinstancestrconfig _name_or_pathsplitrrto_dictr+popr from_pretrainedmodel_init_kwargswarningswarn_create_model_from_pathrr is_peft_modelr%getattrr-r r#r max_lengthtruncation_modeoptimize_device_cachesuper__init__hasattrr!r3 _tag_namescreate_optimizer_and_schedulerr" max_stepsr7prepare optimizer lr_schedulertruncation_sideAttributeErrorr) selfr!r"r#r$r%r&r'r(model_id model_name dict_args __class__r8U/home/ubuntu/.local/lib/python3.10/site-packages/trl/trainer/iterative_sft_trainer.pyrK[sl          zIterativeSFTTrainer.__init__ model_pathreturncCs|jpi}tj|fi|S)z0Creates a model from a path or model identifier.)rArr@)rUr\r"rAr8r8r[rDs z+IterativeSFTTrainer._create_model_from_path input_idsattention_masklabelscs|dur dd|D}jr4ddt|||Djj}|ddd|d|djjk<nddt||Djj}j durwj dkr]fd d | D}|Sj d krofd d | D}|St d j |S)NcSsg|]}t|qSr8)torch ones_like).0idsr8r8r[ z.cSsg|] \}}}|||dqS)r^r_r`r8)rcrdattlabr8r8r[res decoder_input_idsr.r`cSsg|] \}}||dqS))r^r_r8)rcrdrhr8r8r[res keep_startcs i|] \}}||djqSNrGrckvrUr8r[ s z.r4cs"i|] \}}||j dqSrlrmrnrqr8r[rrs"zUnknown truncation mode: ) r-r#ziptor!devicer?r% pad_token_idrGrHitems ValueError)rUr^r_r` input_datar8rqr[prepare_model_inputss4       z(IterativeSFTTrainer.prepare_model_inputstexts texts_labelscCs||durs|dur=tddg||gD]*\}}t|ts%t|dt|t|dtjs;td|dt|dqnztgd|||gD]*\}}t|ts[t|dt|t|dtjsqtd|dt|dqGnDt|tstd t|t|dtstd t|d|durt|tstd t|t|dtstd t|d|||||fS) a Check if the input data is valid for training. Args: input_ids (list[`torch.LongTensor`]): List of tensors containing the input_ids attention_mask (list[`torch.LongTensor`]): List of tensors containing the attention_mask labels (list[`torch.FloatTensor`]): List of tensors containing the labels texts (list[`str`]): List of string containing the text input. texts_labels (list[`str`]): List of string containing the text labels. Returns: `tuple`: The input data. Nr^r`z! must be a list of tensors - got rz Elements in z must be tensors - got rgz''text' must be a list of strings - got z)Elements in 'text' must be strings - got z.'text_labels' must be a list of strings - got z0Elements in 'text_labels' must be strings - got )rsr9listrxtyperaTensorr:)r^r_r`r{r|name tensor_listr8r8r[_step_safety_checkers8    z(IterativeSFTTrainer._step_safety_checkerc sjjjdkrtdjj_ jj_ |dur'|dur't d|dur5|dur5t dt|durD|durDjrDt d|durN|ddnd}|durZ|ddnd}|durf|ddnd}|durr|ddnd}|dur~|ddnd}|||||\}}}}}|durj|jdddd }|d |d }}|durj|jdddd d }|dur|}|||}t|}i}||fd d } t|} | dt| jjd| d} t| D]\} jjmfdd|D}j|} jj dkr| !} | "}j#| jj$r8jj%dur8j&j'jj%j()j(*j+durMj+)jjd7_j |7_ ,Wdn 1skwYqdS)a Run an optimisation step given a list of input_ids, attention_mask, and labels or a list of text and text_labels. Args: input_ids (list[`torch.LongTensor`]): List of tensors containing the input_ids (if not provided, text will be used) attention_mask (list[`torch.LongTensor`], , *optional*): List of tensors containing the attention_mask labels (list[`torch.FloatTensor`], *optional*): List of tensors containing the labels (if set to None, will default to input_ids) texts (list[`str`], *optional*): List of strings containing the text input (if not provided, input_ids will directly be used) texts_labels (list[`str`], *optional*): List of strings containing the text labels (if set to None, will default to text) Returns: `dict[str, Any]`: A summary of the training statistics rgNz@Step should include `input_ids` or `texts` as keyword arguments.ztBoth `input_ids` and `texts` argument are provided. `input_ids` will be ignored. Please provide only one of the two.zNo 'labels' or 'text_labels' are provided. When using an encoder-decoder architecture, 'labels' or 'text_labels' must be passed.Tpt)rG truncationpaddingreturn_tensorsr^r_csFt}|dD]dvr tfdd|Djj|<q|S)Nrrgcsg|]}|qSr8r8)rcdkeyr8r[reisz>IterativeSFTTrainer.step..collator..)dictrastackrtr!ru)data return_dictrqrr[collatores  &z*IterativeSFTTrainer.step..collatorra) batch_sizeshuffle collate_fncsi|]}||qSr8r8)rcro)batchr8r[rrxrfz,IterativeSFTTrainer.step..r)-r!trainstate global_stepratensorrtr"rutr_loss_globalstep_last_loggedrxrBrC UserWarningr-rr%rGrzr}keysupdater from_dict set_formatrper_device_train_batch_size enumerater7 accumulate compute_lossn_gpumeandetachbackwardsync_gradients max_grad_normclip_grad_norm_ parametersrQstep zero_gradrR_maybe_log_save_evaluate)rUr^r_r`r{r| model_inputsmodel_inputs_names batch_dictr batch_datastep_dataloader_loss tr_loss_stepr8)rrUr[rs                 zIterativeSFTTrainer.stepcCs|jjdur|jj|jjdkr|jjdkr||j|jjdurf|jj|jjdkrh|jjdkrji}||j }|j|j8_t ||jj|j d|d<| |d<|jj|_ ||dSdSdSdS)Nrr learning_rate)r" eval_stepsrrevaluater$ logging_steps_nested_gatherrritemroundr_get_learning_ratelog)rUlogstr_loss_scalarr8r8r[rs       z,IterativeSFTTrainer._maybe_log_save_evaluatecsL|jjdurt|jjj}n |jjdd}|j|dt||dS)Nr)r*)rW) r" hub_model_idr output_dirrr=create_model_cardrJ_save_checkpoint)rUr!trialrWrYr8r[rs  z$IterativeSFTTrainer._save_checkpointrW dataset_nametagsc Cs|sdSt|jjdrtj|jjjs|jjj}nd}|dur&t}n t |t r/|h}nt|}t|jjdr?| d| |j t|||j||trXtjdurXtjjndtdd}|tj|jjddS)a Creates a draft of a model card using the information available to the `Trainer`. Args: model_name (`str` or `None`, *optional*, defaults to `None`): Name of the model. dataset_name (`str` or `None`, *optional*, defaults to `None`): Name of the dataset used for training. tags (`str`, `list[str]` or `None`, *optional*, defaults to `None`): Tags to be associated with the model card. Nr<unsloth_versionunslothz Iterative SFT) base_modelrWrrr wandb_url comet_url trainer_namez README.md)is_world_process_zerorLr!r;ospathisdirr<setr9r:addrrMrrrwandbrunurlrsavejoinr"r)rUrWrrr model_cardr8r8r[rs0      z%IterativeSFTTrainer.create_model_card)NNNNr NN)NNNNN)NNN))__name__ __module__ __qualname____doc__rMrr:rrrrr rrrr rrtupleraoptim OptimizerrRLambdaLRrrrrKrDrz staticmethodr} LongTensorrrempty_device_cacherrrr __classcell__r8r8rYr[r6s"   ^  5        r)*rrBpathlibrtypingrrrradatasetsrtorch.utils.datar transformersrr r r r r rrrrrrrtransformers.trainer_utilsrtransformers.utilsrcoreriterative_sft_configrutilsrrpeftrrrr8r8r8r[s$   <