o Ti_@sddlZddlmZmZddlZddlZddlmZddl m Z ddl m Z m Z ddlmZmZmZmZmZmZmZmZddlmZddlmZd d lmZd d lmZd d lm Z m!Z!m"Z"d d l#m$Z$ddl%m&Z&ddl'm(Z(erddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1e rddl2Z2 d-de3e4dedededeede5de3e4fddZ6GdddeZ7Gd d!d!eZ8d"ede3e4d#e3e4d$e3e4dej9f d%d&Z:Gd'd(d(eZ;Gd)d*d*eZ|jr|jdur|dd}|jdt|dSdSdS)N total_flosz[bold green]Status = )rzrppoprvrstr)rFrarbrclogsrd_r.r.rAon_logs zRichProgressCallback.on_logcKsH|jr"|jd|_d|_d|_d|_d|_d|_d|_d|_ dSdSrE) rzrtstoprprqrrrsrurvrwrr.r.rA on_train_ends  z!RichProgressCallback.on_train_endrE) rfrgrhrirGrrerrrrrr.r.r.rArns   rnrbr8winner_indicescCsFt|jgt|}tt||||}dd|D}tj|gddS)NcSs6g|]}|d|d|dd|dd|dfqS)rrrrRr.).0itemr.r.rA s6z,_win_rate_completions_df..)stepr>reference_modelpolicy winner_index)columns)rr]r0rYr4pd DataFrame)rbr"r8rr]rK split_datar.r.rA_win_rate_completions_dfsrc @speZdZdZ    ddededeedeed e d e f d d Z d e de de fddZd e de de fddZdS)WinRateCallbacka A [`~transformers.TrainerCallback`] that computes the win rate of a model based on a reference. It generates completions using prompts from the evaluation dataset and compares the trained model's outputs against a reference. The reference is either the initial version of the model (before training) or the reference model, if available in the trainer. During each evaluation step, a judge determines how often the trained model's completions win against the reference using a judge. The win rate is then logged in the trainer's logs under the key `"eval_win_rate"`. Usage: ```python trainer = DPOTrainer(...) judge = PairRMJudge() win_rate_callback = WinRateCallback(judge=judge, trainer=trainer) trainer.add_callback(win_rate_callback) ``` Args: judge (`BasePairwiseJudge`): The judge to use for comparing completions. trainer (`Trainer`): Trainer to which the callback will be attached. The trainer's evaluation dataset must include a `"prompt"` column containing the prompts for generating completions. If the `Trainer` has a reference model (via the `ref_model` attribute), it will use this reference model for generating the reference completions; otherwise, it defaults to using the initial model. generation_config (`GenerationConfig`, *optional*): The generation config to use for generating completions. num_prompts (`int` or `None`, *optional*, defaults to `None`): The number of prompts to generate completions for. If not provided, defaults to the number of examples in the evaluation dataset. shuffle_order (`bool`, *optional*, defaults to `True`): Whether to shuffle the order of the completions before judging. use_soft_judge (`bool`, *optional*, defaults to `False`): Whether to use a soft judge that returns a win probability between 0 and 1 for the first completion vs the second. NTFjudgetrainerr& num_prompts shuffle_orderuse_soft_judgecCsd||_||_||_||_g|_||_|jjdurtd|jj|_|dur0|jt ||_dSdS)NzCTrainer must have an evaluation dataset to use the WinRateCallback.) rrrr&ref_completionsr eval_dataset ValueErrorselectr/)rFrrr&rrrr.r.rArGs   zWinRateCallback.__init__rarbrcc Ks|d}d|_|jj}t|jdd}|dur|jj}||jdM}t|||||j|j d|_ t t |j |j } |j rS|jj|| |jdd} dd | D} t| } n |j|| |j} t|}t| } t| } Wdn1srwY|jjjrtd d | Dt| } |j rd t| t| } |j| | d n|jd| id|jvrddl}|jdurt||| | d}|d|j|did|jvrt||| | d}td|ddSdSdS)Nprocessing_classleftrDr>r#r$r%r&r'T return_scorescSg|] }|dkr dndqSg?rrr.rscorer.r.rAr;z2WinRateCallback.on_train_begin..cs|]}|dkVqdSrNr.r winner_idxr.r.rA Ez1WinRateCallback.on_train_begin..rHeval_avg_win_prob eval_win_raterwandbrrbr"r8rwin_rate_completions dataframecomet_mlwin_rate_completions.csvnametable) padding_siderr%getattr model_wrappedsplit_between_processesrrBr&per_device_eval_batch_sizerrYr4rrrris_main_processsumr0log report_torrunrTablerrFrarbrcrdr$r%r#r"r8 ref_win_probsrwin_rate avg_win_probrdfr.r.rArsl        zWinRateCallback.on_train_beginc Ks|d}d|_|jj}|jj}||jdK}t|||||j|jd} t t |j | } |j rF|j j || |jdd} dd| D} t| } n |j || |j} t|}t| } t| } Wdn1sewY|jjjrtd d | Dt| } |j rd t| t| } |j| | d n|jd | id|jvrddl}|jdurt||| | d}|d|j|did|jvrt||| | d}td|ddSdSdS)Nrrr>rTrcSrrr.rr.r.rArzrz/WinRateCallback.on_evaluate..csrrr.rr.r.rArrz.WinRateCallback.on_evaluate..rHrrrrrrrrrr)rrr%rrrrBr&rrYr4rrrrrrrr0rrrrrrrrr.r.rArdsh        zWinRateCallback.on_evaluate)NNTF)rfrgrhrirr rrintboolrGrrr rrr.r.r.rArs*) Frc @sFeZdZdZ   d dedeedeedeefddZd d Z dS) LogCompletionsCallbacka A [`~transformers.TrainerCallback`] that logs completions to Weights & Biases and/or Comet. Usage: ```python trainer = DPOTrainer(...) completions_callback = LogCompletionsCallback(trainer=trainer) trainer.add_callback(completions_callback) ``` Args: trainer (`Trainer`): Trainer to which the callback will be attached. The trainer's evaluation dataset must include a `"prompt"` column containing the prompts for generating completions. generation_config (`GenerationConfig`, *optional*): The generation config to use for generating completions. num_prompts (`int` or `None`, *optional*): The number of prompts to generate completions for. If not provided, defaults to the number of examples in the evaluation dataset. freq (`int` or `None`, *optional*): The frequency at which to log completions. If not provided, defaults to the trainer's `eval_steps`. Nrr&rfreqcCs^||_||_||_g|_d|_|jjdurtd|jj|_|dur-|jt||_dSdS)NzJTrainer must have an evaluation dataset to use the LogCompletionsCallback.) rr&rr_last_logged_steprrrr/)rFrr&rrr.r.rArGs  zLogCompletionsCallback.__init__c s>|j|jkrdS|jp |j}|j|dkrdS|dd_|jj}|jj}||j d%}fdd|D}t ||||j |j d} t | } t |}Wdn1sVwY|jjjrt|jgt|} tt| || } |j| tjgd|jd } d |jvrtd | id |jvrtd | d|j|_dS)Nrrrr>csg|] }td|idqS)r>r)rr>r$r.rArsz6LogCompletionsCallback.on_step_end..r)rr>r@)rrKrr8rzcompletions.csvr)r]rr eval_stepsrrr%rrrrBr&rrrrr0rYr4rextendrrrrrr) rFrarbrcrdrr%r#r"r8r]rKrr.rrAresF        z"LogCompletionsCallback.on_step_end)NNN) rfrgrhrir rrrrGrer.r.r.rArs rc@sNeZdZdZ   ddeddedefdd Zd d Zdd d ZdddZ dS)MergeModelCallbacka A [`~transformers.TrainerCallback`] that merges the policy model (the model being trained) with another model based on a merge configuration. Args: merge_config ([`MergeConfig`], *optional*, defaults to `None`): Configuration used for the merging process. If not provided, the default [`MergeConfig`] is used. merge_at_every_checkpoint (`bool`, *optional*, defaults to `False`): Whether to merge the model at every checkpoint. push_to_hub (`bool`, *optional*, defaults to `False`): Whether to push the merged model to the Hub after merging. Example: ```python # pip install mergekit from trl.mergekit_utils import MergeConfig from trl import MergeModelCallback config = MergeConfig() merge_callback = MergeModelCallback(config) trainer = DPOTrainer(..., callbacks=[merge_callback]) ``` NF merge_configrmerge_at_every_checkpoint push_to_hubcCs*tstd|p t|_||_||_dS)NzYMergeModelCallback requires the `mergekit` extra. To install, run `pip install mergekit`.)rrorrrr)rFrrrr.r.rArGs  zMergeModelCallback.__init__cCsztj|d|}||j_|jjdur|jj|j_tj|d}t|j ||j r;|d|d}t ||dSdS)Nz checkpoint-mergedz _checkpoint-_merged) ospathjoinrpolicy_model_pathtarget_model_pathconfig _name_or_pathrcreaterr)rF output_dirr]r#checkpoint_path merge_path repo_namer.r.rA_merge_and_maybe_push)s  z(MergeModelCallback._merge_and_maybe_pushcKs |jr||j|j|dSdSrErrrr]rFrarbrcr#rdr.r.rAon_save6zMergeModelCallback.on_savecKs |js||j|j|dSdSrErrr.r.rAr:rzMergeModelCallback.on_train_end)NFFrE) rfrgrhrirrrGrrrr.r.r.rArs  r)r)>rtypingrrpandasrrj accelerateraccelerate.stateraccelerate.utilsrr transformersrr r r r r rrtransformers.trainer_utilsrtransformers.utilsr data_utilsr import_utilsrmergekit_utilsrrr models.utilsrjudgesrutilsr rich.consolerr rich.liver rich.panelr rich.progressr!rrYrrrBrCrnrrrrrr.r.r.rAsp  (           ')P  E\