o Ti/@sZddlZddlZddlmZmZddlmZddlmZddl m Z eGdddZ dS) N) dataclassfield)Optional)is_bitsandbytes_available) flatten_dictc@s eZdZUdZeejej dde d ddidZ e e d<ed dd idZe e d <eddd idZee d <eddddgddZee e d<eeddidZee d<eeddidZee d<eeddidZee d<edddidZe e d<edddidZe e d<ed dd!idZee d"<ed#dd$idZee d%<ed&dd'idZee d(<ed)dd*idZe e d+<ed,dd-idZee d.<ed dd/idZe e d0<ed1dd2idZ ee d3<ed4dd5idZ!e"e d6<ed7dd8idZ#e"e d9<ed#dd:idZ$ee d;<ed<ed#dd?idZ&ee d@<edAddBidZ'ee dC<edDddEidZ(e"e dF<edGddHidZ)e"e dI<edJddKidZ*e"e dL<edMddNidZ+e"e dO<edPddQidZ,e"e dR<ed#ddSidZ-ee dT<ed4ddUidZ.e"e dV<ed#ddWidZ/ee dX<ed,ddYidZ0ee dZ<ed7dd[idZ1e"e d\<edMdd]idZ2e"e d^<ed4dd_idZ3e"e d`<edAddaidZ4ee db<edcdddidZ5ee de<edcddfidZ6ee dg<edAddhidZ7ee di<edz Adam epsilon.train_adam_epsilonz&Number of gradient accumulation steps.!train_gradient_accumulation_stepsz,Maximum gradient norm for gradient clipping.train_max_grad_normz'Number of inner epochs per outer epoch.train_num_inner_epochsz8Whether to use classifier-free guidance during training. train_cfgzClip advantages to the range.train_adv_clip_maxzPPO clip range.train_clip_rangez"Fraction of timesteps to train on.train_timestep_fractionz7Whether to track statistics for each prompt separately.per_prompt_stat_trackingz?Number of reward values to store in the buffer for each prompt.$per_prompt_stat_tracking_buffer_sizez7Minimum number of reward values to store in the buffer."per_prompt_stat_tracking_min_countz*Whether to compute rewards asynchronously.async_reward_computationz>Maximum number of workers to use for async reward computation. max_workerszr?r@rArJrLrHrHrHrIrsT ] r) rQrT dataclassesrrtypingr transformersrcorerrrHrHrHrIs