o ॵiC @sddlZddlZddlmZddlmZddlmZddlm Z ddl m Z m Z Gdd d e Z e jejd Gd d d eZdS) N)version)Hooks)Hook)HOOKS) OptimizerHookOptimizerProcessorc@s$eZdZddZddZddZdS)ApexOptimizerProcessorcCs ||_dS)N opt_level)selfr r k/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/trainers/hooks/optimizer/apex_optimizer_hook.py__init__s zApexOptimizerProcessor.__init__cCsnddlm}ttjtdkr|jdt d| |j }|j ||j |jd\|_ |_ |j dS)Nrampz1.9.0zApexAMPOptimizerHook is only tested on torch version 1.8.x,if it works abnormally please consider downgrading your torch version to 1.8.x.z open fp16r )apexrrparsetorch __version__loggerwarninglogginginfo unwrap_modulemodel initialize optimizerr zero_grad)r trainerrrr r rinitialize_optimizers    z+ApexOptimizerProcessor.initialize_optimizerc Cs|D] }|j||<qddlm}|D] }||j||j }|Wdn1s1wYqt||rY|durMtj |j fi||j |j dSdS)Nrr) train_outputsrr scale_lossrbackwardr every_n_itersr clip_gradsr parametersstepr)r r loss_keyscumulative_iters grad_clipkr scaled_lossr r rr##s&     zApexOptimizerProcessor.backwardN)__name__ __module__ __qualname__rr r#r r r rr s r ) module_namec@s(eZdZdZejZdddZddZdS) ApexAMPOptimizerHooka Fp16 optimizer, if torch version is less than 1.6.0, you must install apex (https://www.github.com/nvidia/apex) else use torch.cuda.amp by default Args: opt_level (str): "O0" and "O3" are not true mixed precision, but they are useful for establishing accuracy and speed baselines, respectively. "O1" and "O2" are different implementations of mixed precision. Try both, and see what gives the best speedup and accuracy for your model. O1cKs0||_z ddlm}WdStytdw)NrrzPapex not installed, please install apex from https://www.github.com/nvidia/apex.)r rr ImportError ValueError)r r kwargsrr r rrEs zApexAMPOptimizerHook.__init__cCsP|t}t|dkr$t|djtdtfvr&|dt|jdSdSdS)Nr) get_hookrlentype processorr set_processorr r )r roptimizer_hookr r rregister_processorOs z'ApexAMPOptimizerHook.register_processorN)r2)r-r.r/__doc__rPRIORITYrr<r r r rr16s   r1)rr packagingrmodelscope.metainformodelscope.trainers.hooksr!modelscope.trainers.hooks.builderrbaserrr register_moduler1r r r rs     )