o zi@s<ddlZddlmZddlZddlZGdddejjZdS)N)deepcopycsBeZdZdZ       dfdd Zd d Zdd dZZS) AdaHessianzImplements AdamHess algorithm.MbP?g?g+?:0yE>rFc s8t||||d} tt||| ||_||_||_dS)N)lrbetaseps weight_decay)dictsuperr__init__ block_length single_gpu hessian_power) selfparamsrr r r rrrdefaults __class__K/home/ubuntu/.local/lib/python3.10/site-packages/torch_optimizer/adahess.pyr s   zAdaHessian.__init__cCs|jdd}ttdd|}dd|D}|D]}d||dk<d ||dk<qtjj|||d d d }g}t||D]R\}}|} t| d krWt ||d } | | q:t| d krt ||d  d|j } t tj | d gd dt|j } | |j  | } | | q:|S)a  compute the Hessian vector product with v, at the current gradient point or compute the gradient of . :param v: a list of torch tensors :param gradsH: a list of torch variables :return: a list of torch tensors rrcSs|jS)N) requires_grad)xrrr'sz&AdaHessian.get_trace..cSsg|] }tj|ddqS))high)torch randint_like).0prrr )sz(AdaHessian.get_trace..g?rT) grad_outputs only_inputs retain_graphgr)dim) param_groupslistfilterrautogradgradzipsizelenabsappendviewrsumfloatrepeat_interleave)rgradsHrvv_ihvshutchinson_tracehvvi param_size tmp_output tmp_output1 tmp_output2 tmp_output3rrr get_traces@       zAdaHessian.get_traceNcCsd}|dur |}||}|jD]}t|dD]\}}|jdur#qt||j}|jr3td|j} |j |} t | dkrVd| d<t | | d<t | | d<n| d | | d<| d | | d<| d| d} } |d\} }| dd 7<| | d | || |d |||||d | | d}d || d}|jd kr| t||j|d }n| t||d }|d |}|d dkr| |d  |d | | | | ||j| qq|S) zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. NrzPAdaHessian does not support sparse gradients, please consider SparseAdam insteadrstepexp_avgexp_hessian_diag_sqr rr rr )rBr( enumerater,rdatar4 is_sparse RuntimeErrorstater/r zeros_liketype_asmul_add_addcmul_rsqrtmathaddcdiv_copy_)rr6closureloss hut_tracegroupir!r, p_data_fp32rJrDrEbeta1beta2bias_correction1bias_correction2denom step_sizerrrrCGsx               KzAdaHessian.step)rrrrrrF)NN)__name__ __module__ __qualname____doc__rrBrC __classcell__rrrrrs*r)rQcopyrr torch.optimoptim Optimizerrrrrrs