o oi2@s ddlmZmZddlmZddlmZmZmZm Z m Z m Z ddl Z ddl m Z ddlmZddlmZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd l m!Z!ddl"m#Z#ddl$m%Z%eGdddeZ&Gdddee&Z'ee(efZ)GdddeZ*dS)) dataclassfield)partial)AnyCallableDictMappingOptional OrderedDictN)Tensor) Optimizer)override)_Loop)AbstractClosure OutputResult)_OptimizationProgress)_block_parallel_sync_behavior)call)MisconfigurationException) WarningCache) STEP_OUTPUTc@seZdZUdZeeed<edddZeeed<ee dZ e e e fed<dd d Zdd d Zeddeded dfddZed e e e ffddZdS) ClosureResultaUA container to hold the result of a :class:`Closure` call. It is created from the output of :meth:`~lightning.pytorch.core.LightningModule.training_step`. Attributes: closure_loss: The loss with a graph attached. loss: A detached copy of the closure loss. extra: Any keys other than the loss returned. closure_lossFN)initdefaultloss)default_factoryextrareturncCs |dSN) _clone_lossselfr#b/home/ubuntu/.local/lib/python3.10/site-packages/lightning/pytorch/loops/optimization/automatic.py __post_init__3s zClosureResult.__post_init__cCs"|jdur|j|_dSdSr)rdetachclonerr!r#r#r$r 6s zClosureResult._clone_losstraining_step_output normalizecCs~di}}t|tr!|d}|durtddd|D}nt|tr)|}n|dur1td|dur9||}|||dS)NrzbIn automatic_optimization, when `training_step` returns a dict, the 'loss' key needs to be presentcSsi|] \}}|dkr||qSrr#).0kvr#r#r$ Esz;ClosureResult.from_training_step_output..zrIn automatic optimization, `training_step` must return a Tensor, a dict, or None (where the step will be skipped).)r) isinstancergetritemsr )clsr)r*rrr#r#r$from_training_step_output;s"     z'ClosureResult.from_training_step_outputcCsd|ji|jS)Nr)rrr!r#r#r$asdictUszClosureResult.asdictrN)r()__name__ __module__ __qualname____doc__r r __annotations__rrdictrrstrrr%r classmethodrintr4r r5r#r#r#r$r"s    rc seZdZdZeZ  ddegefdeee gdfdeegdfffdd Z e e ded ed efd d Ze ded ed ee fd dZZS)ClosureaAn implementation of a :class:`AbstractClosure` for automatic optimization in Lightning that combines three elementary closures into one: ``training_step``, ``backward`` and ``zero_grad``. The Closure gets created by the training loop(s) and is then passed to the :meth:`torch.optim.Optimizer.step` method. An optimizer is responsible for calling the closure and optionally do something with the output. Args: step_fn: This is typically the :meth:`lightning.pytorch.core.module.LightningModule.training_step wrapped with processing for its outputs backward_fn: A function that takes a loss value as input, performs back-propagation and returns the loss value. Can be set to ``None`` to skip the backward operation. zero_grad_fn: A function that zeroes the gradients. Can be set to ``None`` to skip zero_grad, for example when accumulating gradients. Example: closure = Closure() optimizer = torch.optim.Adam(...) optimizer.step(closure) Nstep_fn backward_fn zero_grad_fncs t||_||_||_dSr)super__init___step_fn _backward_fn _zero_grad_fn)r"rArBrC __class__r#r$rEss  zClosure.__init__argskwargsrcOsT|}|jdur|jd|jdur||jdur(|jdur(||j|S)NzO`training_step` returned `None`. If this was on purpose, ignore this warning...)rFr warning_cachewarnrHrG)r"rKrL step_outputr#r#r$closure~s    zClosure.closurecOs|j|i||_|jjSr)rP_resultr)r"rKrLr#r#r$__call__szClosure.__call__)NN)r7r8r9r:rrMrrr r rEr torch enable_gradrrPrR __classcell__r#r#rIr$r@Zs"  $r@c seZdZdZeZdfdd Zded ed e de fd d Z d e ded ede fd dZ d e degeffddZd ededeegdffddZdedeeegdffddZd edegeefddfddZdejjddfddZd edejjddfddZd e defddZZS)_AutomaticOptimizationzNPerforms automatic optimization (forward, zero grad, backward, optimizer step)trainer pl.TrainerrNcst|t|_d|_dS)NF)rDrEroptim_progress_skip_backward)r"rWrIr#r$rEs  z_AutomaticOptimization.__init__ optimizer batch_idxrLcCs||||}|jjjs.|jjr.t|jjdd |Wdn1s(wYn||||}|j dur?iS| S)zRuns closure (train step + backward) together with optimization if necessary. Args: kwargs: the kwargs passed down to the hooks batch_idx: the current batch index. optimizer: the optimizer T)blockN) _make_closurerWstrategyhandles_gradient_accumulationfit_loop_should_accumulater_optimizer_stepconsume_resultrr5)r"r[r\rLrPresultr#r#r$runs     z_AutomaticOptimization.runcCs.||}||}|||}t|||dS)zBuild a closure object that captures the given arguments and runs the `training_step` function and optionally other functions such as `backward` and `zero_grad`.)rArBrC) _make_step_fn_make_backward_fn_make_zero_grad_fnr@)r"rLr[r\rArBrCr#r#r$r^s   z$_AutomaticOptimization._make_closurecCs t|j|S)zOBuild the step function that runs the `training_step` and processes its output.)r_training_step)r"rLr#r#r$rgs z$_AutomaticOptimization._make_step_fncs8jrdSjjdk}|sdSdfdd }|S)zBuild a `zero_grad` function that zeroes the gradients before back-propagation. Returns ``None`` in the case backward needs to be skipped. NrrcsdSr)_on_before_zero_grad_optimizer_zero_gradr#r\r[r"r#r$rCs z?_AutomaticOptimization._make_zero_grad_fn..zero_grad_fnr6)rZrWaccumulate_grad_batches)r"r\r[is_first_batch_to_accumulaterCr#rmr$risz)_AutomaticOptimization._make_zero_grad_fncs&jrdSdtddffdd }|S)zBuild a `backward` function that handles back-propagation through the output produced by the `training_step` function. Returns ``None`` in the case backward needs to be skipped. Nrrcstjd|dS)Nbackward)r_call_strategy_hookrWr+r[r"r#r$rBsz=_AutomaticOptimization._make_backward_fn..backward_fn)rZr )r"r[rBr#rrr$rhsz(_AutomaticOptimization._make_backward_fntrain_step_and_backward_closurecCs^|j}|jjd}|j}|s|jjjt |d|j ||||s-|jjj dSdS)a?Performs the optimizer step and some sanity checking. Args: batch_idx: the index of the current batch train_step_and_backward_closure: the closure function performing the train step and computing the gradients. By default, called by the optimizer (if possible) roptimizer_stepN) rWr__lightning_optimizersrarbrYr[stepincrement_readyr_call_lightning_module_hook current_epochincrement_completed)r"r\rsrWr[should_accumulater#r#r$rcs    z&_AutomaticOptimization._optimizer_stepcCsB|j}|jjjt|d|t|d||jjjdS)zmCalls the ``on_before_zero_grad`` hook. Args: optimizer: the current optimizer on_before_zero_gradN) rWrYr[ zero_gradrwr_call_callback_hooksrxincrement_started)r"r[rWr#r#r$rks z+_AutomaticOptimization._on_before_zero_gradcCs,|j}t|d|j|||jjjdS)zZeroes out all gradients of parameters optimized by the current optimizer. Args: batch_idx: the index of the current batch optimizer: the current optimizer optimizer_zero_gradN)rWrrxryrYr[r}rz)r"r\r[rWr#r#r$rl%sz+_AutomaticOptimization._optimizer_zero_gradcCsT|j}tj|dg|R}|jj|dur"|jdkr"td|j ||j S)zPerforms the actual train step with the tied hooks. Args: kwargs: the kwargs passed down to the hooks. Returns: A ``ClosureResult`` containing the training step output. training_stepNr(zSkipping the `training_step` by returning None in distributed training is not supported. It is recommended that you rewrite your training logic to avoid having to skip the step in the first place.) rWrrqvaluesr_post_training_step world_size RuntimeErroroutput_result_clsr4rn)r"rLrWr)r#r#r$rj1s  z%_AutomaticOptimization._training_step)rWrXrN)r7r8r9r:rrrEr r?r _OUTPUTS_TYPErfr@r^rrgr rir rhrcrSoptimrkrlrjrUr#r#rIr$rVs&$"  %  rV)+ dataclassesrr functoolsrtypingrrrrr r rSr torch.optimr typing_extensionsr lightning.pytorchpytorchpllightning.pytorch.loops.loopr,lightning.pytorch.loops.optimization.closurerr lightning.pytorch.loops.progressr!lightning.pytorch.loops.utilitiesrlightning.pytorch.trainerr&lightning.pytorch.utilities.exceptionsr%lightning.pytorch.utilities.rank_zeror!lightning.pytorch.utilities.typesrrr@r=rrVr#r#r#r$s*             7 :