o zi, @sddlmZmZmZmZddlZddlmZddl m Z ddl m Z m Z er,ddlmZGdddZ    dd eee d fd ee d ee dedddf ddZdddZdddZdS)) TYPE_CHECKINGLiteralOptionalUnionN)Callback)MisconfigurationException)EVAL_DATALOADERSTRAIN_DATALOADERS) _LRFinderc@s eZdZdZd.ddZ      d/dddeeedfdeedeededde dde de de de de dee fddZ        ! " # $d0dddeeedfdeedeededde dd%e d&e d'e de d(ee d)ed*e ded+fd,d-ZdS)1TunerzTuner class to tune your model.trainer pl.TrainerreturnNcCs ||_dS)N)_trainer)selfr rR/home/ubuntu/.local/lib/python3.10/site-packages/pytorch_lightning/tuner/tuning.py__init__s zTuner.__init__fitpower batch_sizemodelzpl.LightningModuletrain_dataloaderspl.LightningDataModuleval_dataloaders dataloaders datamodulemethodrvalidatetestpredictmodesteps_per_trialinit_val max_trialsbatch_arg_namec st||||t|jddlm} | ||| | | dd_g|jj|j_|dkr5|j||||n)|dkrC|jj|||dn|dkrQ|jj |||dn |d kr^|jj |||dfd d |jjD|j_j S) aIteratively try to find the largest batch size for a given model that does not give an out of memory (OOM) error. Args: model: Model to tune. train_dataloaders: A collection of :class:`torch.utils.data.DataLoader` or a :class:`~pytorch_lightning.core.datamodule.LightningDataModule` specifying training samples. In the case of multiple dataloaders, please see this :ref:`section `. val_dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying validation samples. dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying val/test/predict samples used for running tuner on validation/testing/prediction. datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.LightningDataModule`. method: Method to run tuner on. It can be any of ``("fit", "validate", "test", "predict")``. mode: Search strategy to update the batch size: - ``'power'``: Keep multiplying the batch size by 2, until we get an OOM error. - ``'binsearch'``: Initially keep multiplying by 2 and after encountering an OOM error do a binary search between the last successful batch size and the batch size that failed. steps_per_trial: number of steps to run with a given batch size. Ideally 1 should be enough to test if an OOM error occurs, however in practise a few are needed init_val: initial batch size to start the search with max_trials: max number of increases in batch size done before algorithm is terminated batch_arg_name: name of the attribute that stores the batch size. It is expected that the user has provided a model or datamodule that has a hyperparameter with that name. We will look for this attribute name in the following places - ``model`` - ``model.hparams`` - ``trainer.datamodule`` (the datamodule passed to the tune method) rBatchSizeFinder)r%r&r'r(r)Trr")rr#r$cg|]}|ur|qSrr.0cbbatch_size_finderrr iz*Tuner.scale_batch_size..) _check_tuner_configuration%_check_scale_batch_size_configurationr-pytorch_lightning.callbacks.batch_size_finderr+ _early_exit callbacksrr"r#r$optimal_batch_size) rrrrrrr r%r&r'r(r)r+rr0rscale_batch_sizes,0  zTuner.scale_batch_size:0yE>d exponential@Tmin_lrmax_lr num_trainingearly_stop_threshold update_attr attr_namer c s|dkrtdt||||t|jddlm}|||| | | | | dd_g|jj|j_|j||||fdd|jjD|j_j S) aXEnables the user to do a range test of good initial learning rates, to reduce the amount of guesswork in picking a good starting learning rate. Args: model: Model to tune. train_dataloaders: A collection of :class:`torch.utils.data.DataLoader` or a :class:`~pytorch_lightning.core.datamodule.LightningDataModule` specifying training samples. In the case of multiple dataloaders, please see this :ref:`section `. val_dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying validation samples. dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying val/test/predict samples used for running tuner on validation/testing/prediction. datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.LightningDataModule`. method: Method to run tuner on. It can be any of ``("fit", "validate", "test", "predict")``. min_lr: minimum learning rate to investigate max_lr: maximum learning rate to investigate num_training: number of learning rates to test mode: Search strategy to update learning rate after each batch: - ``'exponential'``: Increases the learning rate exponentially. - ``'linear'``: Increases the learning rate linearly. early_stop_threshold: Threshold for stopping the search. If the loss at any point is larger than early_stop_threshold*best_loss then the search is stopped. To disable, set to None. update_attr: Whether to update the learning rate attribute or not. attr_name: Name of the attribute which stores the learning rate. The names 'learning_rate' or 'lr' get automatically detected. Otherwise, set the name here. Raises: MisconfigurationException: If learning rate/lr in ``model`` or ``model.hparams`` isn't overridden, or if you are using more than one optimizer. rz>method='fit' is the only valid configuration to run lr finder.rLearningRateFinder)rArBnum_training_stepsr%rDrErFTcr,rrr-lr_finder_callbackrrr2r3z!Tuner.lr_find..) rr4_check_lr_find_configurationr%pytorch_lightning.callbacks.lr_finderrHr7r8r optimal_lr)rrrrrrr rArBrCr%rDrErFrHrrJrlr_findls&2   z Tuner.lr_findr r rN) NNNNrrrrrr) NNNNrr;r<r=r>r?Tr@)__name__ __module__ __qualname____doc__rrrr rrstrintr:floatboolrOrrrrr s      P     r rrrrrr r!rcCsjd}||vrtd|d|d|dkr#|dur!td|ddS|dus+|dur3td|d dS) Nr!zmethod z is invalid. Should be one of .rzIn tuner with method=zs, `dataloaders` argument should be None, please consider setting `train_dataloaders` and `val_dataloaders` instead.zIn tuner with `method`=zt, `train_dataloaders` and `val_dataloaders` arguments should be None, please consider setting `dataloaders` instead.) ValueErrorr)rrrr supported_methodsrrrr4s  r4r r cs0ddlmfdd|jD}|rtddS)NrrGcg|] }t|r|qSr isinstancer-rGrrr2z0_check_lr_find_configuration..zqTrainer is already configured with a `LearningRateFinder` callback.Please remove it if you want to use the Tuner.)rMrHr8rZr configured_callbacksrrGrrLs rLcs@|jjrtdddlmfdd|jD}|rtddS)NzMTuning the batch size is currently not supported with distributed strategies.rr*cr\rr]r-r*rrr2r_z9_check_scale_batch_size_configuration..znTrainer is already configured with a `BatchSizeFinder` callback.Please remove it if you want to use the Tuner.)_accelerator_connectoris_distributedrZr6r+r8r`rr*rr5s r5)NNNrrP)typingrrrrpytorch_lightningpl$pytorch_lightning.callbacks.callbackr&pytorch_lightning.utilities.exceptionsr!pytorch_lightning.utilities.typesrr !pytorch_lightning.tuner.lr_finderr r r4rLr5rrrrs4    $