o $i@sddlZddlZddlmZddlmZmZmZmZm Z m Z m Z ddl m Z mZmZddlmZddlmZmZddlmZmZmZmZddlmZmZmZdd lmZdd l m!Z!dd l"m#Z#m$Z$m%Z%dd l&m'Z'e d Z(e dZ)e*e+Z,e'GdddZ-dS)N)Path)AnyCallableDictListOptionalTypeVarUnion)StartTracebackStartTracebackWithWorkerRankskip_exceptions)Dataset) Checkpoint DataConfig)BackendExecutorInactiveWorkerGroupErrorTrainBackendErrorTrainingWorkerError)_TrainingResult _TrainSession get_session) ActorWrapper) BackendConfig) BaseTrainer GenDatasetTrainingFailedError) DeveloperAPITSc@seZdZdZdeeefdedeege fee e e fge ffde e e fde e e fdedeee e eeffd d Zd d Z ddeefddZdefddZddZdeee fddZddZdefddZd S)TrainingIteratorzEAn iterator over Train results. Returned by ``trainer.run_iterator``.backend_executorbackend_config train_funcdatasetsmetadata data_config checkpointcCsL||_||_||_||_||_||_|j||j|j|j|dd|_dS)N)r"r#r$r%r&F) _backend_executor backend_cls_backend _train_func _datasets _metadata _data_config_start_training_finished_training)selfr r!r"r#r$r%r&r1N/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/train/trainer.py__init__'s  zTrainingIterator.__init__cCs|SNr1r0r1r1r2__iter__BszTrainingIterator.__iter__Nc s:t}|s Jd|jfdddS)Nz8`_start_training` should only be called from within TunecsjjdS)N)r"r#r$r%storager&)r'start_trainingr1r&r%r#r$r0r7r"r1r2Rsz2TrainingIterator._start_training..)rr7_run_with_error_handling)r0r"r#r$r%r& tune_sessionr1r9r2r.Es  z TrainingIterator._start_trainingfuncc Cstz|WSty#td||j|j|j|j||YSt y.t ddt y9t ddw)NzSWorkers have been successfully restarted. Resuming training from latest checkpoint.zThis Trainer is not active. It is either shutdown already or never started in the first place. Either create a new Trainer or start this one.zTraining failed. You should not be seeing this error and this is a bug. Please create a new issue at https://github.com/ray-project/ray.) rloggerinfor.r*r+r,r-r;r RuntimeErrorr)r0r=r1r1r2r;\s4   z)TrainingIterator._run_with_error_handlingc Cs|r |jjddtz||j}|dur*|jjdd||jd|_t|WStys}z;t |t r=|j }nd}t |}d tt|||j}|jjd||dz |jjddd|_WtynYwd}~ww)NF)erroredT)rA stack_trace failed_rank)graceful_termination) is_finishedr'report_final_run_status StopIterationr;_fetch_next_result_finish_trainingr/r isinstancer worker_rankr join tracebackformat_exceptiontype __traceback__shutdown Exception)r0 next_resultserDrCr1r1r2__next__|s@    zTrainingIterator.__next__returncCs0|j}|dur dStdd|DsJ|S)aFetch next results produced by ``session.report()`` from each worker. Assumes ``start_training`` has already been called. Returns: A list of dictionaries of values passed to ``session.report()`` from each worker. Each item corresponds to an intermediate result a single worker. If there are no more items to fetch, returns None. Ncss|]}t|tVqdSr4)rKr).0resultr1r1r2 sz6TrainingIterator._fetch_next_result..)r'get_next_resultsall)r0resultsr1r1r2rIs z#TrainingIterator._fetch_next_resultcCs |jS)atFinish training and return final results. Propagate any exceptions. Blocks until training is finished on all workers. Assumes `start_training` has already been called. Returns: A list of return values from calling ``train_func`` on each worker. Each item corresponds to the return value from a single worker. )r'finish_trainingr5r1r1r2rJs z!TrainingIterator._finish_trainingcCs|jSr4)r/r5r1r1r2rFszTrainingIterator.is_finishedr4)__name__ __module__ __qualname____doc__r rrrrrrstrrr rrrrr3r6r.r;rVrrIrJboolrFr1r1r1r2r#s6 $      ' r).loggingrNpathlibrtypingrrrrrrr ray.air._internal.utilr r r ray.datar ray.trainrr$ray.train._internal.backend_executorrrrrray.train._internal.sessionrrrray.train._internal.utilsrray.train.backendrray.train.base_trainerrrrray.util.annotationsrrr getLoggerr_r>rr1r1r1r2s$ $