o `۷i@s ddlZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z ddl mZmZmZddlmZmZddlmZmZmZmZddlmZmZmZddlmZdd lm Z dd l!m"Z"m#Z#m$Z$dd l%m&Z&erndd l'm(Z(e d Z)e dZ*e+e,Z-e&GdddZ.dS)N)Path) TYPE_CHECKINGAnyCallableDictListOptionalTypeVarUnion)StartTracebackStartTracebackWithWorkerRankskip_exceptions) Checkpoint DataConfig)BackendExecutorInactiveWorkerGroupErrorTrainBackendErrorTrainingWorkerError)_TrainingResult _TrainSession get_session) ActorWrapper) BackendConfig) BaseTrainer GenDatasetTrainingFailedError) DeveloperAPI)DatasetTSc@seZdZdZdeeefdedeege fee e e fge ffde e dfde e e fde d eee e eeffd d Zd d Z dd eefddZdefddZddZdeee fddZddZdefddZdS)TrainingIteratorzEAn iterator over Train results. Returned by ``trainer.run_iterator``.backend_executorbackend_config train_funcdatasetsrmetadata data_config checkpointcCsL||_||_||_||_||_||_|j||j|j|j|dd|_dS)N)r#r$r%r&r'F) _backend_executor backend_cls_backend _train_func _datasets _metadata _data_config_start_training_finished_training)selfr!r"r#r$r%r&r'r2G/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/train/trainer.py__init__)s  zTrainingIterator.__init__cCs|SNr2r1r2r2r3__iter__DszTrainingIterator.__iter__Nc s:t}|s Jd|jfdddS)Nz8`_start_training` should only be called from within TunecsjjdS)N)r#r$r%r&storager')r(start_trainingr2r'r&r$r%r1r8r#r2r3Tsz2TrainingIterator._start_training..)rr8_run_with_error_handling)r1r#r$r%r&r' tune_sessionr2r:r3r/Gs  z TrainingIterator._start_trainingfuncc Cstz|WSty#td||j|j|j|j||YSt y.t ddt y9t ddw)NzSWorkers have been successfully restarted. Resuming training from latest checkpoint.zThis Trainer is not active. It is either shutdown already or never started in the first place. Either create a new Trainer or start this one.zTraining failed. You should not be seeing this error and this is a bug. Please create a new issue at https://github.com/ray-project/ray.) rloggerinfor/r+r,r-r.r<r RuntimeErrorr)r1r>r2r2r3r<^s4   z)TrainingIterator._run_with_error_handlingc Cs|r |jjddtz||j}|dur*|jjdd||jd|_t|WStys}z;t |t r=|j }nd}t |}d tt|||j}|jjd||dz |jjddd|_WtynYwd}~ww)NF)erroredT)rB stack_trace failed_rank)graceful_termination) is_finishedr(report_final_run_status StopIterationr<_fetch_next_result_finish_trainingr0r isinstancer worker_rankr join tracebackformat_exceptiontype __traceback__shutdown Exception)r1 next_resultserErDr2r2r3__next__~s@    zTrainingIterator.__next__returncCs0|j}|dur dStdd|DsJ|S)aFetch next results produced by ``session.report()`` from each worker. Assumes ``start_training`` has already been called. Returns: A list of dictionaries of values passed to ``session.report()`` from each worker. Each item corresponds to an intermediate result a single worker. If there are no more items to fetch, returns None. Ncss|]}t|tVqdSr5)rLr).0resultr2r2r3 sz6TrainingIterator._fetch_next_result..)r(get_next_resultsall)r1resultsr2r2r3rJs z#TrainingIterator._fetch_next_resultcCs |jS)atFinish training and return final results. Propagate any exceptions. Blocks until training is finished on all workers. Assumes `start_training` has already been called. Returns: A list of return values from calling ``train_func`` on each worker. Each item corresponds to the return value from a single worker. )r(finish_trainingr6r2r2r3rKs z!TrainingIterator._finish_trainingcCs|jSr5)r0r6r2r2r3rGszTrainingIterator.is_finishedr5)__name__ __module__ __qualname____doc__r rrrrrrstrrrrrrr4r7r/r<rWrrJrKboolrGr2r2r2r3r %s6 $      ' r )/loggingrOpathlibrtypingrrrrrrr r ray.air._internal.utilr r r ray.trainrr$ray.train._internal.backend_executorrrrrray.train._internal.sessionrrrray.train._internal.utilsrray.train.backendrray.train.base_trainerrrrray.util.annotationsrray.datarrr getLoggerr`r?r r2r2r2r3s& (