o `۷iz@szddlZddlmZmZmZmZddlmZddl Z ddl m Z ddl m Z GdddeZe dd Gd d d eZdS) N)DictListOptionalUnion)Callback)TensorflowCheckpoint) PublicAPIcseZdZdZgdZd(deeeefffdd Zde defd d Z d)d d Z d)ddZ d)ddZ d)ddZd)ddZd)ddZd)ddZd)ddZd)ddZd)ddZd)d d!Zd)d"d#Zd)d$d%Zd)d&d'ZZS)* _Callbackz%Base class for Air's Keras callbacks.) epoch_begin epoch_endtrain_batch_begintrain_batch_endtest_batch_begintest_batch_endpredict_batch_beginpredict_batch_end train_begin train_end test_begintest_end predict_begin predict_endvalidation_endoncsPttt|ts|g}tfdd|Dr#td|j|_ dS)Nc3s|]}|jvVqdSN)_allowed).0wselfP/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/air/integrations/keras.py $sz%_Callback.__init__..z4Invalid trigger time selected: {}. Must be one of {}) superr __init__ isinstancelistany ValueErrorformatr_on)rr __class__rr!r$s  z_Callback.__init__logswhencCstr)NotImplementedError)rr-r.r r r!_handle,sz_Callback._handleNcCd|jvr ||ddSdS)Nr r*r0repochr-r r r!on_epoch_begin/ z_Callback.on_epoch_begincCr1)Nr r2r3r r r! on_epoch_end3r6z_Callback.on_epoch_endcCr1)Nr r2rbatchr-r r r!on_train_batch_begin7r6z_Callback.on_train_batch_begincCr1)Nr r2r8r r r!on_train_batch_end;r6z_Callback.on_train_batch_endcCr1)Nrr2r8r r r!on_test_batch_begin?r6z_Callback.on_test_batch_begincCr1)Nrr2r8r r r!on_test_batch_endCr6z_Callback.on_test_batch_endcCr1)Nrr2r8r r r!on_predict_batch_beginGr6z _Callback.on_predict_batch_begincCr1)Nrr2r8r r r!on_predict_batch_endKr6z_Callback.on_predict_batch_endcCd|jvr ||ddSdS)Nrr2rr-r r r!on_train_beginOr6z_Callback.on_train_begincCr@)Nrr2rAr r r! on_train_endSr6z_Callback.on_train_endcCr@)Nrr2rAr r r! on_test_beginWr6z_Callback.on_test_begincCr@)Nrr2rAr r r! on_test_end[r6z_Callback.on_test_endcCr@)Nrr2rAr r r!on_predict_begin_r6z_Callback.on_predict_begincCr@)Nrr2rAr r r!on_predict_endcr6z_Callback.on_predict_end)rr)__name__ __module__ __qualname____doc__rrstrrr$rr0r5r7r:r;r<r=r>r?rBrCrDrErFrG __classcell__r r r+r!r s&              r alpha) stabilityc seZdZdZ   ddeeeefdeeeefdeeeeeeeeffffdd Z d ed efd d Z d ed efddZ Z S)ReportCheckpointCallbackaKeras callback for Ray Train reporting and checkpointing. .. note:: Metrics are always reported with checkpoints, even if the event isn't specified in ``report_metrics_on``. Example: .. code-block:: python ############# Using it in TrainSession ############### from ray.air.integrations.keras import ReportCheckpointCallback def train_loop_per_worker(): strategy = tf.distribute.MultiWorkerMirroredStrategy() with strategy.scope(): model = build_model() model.fit(dataset_shard, callbacks=[ReportCheckpointCallback()]) Args: metrics: Metrics to report. If this is a list, each item describes the metric key reported to Keras, and it's reported under the same name. If this is a dict, each key is the name reported and the respective value is the metric key reported to Keras. If this is None, all Keras logs are reported. report_metrics_on: When to report metrics. Must be one of the Keras event hooks (less the ``on_``), e.g. "train_start" or "predict_end". Defaults to "epoch_end". checkpoint_on: When to save checkpoints. Must be one of the Keras event hooks (less the ``on_``), e.g. "train_start" or "predict_end". Defaults to "epoch_end". r N checkpoint_onreport_metrics_onmetricscsTt|tr|g}t|tr|g}tt||}tj|d||_||_||_dS)N)r) r%rLr&setr#r$_checkpoint_on_report_metrics_on_metrics)rrQrRrSrr+r r!r$s   z!ReportCheckpointCallback.__init__r-r.cCst||jvs ||jvs J||}||jv}|r0t|j}tjj||dt j |j dddStjj|dddS)N) checkpointT) ignore_errors) rUrV_get_reported_metricsr from_modelmodelraytrainreportshutilrmtreepath)rr-r.rSshould_checkpointrXr r r!r0s   z ReportCheckpointCallback._handlereturncst|jtdtttfsJ|jdur}n2t|jtr%|j|ji}n#t|jtr6fdd|jD}nt|jtrHfdd|jD}t|tsOJ|S)Ncsi|]}||qSr r )rmetricr-r r! szBReportCheckpointCallback._get_reported_metrics..csi|] \}}||qSr r )rkeyrerfr r!rgs)r%rWtyperLr&dictitems)rr-reported_metricsr rfr!rZs     z.ReportCheckpointCallback._get_reported_metrics)r r N) rHrIrJrKrrLrrrr$r0rZrMr r r+r!rPhs"rP)r`typingrrrrtensorflow.keras.callbacksr KerasCallbackr]ray.train.tensorflowrray.util.annotationsrr rPr r r r!s   ]