o `۷iS1@sddlZddlZddlmZddlmZmZmZddlm Z ddl m Z er2ddl m Z ddlmZeeZGdd d ZdS) N)deepcopy) TYPE_CHECKINGDictOptional)version) flatten_dict)Run MlflowClientc@sFeZdZdZddZd0ddZ       d1deed eed eed eed eed eedefddZ de de fddZ   d2deedee deddfddZ d3deedee ddfddZ dedefdd Zd4d"d#Zd0d$e deefd%d&Zd0d'e deefd(d)Zd0d*edeefd+d,Zd3d-eefd.d/ZdS)5_MLflowLoggerUtilzUtil class for setting up and logging to MLflow. Use this util for any library that needs MLflow logging/tracking logic such as Ray Tune or Ray Train. cCsddl}||_d|_dS)Nr)mlflow_mlflow experiment_id)selfr rN/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/air/_internal/mlflow.py__init__s z_MLflowLoggerUtil.__init__NcCs@|j}|jdt|j|}t}|j|||_||_|S)Nr )r __dict__poprr update)rmemor dict_copy copied_objectrrr __deepcopy__s   z_MLflowLoggerUtil.__deepcopy__T tracking_uri registry_urirexperiment_nametracking_tokenartifact_locationcreate_experiment_if_not_existsc Cs>|r|tjd<|j||j||dur|ntjd}|durIddlm}z|jj|dt d|d||_ WdS|yHYnw|durO|ntjd }|durs|jj |d rst d |d|j |j |_ dS|r|rt d ||jj ||d |_ dS|rtd|dtd|d|d)a Sets up MLflow. Sets the Mlflow tracking uri & token, and registry URI. Also sets the MLflow experiment that the logger should use, and possibly creates new experiment if it does not exist. Args: tracking_uri: The tracking URI for the MLflow tracking server. registry_uri: The registry URI for the MLflow model registry. experiment_id: The id of an already existing MLflow experiment to use for logging. If None is passed in here and the MFLOW_EXPERIMENT_ID is not set, or the experiment with this id does not exist, ``experiment_name`` will be used instead. This argument takes precedence over ``experiment_name`` if both are passed in. experiment_name: The experiment name to use for logging. If None is passed in here, the MLFLOW_EXPERIMENT_NAME environment variable is used to determine the experiment name. If the experiment with the name already exists with MLflow, it will be reused. If not, a new experiment will be created with the provided name if ``create_experiment_if_not_exists`` is set to True. artifact_location: The location to store run artifacts. If not provided, MLFlow picks an appropriate default. Ignored if experiment already exists. tracking_token: Tracking token used to authenticate with MLflow. create_experiment_if_not_exists: Whether to create an experiment with the provided name if it does not already exist. Defaults to True. Returns: Whether setup is successful. MLFLOW_TRACKING_TOKENNMLFLOW_EXPERIMENT_IDrMlflowException)rzExperiment with provided id z( exists. Setting that as the experiment.MLFLOW_EXPERIMENT_NAME)namezExperiment with provided name zBExisting experiment not found. Creating new experiment with name: )r%rz,Experiment with the provided experiment_id: zZ does not exist and no experiment_name provided. At least one of these has to be provided.z or experiment_name: zV does not exist. Please create an MLflow experiment and provide either its id or name.)osenvironr set_tracking_uriset_registry_urigetmlflow.exceptionsr#get_experimentloggerdebugrget_experiment_by_namecreate_experiment ValueError) rrrrrrrrr#rrr setup_mlflow)sv-         z_MLflowLoggerUtil.setup_mlflow dict_to_logreturnc CsVi}|D]"\}}z t|}|||<Wqttfy(td||Yqw|S)aParses provided dict to convert all values to float. MLflow can only log metrics that are floats. This does not apply to logging parameters or artifacts. Args: dict_to_log: The dictionary containing the metrics to log. Returns: A dictionary containing the metrics to log with all values being converted to floats, or skipped if not able to be converted. zMCannot log key {} with value {} since the value cannot be converted to float.)itemsfloatr1 TypeErrorr-r.format)rr3new_dictkeyvaluerrr _parse_dicts  z_MLflowLoggerUtil._parse_dictFrun_nametags set_activercCsddl}ddlm}|duri}|r|j||dS|}t|jtdkr3|j||j |d}|S|||<|j|j |d}|S)aMStarts a new run and possibly sets it as the active run. Args: tags: Tags to set for the new run. set_active: Whether to set the new run as the active run. If an active run already exists, then that run is returned. Returns: The newly created MLflow run. rN)MLFLOW_RUN_NAME)r=r>z1.30.0r=rr>)rr>) r mlflow.utils.mlflow_tagsr@_start_active_run _get_clientrparse __version__ create_runr)rr=r>r?r r@clientrunrrr start_runs z_MLflowLoggerUtil.start_runcCs&|j}|r |S|jj||j|dS)zStarts a run and sets it as the active run if one does not exist. If an active run already exists, then returns it. rA)r active_runrJr)rr=r>rKrrrrCs z#_MLflowLoggerUtil._start_active_runrun_idcCs6ddlm}z |jj|dWdS|yYdSw)z)Check if run with the provided id exists.rr")rLTF)r+r#r get_run)rrLr#rrr _run_existss  z_MLflowLoggerUtil._run_existsr cCs,|j}|j}ddlm}|||dS)z@Returns an ml.tracking.MlflowClient instance to use for logging.rr )rr)r get_tracking_uriget_registry_urimlflow.trackingr )rrrr rrrrDs    z_MLflowLoggerUtil._get_client params_to_logcCsjt|}|r"||r"|}|D] \}}|j|||dqdS|D] \}}|jj||dq&dS)aLogs the provided parameters to the run specified by run_id. If no ``run_id`` is passed in, then logs to the current active run. If there is not active run, then creates a new run and sets it as the active run. Args: params_to_log: Dictionary of parameters to log. run_id (Optional[str]): The ID of the run to log to. )rLr:r;)r:r;N)rrNrDr5 log_paramr )rrRrLrHr:r;rrr log_paramss z_MLflowLoggerUtil.log_paramsmetrics_to_logcCsxt|}||}|r(||r(|}|D] \}}|j||||dqdS|D] \}}|jj|||dq,dS)a|Logs the provided metrics to the run specified by run_id. If no ``run_id`` is passed in, then logs to the current active run. If there is not active run, then creates a new run and sets it as the active run. Args: metrics_to_log: Dictionary of metrics to log. run_id (Optional[str]): The ID of the run to log to. )rLr:r;step)r:r;rVN)rr<rNrDr5 log_metricr )rrVrUrLrHr:r;rrr log_metricss z_MLflowLoggerUtil.log_metricsdircCs:|r||r|}|j||ddS|jj|ddS)aSaves directory as artifact to the run specified by run_id. If no ``run_id`` is passed in, then saves to the current active run. If there is not active run, then creates a new run and sets it as the active run. Args: dir: Path to directory containing the files to save. run_id (Optional[str]): The ID of the run to log to. )rL local_dir)rZN)rNrD log_artifactsr )rrYrLrHrrrsave_artifacts/s z _MLflowLoggerUtil.save_artifactsstatuscCsV|r"||r"|jr|jjj|ks"|}|j||ddS|jj|ddS)a2Terminates the run specified by run_id. If no ``run_id`` is passed in, then terminates the active run if one exists. Args: status (Optional[str]): The status to set when terminating the run. run_id (Optional[str]): The ID of the run to terminate. )rLr])r]N)rNr rKinforLrDset_terminatedend_run)rr]rLrHrrrr`@s z_MLflowLoggerUtil.end_run)N)NNNNNNT)NNF)NN)r4r )__name__ __module__ __qualname____doc__rrrstrboolr2rr<rJrCrNrDrTrXr\r`rrrrr sj   u '   r )loggingr&copyrtypingrrr packagingrray._private.dictrmlflow.entitiesrrQr getLoggerrar-r rrrrs