o bi+ @sddlZddlZddlZddlmZmZddlZddlZdZej j j j Z dddeide jde jid d d id d id ddddddddgdddddgdddddgdddddgddddZdd iZd!Zd"Ziadaeejdd#Gd$d%d%ZGd&d'd'ejZd(eefd)d*Zd(eefd+d,Zd>d-d.Zd>d/d0Z d(eefd1d2Z!d(ej"fd3d4Z#d5ed(efd6d7Z$d5eed(dfd8d9Z%d5ed(ee&fd:d;Z'd5ed(ee&fd._load_logging_configJSONrr) rDr=environget$RAY_DATA_LOGGING_CONFIG_ENV_VAR_NAME"RAY_DATA_LOG_ENCODING_ENV_VAR_NAMEDEFAULT_CONFIGuppervalues'RAY_DATA_LOG_HANDLER_JSON_SUBSTITUTIONSitemsremoveappend)rLrG log_encodingrKloggerold_handler_namenew_handler_namer%r%r&_get_logging_configs"   r]cCstdi}t|S)Nr)r]rOlistkeys) logger_configr%r%r&_get_logger_namess racCsZtjt}tjt}t}tj||dur)|dur+t t }| ddSdSdS)aConfigure the Python logger named 'ray.data'. This function loads the configration YAML specified by "RAY_DATA_LOGGING_CONFIG" environment variable. If the variable isn't set, this function loads the default "logging.yaml" file that is adjacent to this module. If "RAY_DATA_LOG_ENCODING" is specified as "JSON" we will enable JSON logging mode if using the default logging config. NzMUsing `RAY_DATA_LOG_ENCODING` is not supported with `RAY_DATA_LOGGING_CONFIG`) r=rNrOrPrQr]rArK dictConfig getLoggerr(warning)rGrYrKrZr%r%r&configure_loggings   recCs,td}|j|tjiadadS)zSReset the logger named 'ray.data' to its initial state. Used for testing. rN)rArcrclearsetLevelNOTSET_DATASET_LOGGER_HANDLER_ACTIVE_DATASET)rZr%r%r& reset_loggings   rkcCs.tjjj}|dur dS|}tj|ddS)zyReturn the directory where Ray Data writes log files. If Ray isn't initialized, this function returns ``None``. Nlogszray-data)r_privateworker _global_nodeget_session_dir_pathr=r?r@) global_node session_dirr%r%r&r<s r<cCs0tjt}|dur|dkrtSttS)NrM) r=rNrOrQrSDEFAULT_JSON_FORMATTERrArEDEFAULT_TEXT_FORMATTER)rYr%r%r&_get_default_formatters  ru dataset_idcCs"td|dd}|t|S)zCreate a log handler for a dataset with the given ID. Args: dataset_id: The ID of the dataset. Returns: A log handler for the dataset. z ray-data-z.log)r )r,r:ru)rvhandlerr%r%r&_create_dataset_log_handlers rxcCs>|sdStt|}ddtD}|D]}||qdS)a<Create a log handler for a dataset with the given ID. Switch the dataset logger for the worker to this dataset logger. Note that only the driver keeps track of the active dataset. The worker will just use the handler that the driver tells it to use. Args: dataset_id: The ID of the dataset. NcSg|]}t|qSr%rArc.0namer%r%r& z4update_dataset_logger_for_worker..)rerxra addHandler)rv log_handlerrrZr%r%r& update_dataset_logger_for_workers rcCsddtD}t|}tt}|||d||t|<ts/|a|D]}||q&n||dtd|d| |tS)aCreate a log handler for a dataset with the given ID. Activate the handler if this is the only active dataset. Otherwise, print a warning to that handler and keep it inactive until it becomes the only active dataset. Args: dataset_id: The ID of the dataset. cSryr%rzr{r%r%r&r~ rz+register_dataset_logger..z(Registered dataset logger for dataset %sz- registers for logging while another dataset zJ is also logging. For performance reasons, we will not log to the dataset z% until it is the only active dataset.) rarxrArcr(rinforirj removeHandler)rvrr local_loggerrZr%r%r&register_dataset_loggers(      rcCsdddtD}t|d}t|kr datr tttt|r0|D]}||q$| tS)zmRemove the logger for a dataset with the given ID. Args: dataset_id: The ID of the dataset. cSryr%rzr{r%r%r&r~Arz-unregister_dataset_logger..N) raripoprjrnextiterr_rclose)rvrrrZr%r%r&unregister_dataset_logger9s  r)r9N))rAlogging.configr=typingrrrIrrtrm ray_loggingr JSONFormatterrsr)r(rRrUrQrPrirj addLevelNamerrHandlerr,dictr]rDrarerkr<rErurxrintrrr%r%r%r&s| 2 ,   #