o bi#@sddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z ddl m Z mZerLddlmZddlmZddlmZddlmZdd lmZmZhd Zhd Zhd Zd dhZhdZGdddeZde de e fddZ!d7ddZ"ddZ#de d fddZ$d8d!d"Z%d#d$Z&d%d&Z'd'e ed(d)ee e(ffd*d+Z)d'e ed(d)e*fd,d-Z+d9d0d1Z,d)e*fd2d3Z-d4ed)dfd5d6Z.dS):N)Enum) TYPE_CHECKINGDictListOptionalSetUnion)TagKeyrecord_extra_usage_tag)StorageContext BaseTrainerCallbackTrialSchedulerBasicVariantGeneratorSearcher> TorchTrainerHorovodTrainerXGBoostTrainerLightGBMTrainerTensorflowTrainer>rrrrDataParallelTrainer>AxSearchTuneBOHB HEBOSearch ZOOptSearch OptunaSearchBayesOptSearchHyperOptSearchNevergradSearchConcurrencyLimiterRepeater> PB2 FIFOSchedulerHyperBandForBOHBHyperBandSchedulerMedianStoppingRuleAsyncHyperBandSchedulerPopulationBasedTrainingResourceChangingSchedulerPopulationBasedTrainingReplayc@seZdZdZdZdZdZdS) AirEntrypointz Tuner.fitz Trainer.fitztune.runztune.run_experimentsN)__name__ __module__ __qualname__TUNERTRAINERTUNE_RUNTUNE_RUN_EXPERIMENTSr6r6K/home/ubuntu/.local/lib/python3.10/site-packages/ray/air/_internal/usage.pyr.Bs r.allowed_module_path_prefix whitelistcCs(|j}|jj}||r||vr|SdS)aFind the class name of the object. If the object is not under `allowed_module_path_prefix` or if its class is not in the whitelist, return "Custom". Args: obj: The object under inspection. allowed_module_path_prefix: If the `obj`'s class is not under the `allowed_module_path_prefix`, its class name will be anonymized. whitelist: If the `obj`'s class is not in the `whitelist`, it will be anonymized. Returns: The class name to be tagged with telemetry. Custom)r0 __class__r/ startswith)objr8r9 module_pathcls_namer6r6r7_find_class_nameIs r@trainerr cC6ddlm}t||s Jt|dt}ttj|dS)Nrr ray.train)ray.train.trainerr isinstancer@ AIR_TRAINERSr r AIR_TRAINER)rAr trainer_namer6r6r7tag_air_trainer_  rIcCrB)Nr)rrC)&ray.train.v2.api.data_parallel_trainerrrEr@TRAIN_V2_TRAINERSr r TRAIN_TRAINER)rArrHr6r6r7tag_train_v2_trainergrJrNsearchercCsjddlm}m}t||rttjddSt||r1t|dt t }|t vr)dSttj|dSJd)Nrrrzray.tune.searchFzZNot expecting a non-BasicVariantGenerator, non-Searcher type passed in for `tag_searcher`.) ray.tune.searchrrrEr r TUNE_SEARCHERr@TUNE_SEARCHERSunionTUNE_SEARCHER_WRAPPERS)rOrr searcher_namer6r6r7 tag_searcheros   rV schedulerrcCrB)Nrrzray.tune.schedulers)ray.tune.schedulersrrEr@TUNE_SCHEDULERSr r TUNE_SCHEDULER)rWrscheduler_namer6r6r7 tag_schedulerrJr\cCttjddSN1)r r AIR_SETUP_WANDB_INTEGRATION_USEDr6r6r6r7tag_setup_wandbracCr]r^)r r !AIR_SETUP_MLFLOW_INTEGRATION_USEDr6r6r6r7tag_setup_mlflowrbrd callbacksrreturnc Csddlm}ddlm}ddlm}ddlm}ddlm }ddl m }ddl m }||||f|}d d |D} tt} |pAg}|D]/} t| |sLqD| jj} | | vr]| | d 7<qDt| |rk| d d 7<qD| d d 7<qD| S)zHCreates a map of callback class name -> count given a list of callbacks.r)CometLoggerCallback)MLflowLoggerCallback)WandbLoggerCallbackr)LoggerCallback)AimLoggerCallback)DEFAULT_CALLBACK_CLASSEScSsg|]}|jqSr6)r/).0 callback_clsr6r6r7 sz$_count_callbacks..CustomLoggerCallbackCustomCallback)ray.air.integrations.cometrgray.air.integrations.mlflowrhray.air.integrations.wandbriray.tunerray.tune.loggerrjray.tune.logger.aimrkray.tune.utils.callbackrl collections defaultdictintrEr;r/) rergrhrirrjrkrlbuilt_in_callbackscallback_namescallback_countscallback callback_namer6r6r7_count_callbackss6          rcCs2|sdSt|}|rt|}ttj|dSdS)aRecords built-in callback usage via a JSON str representing a dictionary mapping callback class name -> counts. User-defined callbacks will increment the count under the `CustomLoggerCallback` or `CustomCallback` key depending on which of the provided interfaces they subclass. NOTE: This will NOT track the name of the user-defined callback, nor its implementation. This will NOT report telemetry if no callbacks are provided by the user. Returns: bool: True if usage was recorded, False otherwise. FN)rjsondumpsr r AIR_CALLBACKS)rercallback_counts_strr6r6r7 tag_callbackss rstorager cCs>hd}|jr d}n |jj|vr|jj}nd}ttj|dS)aRecords the storage configuration of an experiment. The storage configuration is set by `RunConfig(storage_path, storage_filesystem)`. The possible storage types (defined by `pyarrow.fs.FileSystem.type_name`) are: - 'local' = pyarrow.fs.LocalFileSystem. This includes NFS usage. - 'mock' = pyarrow.fs._MockFileSystem. This is used for testing. - ('s3', 'gcs', 'abfs', 'hdfs'): Various remote storage schemes with default implementations in pyarrow. - 'custom' = All other storage schemes, which includes ALL cases where a custom `storage_filesystem` is provided. - 'other' = catches any other cases not explicitly handled above. >s3gcsabfshdfsmocklocalcustomotherN)custom_fs_providedstorage_filesystem type_namer r AIR_STORAGE_CONFIGURATION)rr9storage_config_tagr6r6r7tag_storage_types  rcCs|ddlm}ddlm}ddlm}tt|||}g}|D] }|t j vr,| |q |rs@          "&