o $i#@sddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z ddl m Z mZerLddlmZddlmZddlmZddlmZdd lmZmZhd Zhd Zhd Zd dhZhdZGdddeZde de e fddZ!d7ddZ"ddZ#de d fddZ$d8d!d"Z%d#d$Z&d%d&Z'd'e ed(d)ee e(ffd*d+Z)d'e ed(d)e*fd,d-Z+d9d0d1Z,d)e*fd2d3Z-d4ed)dfd5d6Z.dS):N)Enum) TYPE_CHECKINGDictListOptionalSetUnion)TagKeyrecord_extra_usage_tag)StorageContext BaseTrainerCallbackTrialSchedulerBasicVariantGeneratorSearcher> TorchTrainerHorovodTrainerXGBoostTrainerLightGBMTrainerTensorflowTrainer> JaxTrainerrrrrDataParallelTrainer>AxSearchTuneBOHB HEBOSearch ZOOptSearch OptunaSearchBayesOptSearchHyperOptSearchNevergradSearchConcurrencyLimiterRepeater> PB2 FIFOSchedulerHyperBandForBOHBHyperBandSchedulerMedianStoppingRuleAsyncHyperBandSchedulerPopulationBasedTrainingResourceChangingSchedulerPopulationBasedTrainingReplayc@seZdZdZdZdZdZdS) AirEntrypointz Tuner.fitz Trainer.fitztune.runztune.run_experimentsN)__name__ __module__ __qualname__TUNERTRAINERTUNE_RUNTUNE_RUN_EXPERIMENTSr7r7T/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/air/_internal/usage.pyr/Cs r/allowed_module_path_prefix whitelistcCs(|j}|jj}||r||vr|SdS)aFind the class name of the object. If the object is not under `allowed_module_path_prefix` or if its class is not in the whitelist, return "Custom". Args: obj: The object under inspection. allowed_module_path_prefix: If the `obj`'s class is not under the `allowed_module_path_prefix`, its class name will be anonymized. whitelist: If the `obj`'s class is not in the `whitelist`, it will be anonymized. Returns: The class name to be tagged with telemetry. Custom)r1 __class__r0 startswith)objr9r: module_pathcls_namer7r7r8_find_class_nameJs rAtrainerr cC6ddlm}t||s Jt|dt}ttj|dS)Nrr ray.train)ray.train.trainerr isinstancerA AIR_TRAINERSr r AIR_TRAINER)rBr trainer_namer7r7r8tag_air_trainer`  rJcCrC)Nr)rrD)&ray.train.v2.api.data_parallel_trainerrrFrATRAIN_V2_TRAINERSr r TRAIN_TRAINER)rBrrIr7r7r8tag_train_v2_trainerhrKrOsearchercCsjddlm}m}t||rttjddSt||r1t|dt t }|t vr)dSttj|dSJd)Nrrrzray.tune.searchFzZNot expecting a non-BasicVariantGenerator, non-Searcher type passed in for `tag_searcher`.) ray.tune.searchrrrFr r TUNE_SEARCHERrATUNE_SEARCHERSunionTUNE_SEARCHER_WRAPPERS)rPrr searcher_namer7r7r8 tag_searcherps   rW schedulerrcCrC)Nrrzray.tune.schedulers)ray.tune.schedulersrrFrATUNE_SCHEDULERSr r TUNE_SCHEDULER)rXrscheduler_namer7r7r8 tag_schedulerrKr]cCttjddSN1)r r AIR_SETUP_WANDB_INTEGRATION_USEDr7r7r7r8tag_setup_wandbrbcCr^r_)r r !AIR_SETUP_MLFLOW_INTEGRATION_USEDr7r7r7r8tag_setup_mlflowrcre callbacksrreturnc Csddlm}ddlm}ddlm}ddlm}ddlm }ddl m }ddl m }||||f|}d d |D} tt} |pAg}|D]/} t| |sLqD| jj} | | vr]| | d 7<qDt| |rk| d d 7<qD| d d 7<qD| S)zHCreates a map of callback class name -> count given a list of callbacks.r)CometLoggerCallback)MLflowLoggerCallback)WandbLoggerCallbackr)LoggerCallback)AimLoggerCallback)DEFAULT_CALLBACK_CLASSEScSsg|]}|jqSr7)r0).0 callback_clsr7r7r8 sz$_count_callbacks..CustomLoggerCallbackCustomCallback)ray.air.integrations.cometrhray.air.integrations.mlflowriray.air.integrations.wandbrjray.tunerray.tune.loggerrkray.tune.logger.aimrlray.tune.utils.callbackrm collections defaultdictintrFr<r0) rfrhrirjrrkrlrmbuilt_in_callbackscallback_namescallback_countscallback callback_namer7r7r8_count_callbackss6          rcCs2|sdSt|}|rt|}ttj|dSdS)aRecords built-in callback usage via a JSON str representing a dictionary mapping callback class name -> counts. User-defined callbacks will increment the count under the `CustomLoggerCallback` or `CustomCallback` key depending on which of the provided interfaces they subclass. NOTE: This will NOT track the name of the user-defined callback, nor its implementation. This will NOT report telemetry if no callbacks are provided by the user. Returns: bool: True if usage was recorded, False otherwise. FN)rjsondumpsr r AIR_CALLBACKS)rfrcallback_counts_strr7r7r8 tag_callbackss rstorager cCs>hd}|jr d}n |jj|vr|jj}nd}ttj|dS)aRecords the storage configuration of an experiment. The storage configuration is set by `RunConfig(storage_path, storage_filesystem)`. The possible storage types (defined by `pyarrow.fs.FileSystem.type_name`) are: - 'local' = pyarrow.fs.LocalFileSystem. This includes NFS usage. - 'mock' = pyarrow.fs._MockFileSystem. This is used for testing. - ('s3', 'gcs', 'abfs', 'hdfs'): Various remote storage schemes with default implementations in pyarrow. - 'custom' = All other storage schemes, which includes ALL cases where a custom `storage_filesystem` is provided. - 'other' = catches any other cases not explicitly handled above. >s3gcsabfshdfsmocklocalcustomotherN)custom_fs_providedstorage_filesystem type_namer r AIR_STORAGE_CONFIGURATION)rr:storage_config_tagr7r7r8tag_storage_types  rcCs|ddlm}ddlm}ddlm}tt|||}g}|D] }|t j vr,| |q |rs@          "&