o wi0 @sdZddlZddlmZmZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZdd lmZd gZd eeeffddZdededed eeeffddZded eeeffddZdeded eeeffddZd efddZGdd d eeZdS)z OneLogger callback for NeMo training. This module provides a callback that integrates OneLogger telemetry with NeMo training. N)AnyDict)Trainer)ModelCheckpoint)OneLoggerConfig) on_app_start)TrainingTelemetryConfig)TrainingTelemetryProvider)TimeEventCallback) BaseCallbackOneLoggerNeMoCallbackreturncCsLdtjvr tjd}ntjdd}ttjdd}d|t|d}|S)aGenerate minimal configuration for OneLogger initialization. This function provides the absolute minimal configuration needed for OneLogger initialization. It only includes the required fields and uses defaults for everything else to avoid dependencies on exp_manager during early import. Returns: Dictionary containing minimal initialization configuration EXP_NAMESLURM_JOB_NAMEnemo-run WORLD_SIZEnemo)application_namesession_tag_or_fnenable_for_current_rankworld_size_or_fn)osenvirongetint_should_enable_for_current_rank) session_tag world_size init_configr _/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/lightning/one_logger_callback.pyget_one_logger_init_config%s  r"trainerglobal_batch_size seq_lengthc CsVdtjvr tjd}ntjdd}ttjdd}t|dd}t|dd}||}tjd d }|d |d |d |d| } ||} d} d} d} dd|jD}t|dk} t|dd}|dk} t|dr|jdurt |jt r|jddr~d} n t|jdr|jj rd} |D]}t|dr|j rd} nq| ||||| || | | d }|S)aHGenerate base configuration for OneLogger training telemetry. This function provides the common configuration needed for both NeMo v1 and v2. It extracts basic training information from trainer object and uses provided batch size and sequence length values. Args: trainer: PyTorch Lightning trainer instance global_batch_size: Global batch size (calculated by version-specific function) seq_length: Sequence length (calculated by version-specific function) Returns: Dictionary containing base training callback configuration rrrrr max_stepslog_every_n_steps PERF_VERSION_TAGz0.0.0__bf_se_wsFsynccSsg|] }t|tr|qSr ) isinstancer).0cbr r r! osz-_get_base_callback_config..rval_check_intervalstrategyN async_saveasync) perf_tag_or_fnglobal_batch_size_or_fnmicro_batch_size_or_fnseq_length_or_fntrain_iterations_target_or_fntrain_samples_target_or_fnlog_every_n_train_iterations&is_validation_iterations_enabled_or_fn is_save_checkpoint_enabled_or_fnsave_checkpoint_strategy) rrrrgetattr callbackslenhasattrr5r/dictr6)r#r$r%job_namerr&r'micro_batch_sizeperf_version_tagperf_tagtrain_samples_targetis_save_checkpoint_enabled is_validation_iterations_enabledrAcheckpoint_callbacksr3callback base_configr r r!_get_base_callback_configCsR       rQc Csd}d}t|drz|jdurzt|jdrz|jj}t|drk|j}t|dd}|dur8t|ttjdd}n3t|drk|j }t|d rkt |d krkt|d rTt |n|}t |t |}t|ttjdd}t|d rzt|j d rz|j j}t|||d} | S)akGenerate NeMo v1 specific configuration for OneLogger training callback. This function provides NeMo v1 specific configuration by extracting values from the exp_manager_config object and trainer object. Args: trainer: PyTorch Lightning trainer instance Returns: Dictionary containing NeMo v1 training callback configuration rlightning_moduleNcfgtrain_ds batch_sizerbucket_batch_size__len__r__iter__encoderd_modelr#r$r%)rErRrSrTrBrrrrrVrDlistsumrYrZrQ) r#r$r% model_cfgrTrHbucket_batch_sizes bucket_listavg_batch_sizeconfigr r r!get_nemo_v1_callback_configs8      rcdatacCsd}d}|dur8|j}t|drt|ddurtt|d}nt|dd}|dur8ttjdd}t||}t|||d}|S)aGenerate NeMo v2 specific configuration for the OneLogger training callback. This function extracts the global batch size and sequence length from the provided NeMo v2 data module, and uses them to construct the configuration dictionary for the OneLogger training callback. Args: trainer: PyTorch Lightning trainer instance. data: NeMo v2 data module (required). Returns: Dictionary containing the NeMo v2 training callback configuration. rNr$rHrr[)r%rErBrrrrrQ)r#rdr$r%rHrrbr r r!get_nemo_v2_callback_configs   recCsttjdd}|dkS)aDetermine if OneLogger should be enabled for the current rank. Uses environment variables instead of torch.distributed to avoid circular imports. In distributed training, typically only rank 0 (or the last rank) should enable OneLogger to avoid duplicate telemetry data. Returns: True if OneLogger should be enabled for the current rank, False otherwise RANKr4r)rrrr)rankr r r!rs rcsHeZdZdZdZfddZd fdd Zded eddfd d Z Z S) r zAdapter extending OneLogger's PTL callback with init + config update. __init__ configures the provider from meta info, then calls super().__init__. update_config computes TrainingTelemetryConfig and applies it. Ncs|jdur t||_|jS)N) _instancesuper__new__)clsargskwargs __class__r r!rjs zOneLoggerNeMoCallback.__new__r csXt|ddrdSt}tdi|}t|tj tddt dS)N _initializedF)call_on_app_startr ) rBr"rr instancewith_base_configwith_export_configconfigure_providerri__init__r)selfrone_logger_configrnr r!rvs   zOneLoggerNeMoCallback.__init__ nemo_versionr#cKsttjjdur dS|dkrt|d}n|dkr%|dd}t||d}nt|d}tdi|}t|dS)Nv1)r#v2rd)r#rdr ) r rrrbtelemetry_configrcrrerset_training_telemetry_config)rwryr#rmrbrdtraining_telemetry_configr r r! update_config!s   z#OneLoggerNeMoCallback.update_config)r N) __name__ __module__ __qualname____doc__rhrjrvstrrr __classcell__r r rnr!r s   ) rrtypingrrlightning.pytorchr,lightning.pytorch.callbacks.model_checkpointrnv_one_logger.api.configr.nv_one_logger.training_telemetry.api.callbacksr+nv_one_logger.training_telemetry.api.configr@nv_one_logger.training_telemetry.api.training_telemetry_providerr >nv_one_logger.training_telemetry.integration.pytorch_lightningr OneLoggerPTLCallbacknemo.lightning.base_callbackr __all__rr"rrQrcreboolrr r r r r!s@          U5  *