o }oiS@sddlZddlZddlmZmZddlmZddlmZm Z ddl Z ddl Z ddl mZddlmZddlmZddlmZmZmZdd lmZdd lmZdd lmZed \ZZd e jde e j!efddZ"eddGddde j#Z$eddGddde j#Z%eddGddde j#Z&eddGddde j#Z'eddGddde j#Z(eddGddde j#Z)eddGdd d e j#Z*eddGd!d"d"e j#Z+dS)#N) dataclassfield)Path)CallableOptional)Callback) WandbLogger)YamlSerializer)MemoryProfileCallback NsysCallbackPreemptionCallback)MegatronStrategy)logging) safe_importz$nvidia_resiliency_ext.ptl_resiliencypartial callbackscCs\t|dr,t|jdr#|jjr#|D]}||jjvr |jj|qdSt||j_dSdS)Ntrainerr)hasattrrrappendcopydeepcopy)rrcallbackrN/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/run/plugins.py_merge_callbacks(s  rT)kw_onlyc@sZeZdZUdZdZeed<edddZe e j e ed<de j e jBd e jfd d Zd S) PreemptionPlugina` A plugin for setting up Preemption callback and preemption signals. Args: preempt_time (int): The time, in seconds, before the task's time limit at which the executor will send a SIGTERM preemption signal. This allows tasks to be gracefully stopped before reaching their time limit, reducing waste and promoting fair resource usage. The default value is 60 seconds (1 minute). This is only supported for ``run.SlurmExecutor``. callbacks (list[run.Config[Callback]]): A list of callback configurations that the plugin will merge with the task's existing callbacks. By default, the list includes NeMo's preemption callback. < preempt_timecCs ttgSN)runConfigr rrrrCs zPreemptionPlugin.)default_factoryrtaskexecutorcCsnt|tjrtd|jjddSt|tjr.t|jjd|j dd|j |_ t ||j ddS)zSet up the preemption plugin.The B will have no effect on the task as it's an instance of run.ScriptNz will send a SIGTERM z= seconds before the job's time limit for your Slurm executor.zTERM@r) isinstancer Scriptrwarning __class____name__ SlurmExecutorinforsignalrrselfr$r%rrrsetupEs  zPreemptionPlugin.setupN)r- __module__ __qualname____doc__rint__annotations__rrlistr r!rPartialr*Executorr3rrrrr2s    rc@s^eZdZUdZdZeed<dZeed<dZeed<dZ eed <d e j e j Bd e j fd d ZdS)FaultTolerancePlugina A plugin for setting up the fault tolerance callback from nvidia-resiliency-ext. This plugin enables workload hang detection, automatic calculation of timeouts used for hang detection, detection of rank(s) terminated due to an error and workload respawning in case of a failure. Note: FaultTolerancePlugin does not work with the NsysPlugin. Args: num_in_job_restarts (int): Max number of restarts on failure, within the same job. Default is 3. num_job_retries_on_failure (int): Max number of new job restarts on failure. Default is 2. initial_rank_heartbeat_timeout (int): Timeouts are time intervals used by a rank monitor to detect that a rank is not alive. This is the max timeout for the initial heartbeat. Default is 1800. rank_heartbeat_timeout (int): This is the timeout for subsequent hearbeats after the initial heartbeat. Default is 300. num_in_job_restartsnum_job_retries_on_failureiinitial_rank_heartbeat_timeouti,rank_heartbeat_timeoutr$r%cCstsJdtj|j|j|jd|_|j|_t |tj sJtj t j dd|jjdg}|jjr3Jdt|drMt|jdrMttdd |jjsMJd t||d d S) z"Set up the fault tolerance plugin.zQnvidia-resiliency-ext.ptl_resiliency is required to use the FaultTolerancePlugin.) max_restartsrArBT) autoresumecalculate_timeoutsexp_dirz1Nsys not supported with the FaultTolerancePlugin.rrcSsdt|vr |jtk SdS)N __fn_or_cls__T)dirrGr )cbrrrr"sz,FaultTolerancePlugin.setup..z-Nsys not supported with FaultTolerancePlugin.r(N)HAVE_RESr FaultTolerancer>rArBlauncherr@retriesr)r:r! res_moduleFaultToleranceCallbackloglog_dir nsys_profilerrallmaprr)r2r$r%rrrrr3ls. zFaultTolerancePlugin.setupN)r-r4r5r6r>r7r8r@rArBr r:r*r;r3rrrrr<Ws      r<c@sreZdZUdZeed<eed<dZeeeed<dZ eee ed<dZ e ed<d e je jBd e jfd d ZdS) NsysPlugina A plugin for nsys profiling. The NsysPlugin allows you to profile your run using nsys. You can specify when to start and end the profiling, on which ranks to run the profiling, and what to trace during profiling. Args: start_step (int): The step at which to start the nsys profiling. end_step (int): The step at which to end the nsys profiling. ranks (Optional[list[int]]): The ranks on which to run the nsys profiling. If not specified, profiling will be run on rank 0. nsys_trace (Optional[list[str]]): The events to trace during profiling. If not specified, 'nvtx' and 'cuda' events will be traced. start_stepend_stepNranks nsys_traceF gen_shaper$r%cCsxt|tjr tjt|j|j|jpdg|jd}|g}t ||d| }d|_ |j p-ddg|_ t|tj r:d|_dSdS) z!Set up the nsys profiling plugin.r)rVrWrXrZr(TnvtxcudazEprofile_%p_%q{SLURM_JOB_ID}_node%q{SLURM_NODEID}_rank%q{SLURM_PROCID}N)r)r r:r!r rVrWrXrZr get_launcherrRrYr. nsys_filename)r2r$r% nsys_callbackrrLrrrr3s     zNsysPlugin.setup)r-r4r5r6r7r8rXrr9rYstrrZboolr r:r*r;r3rrrrrUs   rUc@sJeZdZUdZeed<dZeee ed<de j e j Bde j fddZdS) MemoryProfilePlugina A plugin for memory profiling. The MemoryProfilePlugin allows you to profile a timeline of memory allocations during you run. The memory profiling plugin creates snapshots during the entire training. You can specify which ranks to run the profiling. Args: dir (str): Directory to store the memory profile dump .pickle files ranks (Optional[list[int]]): The ranks on which to run the memory profiling. If not specified, profiling will be run on rank 0. rHNrXr$r%cCs@t|tjrtjt|j|jpdgd}|g}t||ddSdS)z#Set up the memory profiling plugin.r)rHrXr(N)r)r r:r!r rHrXr)r2r$r%memprof_callbackrrrrr3s  zMemoryProfilePlugin.setup)r-r4r5r6r`r8rXrr9r7r r:r*r;r3rrrrrbs   rbc@sXeZdZUdZeed<edeje fed<dZ e ed<dej ej Bdejfd d Zd S) WandbPlugina A plugin for setting up Weights & Biases. This plugin sets a ``WandbLogger`` to ``NeMoLogger``'s ``wandb`` arg, which in turn initializes the Pytorch Lightning `WandbLogger `_. This plugin is only activated if the ``WANDB_API_KEY`` environment variable is set. The ``WANDB_API_KEY`` environment variables will also be set in the executor's environment variables. Follow https://docs.wandb.ai/quickstart to retrieve your ``WANDB_API_KEY``. If `log_task_config` is True, the plugin will log the task configuration as a config dictionary to the Weights and Biases logger. Args: name (str): The name for the Weights & Biases run. logger_fn (Callable[..., run.Config[WandbLogger]]): A callable that returns a Config of ``WandbLogger`` log_task_config (bool, optional): Whether to log the task configuration to the logger. Defaults to True. Raises: logging.warning: If the task is an instance of `run.Script`, as the plugin has no effect on such tasks. name. logger_fnTlog_task_configr$r%cCst|tjrtd|jjddSdtjvrotjd|j d<t |drit |j drk|j |j d|j _|jrmtt|}|j|j |t|tjrZtj|jjt|jj nd|jd|d <||j j_dSdSdSdStd|jjd dS) zSet up the wandb plugin.r&r'N WANDB_API_KEYrPwandb)re)id task_namer%remote_directorylocal_directory experimentzF will have no effect as WANDB_API_KEY environment variable is not set.)r)r r*rr+r,r-osenvironenv_varsrrPrfrerirgyaml safe_loadr serialize experiment_idr/r.pathjointunneljob_dirrconfig)r2r$r%partial_configrrrr3s2     zWandbPlugin.setupN)r-r4r5r6r`r8rr r!rrgrar:r*r;r3rrrrrds   rdc@sjeZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<d e j e j Bd e jfd d Zd S)ConfigValidationPluginaG A plugin for validating a NeMo task with its executor. This plugin is used to ensure that the NeMo environment, task, and executor meet certain criteria. The validation checks include preemption, checkpoint directory, serialization, and Weights and Biases (wandb) integration. Attributes: validate_preemption (bool): Whether to validate the preemption callback. If set to True, the plugin will assert that the task has a `PreemptionCallback`. Defaults to True. validate_checkpoint_dir (bool): Whether to validate the checkpoint directory. If set to True and the executor is a `SlurmExecutor`, the plugin will assert that the task's log directory exists in the mounts specified in the `SlurmExecutor`. Defaults to True. validate_serialization (bool): Whether to validate task serialization. If set to True, the plugin will assert that the task can be successfully serialized and deserialized using NeMo-Run's `ZlibJSONSerializer`. Defaults to True. validate_wandb (bool): Whether to validate Weights and Biases integration. If set to True, the plugin will assert that the executor's environment variables contain a `WANDB_API_KEY` and that NeMo Logger's `wandb` is set. Defaults to False. validate_nodes_and_devices (bool): Whether to validate the number of devices and nodes. If set to True, the plugin will assert that the task's trainer is configured to use the same number of nodes and devices as the executor. Defaults to True. Tvalidate_preemptionvalidate_checkpoint_dirvalidate_serializationFvalidate_wandbvalidate_nodes_and_devicesr$r%csttjsJtdjjd|jjd|jr-tdt t ddj j s-J|j ret|tjre|jdg}tt dd|}td jjd |jjsXJt t fd d|seJ|jrd d lm}td|}||ksJ|jrtdd|jvsJjjsJ|jrtdt|tjrj j|jksJj j| ksJdSdSdS)z*Set up the plugin to configure validation.z Validating z and .zValidating preemption callbackcSs |jtkSr)rGr )rrrrr";s z.ConfigValidationPlugin.setup..z /nemo_runcSs|ddS)N:)split)mrrrr"@szValidating checkpoint dir z exists in cst|tjjjvSr)rrPrQparents)mountr$rrr"Csr)ZlibJSONSerializerz1Validating serialization/de-serialization of taskz6Validating that Weights and Biases is enabled for taskrhz=Validating that nodes and devices match for task and executorN)!r)r r:rr/rGr5r,r}anyrTrrr~r.container_mountsr9rPrQr%nemo_run.core.serialization.zlib_jsonr deserializertrrqkeysrir num_nodesnodesdevicesnproc_per_node)r2r$r%mountsr serializerrrrr35s:           zConfigValidationPlugin.setupN)r-r4r5r6r}rar8r~rrrr r:r*r;r3rrrrr|s       r|c@seZdZUdZdZeed<dZeed<dZ eed<dZ e eed <dZ eed <dZ eed <d d ZdejejBdejfddZdS) PerfEnvPlugina A plugin for setting up performance optimized environments. Attributes: enable_layernorm_sm_margin (bool): Set SM margin for TransformerEngine's Layernorm, so in order to not block DP level communication overlap. layernorm_sm_margin (int): The SM margin for TransformerEngine Layernorm. enable_vboost (bool): Whether to steer more power towards tensor cores via `sudo nvidia-smi boost-slider --vboost 1`. May not work on all systems. Tenable_layernorm_sm_marginlayernorm_sm_marginF enable_vboostNnccl_pp_comm_chunksizegpu_sm100_or_neweruser_buffer_registrationc CsHddl}dddd|dtj|dd tj|d d |d g }|S) zCCreate the vboost `sudo nvidia-smi boost-slider --vboost 1` commandrN z # Command 0: enable vboost srunz --ntasks=z--outputz vboost.outz--errorz vboost.errzbash -c z'sudo nvidia-smi boost-slider --vboost 1)shlexrwrorvquote)r2rryr vboost_cmdrrrget_vboost_srun_cmdls  z!PerfEnvPlugin.get_vboost_srun_cmdr$r%cCs|jjjtkr|jjj}|jjj}|jr"|dks|dkr"d|jd<n|js2|dks-|dkr2d|jd<|jrEt |j |jd<t |j |jd<|jjj }|dkrh|j durht |j tr^|j dks`Jt |j |jd<d|jd <|jrd|jd <d|jd <d |jvr|jd d }d|vrtd|dd ||jd <|jrt |tjr||j|jj}|jrt|jdkr|j|n||_dSdSdS)z+Enable the performance environment settings32CUDA_DEVICE_MAX_CONNECTIONS1NVTE_FWD_LAYERNORM_SM_MARGINNVTE_BWD_LAYERNORM_SM_MARGINNNCCL_P2P_NET_CHUNKSIZETORCH_NCCL_HIGH_PRIORITYNCCL_NVLS_ENABLENCCL_CTA_POLICYPYTORCH_CUDA_ALLOC_CONF,zexpandable_segments:TruezPYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is not currently compatible withuser buffer registration. Removing expandable_segments:True from the list.r)rstrategyrGr tensor_model_parallel_sizecontext_parallel_sizerrqrr`rpipeline_model_parallel_sizerr)r7rrrr+removerwrr r.rrrxry setup_lineslen)r2r$r%tp_sizecp_sizepp_sizepytorch_cuda_alloc_confrrrrr3sF           zPerfEnvPlugin.setup)r-r4r5r6rrar8rr7rrrrrrr r:r*r;r3rrrrrXs      rc@sXeZdZdZddlmZer(ddlmZedkr*de j e j Bde j fdd Z d Sd Sd S) TritonCacheSetupz A plugin for setting up Triton cache environment variables. This should not be neccessay for Triton 3.2.0 and above. r)TRITON_AVAILABLE) __version__z3.1.0r$r%cCs|jd|jd<d|jd<dS)z.Set up the Triton cache environment variables. triton_cahceTRITON_CACHE_DIRz?megatron.core.ssm.triton_cache_manager:ParallelFileCacheManagerTRITON_CACHE_MANAGERN)ryrqr1rrrr3s zTritonCacheSetup.setupN)r-r4r5r6nemo.core.utils.optional_libsrtritonrtriton_versionr r:r*r;r3rrrrrs   r),rro dataclassesrrpathlibrtypingrrnemo_runr rrlightning.pytorchrlightning.pytorch.loggersr nemo_run.core.serialization.yamlr nemo.lightning.pytorch.callbacksr r r 3nemo.lightning.pytorch.strategies.megatron_strategyr nemo.utilsrnemo.utils.import_utilsrrNrJr:r9r!rPluginrr<rUrbrdr|rrrrrrs@         $3,>B]