o }oi< @sdZddlmZddlmZmZddlmZed\ZZ ej dd   dd ee d ee d ee d ee deejf ddZdS)zCommon callback utilities for LLM training recipes. This module provides factory functions for creating various callbacks used in LLM training, including straggler detection and performance monitoring. )Optional)Configcli) safe_importz$nvidia_resiliency_ext.ptl_resiliencyT)is_target_default,ffffff?straggler_report_time_intervalstop_if_detected_stragglergpu_relative_perf_thresholdgpu_individual_perf_thresholdreturnc Csttj|ddd|||dd S)aCreates a callback for detecting slower ranks in PyTorch distributed workloads. This callback from nvidia-resiliency-ext monitors rank performance using two metrics: 1. Relative performance: Compared to the best-performing rank 2. Individual performance: Compared to the rank's best historical performance Performance scores range from 0.0 (worst) to 1.0 (best). A rank is considered a straggler if its score falls below the configured threshold (default 0.7). The callback reports scores every 5 minutes by default. Args: straggler_report_time_interval: Performance score reporting frequency in seconds. Defaults to 300 seconds. stop_if_detected_straggler: Whether to stop training if a straggler is detected. Defaults to True. gpu_relative_perf_threshold: Relative performance threshold for straggler detection. Defaults to 0.7. gpu_individual_perf_threshold: Individual performance threshold for detection. Defaults to 0.7. T)report_time_intervalcalc_relative_gpu_perfcalc_individual_gpu_perfnum_gpu_perf_scores_to_printr r stop_if_detectedenable_ptl_logging)r res_moduleStragglerDetectionCallback)r r r r ra/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/callbacks/common.pystraggler_det_callbacksrN)rTrr)__doc__typingrnemo_runrrnemo.utils.import_utilsrrHAVE_RESfactoryintboolfloatrrrrrrs*