o TÃi?Ôã @s.ddlZddlZddlZddlZddlZddlZddlmZddlTddl m Z m Z ddl m Z ddlmZddlTdd lmZdd lmZmZmZddl Tdd lmZzdd lmZWn eygdZYnwzddlZd ZWney„ZzdZWYdZ[ndZ[wwdZdZ dZ!e j"Z#Gdd„dƒZ$dS)éNé)Ú"dict_raise_error_on_duplicate_keys)Ú*)ÚZERO_OPTIMIZATIONÚ ZeroStageEnum)Úloggeré)ÚDeepSpeedAutotuningConfig)ÚResourceManager)ÚGridSearchTunerÚ RandomTunerÚModelBasedTuner)Úget_accelerator)ÚtabulateTFÚstageÚoffload_optimizerÚ offload_paramc@seZdZdZdd„Zdd„Zdd„Zdd „Zd d „Zd d „Z dd„Z dd„Z dd„Z dd„Z dd„Zdd„Zdd„Zdd„Zdd„Zd d!„Zd"d#„Zd$d%„ZdFd'd(„Zd)d*„Zd+d,„Zd-d.„Zd/d0„Zd1d2„Zd3d4„Zd5d6„Zd7d8„Zd9d:„Zd;d<„Z d=d>„Z!d?d@„Z"dAdB„Z#dCdD„Z$dES)GÚ AutotuneraxThe DeepSpeed Autotuner automatically discovers the optimal DeepSpeed configuration that delivers good training speed. The Autotuner uses model information, system information, and heuristics to efficiently tune system knobs that affect compute and memory efficiencies, such as ZeRO optimization stages, micro-batch sizes, and many other ZeRO optimization configurations. It not only reduces the time and resources user spend on tuning, but also can discover configurations better than hand-tuned methods. Autotuning with DeepSpeed requires no code change from DeepSpeed users. Please refer to the README for usage details. cCs||_d|_tdusJdƒ‚t d|›¡| |j¡|_|jdus&Jdƒ‚t|jƒ|_ |jt rOt |jt   ¡vr@|jt t =t |jt   ¡vrO|jt t =|j j|_|j jrgtj |j¡rgtj|jddtj |j¡s”ztj|jddt d|j›¡Wnt d|j›d ¡td ƒY|j j|_|j jr¬tj |j¡r¬tj|jddtj |j¡sÙztj|jddt d |j›¡Wnt d|j›d ¡td ƒY| |¡|_| |¡\|_|_|j|jjksóJd ƒ‚|jt |jj!ƒksJdƒ‚i|_"d|_#d|_$d|_%dS)Nz]Missing required package `tabulate`, please install with `pip install deepspeed[autotuning]`.zautotuning args=z'DeepSpeed configuration is not providedT)Ú ignore_errors)Úexist_okz*Created autotuning experiments directory: zFailed to create z`, please check exps_dir in the autotuning config file is accessible by all the nodes in the job.éÿÿÿÿz&Created autotuning results directory: zc, please check results_dir in the autotuning config file is accessible by all the nodes in the job.znnum_gpus in the autotuning configuration must not be less than the --num_gpus value in the train script if anyzpnum_nodes in the autotuning configuration must not be less than the --num_nodes value in the train script if any)&ÚargsÚselected_exp_dirrrÚdebugÚ_get_user_configÚ user_argsÚ user_configr Úautotuning_configÚ AUTOTUNINGÚAUTOTUNING_EXPS_DIRÚkeysÚAUTOTUNING_RESULTS_DIRÚexps_dirÚ overwriteÚosÚpathÚexistsÚshutilÚrmtreeÚmakedirsÚinfoÚerrorÚexitÚ results_dirÚ_get_resource_managerÚrmÚ_get_exp_resourcesÚ exp_num_nodesÚ exp_num_gpusÚnum_gpus_per_nodeÚlenÚnodesÚrecordsÚ optimal_cmdÚoptimal_ds_configÚmlflow_parent_id)ÚselfrÚactive_resources©r<úR/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/autotuning/autotuner.pyÚ__init__/s`      ÿ   ÿ   ÿþ zAutotuner.__init__cCsð| ¡}g}|r| ¡D]J\}}|sq g}| |¡d}|tkr8d}| ¡D]\}} |tkr4|| d7}q&|}n|d}| |¡| |d¡| |dd¡| |¡q t|gd¢dd} t| ƒttj  |j d¡d t d } |   | ¡|   ¡t | ¡Wd ƒn1sŠwYt|vrö|t\} } }| r¬t | d›d | d ›d¡nt d¡tjt ¡|jd}t d|›¡ttj  |j d¡dƒ}|  d|›d|jjd›d¡|  ¡Wd ƒd S1sïwYd Sd S)z8Print the autotuning results in tabular format. rrrÚname)Ú tuning_spaceÚnum_experimentsÚbest_metric_valÚ best_exp_nameÚpipe)ÚheadersÚtablefmtz summary.txtÚw©Ú bufferingNz9 is the optimal setup after tuning. The exp result is at Ú result_dirÚ.zONo optimal setup is found. Please check that experiments were run successfully.)ÚsecondszTuning completed in Úaz Tuning completed in z. Total number of experiments: )Úget_best_space_recordsÚitemsÚappendÚGLOBAL_TUNING_SPACErÚprintÚopenr$r%Újoinr-ÚBUFSIZEÚwriteÚflushÚfsyncrr*ÚdatetimeÚ timedeltaÚtimeÚ start_timer/Úexperiment_count)r:Úbest_space_recordsÚtabÚkeyÚvalÚrowÚnum_expsÚcntÚkÚvÚsummaryÚfdÚbest_exprBÚtotal_num_expsÚtuning_durationÚfr<r<r=Úprint_tuning_resultsls^  €  þ  ýÿ ÿ "üõzAutotuner.print_tuning_resultscCsÀd}d|vr| d¡}d||dvsJdƒ‚||d}nd|vr5| d¡}d||dvr5||d}t d|›¡|dur^tj |¡sNJd |¡ƒ‚tj |¡r^tj t |d ƒt d SdS) züGet DeepSpeed configuration from the user arguments passed to the launcher. Args: user_args ([list]): user arguments passed to the DeepSpeed launcher Returns: [dict]: DeepSpeed configuration dictionary Nz--deepspeed_configú.jsonrzNDeepSpeed --deepspeed_config requires a json file to specify the configurationz --deepspeedzuser_config_file = z8DeepSpeed configuration file: {} is not an existing fileÚr)Úobject_pairs_hook) Úindexrrr$r%ÚisfileÚformatr&ÚjsonÚloadrSr)r:rÚuser_config_fileÚidxr<r<r=ržs*  ÿÿ  ÿ zAutotuner._get_user_configcCspt d|›¡g}d}| ¡D]\}}| |¡tt|ƒ|ƒ}q|dks)Jdƒ‚t|j|||j|j |j j dS)a„Initialize and return a resource manager Args: active_resources ([dict]): A dictionary of hostname and its slots (GPUs), e.g. {"worker-0": "0,1,2,3,4,5,6,7,8"} Raises: RuntimeError: raises the error if no GPU is available Returns: [ResourceManager]: A resource manager that schedules and runs autotuning experiments. zactive_resources = édrzno gpu is available)rÚhostsr3r-r"Ú arg_mappings) rr*rOrPÚminr4r rr-r"rrz)r:r;ryÚngpus_per_nodeÚhostnameÚslotsr<r<r=r.¼s  ûzAutotuner._get_resource_managercCsF|jdkr |j}nt|jjƒ}|jdkr|j}||fS|jj}||fS)zïGet resource requirement for each autotuning experiment Args: args (dict): user args Returns: num_nodes, num_gpus: the number of gpus and number of nodes used in the autotuning experiments r)Ú num_nodesr4r/r5Únum_gpusr3)r:rrr€r<r<r=r0Ùs   þzAutotuner._get_exp_resourcescCó|jjS©N)rÚmetric©r:r<r<r=rƒîózAutotuner.metriccCrr‚)rÚfastr„r<r<r=Ú fast_enabledñr…zAutotuner.fast_enabledcCrr‚)rÚmax_train_batch_sizer„r<r<r=rˆôr…zAutotuner.max_train_batch_sizecCrr‚)rÚmp_sizer„r<r<r=r‰÷r…zAutotuner.mp_sizecCsF| ¡r| ¡dkr| ¡| ¡|j|j}t|jj|ƒS|jjS©Nr)rˆr‰r2r1r{rÚ"max_train_micro_batch_size_per_gpu)r:Úmax_train_micro_batch_sizer<r<r=r‹úsÿ ÿz,Autotuner.max_train_micro_batch_size_per_gpucCrr‚)rÚ"min_train_micro_batch_size_per_gpur„r<r<r=rr…z,Autotuner.min_train_micro_batch_size_per_gpucCrr‚)rÚnum_tuning_micro_batch_sizesr„r<r<r=rŽr…z&Autotuner.num_tuning_micro_batch_sizescCs$t|j ¡vr|jt tt¡SdS)NF)ÚFP16rr ÚgetÚ FP16_ENABLEDÚFP16_ENABLED_DEFAULTr„r<r<r=Ú fp16_enabled szAutotuner.fp16_enabledcCs tƒ ¡Sr‚)rÚ total_memoryr„r<r<r=Úget_gpu_memory_infos zAutotuner.get_gpu_memory_infocCó"|jr d|jvr|jdSdSdS)NÚactivation_mem_per_gpu©Ú model_infor„r<r<r=Úget_activation_memory_per_gpuó ÿz'Autotuner.get_activation_memory_per_gpuc Cs¢| ¡}|j|j}| ¡}|sdS||rdnd}||rdnd}||r'dnd}|tjkr3||}|tjkr<||}|tjkrE||}|||| ¡}|S)Nrrééé) Úget_model_num_paramsr1r2r“rÚoptimizer_statesÚ gradientsÚweightsr‰) r:Ú zero_stageÚ num_paramsÚ total_gpusr“Ú params_memÚ gradients_memÚ optimizer_memÚ mem_per_gpur<r<r=Ú)get_instantiation_memory_required_per_gpus     z3Autotuner.get_instantiation_memory_required_per_gpucCs\g}| ti¡}| tt¡}i}|dkr!t}t t|dƒ¡}d}n\|dkr2t}t t|dƒ¡}d}nK|dkrCt }t t|dƒ¡}d}n:|dkr{t }t t|dƒ¡}|j } | rxd | vrx| d } | | |td <d | | |td <d | |td<d}n|St ||j ttgƒt dt |¡›¡t|dgd} t|ƒ} t d| ›¡t dt| ƒ›¡t| ƒ} t dt| ƒ›¡| D]m}t |¡}t ||ƒ| td¡}|rðt|vrát|tvrá|tt=t|vrðt|tvrð|tt=|t}||}||t<|||j|j| ¡|t<i}t || |ƒ}||d<||t!<|j|d<|j|d<| "|¡q¾|S)a´Generates a list of autotuning experiments given a tuning_space. The corresponding parameter values are replaced by user-defined values in the DeepSpeed configuration file. Args: tuning_space ([dict]): A DeepSpeed configuration dictionary where a value can be a list (called a tuning parameter). For example, { "zero_optimization": { "stage": 1, "reduce_bucket_size": [5e7, 5e8, 1e9], "allgather_bucket_size": [5e7, 5e8, 1e9], } } reduce_bucket_size and allgather_bucket_size are the tuning parameters in this tuning space. Returns: [list]: a list of experiments generated by taking combinations of values of the tuning space. The above tuning space generates 3*3 = 9 experiments if the user DeepSpeed configuration file does not overwrite the two tuning parameters or define more tuning parameters. rroÚz0_rÚz1_rÚz2_éÚ hidden_sizeÚreduce_bucket_sizegÍÌÌÌÌÌì?Ústage3_prefetch_bucket_sizeé Ú"stage3_param_persistence_thresholdÚz3_ztuning_space = Ú optimizer)Ú ignore_keysztuning_keys = zbefore pruning total configs = zafter pruning total configs = Nr?r€r)#rrÚZERO_OPTIMIZATION_STAGEÚZERO_OPTIMIZATION_STAGE_DEFAULTÚDEFAULT_TEMPLATE_PATH_ZERO_0ÚhjsonrurSÚDEFAULT_TEMPLATE_PATH_ZERO_1ÚDEFAULT_TEMPLATE_PATH_ZERO_2ÚDEFAULT_TEMPLATE_PATH_ZERO_3r™Ú replace_dictrÚTRAIN_MICRO_BATCH_SIZE_PER_GPUrrrtÚdumpsÚget_all_configsÚget_tuning_keysr4Ú prune_configsÚcopyÚdeepcopyÚOFFLOAD_OPTIMIZERÚ OFFLOAD_PARAMÚGRADIENT_ACCUMULATION_STEPSr2r1r‰ÚTRAIN_BATCH_SIZEÚcanonical_nameÚ DS_CONFIGrP)r:r@Úmax_train_batch_size_per_gpuÚexpsÚ config_zerorÚtemplate_configÚ template_pathÚprefixr™ÚhsÚ all_configsÚ tuning_keysÚ pruned_listÚconfigÚ exp_configÚmbsÚgasÚexpÚexp_namer<r<r=Ú_generate_experiments0s|        ÿÿÿ    zAutotuner._generate_experimentsc Csötrtjd|_tj|jdt ¡|_| ¡rt   d¡|  ¡}|r'||_ ndSt   dt | ¡ƒ›d¡| ¡|_t   dt|jdd ›d ¡| ¡|_t   d t|jdd ›d ¡|j ti¡ td ¡}t|tƒso|gn|}t   d|›d ¡d }d }d }| tj¡|j}|j|krÄd|vs—tj|vrÃt   dt|dd ›d¡| t¡\}} } | |kr¶| }|}| }trÃt d|  ¡›| ¡nt   dtj›dt|dd ›d¡| tj!¡|j}|j|kr!d|vsîtj!|vr t   dt|dd ›d¡|jt"|||d\}} } | |kr| }|}| }tr t d|  ¡›| ¡nt   dtj!›dt|dd ›d¡| tj#¡|j}|j|krd|vsLtj#|vr~t   dt|dd ›d¡|jt$|||d\}} } | |krp| }|}| }tr~t d|  ¡›| ¡nt   dtj#›dt|dd ›d¡| tj%¡|j}|j|krÒd|vsªtj%|vrÑt   dt|dd ›d¡|jt&|||d\} } } trÑt d|  ¡›| ¡nt   d| ¡›d t|dd ›d!tj%›d"t|jƒ›d# ¡dStrùt '¡dSdS)$z¢ Tunes Zero stages, micro batch size per GPU, and other Zero configurations. Performance metrics of different tuning spaces are recorded in self.records. Ú MLFLOW_RUN_ID)Úrun_idz3Fast mode is enabled. Tuning micro batch size only.NzThe model has z parameters.z Memory per GPU in the system is ÚB)ÚpostfixrKzThe model requires at least z* activation memory for micro batch size 1.rzUser-defined zero stages are Úallz@The model might be runable with ZERO 0 (which requires at least zT memory with mbs = 1), adding DEFAULT_TUNING_SPACE_ZERO_0 to the global tuning spaceÚz0z)The model is not runable with ZERO stage z (which requires at least z memory with mbs = 1)z@The model might be runable with ZERO 1 (which requires at least zG memory), adding DEFAULT_TUNING_SPACE_ZERO_1 to the global tuning space)Ú prev_max_mbsÚ prev_best_mbsÚprev_best_metric_valÚz1z@The model might be runable with ZERO 2 (which requires at least zG memory), adding DEFAULT_TUNING_SPACE_ZERO_2 to the global tuning spaceÚz2z@The model might be runable with ZERO 3 (which requires at least zG memory), adding DEFAULT_TUNING_SPACE_ZERO_3 to the global tuning spaceÚz3z" parameters and requires at least z* memory per GPU with DeepSpeed Zero stage z+ optimization. Memory per GPU in system is z. No tuning is performed.)(Ú has_mlflowr$Úenvironr9ÚmlflowÚ start_runr[r\r‡rr*Úmodel_info_profile_runr™Únumber_to_stringrŸr•Úgpu_memÚmemory_to_stringršÚactivation_memrrrr·Ú isinstanceÚlistrªrÚdisabledÚ tune_spaceÚDEFAULT_TUNING_SPACE_ZERO_0Ú log_metricrƒr ÚDEFAULT_TUNING_SPACE_ZERO_1r¡ÚDEFAULT_TUNING_SPACE_ZERO_2r¢ÚDEFAULT_TUNING_SPACE_ZERO_3Úend_run) r:r™rÚuser_zero_stagesrØÚmax_mbsÚ metric_valÚrequired_gpu_memÚ next_max_mbsÚnext_mbsÚnext_metric_valÚ_r<r<r=Útune”sÞ     ÿ ÿ€ÿÿÿ ÿ ý €ÿÿÿ ÿ ý €ÿ ÿ ý€0ÿ ÿzAutotuner.tunercCs–| ti¡}| td¡}tt|ƒ}g}d} d} t|j| |¡ƒ|j} t   d|›d| ›¡| |krAt   d|›d|›¡dSt |j vrkt |j t tƒrkdd „|j t Dƒ}| ¡} t|ƒ} t|ƒ}|| } d } n&| ||| ¡\} }||kr…t   d|›d|›¡dS|j| || ¡d \}} t   d |›d | ›¡|s¨t   d|›¡dS| || | || ¡}| |¡}|r¼|dnd}|rÈ|dtt nd}t   d|›d|dd›¡| ¡sá|dkrît   d|›¡|||fS|dkr ||ksü||kr t   d|›d¡|||fS||t <t|t|ƒdt|ƒdd d}t   d|›¡t   d|›¡| || ¡}t   d|jj›¡|jjtkrPt||j | !¡|ƒ}n|jjt"krat#||j | !¡ƒ}n t$||j | !¡ƒ}t%|j j&ƒ|j j'|j(|j)}|j*||jj+|jj,d}|j-}|j.}|r—| /||||¡| |¡}|r£|dnd}||krº|}|r·|dtt nd}n|}|}t   d|›¡|||fS)NrFzStart tuning for space z$, calculated_max_micro_batch_size = zNo need to tune Zero stage z. End tuning for space ©rrrcSsg|] }t|tƒr|‘qSr<)ròÚint)Ú.0Úsr<r<r=Ú "s  ÿÿz(Autotuner.tune_space..T)rŽztuning_micro_batch_sizes = z!, max_train_batch_size_per_gpu = zEnd tuning for space rzfast_best_mbs = z , name = r?zEnd tuning for space: z6. No need to tune other Zero configuration parameters.Úzr)rÔrÑÚomit_valzTuning space is zTuning space name is zTuner type is )Ú sample_sizeÚn_trialsÚearly_stoppingr)0rrr·ÚTUNING_MICRO_BATCH_SIZE_PREFIXÚstrrrïrªrñrr*r¿rròróÚget_gas_from_user_configr{ÚmaxÚget_min_max_micro_batch_sizeÚ get_tuning_micro_batch_size_listrŽÚrun_tuning_micro_batch_sizesÚget_best_space_recordrËr‡rÊrÂrÜrÚ tuner_typeÚAUTOTUNING_TUNER_MODELBASEDr r/rƒÚAUTOTUNING_TUNER_RANDOMr r r4r5r3r2r1rÚtuner_num_trialsÚtuner_early_stoppingrirBÚupdate_records)r:r@rãrärårÎrÚtuning_space_nameÚtuning_micro_batch_sizesrÌÚ$tuning_micro_batch_sizes_overwrittenÚcalculated_max_micro_batch_sizerÙÚmin_micro_batch_sizeÚmax_micro_batch_sizeÚfast_best_recordÚfast_best_metric_valÚ fast_best_mbsrÍÚtr rcrÚrþÚfull_best_recordÚfull_best_metric_valrBÚbest_mbsr<r<r=rõ sÀ   ÿÿÿ  ÿÿÿ ýÿý    ÿ ý  þ   zAutotuner.tune_spacec Cs||jvrdS|j|}t|dd„d}d}d}|D])\}}}|r;||kr*|} | S||kr;|||tkr;|} | S|}|tt}q|} | S)NrcSs|dttSrŠ)rËr¿)Úxr<r<r=Ú„sz+Autotuner.get_plateau_mbs..©r`)r6ÚsortedÚMETRIC_PERCENT_DIFF_CONSTrËr¿) r:rÚ space_recordsÚsorted_space_recordsÚprev_metric_valÚprev_micro_batch_sizerÚrþrÚ plateau_mbsr<r<r=Úget_plateau_mbs€s*  úýzAutotuner.get_plateau_mbscCr–)Nr¤r˜r„r<r<r=rŸ“r›zAutotuner.get_model_num_paramsc Csšt d¡|jj}|rt|vr|St |j¡}t|t ƒt j   |j dd¡}d|ddidœ|t<i}d}||d<||t<|j|d<|j|d <|jj|d <t j   |j|›d ¡}t|d td }t ||¡| ¡t  |¡Wdƒn1swwY|j |g¡|j ¡|jj ¡D]\}\} } |j  ¡| r¦t !d| ›¡dSqŽt j  "|¡rËt|dƒ} t# $| ¡}|WdƒS1sÄwYdSdS)a‹Does a model information profiling experiment that collects the number of model parameters and activation memory. The experiment produces a "profile_model_info" folder under self.results_dir. Returns: [dict]: a model information dictionary, e.g., {"num_params": 335144976, "trainable_num_params": 335144976, "activation_mem_per_gpu": 324358144, "rank": 0} z Starting model info profile run.Úprofile_model_infozmodel_info.jsonTÚprofile)ÚenabledÚmodel_info_pathr™r?r€rÚhostfilernrGrHNz6The model is not runnable with DeepSpeed with error = ro)%rr*rr™ÚMODEL_INFO_NUM_PARAMSrÄrÅrr¾ÚDEFAULT_MIN_MEM_CONFIGr$r%rTr-rrËr2r1rr9r"rSrUrtÚdumprWrXr/Úschedule_experimentsÚrunÚfinished_experimentsrOÚclearr+r&rºru) r:r™Ú ds_configr8r×rÛÚexp_pathrhÚexp_idÚexp_jsonÚerrrlr<r<r=rí—sF         ý  þ   $þÿz Autotuner.model_info_profile_runcCs:||jvr|||fg|j|<dS|j| |||f¡dSr‚)r6rP)r:Ú space_namerÚrþrcr<r<r=rÄs zAutotuner.update_recordscCsf||jvrdS|j|}d}d}|D]\}}}||7}|dus%||dkr)||f}q|r1||f}|S)Nrr)r6)r:rFr/Úbest_space_recordÚspace_num_expsrÚrþrcr<r<r=rÊs  € zAutotuner.get_best_space_recordcCsZi}d}|j ¡D]\}}| |¡}|r$|||<|r"|d|dkr$|}q |r+||t<|S©Nr)r6rOrrQ)r:r^Úglobal_best_recordrFr/rGr<r<r=rNØs €z Autotuner.get_best_space_recordsc CsB|sJdƒ‚| ¡|d}d}t|jƒ}t|i|t<tt|ƒ} g} |D]|} | |t<|| } | |t<| | |j |j |  ¡|t <| dt| ƒdt| ƒ} i}| |d<||t <|j |d<|j |d<|jj|d <tj |j| ›d ¡}t|d td }t ||¡| ¡t |¡Wdƒn1s—wY|  |¡q%|j | ¡|j ¡|jj ¡D]Ž\}\}}|r2|t t t!}tj "|¡r)t|d ƒO}t# $|¡}|| %¡}| &| ||d¡||t tkrî|}t'rtj( )d¡t*j+d|dd|D] }t* ,|||¡qt* -¡|j.tj(d<Wdƒn 1s#wYq³| &| |dd¡q³|t t} t/ 0d| ›d¡q³|j 1¡|rL|St2|ƒdkrY|ddn|}|}|}||d}|dkrld}t3t4|||ƒƒD]¡} | |t<|| } | |t<| | |j |j |  ¡|t <| dt| ƒdt| ƒ} | 5|| ¡\}}|r t|d ƒ;}t# $|¡}|| %¡}t'rätj( )d¡t*j+d| d|D] }t* ,|||¡qÎt* -¡|j.tj(d<Wdƒn 1sïwY| &| ||d¡||dt6kr |}| }qtn | &| |dd¡||kr||d<|S)Nz)the tuning micro batch size list is emptyrrÚ_gasÚ_tmbspgr?r€rr9rnrGrHrorrÝT)ÚnestedÚrun_namezmicro batch size = z was not run successfullyéþÿÿÿr®)7ÚsortÚget_first_configrr·rrrr¿rÈr2r1r‰rÉrËrr9r$r%rTr"rSrUrtr<rWrXrPr/r=r>r?rOrÚAUTOTUNING_METRIC_PATHr&rºrurƒrrérêÚpoprërìr÷rûr9rr*r@r4ÚreversedÚrangeÚ run_ds_configr.)r:rrÌr!rrr"Úmax_micro_batch_size_metric_valrArÚ exp_pathsrØrÙrÛr×rBrhrCrÚrEÚ metric_filerlÚresultsrþrƒÚ"min_micro_batch_size_with_same_gasråräÚstrider<r<r=råsØ    ÿÿÿ     ý        €ô€  ÿÿÿ  ÿÿÿ     €÷  z&Autotuner.run_tuning_micro_batch_sizescs(ˆ|krdSg}tt|ƒ}t|jƒ}t|i|t<| ¡}||t<ˆdkrt|jvr;t |jtt ƒr;t |jtƒ}n|  t¡} | rGt | ƒ}nd}|dksQJdƒ‚||t<||t<|||j |j | ¡|t<|dt|ƒdt|ƒ} | || ¡\} } | r‘| || | d¡| |¡|‰n¹| || dd¡t d|›d¡| ¡|krªdS| ¡}||t<||t<|||j |j | ¡|t<|dt|ƒdt|ƒ} | || ¡\} } | sñ| || dd¡t d |›d ¡dS| || | d¡|‰| |¡nIˆ|t<||t<ˆ||j |j | ¡|t<|dt|ƒdtˆƒ} | || ¡\} } | r@| || | d¡| ˆ¡n | || dd¡dSt|| ¡ƒ} t d | ¡| t d | ƒfD]h}|| ¡krjq_||vruˆ|fS||t<|||j |j | ¡|t<|dt|ƒdt|ƒ} | || ¡\} } | r¾t d |›d¡| || | d¡| |¡ˆ|fS| || dd¡q_||jvrÓ|j|ng‰ˆrîtttˆƒƒ‡‡fdd„d}ˆ|d}nd}ˆ}| }||kr€t ||dƒ}t d|›d|›d|›¡||vrw||t<|||j |j | ¡|t<|dt|ƒdt|ƒ} | || ¡\} } | rj|d}| || | d¡| |¡|rg| ||tkrgt d|›¡n| }n| || dd¡|d}n|d}||ksù|d} t dˆ›d| ›d¡ˆ| fS)N)rrrrz4The micro batch size per GPU must be greater than 0.rKrLz(User-specified micro batch size per GPU z does not runz#min_train_micro_batch_size_per_gpu z is not runnable.gÍÌÌÌÌÌð?gffffffî?zmbs = z is found as max mbscstˆ|dttˆƒSrŠ)ÚabsrËr¿)Úi©r!r/r<r=r+±sÿz8Autotuner.get_min_max_micro_batch_size..r,rz trying mbs = z, low = z , high = zperformance plateaus at mbs = úmin_micro_batch_size = ú, max_micro_batch_size = rK)rrrQrr·rrrÈr¿ròrÚget_val_from_user_argsr2r1r‰rÉrVrrPrr*rr{r‹ÚmathÚceilr6rUr4rr.)r:rr!r Úused_micro_batch_sizesrrArÙrØrarÛrÚrþr"Úprev_idxr1ÚlowÚhighÚmidr<r_r=rSs      ÿ  ÿÿÿ  ÿÿÿ ÿÿÿ    ÿÿÿ    ÿ  ÿÿÿ  ÿÿ  êz&Autotuner.get_min_max_micro_batch_sizecCsxd}t|jvr2|jt}t|tƒr|}n|dkr$| t¡}|r#t|ƒ}nt|tƒr2t dt›d¡|dks:Jdƒ‚|S)NrÚautozSpecifying a list of z+ to tune is not supported. 1 would be used.rz-Gradient accumulation steps must be positive.)rÈrròrrbrórr*)r:rÙÚ gas_in_configrar<r<r=r×s     €  ÿz"Autotuner.get_gas_from_user_configcCsV|jj}|jj}|r)||vr)||}||vr)| |¡}||d ¡r)||dSdSrI)rrzrrrqÚ isnumeric)r:Úds_namerzrÚarg_namerwr<r<r=rbçs   z Autotuner.get_val_from_user_argsc Csú|dks|dkrt d|›d|›¡gdfS| ¡r0| ¡dkr0| ¡| ¡|j|j}n | ¡}||| ¡}t d|›¡||dkrN|d}|||}|dkrZd}g}||}t|||ƒD] } || |krs| | ¡qf| |¡||fS)a®Get a list of micro batch sizes to tune based on min and max values, as well as the size of the list. Args: min_micro_batch_size ([int]): min micro batch size per GPU max_micro_batch_size ([int]): max micro batch size per GPU num_tuning_micro_batch_sizes (int): the number of items in the returned list Returns: [list]: a list of micro batch sizes to tune. rr`razmax_train_batch_size_per_gpu = rr) rr*rˆr‰r2r1rrUrP) r:r!r"rŽrÌrÙr\ÚlsÚmin_gasrØr<r<r=ròs6 ÿÿÿ    € z*Autotuner.get_tuning_micro_batch_size_listcCsäi}||d<||t<|j|d<|j|d<|jj|d<tj |j|›d¡}t   d|›¡t |dt d}t  ||¡| ¡t |¡Wdƒn1sNwY|j |g¡|j ¡|j | ¡¡\}}|j ¡||fS) Nr?r€rr9rnzrun_ds_config exp_name = rGrH)rËr2r1rr9r$r%rTr"rrrSrUrtr<rWrXr/r=r>Ú parse_resultsrƒr@)r:rArÛr×rBrhrÚrþr<r<r=rVs$     ý  zAutotuner.run_ds_configc CsH| ¡}t|vr dS|t\}}}|r¢|d}d}ttj |d¡dƒ}dd„| ¡ ¡Dƒ}Wdƒn1s9wYt  ttj |d¡dƒ¡}|  t ¡tj |j d¡} t  |t| dƒ¡tj |j d ¡} t| dƒ} |  d  |¡¡|  d ¡|  ¡Wdƒn1sŠwY||_||_t d | ›d | ›¡dSdS)NrJzcmd.txtrocSsg|]}t|ƒ‘qSr<)r)rr^r<r<r=r <sz2Autotuner.write_optimal_config..zds_config.jsonzds_config_optimal.jsonrGzcmd_optimal.txtú Ú zAWrote the optimal DeepSpeed configuration found by autotuning to z-, and the corresponding DeepSpeed command to )rNrQrSr$r%rTÚreadÚsplitrºrurSrr-rtr<rVrWr7r8rr*) r:r^rirBrÚexp_dirÚcmdrlrAÚds_config_pathÚcmd_pathrhr<r<r=Úwrite_optimal_config3s4ÿ    ýÿízAutotuner.write_optimal_configcCs>|jrt |j¡}| ¡t d|j›¡dSt d¡dS)a Launches the training with the optimal DeepSpeed configuration found through the autotuning process. "ds_config_optimal.json" describing the optimal DeepSpeed configuration as well the command used to launch training "cmd_optimal.txt" are saved to self.results_dir. zrmrr.r0rƒr‡rˆr‰r‹rrŽr“r•ršrªrÜrrõr4rŸrírrrNrrrrbrrVrzrr<r<r<r=r*sH=2 d wu- n , r)%r'r{r[rYrcrºÚruntime.config_utilsrÚruntime.constantsÚruntime.zero.configrrÚutilsrrÖr Ú constantsÚ schedulerr Útunerr r r Údeepspeed.acceleratorrrÚ ImportErrorrëréÚ ExceptionÚer·rÆrÇrôr¸rr<r<r<r=ÚsB      ÿ€ÿ