o TiC@sddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z ddl mZmZmZmZmZmZddlmZddlmZgdZd d Zd d Zd dZ   d ddZ     d!ddZdefddZde fddZ!de fddZ"d"de defddZ#dS)#N)version)ElasticityConfigElasticityConfigErrorElasticityErrorElasticityIncompatibleWorldSize) ELASTICITYENABLEDENABLED_DEFAULTLATEST_ELASTICITY_VERSIONMINIMUM_DEEPSPEED_VERSIONDEEPSPEED_ELASTICITY_CONFIG)logger)&rr $0<xihiiHiii iii`'i;iNipbiHli0iiiDi ii@bi:i`iiui& iP cCstg}|D]%}||kr||q||}ttt|k}|t|d|qtt|}td||S)NrzCandidate batch size: ) appendnpargmaxasarrayHCN_LISTlistsetrinfo) base_listmax_acceptable_batch_sizecandidate_batch_sizebasevalueindexr(S/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/elasticity/elasticity.pyget_candidate_batch_sizess  r*cCsg}|D]?}||dkrC||}||kr|kr!nn||td|ddD]}||kr2n||kr7q*||dkrB||q*qt|}tt|}|S)Nrrr)rranger sortedr) batch_size micro_batchesmin_valid_gpusmax_valid_gpus valid_gpus micro_batchmax_gpusir(r(r)get_valid_gpus)s"     r5c Cstd}d}tt|}|D])}t||||} t| |ks-t| |kr5|r'||ks-|s5||kr5t| }| }|}q ||fSNr)intminr5len) candidate_batch_sizesr.min_gpusr3 prefer_largerr0r1final_batch_sizer-current_valid_gpusr(r(r)get_best_candidates?s" r?Tc s|pd}|p t|}tfdd|Dstdtj|}g}||||t|}t |||||\}} || fS)a*We use two heuristics to compute the batch size 1. We use the Lowest Common Multiple of the micro-batches as the base batch size and scale it by a HCN such that the result is the largest batch size less than the max_acceptable batch size 2. We use each of the micro batches as a base and scale it by a HCN such that the result is the largest batch size less than the max_acceptable batch size. We then use brute force to count the number of compatible GPU count for each of the aforementioned cases, and return the batch size with the most number of compatible GPU counts in the min-max GPU range if provided, other wise we return the batch size with the most number of total compatible GPU counts. Returns: final_batch_size valid_gpus rc3s|]}|kVqdS)Nr().0mbr#r(r) lsz+_get_compatible_gpus_v01..zWAll micro batches must be less than or equal to max_acceptable_batch_size: ) r8all ValueErrorrlcmreduceextendrr*r?) r.r#r;r3r<rFr"r:r=r1r(rBr)_get_compatible_gpus_v01Ss     rIcs||dkrtd|d|fdd}||tt|t||t||d\} } t| } fdd| D} || vrQ|| } | | | fS|} g}D]}|| }t|t|}|||q[d }r{t|}nt|}||} |t| g| fS) zW Returns: final_batch_size valid_gpus micro-batch size rz,In Elasticity v0.2, number of GPUs per node:z, should be divisible by model parallel size cs>d}D]}||dkr|dur|}r||kr|}q|Sr6r()r=candidate_microbatchr2)current_num_gpusr.r<r(r)get_microbatchs z0_get_compatible_gpus_v02..get_microbatch)r<csg|]}|qSr(r()r@r4)dp_size_per_noder(r) sz,_get_compatible_gpus_v02..N) rrIr7mathfloorfloatrmaxr8)r.r#rKr;r3r<num_gpus_per_nodemodel_parallel_sizerLr=valid_world_sizevalid_dp_world_sizerJcurrent_dp_sizer:r2min_batch_sizefactorused_microbatchr$r()rKrMr.r<r)_get_compatible_gpus_v02~sD          r[target_deepspeed_versioncCs:tt}t|}d|dtd}||krt|dS)NzTarget deepspeed version of z( is not compatible with minimum version z supporting elasticity.T) pkg_versionparser r)r\ min_versiontarget_versionerr_strr(r(r)_compatible_ds_version_checks  rb ds_configcCst|vrdS|tttS)NF)rgetr r )rcr(r(r)elasticity_enabledsreruntime_elastic_config_dictcCsttjvrOttjt}t|}t|}d}|j|jkr)t|d|jd|j|j |j kr;t|d|j d|j |j |j krMt|d|j d|j dSt ddS)z[ Ensure the resource scheduler saw the same elastic config we are using at runtime z_Elastic config '{}={}' seen by resource scheduler does not match config passed to runtime {}={}r#r.rzUnable to find DEEPSPEED_ELASTICITY_CONFIG environment variable, cannot guarantee resource scheduler will scale this job using compatible GPU counts.N) r osenvironjsonloadsrr#rformatr.rrwarning)rfscheduler_elastic_config_dictscheduler_elastic_configruntime_elastic_configrar(r(r)ensure_immutable_elastic_configs.      rpFc Cst|tstdt|d|t|vrtdtd|t}|tts+tdt |}|j }|j }|dkrKt |j dkrKtd|j d |t |j tkr]td |j d tt|sktd |d tt |j dkrt|j|j|j|j|jd\}} t|}nOt |j dkr|dkr|} n dtjvrtdrttd} n td} td| t|j|j| |j|j|j||d\}} } t|}ntd|j t d| |dkr'|| vrt!d|d| d} t"t#t$|jddD]}|||dkr |} nq| dus"Jd|d|d|jd|| | fS|rnt |j dkr7|| | fSd} t"t#t$|jddD]}|||dkrS|} nqD| dusiJd|d|d|jd|| | fS|| fS)aCore deepspeed elasticity API. Given an elastic config (similar to the example below) DeepSpeed will compute a total train batch size corresponding valid GPU count list that provides a high level of elasticity. Elasticity in this case means we are safe to scale the training job up/down across the GPU count list *without* any negative impacts on training convergence. This is achievable primarily due to DeepSpeed's gradient accumulation feature which allows us to decompose a global training batch size into: micro-batch-size * gradient-accumulation-steps * world-size. "elasticity": { "enabled": true, "max_train_batch_size": 2000, "micro_batch_sizes": [2,4,6], "min_gpus": 1, "max_gpus" : 10000 "min_time": 20 "version": 0.1 } Intended to be called both by scheduling infrastructure and deepspeed runtime. For the same `ds_config` we should return deterministic results. Args: ds_config (dict): DeepSpeed config dictionary/json target_deepspeed_version (str): When called from scheduling infrastructure we want to ensure that the target deepspeed version is compatible with the elasticity version used in the backend. world_size (int, optional): Intended/current DP world size, will do some sanity checks to ensure world size is actually valid with the config. return_microbatch (bool, optional): whether to return micro batch size or not. Raises: ElasticityConfigError: Missing required elasticity config or elasticity disabled ElasticityError: If target deepspeed version is not compatible with current version Returns: final_batch_size (int): total batch size used for training valid_gpus (list(int)): list of valid GPU counts with this config micro_batch_size (int, optional): if world_size is provided will return specific micro batch size z5Expected ds_config to be a dictionary but received a z, containing: 'zP' is missing from config json, please add it if running an elastic training job.z]Elasticity is disabled, please enable it ('enabled':true) if running an elastic training job.rg?z Elasticity VzF does not support model-parallel training. Given model-parallel size: z%Attempting to run elasticity version z! but runtime only supports up to z8Unable to run elasticity on target deepspeed version of z , currently g?)r.r#r;r3r<r WORLD_SIZEzElasticity V 0.2 needs WORLD_SIZE to compute valid batch size. Either give it as argument to function compute_elastic_config or set it as an environment variable. Value of WORLD_SIZE as environment variable is )r.r#rKr;r3r<rSrTz*Unable to find elastic logic for version: z/Valid World Size (GPUs / Model Parallel Size): z World size (z:) is not valid with the current list of valid GPU counts: NT)reversez5Unable to find divisible micro batch size world_size=z, final_batch_size=z, and micro_batches=.)% isinstancedictrEtyperrrdr r rrTrSrQrr rbr __version__rIr.r#r;r3prefer_larger_batch_sizer7rgrhgetenv isnumericr[NotImplementedErrorrr!rr,rr )rcr\ world_sizereturn_microbatchelastic_config_dictelastic_configrTrSr=r1rKrrcandidate_microbatch_sizemicro_batch_sizembszr(r(r)compute_elastic_configs )            r)NNT)NNTrr)rF)$rgrinumpyrrO packagingrr]configrrrr constantsrr r r r r git_version_inforxutilsrrr*r5r?rIr[strrbrvrerprr(r(r(r)s6     . A