o Ti@sdaiaiaddZddZddZddZd d Zd d Zd dZ ddZ ddZ ddZ ddZ ddZdaddZddZdS)NcCs iaiadS)N) module_names param_namesrrI/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/utils/debug.py"debug_clear_module_and_param_namessrcCs(dd|Dadd|DadS)NcSi|]\}}||qSrr).0namemodulerrr z8debug_extract_module_and_param_names..cSrrr)rrparamrrrr r ) named_modulesrnamed_parametersr)modelrrr$debug_extract_module_and_param_namessrcC|tvrt|SdSNunknown)rr rrrdebug_module2namercCdt|d|jSNname= id=)ridrrrrdebug_module2name_id%rcCsdt|d|jjS)Nr )r __class____name__rrrrdebug_module2name_class)sr!cCrr)rr rrrdebug_param2name-rr#cCrr)r#ds_idr"rrrdebug_param2name_id4rr%cCs dt|d|jd|jjS)Nrr shape=)r#r$datashaper"rrrdebug_param2name_id_shape8 r)cC(dt|d|jd|jjd|jS)Nrrr&z device=)r#r$r'r(devicer"rrr debug_param2name_id_shape_device<(r-cCs dt|d|jd|S)Nrrz numel=)r#r$numelr"rrrdebug_param2name_id_numel@r*r0cCr+)Nrrr&z status=)r#r$r'r( ds_statusr"rrr debug_param2name_id_shape_statusDr.r2c Gsztdurddlattd&}t|tjz t|Wt|tjnt|tjwWddS1s6wYdS)a For printing messages for all concurrent gpus w/o getting interleaved text. This is useful when debugging issues where multi-gpus don't sync. 1. Enable the force debug in say partitioning and zero3 files 2. Override the usual versions with :: def print_rank_0(message, debug=False, force=False): rank = deepspeed.comm.get_rank() printflock(f"[{rank}] {message}") 3. run the program and you get both logs non-interleaved But this makes it very difficult to make sense of the output, so the ``log_rank_file`` helper function might be more useful, as it's easier to send each log stream into a separate file and then compare those. Nr)fcntlopen__file__flockLOCK_EXprintLOCK_UN)msgsfhrrr printflockHs  ""r>cGs@tdur td|dda|D] }t|dqtdS)a+ Print to a log file of the given rank This is useful for debugging hanging in sync processes. Here is a possible workflow: 1. Enable the force debug in say partitioning and zero3 files 2. Override the usual versions of print_rank_0 in those files with :: def print_rank_0(message, debug=False, force=False): rank = deepspeed.comm.get_rank() log_rank_file(rank, message) 3. run the program 4. fix up the expected differences, e.g. different cuda numbers :: perl -pi -e 's|cuda:1|cuda:0|' log_rank_* 5. now diff and see where names and ids diverge - you will find where the gpus don't do the same work (e.g. when some layers get conditionally skipped on one gpu but not all) diff -u log_rank_0.txt log_rank_1.txt | less N log_rank_z.txtw )r=r6writeflush)rankr<mrrr log_rank_fileks  rFcs(fddt|dr|jdSdS)Nc std||jD]B}|drLz#t|dd}t|dtdt|d|jd|d|jWq tyK}z |dWYd}~q d}~wwq dS)NzBackward tensors in r3variablez Tensor - id: z , shape: z, data: z, grad: )r:next_functionsgetattrrr(gradAttributeError)grad_fnfuncstensore_print_bwd_tensorsrrrQs  ,z2print_backward_tensors.._print_bwd_tensorsrL)hasattrrL)rNrrPrprint_backward_tensorss  rS)r5rrrrrrr!r#r%r)r-r0r2r>r=rFrSrrrrs$