o 8wik@sUddlmZddlmZmZmZmZmZmZddl Z ddl m Z ddl m Z mZer7ddlmZddlmZeeeeeffZGdd d ZGd d d eZ dd e jjd ege jfdeee jge jfdefddZide jde jddde j de j!de j"dide jde jddde j de j!de j"dide jde jd dd!e j d"e j!d"e j"d#id$e jd%dd%e j d%e j!d%e j"d&d'd(id)e jd*dd*e j d*e j!d*e j"d+d'd,id-e jd.dd.e j d.e j!d.e j"d/d'd0id1e jd2dd3e j d4e j!d4e j"d5d'd6id7e jd8dd8e j d9e j!d9e j"d:d'd;iddd?e j d@e j!d@e j"dAidBe jdCddDe j dCe j!dCe j"dEd'dFidGe jdHddIe j dHe j!dHe j"dJd'dKidLe jdMddNe j dOe j!dOe j"dPd'dQidRe jdSddSe j dSe j!dSe j"dTd'dUidVe jdWddWe j dWe j!dWe j"dXd'dYidZe jd[dd[e j d[e j!d[e j"d\d'd]id^e jd_dd_e j d_e j!d_e j"d`d'daidbe jdcddce j dce j!dce j"ddd'deie jdfe j!dge j"dhd'diie jdje j!dkie jdje j!dle j"dmd'dnie jdoe j!dpe j"dqd'drie jdse j!dte j"dud'dvie jdwe j!dxe j"dyd'dzie jd{e j!d|e j"d}d'd~ie jde jde j!dOie jde jde j!die jde jde j!dhid Z#eeeeee j$feffe%d<dddddZ&de j'dee j$efdeefddZ(ddde j$fddZ)ededZ*Gddde+e*Z,dS))deque) TYPE_CHECKINGAnyCallableOptionalTypeVarUnionN)override)rank_zero_onlyrank_zero_warn)Fabric) Precisionc @seZdZdZ ddeededed ed df d d Zddd dedededeedeed df ddZ d e fddZ dddZ dS) ThroughputaComputes throughput. +------------------------+-------------------------------------------------------------------------------------+ | Key | Value | +========================+=====================================================================================+ | batches_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of batches | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | samples_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of samples | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | items_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of items | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | flpps_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of flops | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | device/batches_per_sec | batches_per_sec divided by world size | +--------------------------+-----------------------------------------------------------------------------------+ | device/samples_per_sec | samples_per_sec divided by world size | +--------------------------+-----------------------------------------------------------------------------------+ | device/items_per_sec | items_per_sec divided by world size. This may include padding depending on the data | +--------------------------+-----------------------------------------------------------------------------------+ | device/flops_per_sec | flops_per_sec divided by world size. | +--------------------------+-----------------------------------------------------------------------------------+ | device/mfu | device/flops_per_sec divided by world size. | +--------------------------+-----------------------------------------------------------------------------------+ | time | Total elapsed time | +--------------------------+-----------------------------------------------------------------------------------+ | batches | Total batches seen | +--------------------------+-----------------------------------------------------------------------------------+ | samples | Total samples seen | +--------------------------+-----------------------------------------------------------------------------------+ | lengths | Total items seen | +--------------------------+-----------------------------------------------------------------------------------+ Example:: throughput = Throughput() t0 = time() for i in range(1000): do_work() if torch.cuda.is_available(): torch.cuda.synchronize() # required or else time() won't be correct throughput.update(time=time() - t0, samples=i) if i % 10 == 0: print(throughput.compute()) Notes: - The implementation assumes that devices FLOPs are all the same as it normalizes by the world size and only takes a single ``available_flops`` value. - items_per_sec, flops_per_sec and MFU do not account for padding if present. We suggest using samples_per_sec or batches_per_sec to measure throughput under this circumstance. Args: available_flops: Number of theoretical flops available for a single device. world_size: Number of devices available across hosts. Global metrics are not included if the world size is 1. window_size: Number of batches to use for a rolling average. separator: Key separator to use when creating per-device and global metrics. Nd/available_flops world_size window_size separatorreturncCsj||_||_|dks J||_|dksJt|d|_t|d|_t|d|_t|d|_t|d|_ dS)Nrr)maxlen) rrr_MonotonicWindow_time_batches_samples_lengthsr_flops)selfrrrrrb/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lightning_fabric/utilities/throughput.py__init__^s      zThroughput.__init__)lengthsflopstimebatchessamplesr"r#cCs|j|||krtd|d|d|j||j||durU||kr4td|d|d|j|t|jt|jkrUtdt|jdt|jd|durd|j||j dSdS) adUpdate throughput metrics. Args: time: Total elapsed time in seconds. It should monotonically increase by the iteration time with each call. batches: Total batches seen per device. It should monotonically increase with each call. samples: Total samples seen per device. It should monotonically increase by the batch size with each call. lengths: Total length of the samples seen. It should monotonically increase by the lengths of a batch with each call. flops: Flops elapased per device since last ``update()`` call. You can easily compute this by using :func:`measure_flops` and multiplying it by the number of batches that have been processed. The value might be different in each device if the batch size is not the same. zExpected samples (z') to be greater or equal than batches ()NzExpected lengths (z') to be greater or equal than samples (zIf lengths are passed (z1), there needs to be the same number of samples () rappend ValueErrorrrrlen RuntimeErrorrr)rr$r%r&r"r#rrr updateqs$    zThroughput.updatecCs|jd|jd|jdd}|jr|jd|d<|jdk}t|j|jjkr|jd|jd}|jd|jd}|jd|jd}||}||}|d|jd||d|jd|i|rt||j}||||jd t|j|jjkr|jd|jd} | |} | |d|jd <|r| |j} | |d <t|j |j jkrt |j |j d} |jd|jd}| |} | |j}|r| |d <||d|jd <|j r||j |d|jd <|S) zCompute throughput metrics.)r$r%r&r"rrdevicebatches_per_secsamples_per_sec)r/r0 items_per_sec flops_per_secmfu) rrrrrr*rr,rrsumr)rmetricsadd_global_metrics elapsed_timeelapsed_batcheselapsed_samplesdev_samples_per_secdev_batches_per_secr0elapsed_lengthsdev_items_per_secr1 elapsed_flopsr2dev_flops_per_secrrr computesR    zThroughput.computecCs6|j|j|j|j|jdSN)rclearrrrrrrrr resets    zThroughput.reset)Nrrr)rN) __name__ __module__ __qualname____doc__rfloatintstrr!r,_THROUGHPUT_METRICSr@rDrrrr r s>>  )4rcsJeZdZdZdddeddffdd Zd d eededefd d Z Z S) ThroughputMonitoraComputes throughput. This class will automatically keep a count of the number of log calls (``step``). But that can be modified as desired. For manual logging, using :class:`Throughput` directly might be desired. Example:: logger = ... fabric = Fabric(logger=logger) throughput = ThroughputMonitor(fabric) t0 = time() for i in range(1, 100): do_work() if torch.cuda.is_available(): torch.cuda.synchronize() # required or else time() won't be correct throughput.update(time=time() - t0, batches=i, samples=i) if i % 10 == 0: throughput.compute_and_log(step=i) Args: fabric: The Fabric object. \**kwargs: See available parameters in :class:`Throughput` fabricr kwargsrNc s|t|jj}t|j|}tjd||jd|||_ d|_ t |j |_ t |j id|_ t |jid|_t |j|_dS)N)rrr-)defaultr)_validate_launched_plugin_to_compute_dtypestrategy precisionget_available_flopsr.superr!r_fabricstepr r,r@compute_and_logrD)rrNrOdtyper __class__rr r!s   zThroughputMonitor.__init__rXcKs>|dur |jdn||_|jdi|}|jj||jd|S)zSee :meth:`Throughput.compute` Args: step: Can be used to override the logging step. \**kwargs: See available parameters in :meth:`Throughput.compute` Nr)r5rXr)rXr@rWlog_dict)rrXrOr5rrr rYsz!ThroughputMonitor.compute_and_logrA) rErFrGrHrr!rrJrLrY __classcell__rrr[r rMs$ rMmodel forward_fnloss_fnrcCszddlm}|dd}|$|dur|n||Wd|SWd|S1s4wY|S)a-Utility to compute the total number of FLOPs used by a module during training or during inference. It's recommended to create a meta-device model for this: Example:: with torch.device("meta"): model = MyModel() x = torch.randn(2, 32) model_fwd = lambda: model(x) fwd_flops = measure_flops(model, model_fwd) model_loss = lambda y: y.sum() fwd_and_bwd_flops = measure_flops(model, model_fwd, model_loss) Args: model: The model whose FLOPs should be measured. forward_fn: A function that runs ``forward`` on the model and returns the result. loss_fn: A function that computes the loss given the ``forward_fn`` output. If provided, the loss and `backward` FLOPs will be included in the result. r)FlopCounterModeF)displayN)torch.utils.flop_counterrbbackwardget_total_flops)r_r`rarb flop_counterrrr measure_flops s    rhh100 nvlgwBg$^/lBtfloat32gS CgSCg />2,Ch100 sxmgwBg$^/lBgSBg />2C h100 pciegvHBgvHBg|Bg|Cg@.CCzrtx 4090g$BgbCint4gbCzrtx 4080gm%Bg)Bg)Czrtx 4080 superg:Bg:Bg:Cl4gĎBgWHBg$ Bg$ Bg$ Bl40g˓Bg˓Bg˓Bg˓Ca100gꤡBg2Bg2Bg2Bg2Ca6000g\EBg\EBggBg@$Ca40gxBgxBgbcBgCa10ggP`Bg4&kBg4&kBg4&kBg4&kBz rtx 3090 tig@0Bg@0Bg@0Czrtx 3090gPb0Bgq$Bgq$Cz rtx 3080 tigcBgcBgYBzrtx 3080gjZBg.Bg.Bzrtx 3070gIvvBggH|BggH|Bg}wBgBgBgBg h_BgHBgHBgHBgHBg`cԩBgH`Bg3Bg3Bg xHBg xHBg xHBgFBgBgs    77  ' %-5=HPXaiqy  *W&V !