o oij@sUddlmZddlmZmZmZmZmZmZm Z m Z m Z ddl Z ddl mZddlmZmZer=ddlmZddlmZeee eeffZGdd d ZGd d d eZ dd e jjd ege jfde ee jge jfdefddZide j de j!ddde j"de j#de j$dide j de j!ddde j"de j#de j$dide j de j!d dd!e j"d"e j#d"e j$d#id$e j!d%dd%e j"d%e j#d%e j$d&d'd(id)e j!d*dd*e j"d*e j#d*e j$d+d'd,id-e j!d.dd/e j"d0e j#d0e j$d1d'd2id3e j!d4dd4e j"d5e j#d5e j$d6d'd7id8e j d9e j!d:dd;e j"de j!d?dd@e j"d?e j#d?e j$dAd'dBidCe j!dDddEe j"dDe j#dDe j$dFd'dGidHe j!dIddJe j"dKe j#dKe j$dLd'dMidNe j!dOddOe j"dOe j#dOe j$dPd'dQidRe j!dSddSe j"dSe j#dSe j$dTd'dUidVe j!dWddWe j"dWe j#dWe j$dXd'dYidZe j!d[dd[e j"d[e j#d[e j$d\d'd]id^e j!d_dd_e j"d_e j#d_e j$d`d'daidbe j!dce j#dde j$ded'dfie j!dge j#dhie j!dge j#die j$djd'dkie j!dle j#dme j$dnd'doie j!dpe j#dqe j$drd'dsie j!dte j#due j$dvd'dwie j!dxe j#dye j$dzd'd{ie j d|e j!d}e j#dKie j d~e j!de j#die j de j!de j#deid Z%eeee ee j&feffe'd<dddddZ(de j)de e j&efde efddZ*ddde j&fddZ+e dedZ,Gdddee,Z-dS))deque) TYPE_CHECKINGAnyCallableDequeDictListOptionalTypeVarUnionN)override)rank_zero_onlyrank_zero_warn)Fabric) Precisionc @seZdZdZ ddeededed ed df d d Zddd dedededeedeed df ddZ d e fddZ dddZ dS) ThroughputaComputes throughput. +------------------------+-------------------------------------------------------------------------------------+ | Key | Value | +========================+=====================================================================================+ | batches_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of batches | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | samples_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of samples | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | items_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of items | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | flpps_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of flops | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | device/batches_per_sec | batches_per_sec divided by world size | +--------------------------+-----------------------------------------------------------------------------------+ | device/samples_per_sec | samples_per_sec divided by world size | +--------------------------+-----------------------------------------------------------------------------------+ | device/items_per_sec | items_per_sec divided by world size. This may include padding depending on the data | +--------------------------+-----------------------------------------------------------------------------------+ | device/flops_per_sec | flops_per_sec divided by world size. | +--------------------------+-----------------------------------------------------------------------------------+ | device/mfu | device/flops_per_sec divided by world size. | +--------------------------+-----------------------------------------------------------------------------------+ | time | Total elapsed time | +--------------------------+-----------------------------------------------------------------------------------+ | batches | Total batches seen | +--------------------------+-----------------------------------------------------------------------------------+ | samples | Total samples seen | +--------------------------+-----------------------------------------------------------------------------------+ | lengths | Total items seen | +--------------------------+-----------------------------------------------------------------------------------+ Example:: throughput = Throughput() t0 = time() for i in range(1000): do_work() if torch.cuda.is_available(): torch.cuda.synchronize() # required or else time() won't be correct throughput.update(time=time() - t0, samples=i) if i % 10 == 0: print(throughput.compute()) Notes: - The implementation assumes that devices FLOPs are all the same as it normalizes by the world size and only takes a single ``available_flops`` value. - items_per_sec, flops_per_sec and MFU do not account for padding if present. We suggest using samples_per_sec or batches_per_sec to measure throughput under this circumstance. Args: available_flops: Number of theoretical flops available for a single device. world_size: Number of devices available across hosts. Global metrics are not included if the world size is 1. window_size: Number of batches to use for a rolling average. separator: Key separator to use when creating per-device and global metrics. Nd/available_flops world_size window_size separatorreturncCsj||_||_|dks J||_|dksJt|d|_t|d|_t|d|_t|d|_t|d|_ dS)Nrr)maxlen) rrr_MonotonicWindow_time_batches_samples_lengthsr_flops)selfrrrrr"Y/home/ubuntu/.local/lib/python3.10/site-packages/lightning/fabric/utilities/throughput.py__init__^s      zThroughput.__init__)lengthsflopstimebatchessamplesr%r&cCs|j|||krtd|d|d|j||j||durU||kr4td|d|d|j|t|jt|jkrUtdt|jdt|jd|durd|j||j dSdS) adUpdate throughput metrics. Args: time: Total elapsed time in seconds. It should monotonically increase by the iteration time with each call. batches: Total batches seen per device. It should monotonically increase with each call. samples: Total samples seen per device. It should monotonically increase by the batch size with each call. lengths: Total length of the samples seen. It should monotonically increase by the lengths of a batch with each call. flops: Flops elapased per device since last ``update()`` call. You can easily compute this by using :func:`measure_flops` and multiplying it by the number of batches that have been processed. The value might be different in each device if the batch size is not the same. zExpected samples (z') to be greater or equal than batches ()NzExpected lengths (z') to be greater or equal than samples (zIf lengths are passed (z1), there needs to be the same number of samples () rappend ValueErrorrrrlen RuntimeErrorr r)r!r'r(r)r%r&r"r"r#updateqs$    zThroughput.updatecCs|jd|jd|jdd}|jr|jd|d<|jdk}t|j|jjkr|jd|jd}|jd|jd}|jd|jd}||}||}|d|jd||d|jd|i|rt||j}||||jd t|j|jjkr|jd|jd} | |} | |d|jd <|r| |j} | |d <t|j |j jkrt |j |j d} |jd|jd}| |} | |j}|r| |d <||d|jd <|j r||j |d|jd <|S) zCompute throughput metrics.)r'r(r)r%rrdevicebatches_per_secsamples_per_sec)r2r3 items_per_sec flops_per_secmfu) rrrrrr-rr/rr sumr)r!metricsadd_global_metrics elapsed_timeelapsed_batcheselapsed_samplesdev_samples_per_secdev_batches_per_secr3elapsed_lengthsdev_items_per_secr4 elapsed_flopsr5dev_flops_per_secr"r"r#computesR    zThroughput.computecCs6|j|j|j|j|jdSN)rclearrrrr r!r"r"r#resets    zThroughput.reset)Nrrr)rN) __name__ __module__ __qualname____doc__r floatintstrr$r/_THROUGHPUT_METRICSrCrGr"r"r"r#r s>>  )4rcsJeZdZdZdddeddffdd Zd d eededefd d Z Z S) ThroughputMonitoraComputes throughput. This class will automatically keep a count of the number of log calls (``step``). But that can be modified as desired. For manual logging, using :class:`Throughput` directly might be desired. Example:: logger = ... fabric = Fabric(logger=logger) throughput = ThroughputMonitor(fabric) t0 = time() for i in range(1, 100): do_work() if torch.cuda.is_available(): torch.cuda.synchronize() # required or else time() won't be correct throughput.update(time=time() - t0, batches=i, samples=i) if i % 10 == 0: throughput.compute_and_log(step=i) Args: fabric: The Fabric object. \**kwargs: See available parameters in :class:`Throughput` fabricrkwargsrNc s|t|jj}t|j|}tjd||jd|||_ d|_ t |j |_ t |j id|_ t |jid|_t |j|_dS)N)rrr0)defaultr")_validate_launched_plugin_to_compute_dtypestrategy precisionget_available_flopsr1superr$r_fabricstepr r/rCcompute_and_logrG)r!rQrRdtyper __class__r"r#r$s   zThroughputMonitor.__init__r[cKs>|dur |jdn||_|jdi|}|jj||jd|S)zSee :meth:`Throughput.compute` Args: step: Can be used to override the logging step. \**kwargs: See available parameters in :meth:`Throughput.compute` Nr)r8r[r")r[rCrZlog_dict)r!r[rRr8r"r"r#r\sz!ThroughputMonitor.compute_and_logrD) rHrIrJrKrr$r rMrOr\ __classcell__r"r"r^r#rPs$ rPmodel forward_fnloss_fnrcCszddlm}|dd}|$|dur|n||Wd|SWd|S1s4wY|S)a-Utility to compute the total number of FLOPs used by a module during training or during inference. It's recommended to create a meta-device model for this: Example:: with torch.device("meta"): model = MyModel() x = torch.randn(2, 32) model_fwd = lambda: model(x) fwd_flops = measure_flops(model, model_fwd) model_loss = lambda y: y.sum() fwd_and_bwd_flops = measure_flops(model, model_fwd, model_loss) Args: model: The model whose FLOPs should be measured. forward_fn: A function that runs ``forward`` on the model and returns the result. loss_fn: A function that computes the loss given the ``forward_fn`` output. If provided, the loss and `backward` FLOPs will be included in the result. r)FlopCounterModeF)displayN)torch.utils.flop_counterrebackwardget_total_flops)rbrcrdre flop_counterr"r"r# measure_flops s    rkh100 nvlgwBg$^/lBtfloat32gS CgSCg />2,Ch100 sxmgwBg$^/lBgSBg />2C h100 pciegvHBgvHBg|Bg|Cg@.CCzrtx 4090g$BgbCint4gbCzrtx 4080gm%Bg)Bg)Cl4gĎBgWHBg$ Bg$ Bg$ Bl40g˓Bg˓Bg˓Bg˓Ca100gꤡBg2Bg2Bg2Bg2Ca6000g\EBg\EBggBg@$Ca40gxBgxBgbcBgCa10ggP`Bg4&kBg4&kBg4&kBg4&kBz rtx 3090 tig@0Bg@0Bg@0Czrtx 3090gPb0Bgq$Bgq$Cz rtx 3080 tigcBgcBgYBzrtx 3080gjZBg.Bg.Bzrtx 3070gIvvBggH|BggH|Bt4g}wBgBgBgBg h_BgHBgHBgHBgHBg`cԩBgH`Bg3Bg3Bg xHBg xHBg xHBgFBgBgs,   77  ' %-5@HPYaiqy  *O&V !